What makes testing AI systems different?
Almost nothing. The core move is drawing a line between deterministic orchestration (the parsing, routing, permissions, state, and budget code you wrote) and the nondeterministic model boundary, the small part of the system where the same input can produce a different output. Nearly everything falls on your side of that line and tests like ordinary software.
Once you draw the line, a surprising amount stays exactly assertable. You can assert that a model's output parses against a schema, that a permission check blocked a call, that a budget stopped a loop at step 20, that a retry fired with the right backoff, or that a state machine landed in a legal terminal state (the terminals Agent state defines): none of that depends on the model phrasing anything a particular way, and none of it is a test on the model at all.
Sign up free to read, listen and talk to tutor
Want me to explain it differently?
AI concepts can be dense. Tell me what's confusing and I'll find a new analogy.