A sports coach applied for a senior AI management role.
Our committee of AI agents reviewed the résumé and reached a clear conclusion:
The candidate was an excellent fit.
The problem, obviously, was that he was not.
This was not a real candidate. It was a deliberately irrelevant résumé used as a test case. We expected the system to recognize that the applicant had no meaningful AI management experience.
The failure was not caused by a weak model, a poor prompt, or agents that were not capable enough. A new version had broken the connection between the agent committee and the source documents it was supposed to evaluate.
The deployment completed successfully. The services were healthy. The API returned responses. Every agent continued to participate.
But the system was no longer connected to the information on which its decision was supposed to be based.
Instead of stopping and reporting that it lacked evidence, the first agent filled in the missing context. The next agent treated that unsupported claim as a fact. Another agent reinforced it. A full consensus emerged around a candidate who existed only in the committee's shared narrative.
The agents were not reading the résumé. They were reading one another.
What actually failed
It would be easy to label this as hallucination and move on. But the central failure was not inside a single model response. It was a failure of the complete production system.
A modern AI application is not just a model. It is code, prompts, retrieval, memory, identity, permissions, tools, orchestration, configuration, external services, and sometimes several agents exchanging state. Each component can appear healthy while the workflow between them is broken.
In our case, the source-grounding path failed silently. The system did not crash. It degraded into a different system: one that still generated coherent decisions, but no longer had a reliable source of truth.
This is a silent behavioral regression. The software remains available, but the behavior that made it useful or safe has changed.
Could unit tests have caught it?
Yes, if the relevant contract had been tested.
A unit test could have verified that a document was returned by the retrieval function, that empty source text caused an error, or that an agent could not proceed with an empty evidence package. Those tests are essential.
But unit tests alone cannot prove that the deployed system still works end to end.
They often replace real dependencies with mocks. A mock memory service returns the expected document even when the production identity lacks permission to read it. A mocked tool respects the expected schema even when the deployed API changed. Every function may work independently while the output of one component is no longer passed to the next.
This is why the right conclusion is not that conventional tests are irrelevant. It is that they are only the first layer.
The hidden execution path
Ordinary output testing often sees a simple relationship:
An agentic system may actually execute a much longer path:
The final answer is only the visible end of this trajectory. A system can arrive at a plausible answer through an unacceptable path, or reach the correct answer by accident without using the required evidence.
The golden scenario that exposed the regression
A résumé containing sports-coaching experience and no AI-management experience.
The complete agent committee runs against the version that was actually deployed.
- The source document must be retrieved.
- Every participating agent must receive the correct source identifier.
- Material claims must be traceable to the document.
- The candidate must not be recommended for the AI role.
- If the source is unavailable, the system must return insufficient evidence.
- The run must fail if the agents continue without grounded context.
This is a golden scenario: a stable test case whose input, expected behavior, and failure conditions are known in advance. It does not require the model to produce one exact sentence. It verifies the business outcome, the evidence path, and the system's behavior when evidence is missing.
From output testing to trajectory testing
A trajectory is the sequence through which an AI system attempts to complete a task: retrieval, memory reads, model calls, routing decisions, tool use, inter-agent messages, approvals, retries, fallbacks, and the final result.
Trajectory testing asks not only whether the system reached the right destination, but whether it used an acceptable route.
This distinction matters because several trajectories may be valid. The goal is not necessarily to force one exact sequence. It is to define hard boundaries. An agent may use either of two approved search tools, but it may not query a restricted database. It may retry a read operation, but it must not automatically repeat a financial transaction. It may summarize a source, but it must not convert another agent's unsupported statement into evidence.
A practical AI release gate
A reliable release process should combine seven layers.
1. Unit and contract tests
Test deterministic behavior directly: schemas, empty inputs, authorization decisions, expected error states, idempotency, and the contracts between components. Use mocks where useful, but do not confuse a mocked dependency with the production integration.
2. Integration tests with real dependencies
Verify the deployed identity, permissions, network paths, indexes, APIs, and configuration in an environment that resembles production. Confirm that data actually moves between the services, not merely that each service responds independently.
3. End-to-end golden scenarios
Run a representative set of known cases through the complete system. Include clear positive cases, obvious negative cases, ambiguous cases, and cases in which a required source or tool is deliberately unavailable.
4. Grounding and evidence checks
Separate two questions: did the correct evidence reach the system, and did the answer remain faithful to that evidence? A grounded answer based on the wrong document is still wrong. A high-quality response without a required source should still fail.
5. Trace and trajectory evaluation
Record and evaluate retrieval events, tool calls, handoffs, approvals, memory access, retries, and fallbacks. The trace should reveal whether the system followed an acceptable path and where a regression began.
6. Adversarial and long-horizon testing
Test more than one malicious prompt. Evaluate poisoned documents, conflicting instructions, stale memory, compromised tool output, repeated unsupported claims, long sessions, and behavior after dozens of tool calls. Persistent systems need endurance testing for behavior, not only performance.
7. Post-deployment verification
Run synthetic tests against the version that is actually live. Then use shadow mode, constrained canary deployment, and online evaluation to detect failures that only appear with real traffic and real infrastructure.
The AI release process
Fail closed when reality is missing
The remediation required more than restoring the connection to the documents. It also defined what the system is allowed to do when that connection fails.
When evidence is a prerequisite for a decision, absence of evidence must be an explicit system state. The application may remain available, but it must not continue the decision process by substituting the model's general knowledge or the committee's shared journal.
No source should mean no decision, not a more creative decision.
This is fail closed at the behavioral level. It is especially important in systems that influence hiring, healthcare, benefits, finance, legal decisions, infrastructure, or access to public services.
Shared memory is not ground truth
Multi-agent systems often use a shared journal or conversation history to coordinate work. That memory is useful, but it contains agent outputs, not verified facts.
Once an unsupported claim enters shared state, later agents may interpret repetition as confirmation. The resulting consensus is not independent. It is a correlated error amplified by a common context.
This is why adding more agents does not automatically add more scrutiny. Without independent access to evidence, explicit source references, and a role empowered to challenge unsupported claims, a multi-agent committee can become a meeting in which everyone quotes the previous speaker and nobody opens the document.
The broader research signal
Our incident involved missing source context. Research on stateful and multi-agent systems points to a wider problem: even when information and safety instructions are present, accumulated context can gradually reshape behavior.
In persistent systems, risks can emerge through memory, other agents, retrieved content, and tool output rather than through a single direct jailbreak. That means evaluations built around isolated prompt-response pairs cannot represent the full environment in which agents operate.
The engineering response is architectural: least-privilege context, memory isolation, deterministic authorization, observable trajectories, long-horizon tests, and controls that can stop execution independently of the model.
Observability is part of the test architecture
You cannot evaluate a trajectory that you did not record.
A useful trace should identify the deployed configuration, source documents, retrieval results, memory reads and writes, model and tool calls, handoffs, approvals, latency, token use, errors, and final outcome. OpenTelemetry's emerging Generative AI conventions provide a path toward standardizing much of this instrumentation.
But maximum logging is not the goal. Prompts, tool parameters, and memory may themselves contain sensitive information. Teams need a telemetry policy that defines redaction, access, retention, and whether production data may be used by external evaluation services.
The objective is sufficient, secure evidence to reconstruct what happened.
Every detected regression should strengthen the release suite
The sports-coach scenario did more than identify the code defect. It was formalized as a permanent regression test.
Every future release must now prove that the system retrieves the document, exposes it to the correct agents, grounds material claims in it, rejects an irrelevant candidate, and stops when the document is unavailable.
This is how observed failures become versioned engineering controls. Over time, the test suite becomes a practical record of behaviors the system is explicitly prohibited from repeating.
The real deployment question
A successful deployment is not evidence that an AI system still behaves correctly.
Before a release, teams need evidence for three questions:
- Did the system receive the right information?
- Did it use that information through an acceptable trajectory?
- Did it stop when the conditions required for a safe decision were missing?
In conventional software, the obvious failure is when the system stops working.
In AI systems, the more dangerous failure is when the system keeps working, keeps answering, and keeps sounding convincing, while no longer being connected to reality.