Most agent failures are evaluation failures
Teams test their agents, the agents pass, and then they fail in front of customers. The problem usually sits in what the tests asked, which is something a team can fix.
By Nathan Barling Product Marketing Principal 6 min read
Most AI agents that go wrong in production were tested before launch. The trouble is usually what the tests covered.
Take an agent that writes a retail brand's Monday performance summary. Before launch, the team tests it on 40 past weeks. Every number matches the source data, and analysts like the commentary, so it goes live.
In its third week, it tells the client that sales fell 30% on Sunday. They didn't. Sunday's data was still loading when the agent ran, and it read a partial day as a full one. None of the 40 test weeks had a partial day in it, because the team built the test set from finished weeks.
Figure 1
Teams watch their agents more than they test them
Share of organizations building agents, late 2025
Bar chart: 89% have observability for their agents, 62% have detailed tracing, 52.4% run offline evaluations on test sets, and 37.3% run online evaluations
Passing the test and failing the customer
VentureBeat Intelligence's October 2026 survey asked organizations that run evaluations before deployment whether AI that passed went on to fail. Forty-two percent said it had in the past 12 months, and 25% said it happened more than once. In 49% of those cases, the most significant failure reached customers.
The June wave of the same survey asked why teams don't fully trust their automated evaluations. The most common answer, at 29%, was poor alignment with real-world outcomes. Only 5% said they fully trust them. Both waves are small, self-selected samples, so treat them as directional.
LangChain's survey of 1,340 people building agents points the same way. Quality was the top barrier to production, named by 32%. Yet 89% have observability for their agents and 52.4% run offline evaluations on test sets. Teams can see what their agents did. Fewer have checked whether it was right.
A demo is a sample of one
Agents don't give the same answer twice, which changes what testing means. The researchers behind τ-bench, a benchmark of agents handling retail and airline customer requests with real tools and policies, measured this directly. They ran each task several times and counted how often an agent succeeded on every try.
In a single run, gpt-4o completed 61.2% of retail tasks. Asked to succeed on all eight runs of the same task, it managed fewer than 25%. A team that tests each case once is measuring the first number and shipping into the second.
Four ways agent evals go wrong
Four patterns sit behind many of these failures. Each one has a plain fix.
1. The cases are too clean
Test sets get built from tidy examples, because tidy examples are easy to find and easy to grade. Production brings partial days, renamed campaigns, a feed that broke overnight, and questions nobody phrased the way the test did.
The fix: build the evaluation set from real requests, and go looking for the awkward ones. Include missing data, conflicting sources, vague questions, and requests the agent should refuse. If a case made a person stop and check last quarter, it belongs in the set.
2. Only the final answer gets graded
An agent can reach the right number the wrong way: the wrong table, the wrong date range, or a definition of revenue that happens to match this week. The answer passes. Next week it doesn't.
The fix: grade the steps as well as the answer. Check which tools the agent called, with which inputs, and in what order. For function calling, that check is cheap, because every call is already structured data.
3. Each case runs once
A single pass tells you an agent can do a task. It doesn't tell you how often. The τ-bench results show how far those two numbers can sit apart.
The fix: run every case several times and count a case as passed only when every run passes. For tasks that act on budgets or customers, the share that pass every time is the number that matters.
4. The set never changes after launch
Most eval sets are written once, before launch, by the people who built the agent. Then the model updates, a prompt changes, a platform renames a field, and the agent ships against a test that describes last quarter.
The fix: treat the eval set as a regression suite. Every failure found in production becomes a new case. Every change to the model, prompt, tools, or data runs the full set again, and the change ships only if it scores at least as well.
What to test in a marketing agent
Marketing data has its own traps, and generic test sets miss them. These are the cases we'd add first.
- Freshness. Partial days, late conversions, and feeds that missed their window. Platforms keep updating conversions for days after the click, so recent numbers often look worse than they are.
- Definitions. Revenue with and without returns, new versus returning customers, and a campaign renamed mid-flight. The agent should use the governed definition every time, whatever the column is called.
- Permissions. Ask the agent for data the user isn't allowed to see. The right result is a refusal, and that's a test case too.
- Actions. Run anything that changes a budget, a bid, or a live ad against a test copy of the system, and check the exact change it tried to make before it ever touches the real one.
- Handoffs. In a multi-agent system, test what each agent passes to the next. Many failures start as a small mistake one agent hands on and the next treats as fact.
Where Alli fits
Alli Generative Workflows takes a strict line on run-to-run variation. Ask Alli drafts a workflow from a description, and people review the steps. Once it's reviewed, the workflow runs the same steps in the same order every time, on a schedule, from a webhook, or on demand. What the team tested is what runs.
The Testing Suite lets a builder run a single task on its own, see errors as they happen, and ask Ask Alli why a step failed, without rerunning the whole workflow. Each workflow keeps its versions and a history of every run, so a team can see when a result changed and what changed with it.
For outside agents, Alli MCP Server lets teams turn each tool on or off and start with read-only questions before moving to actions. Every request carries the signed-in user's own permissions. That keeps the set of actions an agent can take small enough to test properly.
When evals aren't the answer
Evals tell you whether an agent does its job. They can't tell you whether the job was worth automating. They also don't replace watching live runs or the limits that stop an agent when something goes wrong after launch. For a small task that a person reviews every time anyway, that review is the test, and a formal eval suite may cost more than it saves.
Take the last thing your agent got wrong. Write it down as a test case, with the right answer next to it, and make it the first case in your eval set.
Frequently asked questions
Why do AI agents fail in production?
Often because their tests didn't match real use. Test sets built from clean examples miss partial data, vague requests, and edge cases. Grading only the final answer misses wrong steps, and running each case once misses inconsistency.
What is an AI agent eval?
An eval runs an agent on a set of cases with known good outcomes and scores the results. Good evals use real requests, grade the steps as well as the answer, and run again whenever the model, prompt, tools, or data change.
How often do AI agents pass evaluations and still fail?
In VentureBeat Intelligence's October 2026 survey, 42% of organizations that evaluate agents before deployment said AI that passed went on to fail in the past 12 months. In 49% of those cases, the most significant failure reached customers. The sample was small and self-selected.
What is pass^k?
A measure from the τ-bench paper of how consistently an agent succeeds. It's the chance that all k runs of the same task succeed. In the paper, gpt-4o completed 61.2% of retail tasks in one run but fewer than 25% across all eight runs.
How is evaluation different from observability?
Observability records what an agent did, such as its steps, tool calls, and errors. Evaluation checks whether what it did was right. LangChain found 89% of teams have observability, while 52.4% run offline evaluations.
Sources
- VentureBeat Intelligence, VB Pulse Agentic Reliability and Evals Report, October 2026, October 2026 (n=106 qualified). Self-selected sample. VentureBeat runs events and sells advertising.
- VentureBeat, Enterprise AI is entering an evaluation gap: Agents are gaining autonomy faster than companies can verify them, July 10, 2026, reporting the June 2026 VB Pulse survey (n=157). Self-selected sample.
- LangChain, State of Agent Engineering, June 12, 2026, from a public survey run November 18 to December 2, 2025 (n=1,340). LangChain sells agent evaluation and observability software.
- Yao, Shinn, Razavi, and Narasimhan (Sierra), τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv, June 17, 2024. Sierra sells customer service agents.
Last updated November 3, 2026. Statistics are quoted from the sources above and were checked on that date. Survey results are self-reported unless noted.