What is AI agent evaluation?
Software tests check that code does what it should. Evals do the same job for systems that don't give the same output twice. Instead of checking one exact answer, an eval runs the agent on many cases and measures how often it gets the job right, how it fails when it doesn't, and what each run costs.
Evals matter more for agents than for chat assistants, because agents take actions. A wrong sentence in a chat can be ignored. A wrong tool call can change a budget, send a message, or overwrite a file.
How do you evaluate an AI agent?
- Collect real cases. Build an evaluation dataset from requests people actually make, with the right outcome for each. Include the awkward ones: missing data, vague questions, and requests the agent should refuse.
- Decide what counts as right. Write down what a correct result looks like before running anything. For a report, that's the right numbers. For an action, it's the right tool with the right inputs.
- Grade the run. Score the final answer and the steps that led to it. Code can check exact values and tool calls. A person, or a second model checked against people, judges answers that need judgment.
- Probe for failure. Red teaming tries to break the agent on purpose. Stress tests check whether a small change in wording changes the result.
- Run it on every change. Treat the eval set as a regression suite. A new model or prompt ships only if it scores at least as well as the version it replaces.
Ways to grade an agent
| Method | Best for | Watch out for |
|---|---|---|
| Exact checks in code | Numbers, formats, and which tool was called with which inputs | Misses answers that are right but worded differently |
| Human evaluation | Quality, tone, and judgment calls | Slow and costly, so it covers fewer cases |
| Model as judge | Grading many open-ended answers quickly | Needs checking against human grades, since judge models have their own biases |
| Public benchmarks | Comparing models in general | Rarely match your tasks, data, or definitions |
What is an example of agent evaluation?
Testing a weekly reporting agent
A team builds an agent that writes the Monday performance summary. Before launch, it collects 60 past weeks with the summaries analysts approved, including a week when a feed broke and a holiday week.
The eval checks every number against the source data in code, and an analyst grades the commentary. The agent gets the numbers right every week but calls a normal holiday dip a problem in three of the 60. The team adds the calendar to the agent's context, runs the set again, and ships once those cases pass.
Evals vs. benchmarks vs. a demo
| Demo | Public benchmark | Your own evals | |
|---|---|---|---|
| What it shows | That the agent can work once | How a model does on general tasks | How often the agent does your job right |
| Uses your data | Sometimes | No | Yes |
| Catches regressions | No | No | Yes, when run on every change |
| Who it convinces | The room | Researchers | The people who'll rely on it |
Why does agent evaluation matter?
Many agents stall between pilot and production for one reason: nobody agreed what a good result looks like. Without that, nobody can say whether the agent is ready, or whether last week's change made it worse.
- Launch with evidence. A pass rate on real cases replaces the feeling a good demo leaves.
- Change with confidence. Models update often. Evals show whether a new one helps or hurts before it reaches users.
- Find the failure pattern. Graded runs show where the agent breaks, which tells the team what context, tool, or rule to add.
How does Alli do this?
Alli by PMG's Generative Workflows takes a strict line on this. While a workflow is being built, Ask Alli drafts the steps and people review them. Once it's reviewed, the workflow runs the same steps in the same order on every run, so what was tested is what runs.
The Testing Suite in Generative Workflows lets a builder run one task on its own, see errors as they happen, and ask Ask Alli why a step failed, without rerunning the whole workflow. Each workflow keeps its versions and a history of every run, so a team can see when a result changed and what changed with it.
Explore Alli Generative WorkflowsAI agent evaluation (evals) FAQ
What are evals in AI?
Evals are tests for AI systems. They run a model or agent on cases with known good outcomes and score how often it gets them right, so teams can compare versions and catch regressions.
How many test cases does an agent eval need?
Enough to cover the real variety of requests, including the rare ones. Teams often start with a few dozen real cases and add every failure they find in production as a new case.
What is LLM as a judge?
It's using a second model to grade an agent's answers against a rubric. It's fast enough to grade thousands of answers, but its grades should be checked against a sample graded by people, since judge models favor some styles of answer.
What is the difference between evals and monitoring?
Evals test an agent on known cases before and after a change. Monitoring watches live runs for errors, cost, and drift. Evals decide what ships, and monitoring shows what happens after.
Can you evaluate an agent that takes actions?
Yes. Run it against a test copy of the systems it acts on, then check which tools it called, with which inputs, and in what order, as well as the final result.
Last updated Back to the glossary