Guide · AI agents
How to Evaluate AI Agents Before You Trust Them in Production
By Vikas Goel, Founder of ThinkerWave · Last updated
To evaluate an AI agent, test it on a fixed set of real tasks with answers agreed in advance, compare it with a simple baseline, score the results blind, and measure cost and consistency as well as accuracy. Include questions it should refuse. An agent’s own confidence, or a score from its own evaluator, is useful evidence but not proof.
In brief
- Set the questions, answers and scoring rules before you see any results.
- Always compare with a baseline: the same model in one pass, and the same model checking itself.
- Hide which system produced each answer from whoever scores it.
- Measure cost per correct answer, not just accuracy.
- Treat an internal evaluator’s score as a signal to calibrate, not as independent validation.
Why agents are harder to evaluate than a chatbot
A chatbot gives you one response to judge. An agent plans, calls tools, reads results and decides what to do next, so it can fail in more places. It can reach the right answer by a broken route, or follow a sensible route and still report a false result. Two runs of the same task can also differ, which means one good demo tells you very little.
What to measure
Task accuracy
Did the agent get the right result, according to an answer key written before the test? For numbers, set a tolerance in advance. For text, write down what a correct answer must contain.
Correct refusals
Include tasks with a false premise or missing data. A reliable agent stops and says why. An agent that always produces a confident answer will eventually produce a confident wrong one.
Process and tool use
Check the trail of what the agent actually did: which tools it called, what came back, and whether its final claim matches. This matters most where a wrong step has consequences of its own, such as changing a record or sending a message.
Consistency
Run a sample of tasks several times. If the answer changes from run to run, your accuracy figure is less stable than it looks, and users will see that instability.
Cost and time per correct answer
A multi-step agent can use many times the model calls of a single pass. Divide total spend by the number of correct answers. That shows whether the extra accuracy is worth paying for on this type of task.
A six-step evaluation you can run
- Fix the test set. Use real tasks, selected at random where possible, and record the selection method. Hand-picked questions flatter the system.
- Write the rules first. Commit the answer key, tolerance and pass mark before any run.
- Add baselines. Run the same model in a single pass, then with a self-check, using the same data and tools as the agent.
- Score blind. Whoever judges an answer, person or model, should not know which system produced it.
- Log everything. Keep every output, tool call, token count and cost, including failed runs.
- Report the uncertainty. Give the sample size and margin of error, and show where the agent lost to the baseline.
The independent evaluator, and its limits
An agent that grades its own work is likely to share the blind spots that produced it. A stronger design separates the two: the evaluator receives the problem, the criteria and the output, but not the instructions that shaped the agent. ThinkerWave is built this way, and the same independence caught real errors in our own runs, including a claimed “zero collisions” plan that actually contained 258.
That separation does not make the score true. An evaluator is still a model, and it can give a high score to a polished wrong answer. Scores from ThinkerWave’s evaluator are internal measures, not independent validation. The useful test is calibration: across questions with known answers, do higher scores actually mean more correct answers? If they don’t, the score shouldn’t drive decisions.
Watch for changing criteria too. If an agent or team changes what is being scored, a falling score may mean a harder test rather than worse work. Re-score the earlier best output against the new criteria before you call it a regression.
Frequently asked questions
What is AI agent evaluation?
AI agent evaluation is the process of measuring whether an agent completes real tasks correctly, safely and affordably. Unlike testing a single model response, it also looks at the steps the agent took, the tools it used, and whether it behaves consistently when the same task is run again.
What metrics should I use to evaluate an AI agent?
Start with task accuracy against an answer key, correct refusals, cost per correct answer, time per task, and consistency across repeat runs. Then add checks on tool use and process for any workflow where a wrong step can do damage even when the final answer looks right.
Can an AI agent evaluate itself?
Partly, but do not rely on it alone. Self-review tends to inherit the framing of the original answer. A separate evaluator that cannot see how the agent was instructed is stronger, but it is still a model’s opinion. Calibrate it against answers that were checked independently.
How many test cases do I need?
Enough to separate a real difference from noise. With 50 questions, a measured accuracy has a margin of error of up to about 14 percentage points. That is enough to spot a large problem or decide whether to keep testing, but not enough to claim a small improvement.
Sources
Try it on your own problem
ThinkerWave works a hard problem from several angles, checks its claims against the evidence, and keeps a full record of how it got there. It runs on your machine, with your own model keys.