Agent Evaluation Metrics

Process Metrics:

• Proportion of valid actions;
• Tool call accuracy;
• Path efficiency (steps, redundancy, rollbacks);
• Cost and latency.

Outcome Metrics:

• Success rate;
• Pass@k – at least one of k attempts is successful: ceiling of capability;
• Pass^k – all k are successful: consistency and reliability;
• Best@k.

Difference is critical: with Pass@1=0.6 Pass@5≈99%, and Pass^5≈7.8%.

Safety – zero tolerance: one serious violation cancels the entire evaluation (veto point).

Double coverage: trajectory (“said and did”) and end result (“what happened to the system”) – both are necessary.

Related: Three levels of evaluation system, LLM-as-a-Judge, Statistical significance of evaluation