Agent Evaluation Metrics

Process metrics: rate of valid actions, tool call accuracy, path efficiency (steps, redundancy, rollbacks), cost and latency. Outcome metrics: success rate, Pass@k (at least one of k attempts succeeds – ceiling of capability), Pass^k (all k succeed – stability and reliability), Best@k. The difference is critical: with Pass@1=0.6, Pass@5≈99%, and Pass^5≈7.8%. Safety – zero tolerance: one critical violation voids the entire evaluation (veto point). Double coverage: trajectory (“said and did”) and end result (“what happened to the system”) – both are necessary.

Related: [Three levels of evaluation system], LLM-as-a-Judge, [Statistical significance of evaluation]