Evaluation Environments

Two main paradigms of environments:

• Instrumental (Verifiers) — agent calls tools, verification by executable criteria (tests, answer matching), without human and LLM judge;
• Human-in-the-loop (τ-bench) — another LLM plays the user.

The key principle of human-in-the-loop evaluation is progressive information disclosure: real users do not formulate the need immediately clearly, the agent must clarify with questions, and the simulator reveals information gradually according to the script.

τ²-bench adds double control (the user also acts in the environment) and parameterized task generation.

Related: Three levels of evaluation system, LLM-as-a-Judge, Simulation environment