Evaluation Environments

Two main paradigms of environments: instrumental (Verifiers) – agent calls tools, verification by executable criteria (tests, answer matching), without human and LLM judge; human-in-the-loop (τ-bench) – another LLM plays the user. The key principle of human-in-the-loop evaluation is progressive information disclosure: real users do not formulate the need immediately clearly, the agent must clarify with questions, and the simulator reveals information gradually according to the script. τ²-bench adds double control (the user also acts in the environment) and parameterized task generation.

Related: [Three levels of evaluation system], LLM-as-a-Judge, [Simulation environment]