LLM-as-a-Judge

LLM-as-a-Judge is a model that judges based on an expert rubric: balancing the scale of automation and the quality of human judgment for open-ended tasks. Limitations: bias towards length, instability of repeated evaluations. The solution is multi-source heterogeneous judgment: several LLMs from different families (biases are orthogonal, Goodhart’s Law). Judge calibration is mandatory: a golden dataset with human annotation, agreement threshold (kappa > 0.7), recalibration upon update. Positional bias is cured by evaluating in both orders.

Related: [Four Principles of Rubric], [Agent Evaluation Metrics], [Three Levels of Evaluation System]