Four Principles of a Rubric

Scale AI Principles:

1. Reliance on Expert Knowledge – capturing key facts and steps, typical pitfalls;
2. Comprehensiveness – accuracy, coherence, completeness, safety plus explicit pitfalls;
3. Weighting by Importance – must-have/important/optional items and a veto mechanism (hallucination nullifies the score regardless of other points);
4. Self-Sufficiency – each item is verifiable without domain knowledge: instead of “deep understanding” – “referenced at least two theories”.

Each grade is a concrete verifiable behavior plus examples and edge cases. The rubric is a product of iterative refinement.

Related: LLM-as-a-Judge, Process reward vs outcome reward