Statistical Significance of Evaluation

Observed difference may be sampling noise. Standard error of binomial distribution √(p(1-p)/n): with 100 examples and 70% success, the noise band is ±9 points — a 3% difference is not significant. Correct approach for a single set of tasks is paired analysis (McNemar’s test: look only at examples where results diverged) — more sensitive than subtracting independent proportions. Multiple comparisons trap: with 6 hypotheses, the probability of a false positive conclusion is ~26% — Bonferroni correction or independent verification is needed. Several runs with different seeds are mandatory.

Related: [Evaluation Metrics of Agents], [Three Levels of Evaluation System]