HFEPX · Eval paper review
Justin D. Norman, Michael U. Rivera, D. Alex Hughes
Published
Jun 17, 2026
Citations
0
Trust level
High
Usefulness score
87/100 (High)
Extraction confidence
90% (High)
Derived from extracted protocol signals and abstract evidence.
Rater population
Not reported
Signals refreshed
Jun 17, 2026
This paper has strong direct human-feedback and evaluation protocol signal and is suitable as a primary eval pipeline reference.
Use this as a practical starting point for protocol research, then validate against the original paper.
Best use
Primary benchmark and eval reference
Use if you need
A concrete protocol example with enough signal to inform rater workflow design.
What to verify
Validate the exact study setup in the full paper before operational use.
Main weakness
No major weakness surfaced.
Use this as a primary source when designing or comparing eval protocols.
If you are doing eval pipeline work, start here
LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.
These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.
Pairwise Preference, Rubric Rating
Directly usable for protocol triage.
"LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability."
Llm As Judge, Automatic Metrics
Includes extracted eval setup.
"LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability."
Inter Annotator Agreement Reported
Calibration/adjudication style controls detected.
"LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability."
MT Bench, Judgebench, Rewardbench
Useful for quick benchmark comparison.
"We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments."
Exact match, Kappa, Agreement
Useful for evaluation criteria comparison.
"LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability."
LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability.
Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.
Human feedback protocol is explicit
Detected: Pairwise Preference, Rubric Rating
Evaluation mode is explicit
Detected: Llm As Judge, Automatic Metrics
Quality control reporting appears
Detected: Inter Annotator Agreement Reported
Benchmark or dataset anchors are present
Detected: MT-Bench, Judgebench, Rewardbench
Metric reporting is present
Detected: exact match, kappa, agreement
