Skip to content
OpenTrain AIFor AI Companies

HFEPX · Eval paper review

Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges

Saeid Asgari, Rui Ying, Sara Malvar, Rakshanda Agarwal +4 more

Published

Oct 18, 2026

Citations

0

Trust level

High

Usefulness score

67/100 (Medium)

Extraction confidence

75% (High)

Derived from extracted protocol signals and abstract evidence.

Rater population

Not reported

Signals refreshed

Oct 18, 2026

Open arXiv Find implementation Shortlist (0) RSS feed

Tags

Critique Edit Law Llm As Judge Rubric Rating

Should you rely on this paper?

This paper has useful evaluation signal, but protocol completeness is partial; pair it with related papers before deciding implementation strategy.

Use this as a practical starting point for protocol research, then validate against the original paper.

Best use

Secondary protocol comparison source

Use if you need

A secondary eval reference to pair with stronger protocol papers.

What to verify

Validate the evaluation procedure and quality controls in the full paper before operational use.

Main weakness

No major weakness surfaced.

Human feedback signal
Detected
From extracted signals
Evaluation signal
Detected
Eval setup described
Usefulness for eval research
67/100
Moderate-confidence candidate

Useful as a secondary reference; validate protocol details against neighboring papers.

If you are doing eval pipeline work, start here

Human Eval Papers LLM-as-Judge Papers Pairwise Preference Papers Tool-Use Evaluation Search

Abstract

Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training. Two assumptions surround this practice: that leniency or strictness is a property of the judge model, and that a rubric can be improved by asking a model to review it. We test both in a controlled grading task with planted flaws and on two human-labeled datasets, SummEdits and RAGTruth. Holding the judge and the outputs fixed, rubric content moves the pass rate of correct outputs from 2% to 100%, whereas changing the judge model or the rendering of the same checks moves it by a few points. Model self-review then drifts the rubric toward the reviser's own standard: it adds requirements the task never stated and deletes checks that human labels had taught, with three reviser models and on all three datasets. To measure and control this, we show that under all-must-pass aggregation the Shapley value of each check has an exact closed form, which yields per-check catch credit, false-fail blame and unique catches in linear time. Attribution shows that self-review deletes the checks that do unique work and keeps the checks that cause false fails. A change-control gate that combines these quantities with a provenance freeze on label-derived checks holds nearly every drifting revision chain at its starting quality, with identical decisions under three judge models.

What we could verify

These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.

Human Feedback Types

strong

Rubric Rating, Critique Edit

Directly usable for protocol triage.

"Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training."

Evaluation Modes

strong

Llm As Judge

Includes extracted eval setup.

"Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training."

Quality Controls

missing

Not reported

No explicit QC controls found.

"Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training."

Benchmarks / Datasets

strong

RAGTruth

Useful for quick benchmark comparison.

"We test both in a controlled grading task with planted flaws and on two human-labeled datasets, SummEdits and RAGTruth."

Reported Metrics

missing

Not extracted

No metric anchors detected.

"Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training."

Benchmarks and datasets

RAGTruth

Reported metrics

No metric terms were extracted from the available abstract.

Human feedback details
Uses human feedback
Yes
Feedback types
Rubric Rating, Critique Edit
Rater population
Not reported
Unit of annotation
Multi Dim Rubric
Expertise required
General
Evaluation details
Evaluation modes
Llm As Judge
Agentic eval
None
Quality controls
Not reported
Evidence quality
High
Use this page as
Secondary protocol comparison source

Research brief

Metadata summary

Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training.

Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.

Key takeaways

  • Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training.
  • Two assumptions surround this practice: that leniency or strictness is a property of the judge model, and that a rubric can be improved by asking a model to review it.
  • We test both in a controlled grading task with planted flaws and on two human-labeled datasets, SummEdits and RAGTruth.

Researcher actions

  • Compare this paper against nearby papers in the same arXiv category before using it for protocol decisions.
  • Validate inferred eval signals (LLM-as-judge) against the full paper.
  • Use related-paper links to find stronger protocol-specific references.

Caveats

  • Generated from abstract + metadata only; no PDF parsing.
  • Signals below are heuristic and may miss details reported outside the abstract.

Recommended queries

Rubric rating evaluation

Contribution summary

  • Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training.
  • Holding the judge and the outputs fixed, rubric content moves the pass rate of correct outputs from 2% to 100%, whereas changing the judge model or the rendering of the same checks moves it by a few points.
  • To measure and control this, we show that under all-must-pass aggregation the Shapley value of each check has an exact closed form, which yields per-check catch credit, false-fail blame and unique catches in linear time.

Why it matters for eval

  • Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training.
  • Holding the judge and the outputs fixed, rubric content moves the pass rate of correct outputs from 2% to 100%, whereas changing the judge model or the rendering of the same checks moves it by a few points.

Researcher checklist

  • Human feedback protocol is explicit

    Detected: Rubric Rating, Critique Edit

  • Evaluation mode is explicit

    Detected: Llm As Judge

  • Quality control reporting appears

    No calibration/adjudication/IAA control explicitly detected.

  • Benchmark or dataset anchors are present

    Detected: RAGTruth

  • Metric reporting is present

    No metric terms extracted.