Rubric Rating Papers (Last 45 Days)

Why This Matters For Eval Research

• 100% of papers report explicit human-feedback signals, led by rubric ratings.
• automatic metrics appears in 26.7% of papers in this hub.
• Healthbench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is inter-annotator agreement reporting (6.7% of papers).
• Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.
• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.

Benchmark Interpretation

• Healthbench appears in 6.7% of hub papers (1/15); use this cohort for benchmark-matched comparisons.
• LongBench appears in 6.7% of hub papers (1/15); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 13.3% of hub papers (2/15); compare with a secondary metric before ranking methods.
• agreement is reported in 6.7% of hub papers (1/15); compare with a secondary metric before ranking methods.

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (6.7% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (20% vs 35% target).

• Moderate: Papers naming evaluation metrics

  Coverage is usable but incomplete (33.3% vs 35% target).

• Strong: Papers with known rater population

  Coverage is strong (40% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (100% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).
• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.
• Agentic evaluation appears in 26.7% of papers.

Known Gaps

• Only 6.7% of papers report quality controls; prioritize calibration/adjudication evidence.

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (Healthbench vs LongBench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and agreement.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Decomposing Physician Disagreement in HealthBench

   Start here for detailed protocol reporting and quality-control evidence. Signals: rubric ratings. Focus: Healthbench. Abstract: Rubric identity accounts for 15.8% of met/not-met label variance but only 3.6-6.9% of.

2. RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoning

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + rubric ratings. Focus: accuracy. Abstract: While recent rubric-based approaches offer fine-grained supervision signals, they suffer.

3. SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probing

   Start here for detailed protocol reporting and quality-control evidence. Signals: rubric ratings. Abstract: Rubrics provide structured, interpretable supervision, but scaling rubric construction is difficult: expert rubrics are costly,.

4. Discovering Implicit Large Language Model Alignment Objectives

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + rubric ratings. Abstract: Existing interpretation methods typically rely on pre-defined rubrics, risking the omission.

5. APEX-Agents

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + rubric ratings. Focus: pass@1. Abstract: We open source the APEX-Agents benchmark (n=480) with all.

6. Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge + pairwise preferences. Abstract: Scalar reward models compress multi-dimensional human preferences into a single opaque.

7. Small Reward Models via Backward Inference

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge + rubric ratings. Abstract: However, the dominant LLM-as-a-Judge paradigm relies on the strong reasoning capabilities.

8. Document Reconstruction Unlocks Scalable Long-Context RLVR

   Adds automatic metrics with rubric ratings for broader protocol coverage within this hub. Signals: automatic metrics + rubric ratings. Focus: LongBench / coherence. Abstract: However, it often relies.

Known Limitations

• Only 6.7% of papers report quality controls; prioritize calibration/adjudication evidence.
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
• Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.

Research Utility Snapshot