Human Eval Papers (Last 30 Days)

Why This Matters For Eval Research

• 40% of papers report explicit human-feedback signals, led by pairwise preferences.
• human evaluation appears in 71.4% of papers in this hub.
• AIME is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• 1 sampled papers report both human evaluation and LLM-as-judge, supporting direct agreement checks.
• Most common quality-control signal is adjudication (7.1% of papers).
• Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.

Benchmark Interpretation

• AIME appears in 10% of hub papers (1/14); use this cohort for benchmark-matched comparisons.
• Correctbench appears in 10% of hub papers (1/14); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 30% of hub papers (3/14); compare with a secondary metric before ranking methods.
• f1 is reported in 20% of hub papers (2/14); compare with a secondary metric before ranking methods.

Researcher Checklist

• Moderate: Papers with explicit human feedback

  Coverage is usable but incomplete (40% vs 45% target).

• Moderate: Papers reporting quality controls

  Coverage is usable but incomplete (20% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (10% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (50% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (20% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (30% vs 35% target).

Strengths

• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.

Known Gaps

• Rater population is under-specified (20% coverage).
• Benchmark coverage is thin (10% of papers mention benchmarks/datasets).

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (AIME vs Correctbench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and f1.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Distill and Align Decomposition for Enhanced Claim Verification

   Start here for detailed protocol reporting and quality-control evidence. Signals: human evaluation. Focus: accuracy. Abstract: Human evaluation confirms the high quality of the generated subclaims.

2. Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

   Start here for detailed protocol reporting and quality-control evidence. Signals: human evaluation + pairwise preferences. Abstract: Preference-based RL offers an appealing alternative by learning from human preferences over.

3. CARE: An Explainable Computational Framework for Assessing Client-Perceived Therapeutic Alliance Using Large Language Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: human evaluation. Focus: accuracy. Abstract: CARE also produces high-quality, contextually grounded rationales, validated by both automatic and human.

4. Validating Political Position Predictions of Arguments

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: Real-world knowledge representation often requires capturing subjective, continuous attributes.

5. Think$^{2}$: Grounded Metacognitive Reasoning in Large Language Models

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: GSM8K. Abstract: Blinded human evaluations over 580 query pairs show an.

6. AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summarization

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: human evaluation. Abstract: We evaluate AgenticSum on two public datasets, using reference-based metrics, LLM-as-a-judge assessment, and.

7. Discovering Implicit Large Language Model Alignment Objectives

   Adds human evaluation with rubric ratings for broader protocol coverage within this hub. Signals: human evaluation + rubric ratings. Abstract: Existing interpretation methods typically rely on pre-defined rubrics,.

8. FrameRef: A Framing Dataset and Simulation Testbed for Modeling Bounded Rational Information Health

   Adds human evaluation for broader protocol coverage within this hub. Signals: human evaluation. Abstract: Human evaluation further confirms that FrameRef's generated framings measurably affect human judgment.

Known Limitations

• Rater population is under-specified (20% coverage).
• Benchmark coverage is thin (10% of papers mention benchmarks/datasets).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot