Human Eval Papers (Last 90 Days)

Why This Matters For Eval Research

• 50% of papers report explicit human-feedback signals, led by pairwise preferences.
• human evaluation appears in 70.6% of papers in this hub.
• AIME is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• 2 sampled papers report both human evaluation and LLM-as-judge, supporting direct agreement checks.
• Most common quality-control signal is adjudication (5.9% of papers).
• Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.

Benchmark Interpretation

• AIME appears in 8.3% of hub papers (1/17); use this cohort for benchmark-matched comparisons.
• Correctbench appears in 8.3% of hub papers (1/17); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 25% of hub papers (3/17); compare with a secondary metric before ranking methods.
• agreement is reported in 16.7% of hub papers (2/17); compare with a secondary metric before ranking methods.

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (50% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (16.7% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (16.7% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (50% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (16.7% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (33.3% vs 35% target).

Strengths

• Strong human-feedback signal (50% of papers).
• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.

Known Gaps

• Only 16.7% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (16.7% coverage).
• Benchmark coverage is thin (16.7% of papers mention benchmarks/datasets).

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (AIME vs Correctbench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and agreement.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Distill and Align Decomposition for Enhanced Claim Verification

   Start here for detailed protocol reporting and quality-control evidence. Signals: human evaluation. Focus: accuracy. Abstract: Human evaluation confirms the high quality of the generated subclaims.

2. Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

   Start here for detailed protocol reporting and quality-control evidence. Signals: human evaluation + pairwise preferences. Abstract: Preference-based RL offers an appealing alternative by learning from human preferences over.

3. CARE: An Explainable Computational Framework for Assessing Client-Perceived Therapeutic Alliance Using Large Language Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: human evaluation. Focus: accuracy. Abstract: CARE also produces high-quality, contextually grounded rationales, validated by both automatic and human.

4. Validating Political Position Predictions of Arguments

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: Real-world knowledge representation often requires capturing subjective, continuous attributes.

5. HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: For each dialogue history, we pair human and model.

6. AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summarization

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: human evaluation. Abstract: We evaluate AgenticSum on two public datasets, using reference-based metrics, LLM-as-a-judge assessment, and.

7. Think$^{2}$: Grounded Metacognitive Reasoning in Large Language Models

   Adds human evaluation with pairwise preferences for broader protocol coverage within this hub. Signals: human evaluation + pairwise preferences. Focus: GSM8K. Abstract: Blinded human evaluations over 580 query.

8. RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mind

   Adds human evaluation with pairwise preferences for broader protocol coverage within this hub. Signals: human evaluation + pairwise preferences. Focus: Rebuttalbench. Abstract: For reliable and efficient automated evaluation,.

Known Limitations

• Only 16.7% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (16.7% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot