Human Eval + Coding Papers

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (53.8% vs 45% target).

• Strong: Papers reporting quality controls

  Coverage is strong (30.8% vs 30% target).

• Moderate: Papers naming benchmarks/datasets

  Coverage is usable but incomplete (30.8% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (46.2% vs 35% target).

• Moderate: Papers with known rater population

  Coverage is usable but incomplete (30.8% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (46.2% vs 35% target).

Strengths

• Strong human-feedback signal (53.8% of papers).
• Quality-control evidence appears in 30.8% of papers.
• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.

Known Gaps

• No dominant metadata gap detected in current extraction coverage.

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (APPS vs Paperbench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and agreement.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. CounselReflect: A Toolkit for Auditing Mental-Health Dialogues

   Start here for detailed protocol reporting and quality-control evidence. Signals: human evaluation + rubric ratings. Abstract: The system integrates two families of evaluation signals: (i) 12 model-based metrics.

2. Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

   Start here for detailed protocol reporting and quality-control evidence. Signals: human evaluation + rubric ratings. Focus: Rinobench. Abstract: Yet, evaluation of these approaches remains largely inconsistent and is.

3. Learning to Predict Future-Aligned Research Proposals with Language Models

   High citation traction makes this a strong baseline for protocol comparison. Signals: human evaluation. Focus: accuracy. Abstract: Large language models (LLMs) are increasingly used to assist ideation in.

4. Grounding Arabic LLMs in the Doha Historical Dictionary: Retrieval-Augmented Understanding of Quran and Hadith

   High citation traction makes this a strong baseline for protocol comparison. Signals: human evaluation. Focus: accuracy. Abstract: Gemini also serves as an LLM-as-a-judge system for automatic evaluation in.

5. IntelliAsk: Learning to Ask High-Quality Research Questions via RLVR

   Adds human evaluation with pairwise preferences for broader protocol coverage within this hub. Signals: human evaluation + pairwise preferences. Focus: Writingbench. Abstract: To address this gap, we curate.

6. Automated Coding of Communication Data Using ChatGPT: Consistency Across Subgroups

   Adds human evaluation with rubric ratings for broader protocol coverage within this hub. Signals: human evaluation + rubric ratings. Focus: accuracy. Abstract: Prior research has established that ChatGPT.

7. EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagation

   Adds human evaluation with pairwise preferences for broader protocol coverage within this hub. Signals: human evaluation + pairwise preferences. Focus: APPS. Abstract: To enhance video generation quality, we.

8. XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaboration

   Adds human evaluation with pairwise preferences for broader protocol coverage within this hub. Signals: human evaluation + pairwise preferences. Focus: coherence. Abstract: Both automated preference assessments and human.

Known Limitations

• No dominant metadata gap detected in current extraction coverage.
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
• Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.

Research Utility Snapshot