Coding + Rubric Rating Papers

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Moderate: Papers reporting quality controls

  Coverage is usable but incomplete (18.2% vs 30% target).

• Strong: Papers naming benchmarks/datasets

  Coverage is strong (36.4% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (36.4% vs 35% target).

• Strong: Papers with known rater population

  Coverage is strong (45.5% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (100% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).
• Most papers provide measurable evaluation context (36.4% benchmarks, 36.4% metrics).
• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.

Known Gaps

• Only 18.2% of papers report quality controls; prioritize calibration/adjudication evidence.
• LLM-as-judge appears without enough inter-annotator agreement reporting.

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (Kernelbench vs LongBench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and coherence.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Beyond the Resumé: A Rubric-Aware Automatic Interview System for Information Elicitation

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + rubric ratings. Focus: cost. Abstract: We present a system that leverages an LLM interviewer to.

2. Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: rubric ratings. Abstract: Existing benchmarks assess perception without interpretation, and common evaluation proxies, such as automated metrics and.

3. StitchCUDA: An Automated Multi-Agents End-to-End GPU Programing Framework with Rubric-based Agentic Reinforcement Learning

   High citation traction makes this a strong baseline for protocol comparison. Signals: automatic metrics + rubric ratings. Focus: Kernelbench / success rate. Abstract: To fundamentally improve the Coder's.

4. KLong: Training LLM Agent for Extremely Long-horizon Tasks

   High citation traction makes this a strong baseline for protocol comparison. Signals: rubric ratings. Focus: SWE-bench. Abstract: Then, we introduce Research-Factory, an automated pipeline that generates high-quality training.

5. Automated Coding of Communication Data Using ChatGPT: Consistency Across Subgroups

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + rubric ratings. Focus: accuracy. Abstract: Prior research has established that ChatGPT can be directly.

6. Small Reward Models via Backward Inference

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge + rubric ratings. Abstract: However, the dominant LLM-as-a-Judge paradigm relies on the strong reasoning capabilities.

7. Document Reconstruction Unlocks Scalable Long-Context RLVR

   Adds automatic metrics with rubric ratings for broader protocol coverage within this hub. Signals: automatic metrics + rubric ratings. Focus: LongBench / coherence. Abstract: However, it often relies.

8. ScholarEval: Research Idea Evaluation Grounded in Literature

   Adds evaluation protocol evidence with rubric ratings for broader protocol coverage within this hub. Signals: rubric ratings. Focus: Scholareval. Abstract: Our evaluation shows that ScholarEval achieves significantly higher.

Known Limitations

• Only 18.2% of papers report quality controls; prioritize calibration/adjudication evidence.
• LLM-as-judge appears without enough inter-annotator agreement reporting.
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot