General + Rubric Rating (Last 120 Days)

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (9.1% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (0% vs 35% target).

• Moderate: Papers naming evaluation metrics

  Coverage is usable but incomplete (27.3% vs 35% target).

• Moderate: Papers with known rater population

  Coverage is usable but incomplete (27.3% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (100% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).
• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.

Known Gaps

• Only 9.1% of papers report quality controls; prioritize calibration/adjudication evidence.
• Benchmark coverage is thin (0% of papers mention benchmarks/datasets).

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Track metric sensitivity by reporting both accuracy and agreement.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Optimizing In-Context Demonstrations for LLM-based Automated Grading

   Start here for detailed protocol reporting and quality-control evidence. Signals: rubric ratings. Abstract: Standard retrieval methods typically select examples based on semantic similarity, which often fails to capture.

2. LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: We present LFQA-HP-1M, a large-scale dataset comprising 1.3M human pairwise preference annotations for LFQA.

3. RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoning

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + rubric ratings. Focus: accuracy. Abstract: While recent rubric-based approaches offer fine-grained supervision signals, they suffer.

4. HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: For each dialogue history, we pair human and model.

5. Discovering Implicit Large Language Model Alignment Objectives

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + rubric ratings. Abstract: Existing interpretation methods typically rely on pre-defined rubrics, risking the omission.

6. Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge + pairwise preferences. Abstract: Scalar reward models compress multi-dimensional human preferences into a single opaque.

7. Personalized Prediction of Perceived Message Effectiveness Using Large Language Model Based Digital Twins

   Adds automatic metrics with rubric ratings for broader protocol coverage within this hub. Signals: automatic metrics + rubric ratings. Focus: accuracy. Abstract: Perceived message effectiveness (PME) by potential.

8. Multi-Agent Comedy Club: Investigating Community Discussion Effects on LLM Humor Generation

   Adds evaluation protocol evidence with pairwise preferences for broader protocol coverage within this hub. Signals: pairwise preferences. Abstract: Across 50 rounds (250 paired monologues) judged by five expert.

Known Limitations

• Only 9.1% of papers report quality controls; prioritize calibration/adjudication evidence.
• Benchmark coverage is thin (0% of papers mention benchmarks/datasets).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot