Llm As Judge Papers (Last 45 Days)

Why This Matters For Eval Research

• 18.2% of papers report explicit human-feedback signals, led by rubric ratings.
• LLM-as-judge appears in 68.8% of papers in this hub.
• Innoeval is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• 1 sampled papers report both human evaluation and LLM-as-judge, supporting direct agreement checks.
• Most common quality-control signal is adjudication (6.3% of papers).
• Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.

Benchmark Interpretation

• Innoeval appears in 9.1% of hub papers (1/16); use this cohort for benchmark-matched comparisons.
• Mind2Web appears in 9.1% of hub papers (1/16); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 36.4% of hub papers (4/16); compare with a secondary metric before ranking methods.
• bleu is reported in 9.1% of hub papers (1/16); compare with a secondary metric before ranking methods.

Researcher Checklist

• Gap: Papers with explicit human feedback

  Coverage is a replication risk (18.2% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (9.1% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (18.2% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (54.5% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (18.2% vs 35% target).

• Gap: Papers with known annotation unit

  Coverage is a replication risk (18.2% vs 35% target).

Strengths

• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.
• Agentic evaluation appears in 27.3% of papers.

Known Gaps

• Only 9.1% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (18.2% coverage).
• Annotation unit is under-specified (18.2% coverage).

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (Innoeval vs Mind2Web) before comparing methods.
• Track metric sensitivity by reporting both accuracy and bleu.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Overton Pluralistic Reinforcement Learning for Large Language Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge. Focus: accuracy. Abstract: Additional evaluations using GPT-4.1 as a large language model judge further confirm the robustness.

2. Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge. Focus: coherence. Abstract: Evaluations show STAR-LDM significantly outperforms similar-sized models on language understanding benchmarks and achieves $>70\%$.

3. AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summarization

   Start here for detailed protocol reporting and quality-control evidence. Signals: human evaluation. Abstract: We evaluate AgenticSum on two public datasets, using reference-based metrics, LLM-as-a-judge assessment, and human evaluation.

4. InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: LLM-as-judge. Focus: Innoeval. Abstract: However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation.

5. Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge + pairwise preferences. Abstract: Scalar reward models compress multi-dimensional human preferences into a single opaque.

6. Small Reward Models via Backward Inference

   Adds LLM-as-judge with rubric ratings for broader protocol coverage within this hub. Signals: LLM-as-judge + rubric ratings. Abstract: However, the dominant LLM-as-a-Judge paradigm relies on the strong reasoning.

7. World-Model-Augmented Web Agents with Action Correction

   Adds LLM-as-judge for broader protocol coverage within this hub. Signals: LLM-as-judge. Focus: VisualWebArena. Abstract: A world model, specialized in environmental state transitions, simulates action outcomes, which a judge.

8. The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

   Adds LLM-as-judge for broader protocol coverage within this hub. Signals: LLM-as-judge. Focus: accuracy. Abstract: As Large Language Models (LLMs) transition from standalone chat interfaces to foundational reasoning layers.

Known Limitations

• Only 9.1% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (18.2% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot