Llm As Judge + General (Last 90 Days)

Why This Matters For Eval Research

• 30% of papers report explicit human-feedback signals, led by pairwise preferences.
• LLM-as-judge appears in 100% of papers in this hub.
• Innoeval is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• 1 sampled papers report both human evaluation and LLM-as-judge, supporting direct agreement checks.
• Most common quality-control signal is adjudication (10% of papers).
• Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.

Benchmark Interpretation

• Innoeval appears in 10% of hub papers (1/10); use this cohort for benchmark-matched comparisons.
• Jailbreakbench appears in 10% of hub papers (1/10); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 30% of hub papers (3/10); compare with a secondary metric before ranking methods.
• agreement is reported in 10% of hub papers (1/10); compare with a secondary metric before ranking methods.

Researcher Checklist

• Moderate: Papers with explicit human feedback

  Coverage is usable but incomplete (30% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (10% vs 30% target).

• Moderate: Papers naming benchmarks/datasets

  Coverage is usable but incomplete (30% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (60% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (20% vs 35% target).

• Gap: Papers with known annotation unit

  Coverage is a replication risk (20% vs 35% target).

Strengths

• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.
• Agentic evaluation appears in 30% of papers.

Known Gaps

• Only 10% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (20% coverage).
• Annotation unit is under-specified (20% coverage).

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (Innoeval vs Jailbreakbench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and agreement.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Overton Pluralistic Reinforcement Learning for Large Language Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge. Focus: accuracy. Abstract: Additional evaluations using GPT-4.1 as a large language model judge further confirm the robustness.

2. Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge. Focus: coherence. Abstract: Evaluations show STAR-LDM significantly outperforms similar-sized models on language understanding benchmarks and achieves $>70\%$.

3. Luna-2: Scalable Single-Token Evaluation with Small Language Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge. Focus: accuracy. Abstract: Real-time guardrails require evaluation that is accurate, cheap, and fast - yet today's default,.

4. HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: For each dialogue history, we pair human and model.

5. InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: LLM-as-judge. Focus: Innoeval. Abstract: However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation.

6. Refusal Steering: Fine-grained Control over LLM Refusal Behaviour for Sensitive Topics

   Adds LLM-as-judge with red-team protocols for broader protocol coverage within this hub. Signals: LLM-as-judge + red-team protocols. Focus: Jailbreakbench. Abstract: We replace fragile pattern-based refusal detection with an.

7. Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

   Adds LLM-as-judge with pairwise preferences for broader protocol coverage within this hub. Signals: LLM-as-judge + pairwise preferences. Abstract: Scalar reward models compress multi-dimensional human preferences into a single.

8. World-Model-Augmented Web Agents with Action Correction

   Adds LLM-as-judge for broader protocol coverage within this hub. Signals: LLM-as-judge. Focus: VisualWebArena. Abstract: A world model, specialized in environmental state transitions, simulates action outcomes, which a judge.

Known Limitations

• Only 10% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (20% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot