Llm As Judge + General (Last 45 Days)

Researcher Checklist

• Moderate: Papers with explicit human feedback

  Coverage is usable but incomplete (28.6% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (7.1% vs 30% target).

• Moderate: Papers naming benchmarks/datasets

  Coverage is usable but incomplete (28.6% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (35.7% vs 35% target).

• Moderate: Papers with known rater population

  Coverage is usable but incomplete (21.4% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (42.9% vs 35% target).

Strengths

• Agentic evaluation appears in 42.9% of papers.

Known Gaps

• Only 7.1% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (21.4% coverage).
• LLM-as-judge appears without enough inter-annotator agreement reporting.

Suggested Next Analyses

• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
• Stratify by benchmark (Emobench vs Eq-Bench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and cost.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge + pairwise preferences. Abstract: Checklists have emerged as a popular approach for interpretable and fine-grained evaluation, particularly.

2. Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge + pairwise preferences. Abstract: This has prompted evaluation frameworks that use LLM-as-judge protocols and claim verification, along.

3. MAPO: Mixed Advantage Policy Optimization for Long-Horizon Multi-Turn Dialogue

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge. Focus: Emobench. Abstract: We propose a critic-free and efficient RL algorithm named MAPO that leverages dense process.

4. Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: LLM-as-judge + pairwise preferences. Abstract: Grocery shopping further amplifies these difficulties, as user requests are often underspecified,.

5. InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: LLM-as-judge. Focus: Innoeval. Abstract: However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation.

6. Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

   Adds LLM-as-judge with pairwise preferences for broader protocol coverage within this hub. Signals: LLM-as-judge + pairwise preferences. Abstract: Scalar reward models compress multi-dimensional human preferences into a single.

7. World-Model-Augmented Web Agents with Action Correction

   Adds LLM-as-judge for broader protocol coverage within this hub. Signals: LLM-as-judge. Focus: VisualWebArena. Abstract: A world model, specialized in environmental state transitions, simulates action outcomes, which a judge.

8. The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

   Adds LLM-as-judge for broader protocol coverage within this hub. Signals: LLM-as-judge. Focus: accuracy. Abstract: As Large Language Models (LLMs) transition from standalone chat interfaces to foundational reasoning layers.

Known Limitations

• Only 7.1% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (21.4% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot