Llm As Judge + Pairwise Preference (Last 45 Days)

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (0% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (20% vs 35% target).

• Gap: Papers naming evaluation metrics

  Coverage is a replication risk (10% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (20% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (50% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).

Known Gaps

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (20% coverage).
• Metric coverage is thin (10% of papers mention reported metrics).

Suggested Next Analyses

• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
• Stratify by benchmark (Healthbench vs if-rewardbench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and agreement.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge + pairwise preferences. Focus: IFEval. Abstract: LLM-as-a-judge has become the de facto approach for evaluating LLM outputs.

2. Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge + pairwise preferences. Abstract: Large language models (LLMs) are increasingly used as automated evaluators (LLM-as-a-Judge).

3. Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge + pairwise preferences. Focus: accuracy. Abstract: Objective: To evaluate the educational suitability of LLM-generated Japanese translations of.

4. Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: LLM-as-judge + pairwise preferences. Abstract: Grocery shopping further amplifies these difficulties, as user requests are often underspecified,.

5. IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation

   Adds LLM-as-judge with pairwise preferences for broader protocol coverage within this hub. Signals: LLM-as-judge + pairwise preferences. Focus: if-rewardbench. Abstract: Instruction-following is a foundational capability of large language.

6. Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks

   Adds LLM-as-judge with pairwise preferences for broader protocol coverage within this hub. Signals: LLM-as-judge + pairwise preferences. Abstract: This has prompted evaluation frameworks that use LLM-as-judge protocols and.

7. Text-to-Stage: Spatial Layouts from Long-form Narratives

   Adds LLM-as-judge with pairwise preferences for broader protocol coverage within this hub. Signals: LLM-as-judge + pairwise preferences. Abstract: Experiments on a text-only corpus of classical English literature demonstrate.

8. Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge

   Adds LLM-as-judge with pairwise preferences for broader protocol coverage within this hub. Signals: LLM-as-judge + pairwise preferences. Abstract: Multimodal Large Language Models (MLLMs) have been widely adopted as.

Known Limitations

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (20% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot