Llm As Judge + Pairwise Preference (Last 120 Days)

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (0% vs 30% target).

• Moderate: Papers naming benchmarks/datasets

  Coverage is usable but incomplete (21.4% vs 35% target).

• Moderate: Papers naming evaluation metrics

  Coverage is usable but incomplete (21.4% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (14.3% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (57.1% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).
• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.

Known Gaps

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (14.3% coverage).
• LLM-as-judge appears without enough inter-annotator agreement reporting.

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (APPS vs Healthbench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and agreement.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. HyperMem: Hypergraph Memory for Long-Term Conversations

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge + pairwise preferences. Focus: accuracy. Abstract: However, existing approaches as Retrieval-Augmented Generation (RAG) and graph-based memory mostly.

2. Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge + pairwise preferences. Focus: IFEval. Abstract: LLM-as-a-judge has become the de facto approach for evaluating LLM outputs.

3. Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

   Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge + pairwise preferences. Abstract: Large language models (LLMs) are increasingly used as automated evaluators (LLM-as-a-Judge).

4. HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: For each dialogue history, we pair human and model.

5. Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: LLM-as-judge + pairwise preferences. Focus: accuracy. Abstract: Objective: To evaluate the educational suitability of LLM-generated Japanese translations.

6. Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge + pairwise preferences. Abstract: Grocery shopping further amplifies these difficulties, as user requests are often.

7. IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation

   Adds LLM-as-judge with pairwise preferences for broader protocol coverage within this hub. Signals: LLM-as-judge + pairwise preferences. Focus: if-rewardbench. Abstract: Instruction-following is a foundational capability of large language.

8. WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics

   Adds LLM-as-judge with pairwise preferences for broader protocol coverage within this hub. Signals: LLM-as-judge + pairwise preferences. Focus: LMSYS Chatbot Arena. Abstract: WebCoderBench provides 24 fine-grained evaluation metrics.

Known Limitations

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (14.3% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot