Pairwise Preference Papers (Last 45 Days)

Why This Matters For Eval Research

• 100% of papers report explicit human-feedback signals, led by pairwise preferences.
• automatic metrics appears in 36.5% of papers in this hub.
• LiveCodeBench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is inter-annotator agreement reporting (5.8% of papers).
• Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.
• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.

Benchmark Interpretation

• LiveCodeBench appears in 3.8% of hub papers (2/52); use this cohort for benchmark-matched comparisons.
• AIME appears in 1.9% of hub papers (1/52); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• cost is reported in 9.6% of hub papers (5/52); compare with a secondary metric before ranking methods.
• accuracy is reported in 7.7% of hub papers (4/52); compare with a secondary metric before ranking methods.

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (9.6% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (15.4% vs 35% target).

• Moderate: Papers naming evaluation metrics

  Coverage is usable but incomplete (32.7% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (17.3% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (36.5% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).
• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.
• Agentic evaluation appears in 25% of papers.

Known Gaps

• Only 9.6% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (17.3% coverage).
• Benchmark coverage is thin (15.4% of papers mention benchmarks/datasets).

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (LiveCodeBench vs AIME) before comparing methods.
• Track metric sensitivity by reporting both cost and accuracy.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Toward Expert Investment Teams:A Multi-Agent LLM System with Fine-Grained Trading Tasks

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: Crucially, further analysis of intermediate agent outputs suggests that alignment between analytical outputs and downstream.

2. ReCoN-Ipsundrum: An Inspectable Recurrent Persistence Loop Agent with Affect-Coupled Control and Mechanism-Linked Consciousness Indicator Assays

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Focus: DROP. Abstract: Across fixed-parameter ablations (ReCoN, Ipsundrum, Ipsundrum+affect), we operationalize Humphrey's qualiaphilia (preference for sensory.

3. Decentralized Ranking Aggregation: Gossip Algorithms for Borda and Copeland Consensus

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: The concept of ranking aggregation plays a central role in preference analysis, and numerous algorithms.

4. Validating Political Position Predictions of Arguments

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: Real-world knowledge representation often requires capturing subjective, continuous attributes.

5. Think$^{2}$: Grounded Metacognitive Reasoning in Large Language Models

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: GSM8K. Abstract: Blinded human evaluations over 580 query pairs show an.

6. Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge + pairwise preferences. Abstract: Scalar reward models compress multi-dimensional human preferences into a single opaque.

7. SCOPE: Selective Conformal Optimized Pairwise LLM Judging

   Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: MT-Bench / error rate. Abstract: Large language models.

8. Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters

   Adds evaluation protocol evidence with pairwise preferences for broader protocol coverage within this hub. Signals: pairwise preferences. Focus: LiveCodeBench / latency. Abstract: To reach frontier-level intelligence, we design.

Known Limitations

• Only 9.6% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (17.3% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot