General + Pairwise Preference (Last 90 Days)

Why This Matters For Eval Research

• 100% of papers report explicit human-feedback signals, led by pairwise preferences.
• automatic metrics appears in 31.6% of papers in this hub.
• DROP is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• 1 sampled papers report both human evaluation and LLM-as-judge, supporting direct agreement checks.
• Most common quality-control signal is inter-annotator agreement reporting (7.9% of papers).
• Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.

Benchmark Interpretation

• DROP appears in 2.6% of hub papers (1/38); use this cohort for benchmark-matched comparisons.
• LMSYS Chatbot Arena appears in 2.6% of hub papers (1/38); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• agreement is reported in 10.5% of hub papers (4/38); compare with a secondary metric before ranking methods.
• accuracy is reported in 7.9% of hub papers (3/38); compare with a secondary metric before ranking methods.

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (13.2% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (10.5% vs 35% target).

• Moderate: Papers naming evaluation metrics

  Coverage is usable but incomplete (31.6% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (13.2% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (31.6% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).
• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.

Known Gaps

• Only 13.2% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (13.2% coverage).
• Benchmark coverage is thin (10.5% of papers mention benchmarks/datasets).

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (DROP vs LMSYS Chatbot Arena) before comparing methods.
• Track metric sensitivity by reporting both agreement and accuracy.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Toward Expert Investment Teams:A Multi-Agent LLM System with Fine-Grained Trading Tasks

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: Crucially, further analysis of intermediate agent outputs suggests that alignment between analytical outputs and downstream.

2. ReCoN-Ipsundrum: An Inspectable Recurrent Persistence Loop Agent with Affect-Coupled Control and Mechanism-Linked Consciousness Indicator Assays

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Focus: DROP. Abstract: Across fixed-parameter ablations (ReCoN, Ipsundrum, Ipsundrum+affect), we operationalize Humphrey's qualiaphilia (preference for sensory.

3. Decentralized Ranking Aggregation: Gossip Algorithms for Borda and Copeland Consensus

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: The concept of ranking aggregation plays a central role in preference analysis, and numerous algorithms.

4. Validating Political Position Predictions of Arguments

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: Real-world knowledge representation often requires capturing subjective, continuous attributes.

5. HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: For each dialogue history, we pair human and model.

6. Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge + pairwise preferences. Abstract: Scalar reward models compress multi-dimensional human preferences into a single opaque.

7. SCOPE: Selective Conformal Optimized Pairwise LLM Judging

   Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: MT-Bench / error rate. Abstract: Large language models.

8. Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning

   Adds simulation environments with pairwise preferences for broader protocol coverage within this hub. Signals: simulation environments + pairwise preferences. Focus: latency. Abstract: Fast-ThinkAct learns to reason efficiently with.

Known Limitations

• Only 13.2% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (13.2% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot