Automatic Metrics + Pairwise Preference (Last 45 Days)

Why This Matters For Eval Research

• 100% of papers report explicit human-feedback signals, led by pairwise preferences.
• automatic metrics appears in 100% of papers in this hub.
• Charteditbench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is inter-annotator agreement reporting (10.5% of papers).
• Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.
• Stratify by benchmark (Charteditbench vs LiveCodeBench) before comparing methods.

Benchmark Interpretation

• Charteditbench appears in 5.3% of hub papers (1/19); use this cohort for benchmark-matched comparisons.
• LiveCodeBench appears in 5.3% of hub papers (1/19); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 21.1% of hub papers (4/19); compare with a secondary metric before ranking methods.
• agreement is reported in 15.8% of hub papers (3/19); compare with a secondary metric before ranking methods.

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (15.8% vs 30% target).

• Moderate: Papers naming benchmarks/datasets

  Coverage is usable but incomplete (21.1% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (68.4% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (15.8% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (42.1% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).

Known Gaps

• Only 15.8% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (15.8% coverage).

Suggested Next Analyses

• Stratify by benchmark (Charteditbench vs LiveCodeBench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and agreement.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. RLHFless: Serverless Computing for Efficient RLHF

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + pairwise preferences. Focus: cost. Abstract: Reinforcement Learning from Human Feedback (RLHF) has been widely applied.

2. Same Words, Different Judgments: Modality Effects on Preference Alignment

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + pairwise preferences. Focus: agreement. Abstract: Preference-based reinforcement learning (PbRL) is the dominant framework for aligning.

3. DynamicGTR: Leveraging Graph Topology Representation Preferences to Boost VLM Capabilities on Graph QAs

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + pairwise preferences. Focus: accuracy. Abstract: This ``one-size-fits-all'' strategy often neglects model-specific and task-specific preferences, resulting.

4. SCOPE: Selective Conformal Optimized Pairwise LLM Judging

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + pairwise preferences. Focus: MT-Bench / error rate. Abstract: Large language models (LLMs) are increasingly.

5. MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + pairwise preferences. Focus: Memoryarena / recall. Abstract: MemoryArena supports evaluation across web navigation, preference-constrained.

6. Decoupling Strategy and Execution in Task-Focused Dialogue via Goal-Oriented Preference Optimization

   Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: task success. Abstract: Large language models show potential.

7. Yor-Sarc: A gold-standard dataset for sarcasm detection in a low-resource African language

   Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: agreement. Abstract: The dataset comprises 436 instances annotated.

8. Duel-Evolve: Reward-Free Test-Time Scaling via LLM Self-Preferences

   Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: LiveCodeBench / accuracy. Abstract: Pairwise comparisons, by contrast,.

Known Limitations

• Only 15.8% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (15.8% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot