Coding + Pairwise Preference (Last 45 Days)

Why This Matters For Eval Research

• 100% of papers report explicit human-feedback signals, led by pairwise preferences.
• automatic metrics appears in 50% of papers in this hub.
• BrowseComp is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
• Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.
• Stratify by benchmark (BrowseComp vs Charteditbench) before comparing methods.

Benchmark Interpretation

• BrowseComp appears in 10% of hub papers (1/10); use this cohort for benchmark-matched comparisons.
• Charteditbench appears in 10% of hub papers (1/10); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• cost is reported in 10% of hub papers (1/10); compare with a secondary metric before ranking methods.
• helpfulness is reported in 10% of hub papers (1/10); compare with a secondary metric before ranking methods.

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (0% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (20% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (40% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (20% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (40% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).
• Agentic evaluation appears in 40% of papers.

Known Gaps

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (20% coverage).

Suggested Next Analyses

• Stratify by benchmark (BrowseComp vs Charteditbench) before comparing methods.
• Track metric sensitivity by reporting both cost and helpfulness.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. gencat: Generative computerized adaptive testing

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: We train the model in a two-step process, first via Supervised Fine-Tuning and then via.

2. ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + pairwise preferences. Focus: Charteditbench. Abstract: In practice, users iteratively refine visualizations through multi-turn interactions that.

3. Rethinking Metrics for Lexical Semantic Change Detection

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + pairwise preferences. Abstract: Lexical semantic change detection (LSCD) increasingly relies on contextualised language model embeddings,.

4. Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: pairwise preferences. Focus: LiveCodeBench / latency. Abstract: To reach frontier-level intelligence, we design a scalable reinforcement learning.

5. Decoupling Strategy and Execution in Task-Focused Dialogue via Goal-Oriented Preference Optimization

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + pairwise preferences. Focus: task success. Abstract: Large language models show potential in task-oriented dialogue.

6. PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training

   Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: helpfulness. Abstract: By embedding privacy preferences into each.

7. The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

   Adds evaluation protocol evidence with pairwise preferences for broader protocol coverage within this hub. Signals: pairwise preferences. Abstract: Our framework adopts a hub-and-spoke topology to reduce pairwise alignment.

8. Bridging the Multilingual Safety Divide: Efficient, Culturally-Aware Alignment for Global South Languages

   Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: toxicity. Abstract: In response, we outline a practical.

Known Limitations

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (20% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot