LiveCodeBench + Coding Benchmark Papers

Why This Matters For Eval Research

• 100% of papers report explicit human-feedback signals, led by pairwise preferences.
• automatic metrics appears in 25% of papers in this hub.
• LiveCodeBench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is rater calibration (25% of papers).
• Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.
• Stratify by benchmark (LiveCodeBench vs AIME) before comparing methods.

Benchmark Interpretation

• LiveCodeBench appears in 100% of hub papers (4/4); use this cohort for benchmark-matched comparisons.
• AIME appears in 25% of hub papers (1/4); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• cost is reported in 25% of hub papers (1/4); compare with a secondary metric before ranking methods.
• latency is reported in 25% of hub papers (1/4); compare with a secondary metric before ranking methods.

Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Moderate: Papers reporting quality controls

  Coverage is usable but incomplete (25% vs 30% target).

• Strong: Papers naming benchmarks/datasets

  Coverage is strong (100% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (50% vs 35% target).

• Strong: Papers with known rater population

  Coverage is strong (50% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (25% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).
• Most papers provide measurable evaluation context (100% benchmarks, 50% metrics).
• Agentic evaluation appears in 50% of papers.

Known Gaps

• No dominant metadata gap detected in current extraction coverage.

Suggested Next Analyses

• Stratify by benchmark (LiveCodeBench vs AIME) before comparing methods.
• Track metric sensitivity by reporting both cost and latency.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

• No dominant metadata gap detected in current extraction coverage.
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
• Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.