HFEPX Benchmark Hub

LiveCodeBench In CS.CL Papers

Updated from current HFEPX corpus (Mar 17, 2026). 5 papers are grouped in this benchmark page.

Read Full Context

Updated from current HFEPX corpus (Mar 17, 2026). 5 papers are grouped in this benchmark page. Common evaluation modes: Automatic Metrics. Most common rater population: Domain Experts. Common annotation unit: Pairwise. Frequent quality control: Calibration. Frequently cited benchmark: LiveCodeBench. Common metric signal: accuracy. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 18, 2026.

Papers: 5 Last published: Feb 18, 2026 Global RSS

Researcher Quick Triage

Use this page for benchmark-matched method comparisons and eval protocol selection. Quality band: Developing .

High-Signal Coverage

100.0%

5 / 5 sampled papers are not low-signal flagged.

Replication-Ready Set

Papers with explicit benchmark + metric + eval mode fields.

Quality Controls

20.0%

1 papers report calibration/adjudication/IAA controls.

5 papers explicitly name benchmark datasets in the sampled set.
3 papers report at least one metric term in metadata extraction.
Start with the ranked shortlist below before reading all papers.

Primary action: Use this page to map benchmark mentions first; wait for stronger metric/QC coverage before strict comparisons.

Why This Matters (Expanded)

Why This Matters For Eval Research

100% of papers report explicit human-feedback signals, led by pairwise preferences.
automatic metrics appears in 40% of papers in this hub.
LiveCodeBench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Notes (Expanded)

Protocol Takeaways

Most common quality-control signal is rater calibration (20% of papers).
Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.
Stratify by benchmark (LiveCodeBench vs AIME) before comparing methods.

Benchmark Interpretation

LiveCodeBench appears in 100% of hub papers (5/5); use this cohort for benchmark-matched comparisons.
AIME appears in 20% of hub papers (1/5); use this cohort for benchmark-matched comparisons.

Metric Interpretation

accuracy is reported in 20% of hub papers (1/5); compare with a secondary metric before ranking methods.
cost is reported in 20% of hub papers (1/5); compare with a secondary metric before ranking methods.

Start Here (Benchmark-Matched First 6)

Ranked by protocol completeness so you can quickly find papers suitable for comparison studies.

$V_1$: Unifying Generation and Self-Verification for Parallel Reasoners
Mar 4, 2026 · Citations: 0 · Score: 8.5

Eval: Automatic Metrics · Metrics: Pass@1
Duel-Evolve: Reward-Free Test-Time Scaling via LLM Self-Preferences
Feb 25, 2026 · Citations: 0 · Score: 8.5

Eval: Automatic Metrics · Metrics: Accuracy
Team of Thoughts: Efficient Test-time Scaling of Agentic Systems through Orchestrated Tool Calling
Feb 18, 2026 · Citations: 0 · Score: 8.0

Eval: Not reported · Metrics: Not Reported
Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters
Feb 11, 2026 · Citations: 0 · Score: 8.0

Eval: Not reported · Metrics: Latency
Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning
Sep 26, 2025 · Citations: 0 · Score: 5.5

Eval: Not reported · Metrics: Not Reported

Protocol Matrix (Top 10)

Compare protocol ingredients quickly before deep-reading full papers.

Paper	Eval Modes	Human Feedback	Metrics	Quality Controls
$V_1$: Unifying Generation and Self-Verification for Parallel Reasoners Mar 4, 2026	Automatic Metrics	Pairwise Preference	Pass@1	Not reported
Duel-Evolve: Reward-Free Test-Time Scaling via LLM Self-Preferences Feb 25, 2026	Automatic Metrics	Pairwise Preference	Accuracy	Not reported
Team of Thoughts: Efficient Test-time Scaling of Agentic Systems through Orchestrated Tool Calling Feb 18, 2026	Not reported	Expert Verification	Not reported	Calibration
Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters Feb 11, 2026	Not reported	Pairwise Preference	Latency, Cost	Not reported
Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning Sep 26, 2025	Not reported	Critique Edit	Not reported	Not reported

Researcher Workflow (Detailed)

Checklist

Strong: Papers with explicit human feedback

Coverage is strong (100% vs 45% target).
Moderate: Papers reporting quality controls

Coverage is usable but incomplete (20% vs 30% target).
Strong: Papers naming benchmarks/datasets

Coverage is strong (100% vs 35% target).
Strong: Papers naming evaluation metrics

Coverage is strong (60% vs 35% target).
Strong: Papers with known rater population

Coverage is strong (40% vs 35% target).
Strong: Papers with known annotation unit

Coverage is strong (40% vs 35% target).

Strengths

Strong human-feedback signal (100% of papers).
Most papers provide measurable evaluation context (100% benchmarks, 60% metrics).
Agentic evaluation appears in 40% of papers.

Known Gaps

No dominant metadata gap detected in current extraction coverage.

Suggested Next Analyses

Stratify by benchmark (LiveCodeBench vs AIME) before comparing methods.
Track metric sensitivity by reporting both accuracy and cost.
Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

Benchmark Slice: LiveCodeBench Metric Slice: accuracy Recent High-Signal Papers

Known Limitations

No dominant metadata gap detected in current extraction coverage.
Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.

Research Utility Snapshot (Detailed)

Evaluation Modes

Automatic Metrics (2)

Human Feedback Mix

Pairwise Preference (3)
Critique Edit (1)
Expert Verification (1)

Top Benchmarks

LiveCodeBench (5)
AIME (1)
BrowseComp (1)
CodeContests (1)

Top Metrics

Accuracy (1)
Cost (1)
Latency (1)
Pass@1 (1)

Top Papers On This Benchmark

Team of Thoughts: Efficient Test-time Scaling of Agentic Systems through Orchestrated Tool Calling
Jeffrey T. H. Wong, Zixi Zhang, Junyi Liu, Yiren Zhao · Feb 18, 2026 · Citations: 0

Expert Verification

Existing Multi-Agent Systems (MAS) typically rely on static, homogeneous model configurations, limiting their ability to exploit the distinct strengths of differently post-trained models.
Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters
Ailin Huang, Ang Li, Aobo Kong, Bin Wang, Binxing Jiao · Feb 11, 2026 · Citations: 0

Pairwise Preference

We introduce Step 3.5 Flash, a sparse Mixture-of-Experts (MoE) model that bridges frontier-level agentic intelligence and computational efficiency.
$V_1$: Unifying Generation and Self-Verification for Parallel Reasoners
Harman Singh, Xiuyu Li, Kusha Sareen, Monishwaran Maheswaran, Sijun Tan · Mar 4, 2026 · Citations: 0

Pairwise Preference Automatic Metrics

On code generation (LiveCodeBench, CodeContests, SWE-Bench) and math reasoning (AIME, HMMT) benchmarks, V_1-Infer improves Pass@1 by up to 10% over pointwise verification and outperforms recent test-time scaling methods while being…
Duel-Evolve: Reward-Free Test-Time Scaling via LLM Self-Preferences
Sweta Karlekar, Carolina Zheng, Magnus Saebo, Nicolas Beltran-Velez, Shuyang Yu · Feb 25, 2026 · Citations: 0

Pairwise Preference Automatic Metrics

Building on this observation, we introduce Duel-Evolve, an evolutionary optimization algorithm that replaces external scalar rewards with pairwise preferences elicited from the same LLM used to generate candidates.
Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning
Chi Ruan, Dongfu Jiang, Yubo Wang, Wenhu Chen · Sep 26, 2025 · Citations: 0

Critique Edit

We fine-tune multiple models (Critique-Coder) and evaluate them on different benchmarks to show their advantages over RL-only models.

Related Benchmark Hubs

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote