HFEPX Benchmark Hub

SWE-Bench Ecosystem Benchmark Papers

Updated from current HFEPX corpus (Mar 8, 2026). 10 papers are grouped in this benchmark page.

Read Full Context

Updated from current HFEPX corpus (Mar 8, 2026). 10 papers are grouped in this benchmark page. Common evaluation modes: Automatic Metrics. Most common rater population: Domain Experts. Common annotation unit: Multi Dim Rubric. Frequent quality control: Calibration. Frequently cited benchmark: LiveCodeBench. Common metric signal: cost. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 18, 2026.

Papers: 10 Last published: Feb 18, 2026 Global RSS

Researcher Quick Triage

Use this page for benchmark-matched method comparisons and eval protocol selection. Quality band: Developing .

High-Signal Coverage

100.0%

10 / 10 sampled papers are not low-signal flagged.

Replication-Ready Set

Papers with explicit benchmark + metric + eval mode fields.

Quality Controls

10.0%

1 papers report calibration/adjudication/IAA controls.

6 papers explicitly name benchmark datasets in the sampled set.
3 papers report at least one metric term in metadata extraction.
Start with the ranked shortlist below before reading all papers.

Primary action: Use this page to map benchmark mentions first; wait for stronger metric/QC coverage before strict comparisons.

Why This Matters (Expanded)

Why This Matters For Eval Research

66.7% of papers report explicit human-feedback signals, led by pairwise preferences.
automatic metrics appears in 30% of papers in this hub.
LiveCodeBench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Notes (Expanded)

Protocol Takeaways

Most common quality-control signal is rater calibration (10% of papers).
Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.
Stratify by benchmark (LiveCodeBench vs SWE-bench) before comparing methods.

Benchmark Interpretation

LiveCodeBench appears in 50% of hub papers (3/10); use this cohort for benchmark-matched comparisons.
SWE-bench appears in 50% of hub papers (3/10); use this cohort for benchmark-matched comparisons.

Metric Interpretation

cost is reported in 33.3% of hub papers (2/10); compare with a secondary metric before ranking methods.
latency is reported in 33.3% of hub papers (2/10); compare with a secondary metric before ranking methods.

Start Here (Benchmark-Matched First 6)

Ranked by protocol completeness so you can quickly find papers suitable for comparison studies.

Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters
Feb 11, 2026 · Citations: 0 · Score: 8.5

Eval: Not reported · Metrics: Latency
Duel-Evolve: Reward-Free Test-Time Scaling via LLM Self-Preferences
Feb 25, 2026 · Citations: 0 · Score: 8.5

Eval: Automatic Metrics · Metrics: Accuracy
Team of Thoughts: Efficient Test-time Scaling of Agentic Systems through Orchestrated Tool Calling
Feb 18, 2026 · Citations: 0 · Score: 8.0

Eval: Not reported · Metrics: Not Reported
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Feb 25, 2026 · Citations: 0 · Score: 7.0

Eval: Automatic Metrics · Metrics: Pass@1
KLong: Training LLM Agent for Extremely Long-horizon Tasks
Feb 19, 2026 · Citations: 0 · Score: 6.5

Eval: Not reported · Metrics: Not Reported
Structurally Aligned Subtask-Level Memory for Software Engineering Agents
Feb 25, 2026 · Citations: 0 · Score: 5.0

Eval: Automatic Metrics · Metrics: Not Reported

Protocol Matrix (Top 10)

Compare protocol ingredients quickly before deep-reading full papers.

Paper	Eval Modes	Human Feedback	Metrics	Quality Controls
Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters Feb 11, 2026	Not reported	Pairwise Preference	Latency, Cost	Not reported
Duel-Evolve: Reward-Free Test-Time Scaling via LLM Self-Preferences Feb 25, 2026	Automatic Metrics	Pairwise Preference	Accuracy	Not reported
Team of Thoughts: Efficient Test-time Scaling of Agentic Systems through Orchestrated Tool Calling Feb 18, 2026	Not reported	Expert Verification	Not reported	Calibration
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents Feb 25, 2026	Automatic Metrics	Not reported	Pass@1, Latency	Not reported
KLong: Training LLM Agent for Extremely Long-horizon Tasks Feb 19, 2026	Not reported	Rubric Rating	Not reported	Not reported
Structurally Aligned Subtask-Level Memory for Software Engineering Agents Feb 25, 2026	Automatic Metrics	Not reported	Not reported	Not reported
$V_1$: Unifying Generation and Self-Verification for Parallel Reasoners Mar 4, 2026	Not reported	Not reported	Not reported	Not reported
SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration Mar 4, 2026	Not reported	Not reported	Not reported	Not reported
Qwen3-Coder-Next Technical Report Feb 28, 2026	Not reported	Not reported	Not reported	Not reported
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale Feb 27, 2026	Not reported	Not reported	Not reported	Not reported

Researcher Workflow (Detailed)

Checklist

Strong: Papers with explicit human feedback

Coverage is strong (66.7% vs 45% target).
Gap: Papers reporting quality controls

Coverage is a replication risk (16.7% vs 30% target).
Strong: Papers naming benchmarks/datasets

Coverage is strong (100% vs 35% target).
Strong: Papers naming evaluation metrics

Coverage is strong (66.7% vs 35% target).
Strong: Papers with known rater population

Coverage is strong (50% vs 35% target).
Strong: Papers with known annotation unit

Coverage is strong (50% vs 35% target).

Strengths

Strong human-feedback signal (66.7% of papers).
Most papers provide measurable evaluation context (100% benchmarks, 66.7% metrics).
Agentic evaluation appears in 83.3% of papers.

Known Gaps

Only 16.7% of papers report quality controls; prioritize calibration/adjudication evidence.

Suggested Next Analyses

Stratify by benchmark (LiveCodeBench vs SWE-bench) before comparing methods.
Track metric sensitivity by reporting both cost and latency.
Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

Benchmark Slice: LiveCodeBench Metric Slice: cost Recent High-Signal Papers

Known Limitations

Only 16.7% of papers report quality controls; prioritize calibration/adjudication evidence.
Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.

Research Utility Snapshot (Detailed)

Evaluation Modes

Automatic Metrics (3)

Human Feedback Mix

Pairwise Preference (2)
Expert Verification (1)
Rubric Rating (1)

Top Benchmarks

LiveCodeBench (3)
SWE Bench (3)
SWE Bench Verified (3)
BrowseComp (1)

Top Metrics

Cost (2)
Latency (2)
Pass@1 (2)
Accuracy (1)

Top Papers On This Benchmark

Team of Thoughts: Efficient Test-time Scaling of Agentic Systems through Orchestrated Tool Calling
Jeffrey T. H. Wong, Zixi Zhang, Junyi Liu, Yiren Zhao · Feb 18, 2026 · Citations: 0

Expert Verification

Existing Multi-Agent Systems (MAS) typically rely on static, homogeneous model configurations, limiting their ability to exploit the distinct strengths of differently post-trained models.
Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters
Ailin Huang, Ang Li, Aobo Kong, Bin Wang, Binxing Jiao · Feb 11, 2026 · Citations: 0

Pairwise Preference

We introduce Step 3.5 Flash, a sparse Mixture-of-Experts (MoE) model that bridges frontier-level agentic intelligence and computational efficiency.
Duel-Evolve: Reward-Free Test-Time Scaling via LLM Self-Preferences
Sweta Karlekar, Carolina Zheng, Magnus Saebo, Nicolas Beltran-Velez, Shuyang Yu · Feb 25, 2026 · Citations: 0

Pairwise Preference Automatic Metrics

Building on this observation, we introduce Duel-Evolve, an evolutionary optimization algorithm that replaces external scalar rewards with pairwise preferences elicited from the same LLM used to generate candidates.
KLong: Training LLM Agent for Extremely Long-horizon Tasks
Yue Liu, Zhiyuan Hu, Flood Sung, Jiaheng Zhang, Bryan Hooi · Feb 19, 2026 · Citations: 0

Rubric Rating

Then, we introduce Research-Factory, an automated pipeline that generates high-quality training data by collecting research papers and constructing evaluation rubrics.
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Patrick Tser Jern Kon, Archana Pradeep, Ang Chen, Alexander P. Ellis, Warren Hunt · Feb 25, 2026 · Citations: 0

Automatic Metrics

Our approach combines supervised fine-tuning on expert-augmented trajectories with agentic reinforcement learning that explicitly discourages degenerative looping and unproductive expert collaboration.
Structurally Aligned Subtask-Level Memory for Software Engineering Agents
Kangning Shen, Jingyuan Zhang, Chenxi Sun, Wencong Zeng, Yang Yue · Feb 25, 2026 · Citations: 0

Automatic Metrics

Large Language Models (LLMs) have demonstrated significant potential as autonomous software engineering (SWE) agents.
$V_1$: Unifying Generation and Self-Verification for Parallel Reasoners
Harman Singh, Xiuyu Li, Kusha Sareen, Monishwaran Maheswaran, Sijun Tan · Mar 4, 2026 · Citations: 0
SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
Jialong Chen, Xander Xu, Hu Wei, Chuan Chen, Bing Zhao · Mar 4, 2026 · Citations: 0
Qwen3-Coder-Next Technical Report
Ruisheng Cao, Mouxiang Chen, Jiawei Chen, Zeyu Cui, Yunlong Feng · Feb 28, 2026 · Citations: 0
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Alexander Golubev · Feb 27, 2026 · Citations: 0

Related Benchmark Hubs

SWE-bench Benchmark Papers SWE-bench In CS.CL Papers HFEPX Quarterly Archive: 2026-Q1 CS.CL Papers (Last 30 Days) CS.CL Papers (Last 60 Days) CS.CL Papers (Last 90 Days) WebArena Ecosystem Benchmark Papers (15) Reasoning & Math Suite Benchmark Papers (16) WebArena Ecosystem Benchmark Papers In CS.CL (10) Reasoning & Math Suite Benchmark Papers In CS.CL (15) Arena/Judge Suite Benchmark Papers (10) Reasoning & Math Suite Benchmark Papers In CS.AI (13)

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote