HFEPX Metric Hub

Pass@1 Metric Papers

Updated from current HFEPX corpus (Mar 8, 2026). 12 papers are grouped in this metric page.

Read Full Context

Updated from current HFEPX corpus (Mar 8, 2026). 12 papers are grouped in this metric page. Common evaluation modes: Automatic Metrics, Simulation Env. Most common rater population: Domain Experts. Common annotation unit: Trajectory. Frequently cited benchmark: SWE-bench. Common metric signal: pass@1. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 15, 2026.

Papers: 12 Last published: Feb 15, 2026 Global RSS

Researcher Quick Triage

Use this page to compare metric behavior across protocols and benchmarks before selecting your reporting stack. Quality band: Medium .

Metric Coverage

50.0%

6 sampled papers include metric names.

Benchmark Anchoring

41.7%

Papers with explicit dataset/benchmark anchors for fair comparison.

Quality Controls

0.0%

0 papers report calibration/adjudication/IAA controls.

12 papers are not low-signal flagged in this sample.
Use the protocol matrix below to avoid comparing metrics across incompatible eval setups.

Primary action: Treat this as directional signal only; metric reporting is present but benchmark anchoring is still thin.

Why This Matters (Expanded)

Why This Matters For Eval Research

42.9% of papers report explicit human-feedback signals, led by expert verification.
automatic metrics appears in 33.3% of papers in this hub.
SWE-bench is a recurring benchmark anchor for cross-paper comparisons in this page.

Metric Notes (Expanded)

Metric-Driven Protocol Takeaways

Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
Rater context is mostly domain experts, and annotation is commonly trajectory-level annotation; use this to scope replication staffing.
Stratify by benchmark (SWE-bench vs SWE-bench Verified) before comparing methods.

Metric Interpretation

pass@1 is reported in 100% of hub papers (7/12); compare with a secondary metric before ranking methods.
cost is reported in 28.6% of hub papers (2/12); compare with a secondary metric before ranking methods.

Benchmark Context

SWE-bench appears in 28.6% of hub papers (2/12); use this cohort for benchmark-matched comparisons.
SWE-bench Verified appears in 28.6% of hub papers (2/12); use this cohort for benchmark-matched comparisons.

Start Here (Metric-Reliable First 6)

Ranked for metric reporting completeness and comparability.

AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents
Feb 15, 2026 · Citations: 0 · Score: 9.0

Metrics: Pass@1, Pass@3 · Eval: Simulation Env
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Feb 25, 2026 · Citations: 0 · Score: 8.0

Metrics: Pass@1, Latency · Eval: Automatic Metrics
Embodied Task Planning via Graph-Informed Action Generation with Large Language Model
Jan 29, 2026 · Citations: 0 · Score: 7.5

Metrics: Pass@1, Cost · Eval: Simulation Env
Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
Jun 3, 2025 · Citations: 0 · Score: 7.5

Metrics: Pass@1 · Eval: Automatic Metrics
APEX-Agents
Jan 20, 2026 · Citations: 0 · Score: 7.0

Metrics: Pass@1 · Eval: Automatic Metrics
Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards
Oct 28, 2025 · Citations: 0 · Score: 5.5

Metrics: Pass@1 · Eval: Simulation Env

Metric Protocol Matrix (Top 10)

Compare metric, benchmark, and evaluation context side by side.

Paper	Metrics	Benchmarks	Eval Modes	Quality Controls
AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents Feb 15, 2026	Pass@1, Pass@3	Ad Bench	Simulation Env	Not reported
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents Feb 25, 2026	Pass@1, Latency	SWE Bench, SWE Bench Verified	Automatic Metrics	Not reported
Embodied Task Planning via Graph-Informed Action Generation with Large Language Model Jan 29, 2026	Pass@1, Cost	ALFWorld	Simulation Env	Not reported
Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback Jun 3, 2025	Pass@1	AIME	Automatic Metrics	Not reported
APEX-Agents Jan 20, 2026	Pass@1	Not reported	Automatic Metrics	Not reported
Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards Oct 28, 2025	Pass@1	Not reported	Simulation Env	Not reported
Structurally Aligned Subtask-Level Memory for Software Engineering Agents Feb 25, 2026	Not reported	SWE Bench	Automatic Metrics	Not reported
$V_1$: Unifying Generation and Self-Verification for Parallel Reasoners Mar 4, 2026	Not reported	Not reported	Not reported	Not reported
BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning Mar 4, 2026	Not reported	Not reported	Not reported	Not reported
Confidence-Calibrated Small-Large Language Model Collaboration for Cost-Efficient Reasoning Mar 4, 2026	Not reported	Not reported	Not reported	Not reported

Researcher Workflow (Detailed)

Checklist

Moderate: Papers with explicit human feedback

Coverage is usable but incomplete (42.9% vs 45% target).
Gap: Papers reporting quality controls

Coverage is a replication risk (0% vs 30% target).
Strong: Papers naming benchmarks/datasets

Coverage is strong (71.4% vs 35% target).
Strong: Papers naming evaluation metrics

Coverage is strong (100% vs 35% target).
Strong: Papers with known rater population

Coverage is strong (42.9% vs 35% target).
Strong: Papers with known annotation unit

Coverage is strong (71.4% vs 35% target).

Strengths

Most papers provide measurable evaluation context (71.4% benchmarks, 100% metrics).
Agentic evaluation appears in 85.7% of papers.
Rater population and annotation-unit details are frequently specified.

Known Gaps

Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.

Suggested Next Analyses

Stratify by benchmark (SWE-bench vs SWE-bench Verified) before comparing methods.
Track metric sensitivity by reporting both pass@1 and cost.

Recommended Queries

Benchmark Slice: SWE-bench Metric Slice: pass@1 Recent High-Signal Papers

Known Limitations

Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.

Research Utility Snapshot (Detailed)

Top Metrics

Pass@1 (7)
Cost (2)
Coherence (1)
Latency (1)

Evaluation Modes

Automatic Metrics (4)
Simulation Env (3)

Top Benchmarks

SWE Bench (2)
SWE Bench Verified (2)
Ad Bench (1)
AIME (1)

Agentic Mix

Long Horizon (6)

Top Papers Reporting This Metric

AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents
Lingxiang Hu, Yiding Sun, Tianle Xia, Wenwei Li, Ming Xu · Feb 15, 2026 · Citations: 0

Simulation Env Coding

While Large Language Model (LLM) agents have achieved remarkable progress in complex reasoning tasks, evaluating their performance in real-world environments has become a critical problem.
APEX-Agents
Bertie Vidgen, Austin Mann, Abby Fennelly, John Wright Stanly, Lucas Rothman · Jan 20, 2026 · Citations: 0

Automatic Metrics Law

We introduce the AI Productivity Index for Agents (APEX-Agents), a benchmark for assessing whether AI agents can execute long-horizon, cross-application tasks created by investment banking analysts, management consultants, and corporate…
Embodied Task Planning via Graph-Informed Action Generation with Large Language Model
Xiang Li, Ning Yan, Masood Mortazavi · Jan 29, 2026 · Citations: 0

Simulation Env General

We propose GiG, a novel planning framework that structures embodied agents' memory using a Graph-in-Graph architecture.
Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
Xiaoying Zhang, Yipeng Zhang, Hao Sun, Kaituo Feng, Chaochao Lu · Jun 3, 2025 · Citations: 0

Automatic Metrics General

We show that plateaued RL models can successfully refine failed solutions when given natural language critiques.
Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards
Shangyu Xing, Siyuan Wang, Chenyuan Yang, Xinyu Dai, Xiang Ren · Oct 28, 2025 · Citations: 0

Simulation Env Coding

To address this limitation, we propose Lookahead Tree-Based Rollouts (LATR), a novel rollout strategy designed to explicitly promotes trajectory-level diversity by enforcing branching into different candidate tokens likely to yield distinct…
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Patrick Tser Jern Kon, Archana Pradeep, Ang Chen, Alexander P. Ellis, Warren Hunt · Feb 25, 2026 · Citations: 0

Automatic Metrics Coding

Our approach combines supervised fine-tuning on expert-augmented trajectories with agentic reinforcement learning that explicitly discourages degenerative looping and unproductive expert collaboration.
Structurally Aligned Subtask-Level Memory for Software Engineering Agents
Kangning Shen, Jingyuan Zhang, Chenxi Sun, Wencong Zeng, Yang Yue · Feb 25, 2026 · Citations: 0

Automatic Metrics Coding

Large Language Models (LLMs) have demonstrated significant potential as autonomous software engineering (SWE) agents.
$V_1$: Unifying Generation and Self-Verification for Parallel Reasoners
Harman Singh, Xiuyu Li, Kusha Sareen, Monishwaran Maheswaran, Sijun Tan · Mar 4, 2026 · Citations: 0
BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning
Tarjei Paule Hage, Markus J. Buehler · Mar 4, 2026 · Citations: 0
Confidence-Calibrated Small-Large Language Model Collaboration for Cost-Efficient Reasoning
Chuang Zhang, Zizhen Zhu, Yihao Wei, Bing Tian, Junyi Liu · Mar 4, 2026 · Citations: 0
Rewards as Labels: Revisiting RLVR from a Classification Perspective
Zepeng Zhai, Meilin Chen, Jiaxuan Zhao, Junlang Qian, Lei Shen · Feb 5, 2026 · Citations: 0
Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
Xichen Zhang, Sitong Wu, Yinghao Zhu, Haoru Tan, Shaozuo Yu · Oct 22, 2025 · Citations: 0

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote