HFEPX Metric Hub

Pass@1 In CS.LG Papers

Updated from current HFEPX corpus (Mar 8, 2026). 6 papers are grouped in this metric page.

Read Full Context

Updated from current HFEPX corpus (Mar 8, 2026). 6 papers are grouped in this metric page. Common evaluation modes: Automatic Metrics, Simulation Env. Most common rater population: Domain Experts. Common annotation unit: Multi Dim Rubric. Frequently cited benchmark: Ad-Bench. Common metric signal: pass@1. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 15, 2026.

Papers: 6 Last published: Feb 15, 2026 Global RSS

Researcher Quick Triage

Use this page to compare metric behavior across protocols and benchmarks before selecting your reporting stack. Quality band: Developing .

Metric Coverage

50.0%

3 sampled papers include metric names.

Benchmark Anchoring

33.3%

Papers with explicit dataset/benchmark anchors for fair comparison.

Quality Controls

0.0%

0 papers report calibration/adjudication/IAA controls.

6 papers are not low-signal flagged in this sample.
Use the protocol matrix below to avoid comparing metrics across incompatible eval setups.

Primary action: Treat this as directional signal only; metric reporting is present but benchmark anchoring is still thin.

Why This Matters (Expanded)

Why This Matters For Eval Research

33.3% of papers report explicit human-feedback signals, led by expert verification.
automatic metrics appears in 33.3% of papers in this hub.
Ad-Bench is a recurring benchmark anchor for cross-paper comparisons in this page.

Metric Notes (Expanded)

Metric-Driven Protocol Takeaways

Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.
Stratify by benchmark (Ad-Bench vs SWE-bench) before comparing methods.

Metric Interpretation

pass@1 is reported in 100% of hub papers (6/6); compare with a secondary metric before ranking methods.
cost is reported in 16.7% of hub papers (1/6); compare with a secondary metric before ranking methods.

Benchmark Context

Ad-Bench appears in 16.7% of hub papers (1/6); use this cohort for benchmark-matched comparisons.
SWE-bench appears in 16.7% of hub papers (1/6); use this cohort for benchmark-matched comparisons.

Start Here (Metric-Reliable First 6)

Ranked for metric reporting completeness and comparability.

AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents
Feb 15, 2026 · Citations: 0 · Score: 9.0

Metrics: Pass@1, Pass@3 · Eval: Simulation Env
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Feb 25, 2026 · Citations: 0 · Score: 8.0

Metrics: Pass@1, Latency · Eval: Automatic Metrics
APEX-Agents
Jan 20, 2026 · Citations: 0 · Score: 7.0

Metrics: Pass@1 · Eval: Automatic Metrics
BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning
Mar 4, 2026 · Citations: 0 · Score: 0.0

Metrics: Not reported · Eval: Not reported
Rewards as Labels: Revisiting RLVR from a Classification Perspective
Feb 5, 2026 · Citations: 0 · Score: 0.0

Metrics: Not reported · Eval: Not reported
Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
Oct 22, 2025 · Citations: 0 · Score: 0.0

Metrics: Not reported · Eval: Not reported

Metric Protocol Matrix (Top 10)

Compare metric, benchmark, and evaluation context side by side.

Paper	Metrics	Benchmarks	Eval Modes	Quality Controls
AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents Feb 15, 2026	Pass@1, Pass@3	Ad Bench	Simulation Env	Not reported
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents Feb 25, 2026	Pass@1, Latency	SWE Bench, SWE Bench Verified	Automatic Metrics	Not reported
APEX-Agents Jan 20, 2026	Pass@1	Not reported	Automatic Metrics	Not reported
BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning Mar 4, 2026	Not reported	Not reported	Not reported	Not reported
Rewards as Labels: Revisiting RLVR from a Classification Perspective Feb 5, 2026	Not reported	Not reported	Not reported	Not reported
Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning Oct 22, 2025	Not reported	Not reported	Not reported	Not reported

Researcher Workflow (Detailed)

Checklist

Moderate: Papers with explicit human feedback

Coverage is usable but incomplete (33.3% vs 45% target).
Gap: Papers reporting quality controls

Coverage is a replication risk (0% vs 30% target).
Moderate: Papers naming benchmarks/datasets

Coverage is usable but incomplete (33.3% vs 35% target).
Strong: Papers naming evaluation metrics

Coverage is strong (100% vs 35% target).
Strong: Papers with known rater population

Coverage is strong (50% vs 35% target).
Moderate: Papers with known annotation unit

Coverage is usable but incomplete (33.3% vs 35% target).

Strengths

Agentic evaluation appears in 50% of papers.

Known Gaps

Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.

Suggested Next Analyses

Stratify by benchmark (Ad-Bench vs SWE-bench) before comparing methods.
Track metric sensitivity by reporting both pass@1 and cost.

Recommended Queries

Benchmark Slice: Ad-Bench Metric Slice: pass@1 Recent High-Signal Papers

Known Limitations

Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.

Research Utility Snapshot (Detailed)

Top Metrics

Pass@1 (6)
Cost (1)
Latency (1)
Pass@3 (1)

Evaluation Modes

Automatic Metrics (2)
Simulation Env (1)

Top Benchmarks

Ad Bench (1)
SWE Bench (1)
SWE Bench Verified (1)

Agentic Mix

Long Horizon (3)

Top Papers Reporting This Metric

AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents
Lingxiang Hu, Yiding Sun, Tianle Xia, Wenwei Li, Ming Xu · Feb 15, 2026 · Citations: 0

Simulation Env Coding

While Large Language Model (LLM) agents have achieved remarkable progress in complex reasoning tasks, evaluating their performance in real-world environments has become a critical problem.
APEX-Agents
Bertie Vidgen, Austin Mann, Abby Fennelly, John Wright Stanly, Lucas Rothman · Jan 20, 2026 · Citations: 0

Automatic Metrics Law

We introduce the AI Productivity Index for Agents (APEX-Agents), a benchmark for assessing whether AI agents can execute long-horizon, cross-application tasks created by investment banking analysts, management consultants, and corporate…
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Patrick Tser Jern Kon, Archana Pradeep, Ang Chen, Alexander P. Ellis, Warren Hunt · Feb 25, 2026 · Citations: 0

Automatic Metrics Coding

Our approach combines supervised fine-tuning on expert-augmented trajectories with agentic reinforcement learning that explicitly discourages degenerative looping and unproductive expert collaboration.
BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning
Tarjei Paule Hage, Markus J. Buehler · Mar 4, 2026 · Citations: 0
Rewards as Labels: Revisiting RLVR from a Classification Perspective
Zepeng Zhai, Meilin Chen, Jiaxuan Zhao, Junlang Qian, Lei Shen · Feb 5, 2026 · Citations: 0
Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
Xichen Zhang, Sitong Wu, Yinghao Zhu, Haoru Tan, Shaozuo Yu · Oct 22, 2025 · Citations: 0

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote