HFEPX Hub

CS.SE + Automatic Metrics Papers

Updated from current HFEPX corpus (Mar 8, 2026). 10 papers are grouped in this hub page.

Read Full Context

Updated from current HFEPX corpus (Mar 8, 2026). 10 papers are grouped in this hub page. Common evaluation modes: Automatic Metrics. Most common rater population: Domain Experts. Common annotation unit: Trajectory. Frequently cited benchmark: SWE-bench. Common metric signal: pass@1. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 25, 2026.

Papers: 10 Last published: Feb 25, 2026 Global RSS Tag RSS

Cs.SEAutomatic Metrics

Researcher Quick Triage

This hub is best used for protocol triage and replication planning from abstract-level evidence. Quality band: Developing .

All Sampled Papers (10) Replication-Ready Only (2)

High-Signal Coverage

100.0%

10 / 10 sampled papers are not low-signal flagged.

Replication-Ready Set

Benchmark + metric + eval mode explicitly present.

Judge/Human Comparability

Papers containing both `human_eval` and `llm_as_judge`.

2 papers are replication-ready (benchmark + metric + explicit evaluation mode).
0 papers support judge-vs-human agreement analysis.
0 papers report explicit quality controls (calibration/adjudication/IAA).

Primary action: Use this page for scouting only; collect additional papers before attempting replication-critical comparisons.

Why This Matters For Eval Research

automatic metrics appears in 100% of papers in this hub.
SWE-bench is a recurring benchmark anchor for cross-paper comparisons in this page.
long-horizon tasks appears in 40% of papers, indicating agentic evaluation demand.

Protocol Takeaways

Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
Rater context is mostly domain experts, and annotation is commonly trajectory-level annotation; use this to scope replication staffing.
Stratify by benchmark (SWE-bench vs SWE-bench Verified) before comparing methods.

Benchmark Interpretation

SWE-bench appears in 20% of hub papers (2/10); use this cohort for benchmark-matched comparisons.
SWE-bench Verified appears in 20% of hub papers (2/10); use this cohort for benchmark-matched comparisons.

Metric Interpretation

pass@1 is reported in 20% of hub papers (2/10); compare with a secondary metric before ranking methods.
coherence is reported in 10% of hub papers (1/10); compare with a secondary metric before ranking methods.

Researcher Checklist (Expanded)

Researcher Checklist

Gap: Papers with explicit human feedback

Coverage is a replication risk (0% vs 45% target).
Gap: Papers reporting quality controls

Coverage is a replication risk (0% vs 30% target).
Gap: Papers naming benchmarks/datasets

Coverage is a replication risk (20% vs 35% target).
Strong: Papers naming evaluation metrics

Coverage is strong (40% vs 35% target).
Gap: Papers with known rater population

Coverage is a replication risk (10% vs 35% target).
Gap: Papers with known annotation unit

Coverage is a replication risk (10% vs 35% target).

Strengths

Agentic evaluation appears in 40% of papers.

Known Gaps

Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
Rater population is under-specified (10% coverage).
Annotation unit is under-specified (10% coverage).

Suggested Next Analyses

Stratify by benchmark (SWE-bench vs SWE-bench Verified) before comparing methods.
Track metric sensitivity by reporting both pass@1 and coherence.

Recommended Queries (Expanded)

Recommended Queries

Benchmark Slice: SWE-bench Metric Slice: pass@1 Recent High-Signal Papers

Start with These 3

Use these when you need one protocol anchor, one benchmark anchor, and one recent comparison point before reading the wider hub.

Strongest protocol reference

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unloc…

Highest protocol score with explicit human/eval signal plus SWE-bench.

Strongest benchmark reference

Structurally Aligned Subtask-Level Memory for Software Engineering Ag…

SWE-bench gives a fast comparison anchor.

Strongest recent paper

SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?

Useful for current practice scanning; published Feb 28, 2026.

Start Here (Best First 6)

Ranked for protocol completeness (human signal, benchmark + metric anchors, quality controls, and judge/human overlap).

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Feb 25, 2026 · Citations: 0 · Score: 6.0

HF: Not reported · Eval: Automatic Metrics · Benchmark: SWE Bench · Metric: Pass@1
Structurally Aligned Subtask-Level Memory for Software Engineering Agents
Feb 25, 2026 · Citations: 0 · Score: 4.5

HF: Not reported · Eval: Automatic Metrics · Benchmark: SWE Bench · Metric: Not Reported
SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?
Feb 28, 2026 · Citations: 0 · Score: 4.0

HF: Not reported · Eval: Automatic Metrics · Benchmark: Not Reported · Metric: Success rate
ToolMATH: A Math Tool Benchmark for Realistic Long-Horizon Multi-Tool Reasoning
Feb 24, 2026 · Citations: 0 · Score: 4.0

HF: Not reported · Eval: Automatic Metrics · Benchmark: Not Reported · Metric: Coherence
EVALOOOP: A Self-Consistency-Centered Framework for Assessing Large Language Model Robustness in Programming
May 18, 2025 · Citations: 0 · Score: 2.0

HF: Not reported · Eval: Automatic Metrics · Benchmark: MBPP+ · Metric: Accuracy
SpecMind: Cognitively Inspired, Interactive Multi-Turn Framework for Postcondition Inference
Feb 24, 2026 · Citations: 0 · Score: 1.5

HF: Not reported · Eval: Automatic Metrics · Benchmark: Not Reported · Metric: Accuracy

Protocol Matrix (Top 12)

Use this to quickly compare protocol ingredients instead of scanning long prose.

Paper	HF Signal	Eval Modes	Benchmarks	Metrics	QC
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents Feb 25, 2026	No Not Reported	Automatic Metrics	SWE Bench , SWE Bench Verified	Pass@1 , Latency	Not Reported
Structurally Aligned Subtask-Level Memory for Software Engineering Agents Feb 25, 2026	No Not Reported	Automatic Metrics	SWE Bench	Not Reported	Not Reported
SkillCraft: Can LLM Agents Learn to Use Tools Skillfully? Feb 28, 2026	No Not Reported	Automatic Metrics	Not Reported	Success rate	Not Reported
ToolMATH: A Math Tool Benchmark for Realistic Long-Horizon Multi-Tool Reasoning Feb 24, 2026	No Not Reported	Automatic Metrics	Not Reported	Coherence	Not Reported
EVALOOOP: A Self-Consistency-Centered Framework for Assessing Large Language Model Robustness in Programming May 18, 2025	No Not Reported	Automatic Metrics	MBPP+ , DROP	Accuracy , Pass@1	Not Reported
SpecMind: Cognitively Inspired, Interactive Multi-Turn Framework for Postcondition Inference Feb 24, 2026	No Not Reported	Automatic Metrics	Not Reported	Accuracy	Not Reported
Exploring LLMs for User Story Extraction from Mockups Feb 19, 2026	No Not Reported	Automatic Metrics	Not Reported	Accuracy	Not Reported
The Invisible Hand of AI Libraries Shaping Open Source Projects and Communities Jan 5, 2026	No Not Reported	Automatic Metrics	Not Reported	Relevance	Not Reported
Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agent Dec 17, 2025	No Not Reported	Automatic Metrics	Not Reported	Success rate	Not Reported
On the Inference (In-)Security of Vertical Federated Learning: Efficient Auditing against Inference Tampering Attack Jul 3, 2025	No Not Reported	Automatic Metrics	Not Reported	Accuracy , Latency	Not Reported

Protocol Diff (Top Papers)

Fast side-by-side comparison for the highest-ranked papers in this hub.

Signal	SWE-Protégé: Learning to Selectively Collaborate Wi…	Structurally Aligned Subtask-Level Memory for Softw…	SkillCraft: Can LLM Agents Learn to Use Tools Skill…
Human Feedback	Not reported	Not reported	Not reported
Evaluation Modes	Automatic Metrics	Automatic Metrics	Automatic Metrics
Benchmarks	SWE Bench, SWE Bench Verified	SWE Bench	Not reported
Metrics	Pass@1, Latency	Not reported	Success rate
Quality Controls	Not reported	Not reported	Not reported
Rater Population	Domain Experts	Unknown	Unknown
Annotation Unit	Unknown	Trajectory	Unknown

Research Utility Snapshot

Human Feedback Mix

Evaluation Modes

Automatic Metrics (10)

Top Benchmarks

SWE Bench (2)
SWE Bench Verified (2)

Top Metrics

Pass@1 (2)
Coherence (1)
Cost (1)
Latency (1)

Rater Population Mix

Domain Experts (1)

Quality Controls

Coverage diagnostics (sample-based): human-feedback 0.0% · benchmarks 30.0% · metrics 90.0% · quality controls 0.0%.

Top Papers

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Patrick Tser Jern Kon, Archana Pradeep, Ang Chen, Alexander P. Ellis, Warren Hunt · Feb 25, 2026 · Citations: 0

Automatic Metrics Long Horizon

Our approach combines supervised fine-tuning on expert-augmented trajectories with agentic reinforcement learning that explicitly discourages degenerative looping and unproductive expert collaboration.
Structurally Aligned Subtask-Level Memory for Software Engineering Agents
Kangning Shen, Jingyuan Zhang, Chenxi Sun, Wencong Zeng, Yang Yue · Feb 25, 2026 · Citations: 0

Automatic Metrics Long Horizon

Large Language Models (LLMs) have demonstrated significant potential as autonomous software engineering (SWE) agents.
SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?
Shiqi Chen, Jingze Gai, Ruochen Zhou, Jinghan Zhang, Tongyao Zhu · Feb 28, 2026 · Citations: 0

Automatic Metrics Long Horizon

Real-world tool-using agents operate over long-horizon workflows with recurring structure and diverse demands, where effective behavior requires not only invoking atomic tools but also abstracting, and reusing higher-level tool…
ToolMATH: A Math Tool Benchmark for Realistic Long-Horizon Multi-Tool Reasoning
Hyeonje Choi, Jeongsoo Lee, Hyojun Lee, Jay-Yoon Lee · Feb 24, 2026 · Citations: 0

Automatic Metrics Long Horizon

We introduce \ToolMATH, a math-grounded benchmark that evaluates tool-augmented language models in realistic multi-tool environments where the output depends on calling schema-specified tools and sustaining multi-step execution.
SpecMind: Cognitively Inspired, Interactive Multi-Turn Framework for Postcondition Inference
Cuong Chi Le, Minh V. T Pham, Tung Vu Duy, Cuong Duc Van, Huy N. Phan · Feb 24, 2026 · Citations: 0

Automatic Metrics

Our empirical evaluation shows that SpecMind significantly outperforms state-of-the-art approaches in both accuracy and completeness of generated postconditions.
Exploring LLMs for User Story Extraction from Mockups
Diego Firmenich, Leandro Antonelli, Bruno Pazos, Fabricio Lozada, Leonardo Morales · Feb 19, 2026 · Citations: 0

Automatic Metrics

Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
The Invisible Hand of AI Libraries Shaping Open Source Projects and Communities
Matteo Esposito, Andrea Janes, Valentina Lenarduzzi, Davide Taibi · Jan 5, 2026 · Citations: 0

Automatic Metrics

Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agent
Mehil B Shah, Mohammad Masudur Rahman, Foutse Khomh · Dec 17, 2025 · Citations: 0

Automatic Metrics

Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
On the Inference (In-)Security of Vertical Federated Learning: Efficient Auditing against Inference Tampering Attack
Chung-ju Huang, Ziqi Zhang, Yinggui Wang, Binghui Wang, Tao Wei · Jul 3, 2025 · Citations: 0

Automatic Metrics

Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
EVALOOOP: A Self-Consistency-Centered Framework for Assessing Large Language Model Robustness in Programming
Sen Fang, Weiyuan Ding, Mengshi Zhang, Zihao Chen, Bowen Xu · May 18, 2025 · Citations: 0

Automatic Metrics

However, adversarial attacks exhibit fundamental limitations that compromise fair robustness assessment: they demonstrate contradictory evaluation outcomes where different attack strategies tend to favor different models, and more…

Related Hubs

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote