HFEPX Hub

CS.SE + Automatic Metrics Papers

Updated from current HFEPX corpus (Mar 8, 2026). 10 papers are grouped in this hub page.

Read Full Context

Updated from current HFEPX corpus (Mar 8, 2026). 10 papers are grouped in this hub page. Common evaluation modes: Automatic Metrics. Most common rater population: Domain Experts. Common annotation unit: Trajectory. Frequently cited benchmark: SWE-bench. Common metric signal: pass@1. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 25, 2026.

Papers: 10 Last published: Feb 25, 2026 Global RSS Tag RSS

Cs.SEAutomatic Metrics

Researcher Quick Triage

This hub is best used for protocol triage and replication planning from abstract-level evidence. Quality band: Developing .

All Sampled Papers (10) Replication-Ready Only (2)

High-Signal Coverage

100.0%

10 / 10 sampled papers are not low-signal flagged.

Replication-Ready Set

Benchmark + metric + eval mode explicitly present.

Judge/Human Comparability

Papers containing both `human_eval` and `llm_as_judge`.

2 papers are replication-ready (benchmark + metric + explicit evaluation mode).
0 papers support judge-vs-human agreement analysis.
0 papers report explicit quality controls (calibration/adjudication/IAA).

Primary action: Use this page for scouting only; collect additional papers before attempting replication-critical comparisons.

Currently showing only replication-ready papers in ranking and matrix sections (2 papers).

Why This Matters For Eval Research

automatic metrics appears in 100% of papers in this hub.
SWE-bench is a recurring benchmark anchor for cross-paper comparisons in this page.
long-horizon tasks appears in 40% of papers, indicating agentic evaluation demand.

Protocol Takeaways

Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
Rater context is mostly domain experts, and annotation is commonly trajectory-level annotation; use this to scope replication staffing.
Stratify by benchmark (SWE-bench vs SWE-bench Verified) before comparing methods.

Benchmark Interpretation

SWE-bench appears in 20% of hub papers (2/10); use this cohort for benchmark-matched comparisons.
SWE-bench Verified appears in 20% of hub papers (2/10); use this cohort for benchmark-matched comparisons.

Metric Interpretation

pass@1 is reported in 20% of hub papers (2/10); compare with a secondary metric before ranking methods.
coherence is reported in 10% of hub papers (1/10); compare with a secondary metric before ranking methods.

Researcher Checklist (Expanded)

Researcher Checklist

Gap: Papers with explicit human feedback

Coverage is a replication risk (0% vs 45% target).
Gap: Papers reporting quality controls

Coverage is a replication risk (0% vs 30% target).
Gap: Papers naming benchmarks/datasets

Coverage is a replication risk (20% vs 35% target).
Strong: Papers naming evaluation metrics

Coverage is strong (40% vs 35% target).
Gap: Papers with known rater population

Coverage is a replication risk (10% vs 35% target).
Gap: Papers with known annotation unit

Coverage is a replication risk (10% vs 35% target).

Strengths

Agentic evaluation appears in 40% of papers.

Known Gaps

Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
Rater population is under-specified (10% coverage).
Annotation unit is under-specified (10% coverage).

Suggested Next Analyses

Stratify by benchmark (SWE-bench vs SWE-bench Verified) before comparing methods.
Track metric sensitivity by reporting both pass@1 and coherence.

Recommended Queries (Expanded)

Recommended Queries

Benchmark Slice: SWE-bench Metric Slice: pass@1 Recent High-Signal Papers

Start with These 3

Use these when you need one protocol anchor, one benchmark anchor, and one recent comparison point before reading the wider hub.

Strongest protocol reference

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unloc…

Highest protocol score with explicit human/eval signal plus SWE-bench.

Strongest benchmark reference

EVALOOOP: A Self-Consistency-Centered Framework for Assessing Large L…

MBPP+ with accuracy gives a fast comparison anchor.

Start Here (Best First 6)

Ranked for protocol completeness (human signal, benchmark + metric anchors, quality controls, and judge/human overlap).

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Feb 25, 2026 · Citations: 0 · Score: 6.0

HF: Not reported · Eval: Automatic Metrics · Benchmark: SWE Bench · Metric: Pass@1
EVALOOOP: A Self-Consistency-Centered Framework for Assessing Large Language Model Robustness in Programming
May 18, 2025 · Citations: 0 · Score: 2.0

HF: Not reported · Eval: Automatic Metrics · Benchmark: MBPP+ · Metric: Accuracy

Protocol Matrix (Top 12)

Use this to quickly compare protocol ingredients instead of scanning long prose.

Paper	HF Signal	Eval Modes	Benchmarks	Metrics	QC
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents Feb 25, 2026	No Not Reported	Automatic Metrics	SWE Bench , SWE Bench Verified	Pass@1 , Latency	Not Reported
EVALOOOP: A Self-Consistency-Centered Framework for Assessing Large Language Model Robustness in Programming May 18, 2025	No Not Reported	Automatic Metrics	MBPP+ , DROP	Accuracy , Pass@1	Not Reported

Protocol Diff (Top Papers)

Fast side-by-side comparison for the highest-ranked papers in this hub.

Signal	SWE-Protégé: Learning to Selectively Collaborate Wi…	EVALOOOP: A Self-Consistency-Centered Framework for…
Human Feedback	Not reported	Not reported
Evaluation Modes	Automatic Metrics	Automatic Metrics
Benchmarks	SWE Bench, SWE Bench Verified	MBPP+, DROP
Metrics	Pass@1, Latency	Accuracy, Pass@1
Quality Controls	Not reported	Not reported
Rater Population	Domain Experts	Unknown
Annotation Unit	Unknown	Unknown

Research Utility Snapshot

Human Feedback Mix

Evaluation Modes

Automatic Metrics (10)

Top Benchmarks

SWE Bench (2)
SWE Bench Verified (2)

Top Metrics

Pass@1 (2)
Coherence (1)
Cost (1)
Latency (1)

Rater Population Mix

Domain Experts (1)

Quality Controls

Coverage diagnostics (sample-based): human-feedback 0.0% · benchmarks 30.0% · metrics 90.0% · quality controls 0.0%.

Top Papers

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Patrick Tser Jern Kon, Archana Pradeep, Ang Chen, Alexander P. Ellis, Warren Hunt · Feb 25, 2026 · Citations: 0

Automatic Metrics Long Horizon

Our approach combines supervised fine-tuning on expert-augmented trajectories with agentic reinforcement learning that explicitly discourages degenerative looping and unproductive expert collaboration.
EVALOOOP: A Self-Consistency-Centered Framework for Assessing Large Language Model Robustness in Programming
Sen Fang, Weiyuan Ding, Mengshi Zhang, Zihao Chen, Bowen Xu · May 18, 2025 · Citations: 0

Automatic Metrics

However, adversarial attacks exhibit fundamental limitations that compromise fair robustness assessment: they demonstrate contradictory evaluation outcomes where different attack strategies tend to favor different models, and more…

Related Hubs

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote