Pass@1 + Automatic Metrics Metric Papers

Why This Matters For Eval Research

• 50% of papers report explicit human-feedback signals, led by critique/edit feedback.
• automatic metrics appears in 100% of papers in this hub.
• SWE-bench is a recurring benchmark anchor for cross-paper comparisons in this page.

Metric-Driven Protocol Takeaways

• Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
• Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.
• Stratify by benchmark (SWE-bench vs SWE-bench Verified) before comparing methods.

Metric Interpretation

• pass@1 is reported in 100% of hub papers (4/4); compare with a secondary metric before ranking methods.
• cost is reported in 25% of hub papers (1/4); compare with a secondary metric before ranking methods.

Benchmark Context

• SWE-bench appears in 50% of hub papers (2/4); use this cohort for benchmark-matched comparisons.
• SWE-bench Verified appears in 50% of hub papers (2/4); use this cohort for benchmark-matched comparisons.

Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (50% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (0% vs 30% target).

• Strong: Papers naming benchmarks/datasets

  Coverage is strong (75% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (100% vs 35% target).

• Strong: Papers with known rater population

  Coverage is strong (50% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (50% vs 35% target).

Strengths

• Strong human-feedback signal (50% of papers).
• Most papers provide measurable evaluation context (75% benchmarks, 100% metrics).
• Agentic evaluation appears in 75% of papers.

Known Gaps

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.

Suggested Next Analyses

• Stratify by benchmark (SWE-bench vs SWE-bench Verified) before comparing methods.
• Track metric sensitivity by reporting both pass@1 and cost.

Recommended Queries

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
• Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.