Skip to content
← Back to explorer

HFEPX Benchmark Hub

APPS Benchmark Papers

Updated from current HFEPX corpus (Mar 17, 2026). 4 papers are grouped in this benchmark page.

Read Full Context

Updated from current HFEPX corpus (Mar 17, 2026). 4 papers are grouped in this benchmark page. Common evaluation modes: Automatic Metrics, Human Eval. Frequently cited benchmark: APPS. Common metric signal: cost. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Jan 5, 2026.

Papers: 4 Last published: Jan 5, 2026 Global RSS

Researcher Quick Triage

Use this page for benchmark-matched method comparisons and eval protocol selection. Quality band: Developing .

High-Signal Coverage

100.0%

4 / 4 sampled papers are not low-signal flagged.

Replication-Ready Set

2

Papers with explicit benchmark + metric + eval mode fields.

Quality Controls

0.0%

0 papers report calibration/adjudication/IAA controls.

  • 4 papers explicitly name benchmark datasets in the sampled set.
  • 2 papers report at least one metric term in metadata extraction.
  • Start with the ranked shortlist below before reading all papers.

Primary action: Use this page to map benchmark mentions first; wait for stronger metric/QC coverage before strict comparisons.

Why This Matters (Expanded)

Why This Matters For Eval Research

  • 50% of papers report explicit human-feedback signals, led by pairwise preferences.
  • automatic metrics appears in 50% of papers in this hub.
  • APPS is a recurring benchmark anchor for cross-paper comparisons in this page.
Protocol Notes (Expanded)

Protocol Takeaways

  • Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
  • Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
  • Stratify by benchmark (APPS vs LMSYS Chatbot Arena) before comparing methods.

Benchmark Interpretation

  • APPS appears in 100% of hub papers (4/4); use this cohort for benchmark-matched comparisons.
  • LMSYS Chatbot Arena appears in 25% of hub papers (1/4); use this cohort for benchmark-matched comparisons.

Metric Interpretation

  • cost is reported in 25% of hub papers (1/4); compare with a secondary metric before ranking methods.
  • success rate is reported in 25% of hub papers (1/4); compare with a secondary metric before ranking methods.

Start Here (Benchmark-Matched First 6)

Ranked by protocol completeness so you can quickly find papers suitable for comparison studies.

Protocol Matrix (Top 10)

Compare protocol ingredients quickly before deep-reading full papers.

Paper Eval Modes Human Feedback Metrics Quality Controls
RLShield: Practical Multi-Agent RL for Financial Cyber Defense with Attack-Surface MDPs and Real-Time Response Orchestration

Feb 26, 2026

 Automatic Metrics Not reported Cost Not reported
WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics

Jan 5, 2026

 Llm As Judge Pairwise Preference Not reported Not reported
The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

Oct 29, 2025

 Automatic Metrics Not reported Success rate Not reported
EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagation

May 29, 2024

 Human Eval Pairwise Preference Not reported Not reported
Researcher Workflow (Detailed)

Checklist

  • Strong: Papers with explicit human feedback

    Coverage is strong (50% vs 45% target).

  • Gap: Papers reporting quality controls

    Coverage is a replication risk (0% vs 30% target).

  • Strong: Papers naming benchmarks/datasets

    Coverage is strong (100% vs 35% target).

  • Strong: Papers naming evaluation metrics

    Coverage is strong (50% vs 35% target).

  • Gap: Papers with known rater population

    Coverage is a replication risk (0% vs 35% target).

  • Gap: Papers with known annotation unit

    Coverage is a replication risk (0% vs 35% target).

Strengths

  • Strong human-feedback signal (50% of papers).
  • Most papers provide measurable evaluation context (100% benchmarks, 50% metrics).
  • Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.

Known Gaps

  • Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Rater population is under-specified (0% coverage).
  • Annotation unit is under-specified (0% coverage).

Suggested Next Analyses

  • Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
  • Stratify by benchmark (APPS vs LMSYS Chatbot Arena) before comparing methods.
  • Track metric sensitivity by reporting both cost and success rate.

Recommended Queries

Judge vs Human AgreementBenchmark Slice: APPSMetric Slice: costRecent High-Signal Papers
Known Limitations
  • Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Rater population is under-specified (0% coverage).
  • Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
Research Utility Snapshot (Detailed)

Evaluation Modes

  • Automatic Metrics (2)
  • Human Eval (1)
  • Llm As Judge (1)

Human Feedback Mix

  • Pairwise Preference (2)

Top Benchmarks

  • APPS (4)
  • LMSYS Chatbot Arena (1)
  • Webcoderbench (1)

Top Metrics

  • Cost (1)
  • Success rate (1)

Top Papers On This Benchmark

Related Benchmark Hubs

APPS In CS.AI Papers APPS In CS.CL Papers Coding Evaluation Suite Benchmark Papers Coding Or General Papers APPS + Automatic Metrics Benchmark Papers APPS + Coding Benchmark Papers DROP Benchmark Papers (Last 45 Days) (11) DROP Benchmark Papers (Last 60 Days) (11) DROP Benchmark Papers (Last 75 Days) (11) Reasoning & Math Suite Benchmark Papers + Automatic Metrics (10) Reasoning & Math Suite Benchmark Papers (23) Reasoning & Math Suite Benchmark Papers In CS.CL (21) Reasoning & Math Suite Benchmark Papers In CS.AI (15) Coding Evaluation Suite Benchmark Papers (11) Coding Evaluation Suite Benchmark Papers In CS.CL (10) SWE-Bench Ecosystem Benchmark Papers (13) SWE-Bench Ecosystem Benchmark Papers In CS.CL (11)

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote