HFEPX Hub

Multi Agent + Automatic Metrics (Last 60 Days)

Updated from current HFEPX corpus (Apr 27, 2026). 37 papers are grouped in this hub page.

Read Full Context

Updated from current HFEPX corpus (Apr 27, 2026). 37 papers are grouped in this hub page. Common evaluation modes: Automatic Metrics, Llm As Judge. Most common rater population: Domain Experts. Common annotation unit: Multi Dim Rubric. Frequent quality control: Calibration. Frequently cited benchmark: APPS. Common metric signal: accuracy. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Mar 29, 2026.

Papers: 37 Last published: Mar 29, 2026 Global RSS Tag RSS

Multi AgentAutomatic MetricsLast 60d

Researcher Quick Triage

This hub is best used for protocol triage and replication planning from abstract-level evidence. Quality band: Medium .

All Sampled Papers (37) Replication-Ready Only (10)

High-Signal Coverage

100.0%

37 / 37 sampled papers are not low-signal flagged.

Replication-Ready Set

Benchmark + metric + eval mode explicitly present.

Judge/Human Comparability

Papers containing both `human_eval` and `llm_as_judge`.

10 papers are replication-ready (benchmark + metric + explicit evaluation mode).
0 papers support judge-vs-human agreement analysis.
2 papers report explicit quality controls (calibration/adjudication/IAA).

Primary action: Start with the top 2 papers in “Start Here”, then validate assumptions in the protocol matrix.

Currently showing only replication-ready papers in ranking and matrix sections (10 papers).

Need evaluators for this research workflow?

Post a Job →

Why This Matters For Eval Research

16.2% of papers report explicit human-feedback signals, led by expert verification.
automatic metrics appears in 100% of papers in this hub.
APPS is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

Most common quality-control signal is rater calibration (5.4% of papers).
Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.
Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.

Benchmark Interpretation

APPS appears in 2.7% of hub papers (1/37); use this cohort for benchmark-matched comparisons.
HLE appears in 2.7% of hub papers (1/37); use this cohort for benchmark-matched comparisons.

Metric Interpretation

accuracy is reported in 64.9% of hub papers (24/37); compare with a secondary metric before ranking methods.
cost is reported in 18.9% of hub papers (7/37); compare with a secondary metric before ranking methods.

Researcher Checklist (Expanded)

Researcher Checklist

Gap: Papers with explicit human feedback

Coverage is a replication risk (16.2% vs 45% target).
Gap: Papers reporting quality controls

Coverage is a replication risk (5.4% vs 30% target).
Moderate: Papers naming benchmarks/datasets

Coverage is usable but incomplete (27% vs 35% target).
Strong: Papers naming evaluation metrics

Coverage is strong (100% vs 35% target).
Moderate: Papers with known rater population

Coverage is usable but incomplete (29.7% vs 35% target).
Gap: Papers with known annotation unit

Coverage is a replication risk (5.4% vs 35% target).

Strengths

Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.
Agentic evaluation appears in 100% of papers.

Known Gaps

Only 5.4% of papers report quality controls; prioritize calibration/adjudication evidence.
Annotation unit is under-specified (5.4% coverage).
LLM-as-judge appears without enough inter-annotator agreement reporting.

Suggested Next Analyses

Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
Stratify by benchmark (APPS vs HLE) before comparing methods.
Track metric sensitivity by reporting both accuracy and cost.
Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries (Expanded)

Recommended Queries

Judge vs Human Agreement Benchmark Slice: APPS Metric Slice: accuracy Recent High-Signal Papers

Start with These 3

Use these when you need one protocol anchor, one benchmark anchor, and one recent comparison point before reading the wider hub.

Strongest protocol reference

SODIUM: From Open Web Data to Queryable Databases

Highest protocol score with explicit human/eval signal plus Sodium-Bench.

Strongest benchmark reference

StitchCUDA: An Automated Multi-Agents End-to-End GPU Programing Frame…

Kernelbench with success rate gives a fast comparison anchor.

Strongest recent paper

Navigating Large-Scale Document Collections: MuDABench for Multi-Docu…

Useful for current practice scanning; published Apr 24, 2026.

Start Here (Best First 6)

Ranked for protocol completeness (human signal, benchmark + metric anchors, quality controls, and judge/human overlap).

SODIUM: From Open Web Data to Queryable Databases
Mar 19, 2026 · Citations: 0 · Score: 7.5

HF: Expert Verification · Eval: Automatic Metrics · Benchmark: Sodium Bench · Metric: Accuracy
StitchCUDA: An Automated Multi-Agents End-to-End GPU Programing Framework with Rubric-based Agentic Reinforcement Learning
Mar 3, 2026 · Citations: 0 · Score: 7.5

HF: Rubric Rating · Eval: Automatic Metrics · Benchmark: Kernelbench · Metric: Success rate
Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA
Apr 24, 2026 · Citations: 0 · Score: 6.0

HF: Not reported · Eval: Automatic Metrics · Benchmark: Mudabench · Metric: Accuracy
From Skills to Talent: Organising Heterogeneous Agents as a Real-World Company
Apr 24, 2026 · Citations: 0 · Score: 6.0

HF: Not reported · Eval: Automatic Metrics · Benchmark: Prdbench · Metric: Success rate
MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-End Question Answering
Mar 15, 2026 · Citations: 0 · Score: 5.5

HF: Not reported · Eval: Automatic Metrics · Benchmark: Medpriv Bench · Metric: Accuracy
Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learning
Apr 1, 2026 · Citations: 0 · Score: 5.5

HF: Not reported · Eval: Automatic Metrics · Benchmark: HLE · Metric: Accuracy

Protocol Matrix (Top 12)

Use this to quickly compare protocol ingredients instead of scanning long prose.

Paper	HF Signal	Eval Modes	Benchmarks	Metrics	QC
SODIUM: From Open Web Data to Queryable Databases Mar 19, 2026	Yes Expert Verification	Automatic Metrics	Sodium Bench	Accuracy	Not Reported
StitchCUDA: An Automated Multi-Agents End-to-End GPU Programing Framework with Rubric-based Agentic Reinforcement Learning Mar 3, 2026	Yes Rubric Rating	Automatic Metrics	Kernelbench	Success rate	Not Reported
Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA Apr 24, 2026	No Not Reported	Automatic Metrics	Mudabench	Accuracy	Not Reported
From Skills to Talent: Organising Heterogeneous Agents as a Real-World Company Apr 24, 2026	No Not Reported	Automatic Metrics	Prdbench	Success rate	Not Reported
MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-End Question Answering Mar 15, 2026	No Not Reported	Automatic Metrics	Medpriv Bench	Accuracy	Not Reported
Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learning Apr 1, 2026	No Not Reported	Automatic Metrics	HLE	Accuracy , Token cost	Not Reported
Effective Strategies for Asynchronous Software Engineering Agents Mar 23, 2026	No Not Reported	Automatic Metrics	Paperbench	Accuracy	Not Reported
Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis Mar 23, 2026	No Not Reported	Automatic Metrics	SWE Bench , SWE Bench Verified	Accuracy , Recall	Not Reported
QChunker: Learning Question-Aware Text Chunking for Domain RAG via Multi-Agent Debate Mar 12, 2026	No Not Reported	Automatic Metrics	Understanding Retrieval	Coherence	Not Reported
RLShield: Practical Multi-Agent RL for Financial Cyber Defense with Attack-Surface MDPs and Real-Time Response Orchestration Feb 26, 2026	No Not Reported	Automatic Metrics	APPS	Not Reported	Not Reported

Protocol Diff (Top Papers)

Fast side-by-side comparison for the highest-ranked papers in this hub.

Signal	SODIUM: From Open Web Data to Queryable Databases	StitchCUDA: An Automated Multi-Agents End-to-End GP…	Navigating Large-Scale Document Collections: MuDABe…
Human Feedback	Expert Verification	Rubric Rating	Not reported
Evaluation Modes	Automatic Metrics	Automatic Metrics	Automatic Metrics
Benchmarks	Sodium Bench	Kernelbench	Mudabench
Metrics	Accuracy	Success rate	Accuracy
Quality Controls	Not reported	Not reported	Not reported
Rater Population	Domain Experts	Unknown	Domain Experts
Annotation Unit	Unknown	Multi Dim Rubric	Unknown

Research Utility Snapshot

Human Feedback Mix

Expert Verification (4)
Red Team (1)
Rubric Rating (1)

Evaluation Modes

Automatic Metrics (37)
Llm As Judge (2)
Simulation Env (2)
Human Eval (1)

Top Benchmarks

APPS (1)
HLE (1)
Kernelbench (1)
Medpriv Bench (1)

Top Metrics

Accuracy (24)
Cost (7)
Recall (3)
F1 (2)

Rater Population Mix

Domain Experts (11)

Quality Controls

Calibration (2)

Coverage diagnostics (sample-based): human-feedback 16.2% · benchmarks 27.0% · metrics 100.0% · quality controls 5.4%.

Top Papers

SODIUM: From Open Web Data to Queryable Databases
Chuxuan Hu, Philip Li, Maxwell Yang, Daniel Kang · Mar 19, 2026 · Citations: 0

Expert Verification Automatic Metrics Multi Agent

Existing systems struggle with SODIUM tasks: we evaluate 6 advanced AI agents on SODIUM-Bench, with the strongest baseline achieving only 46.5% accuracy.
StitchCUDA: An Automated Multi-Agents End-to-End GPU Programing Framework with Rubric-based Agentic Reinforcement Learning
Shiyang Li, Zijian Zhang, Winson Chen, Yuebo Luo, Mingyi Hong · Mar 3, 2026 · Citations: 0

Rubric Rating Automatic Metrics Multi Agent

To address the challenge, in this work, we propose StitchCUDA, a multi-agent framework for end-to-end GPU program generation, with three specialized agents: a Planner to orchestrate whole system design, a Coder dedicated to implementing it…
Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA
Zhanli Li, Yixuan Cao, Lvzhou Luo, Ping Luo · Apr 24, 2026 · Citations: 0

Automatic Metrics Multi Agent

We present MuDABench, a benchmark for multi-document analytical QA, where questions require extracting and synthesizing information across numerous documents to perform quantitative analysis.
MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-End Question Answering
Shaowei Guan, Yu Zhai, Hin Chi Kwok, Jiawei Du, Xinyu Feng · Mar 15, 2026 · Citations: 0

Automatic Metrics Multi Agent

To fill this gap, we present MedPriv-Bench, the first benchmark specifically designed to jointly evaluate privacy preservation and clinical utility in medical open-ended question answering.
From Skills to Talent: Organising Heterogeneous Agents as a Real-World Company
Zhengxu Yu, Yu Fu, Zhiyuan He, Yuxuan Huang, Lee Ka Yiu · Apr 24, 2026 · Citations: 0

Automatic Metrics Multi Agent

Individual agent capabilities have advanced rapidly through modular skills and tool integrations, yet multi-agent systems remain constrained by fixed team structures, tightly coupled coordination logic, and session-bound learning.
Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learning
Eric Hanchen Jiang, Levina Li, Rui Sun, Xiao Liang, Yubei Li · Apr 1, 2026 · Citations: 0

Automatic Metrics Multi Agent

In this paper, we propose Agent Q-Mix, a reinforcement learning framework that reformulates topology selection as a cooperative Multi-Agent Reinforcement Learning (MARL) problem.
Effective Strategies for Asynchronous Software Engineering Agents
Jiayi Geng, Graham Neubig · Mar 23, 2026 · Citations: 0

Automatic Metrics Long Horizon

Inspired by these collaboration primitives, we introduce Centralized Asynchronous Isolated Delegation (CAID), a structured multi-agent coordination paradigm grounded in three core SWE primitives: centralized task delegation, asynchronous…
Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis
Tae-Eun Song · Mar 23, 2026 · Citations: 0

Automatic Metrics Multi Agent

LLM coding benchmarks face a credibility crisis: widespread solution leakage and test quality issues undermine SWE-bench Verified, while existing detection methods--paraphrase consistency, n-gram overlap, perplexity analysis--never directly…
QChunker: Learning Question-Aware Text Chunking for Domain RAG via Multi-Agent Debate
Jihao Zhao, Daixuan Li, Pengfei Li, Shuaishuai Zu, Biao Qin · Mar 12, 2026 · Citations: 0

Automatic Metrics Multi Agent

Drawing inspiration from Hal Gregersen's "Questions Are the Answer" theory, we design a multi-agent debate framework comprising four specialized components: a question outline generator, text segmenter, integrity reviewer, and knowledge…
RLShield: Practical Multi-Agent RL for Financial Cyber Defense with Attack-Surface MDPs and Real-Time Response Orchestration
Srikumar Nayak · Feb 26, 2026 · Citations: 0

Automatic Metrics Multi Agent

This paper proposes RLShield, a practical multi-agent RL pipeline for financial cyber defense.

Related Hubs

Get Started

Join the #1 Platform for AI Training Talent

Where top AI builders and expert AI Trainers connect to build the future of AI.

Self-Service

Post a Job

Post your project and get a shortlist of qualified AI Trainers and Data Labelers. Hire and manage your team in the tools you already use.

Create Account & Post a Job

Managed Service

For Large Projects

Done-for-You

We recruit, onboard, and manage a dedicated team inside your tools. End-to-end operations for large or complex projects.

Learn About Managed Service

For Freelancers

Join as an AI Trainer

Find AI training and data labeling projects across platforms, all in one place. One profile, one application process, more opportunities.

Join Now