Skip to content
← Back to explorer

HFEPX Hub

Multi Agent + Automatic Metrics (Last 60 Days)

Updated from current HFEPX corpus (Mar 1, 2026). 12 papers are grouped in this hub page.

Read Full Context

Updated from current HFEPX corpus (Mar 1, 2026). 12 papers are grouped in this hub page. Common evaluation modes: Automatic Metrics, Llm As Judge. Most common rater population: Domain Experts. Common annotation unit: Ranking. Common metric signal: accuracy. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 24, 2026.

Papers: 12 Last published: Feb 24, 2026 Global RSS Tag RSS
Multi AgentAutomatic MetricsLast 60d

Researcher Quick Triage

This hub is best used for protocol triage and replication planning from abstract-level evidence. Quality band: Developing .

High-Signal Coverage

100.0%

12 / 12 sampled papers are not low-signal flagged.

Replication-Ready Set

0

Benchmark + metric + eval mode explicitly present.

Judge/Human Comparability

0

Papers containing both `human_eval` and `llm_as_judge`.

  • 0 papers are replication-ready (benchmark + metric + explicit evaluation mode).
  • 0 papers support judge-vs-human agreement analysis.
  • 0 papers report explicit quality controls (calibration/adjudication/IAA).

Primary action: Use this page for scouting only; collect additional papers before attempting replication-critical comparisons.

Why This Matters (Expanded)

Why This Matters For Eval Research

  • 16.7% of papers report explicit human-feedback signals, led by expert verification.
  • automatic metrics appears in 100% of papers in this hub.
  • multi-agent setups appears in 100% of papers, indicating agentic evaluation demand.
Protocol Notes (Expanded)

Protocol Takeaways

  • Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
  • Rater context is mostly domain experts, and annotation is commonly ranking annotation; use this to scope replication staffing.
  • Pair this hub with a human_eval-heavy hub to validate judge-model calibration.

Metric Interpretation

  • accuracy is reported in 50% of hub papers (6/12); compare with a secondary metric before ranking methods.
  • cost is reported in 16.7% of hub papers (2/12); compare with a secondary metric before ranking methods.
Researcher Checklist (Expanded)

Researcher Checklist

  • Gap: Papers with explicit human feedback

    Coverage is a replication risk (16.7% vs 45% target).

  • Gap: Papers reporting quality controls

    Coverage is a replication risk (0% vs 30% target).

  • Gap: Papers naming benchmarks/datasets

    Coverage is a replication risk (0% vs 35% target).

  • Strong: Papers naming evaluation metrics

    Coverage is strong (83.3% vs 35% target).

  • Gap: Papers with known rater population

    Coverage is a replication risk (8.3% vs 35% target).

  • Gap: Papers with known annotation unit

    Coverage is a replication risk (16.7% vs 35% target).

Strengths

  • Agentic evaluation appears in 100% of papers.

Known Gaps

  • Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Rater population is under-specified (8.3% coverage).
  • Annotation unit is under-specified (16.7% coverage).

Suggested Next Analyses

  • Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
  • Track metric sensitivity by reporting both accuracy and cost.
Recommended Queries (Expanded)

Recommended Queries

Start Here (Best First 6)

Ranked for protocol completeness (human signal, benchmark + metric anchors, quality controls, and judge/human overlap).

Protocol Matrix (Top 12)

Use this to quickly compare protocol ingredients instead of scanning long prose.

Paper HF Signal Eval Modes Benchmarks Metrics QC
SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery

Feb 24, 2026

Yes Automatic Metrics Not Reported Cost Not Reported
PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training

Feb 14, 2026

Yes Automatic Metrics Not Reported Helpfulness Not Reported
The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

Feb 19, 2026

No
Not Reported
Llm As Judge , Automatic Metrics Not Reported Accuracy Not Reported
Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability

Feb 19, 2026

No
Not Reported
Automatic Metrics Not Reported Accuracy Not Reported
AgentDropoutV2: Optimizing Information Flow in Multi-Agent Systems via Test-Time Rectify-or-Reject Pruning

Feb 26, 2026

No
Not Reported
Automatic Metrics Not Reported Accuracy Not Reported
A Multi-Agent Framework for Medical AI: Leveraging Fine-Tuned GPT, LLaMA, and DeepSeek R1 for Evidence-Based and Bias-Aware Clinical Query Processing

Feb 15, 2026

No
Not Reported
Automatic Metrics Not Reported Accuracy , Bleu Not Reported
Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

Feb 25, 2026

No
Not Reported
Automatic Metrics Not Reported Accuracy , Success rate Not Reported
The Headless Firm: How AI Reshapes Enterprise Boundaries

Feb 24, 2026

No
Not Reported
Automatic Metrics Not Reported Throughput , Cost Not Reported
SAMAS: A Spectrum-Guided Multi-Agent System for Achieving Style Fidelity in Literary Translation

Feb 23, 2026

No
Not Reported
Automatic Metrics Not Reported Accuracy Not Reported
Whisper: Courtside Edition Enhancing ASR Performance Through LLM-Driven Context Generation

Feb 21, 2026

No
Not Reported
Automatic Metrics Not Reported Error rate , Wer Not Reported
Training Generalizable Collaborative Agents via Strategic Risk Aversion

Feb 25, 2026

No
Not Reported
Automatic Metrics Not Reported Not Reported Not Reported
A Hierarchical Multi-Agent System for Autonomous Discovery in Geoscientific Data Archives

Feb 24, 2026

No
Not Reported
Automatic Metrics Not Reported Not Reported Not Reported

Protocol Diff (Top Papers)

Fast side-by-side comparison for the highest-ranked papers in this hub.

Signal SparkMe: Adaptive Semi-Structured Interviewing for… PrivAct: Internalizing Contextual Privacy Preservat… The Emergence of Lab-Driven Alignment Signatures: A…
Human Feedback Expert VerificationPairwise PreferenceNot reported
Evaluation Modes Automatic MetricsAutomatic MetricsLlm As Judge, Automatic Metrics
Benchmarks Not reportedNot reportedNot reported
Metrics CostHelpfulnessAccuracy
Quality Controls Not reportedNot reportedNot reported
Rater Population Domain ExpertsUnknownUnknown
Annotation Unit TrajectoryUnknownUnknown
Suggested Reading Order (Extended)

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

  1. AgentDropoutV2: Optimizing Information Flow in Multi-Agent Systems via Test-Time Rectify-or-Reject Pruning

    Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: While Multi-Agent Systems (MAS) excel in complex reasoning, they suffer from the cascading.

  2. Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

    Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: Multi-robot task planning requires decomposing natural-language instructions into executable actions for heterogeneous robot.

  3. Training Generalizable Collaborative Agents via Strategic Risk Aversion

    Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Abstract: Many emerging agentic paradigms require agents to collaborate with one another (or people) to achieve.

  4. SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery

    Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + expert verification. Focus: cost. Abstract: The code, datasets, and evaluation protocols for SparkMe are.

  5. PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training

    Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + pairwise preferences. Focus: helpfulness. Abstract: By embedding privacy preferences into each agent, PrivAct enhances.

  6. The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

    Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: accuracy. Abstract: As Large Language Models (LLMs) transition from standalone chat interfaces to foundational.

  7. Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability

    Adds automatic metrics for broader protocol coverage within this hub. Signals: automatic metrics. Focus: accuracy. Abstract: In multi-agent IR pipelines for tasks such as search and ranking, LLM-based.

  8. A Multi-Agent Framework for Medical AI: Leveraging Fine-Tuned GPT, LLaMA, and DeepSeek R1 for Evidence-Based and Bias-Aware Clinical Query Processing

    Adds automatic metrics for broader protocol coverage within this hub. Signals: automatic metrics. Focus: accuracy. Abstract: Large language models (LLMs) show promise for healthcare question answering, but clinical.

Known Limitations

Known Limitations

  • Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Rater population is under-specified (8.3% coverage).
  • Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
Research Utility Snapshot (Detailed)

Research Utility Snapshot

Human Feedback Mix

  • Expert Verification (1)
  • Pairwise Preference (1)

Evaluation Modes

  • Automatic Metrics (12)
  • Llm As Judge (1)

Top Benchmarks

Top Metrics

  • Accuracy (6)
  • Cost (2)
  • Bleu (1)
  • Error rate (1)

Rater Population Mix

  • Domain Experts (1)

Quality Controls

Coverage diagnostics (sample-based): human-feedback 16.7% · benchmarks 0.0% · metrics 83.3% · quality controls 0.0%.

Top Papers

Related Hubs

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.