Multi Agent + Automatic Metrics (Last 45 Days)

Why This Matters For Eval Research

• 16.7% of papers report explicit human-feedback signals, led by expert verification.
• automatic metrics appears in 100% of papers in this hub.
• multi-agent setups appears in 100% of papers, indicating agentic evaluation demand.

Protocol Takeaways

• Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
• Rater context is mostly domain experts, and annotation is commonly ranking annotation; use this to scope replication staffing.
• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.

Metric Interpretation

• accuracy is reported in 50% of hub papers (6/12); compare with a secondary metric before ranking methods.
• cost is reported in 16.7% of hub papers (2/12); compare with a secondary metric before ranking methods.

Researcher Checklist

• Gap: Papers with explicit human feedback

  Coverage is a replication risk (16.7% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (0% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (0% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (83.3% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (8.3% vs 35% target).

• Gap: Papers with known annotation unit

  Coverage is a replication risk (16.7% vs 35% target).

Strengths

• Agentic evaluation appears in 100% of papers.

Known Gaps

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (8.3% coverage).
• Annotation unit is under-specified (16.7% coverage).

Suggested Next Analyses

• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
• Track metric sensitivity by reporting both accuracy and cost.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. AgentDropoutV2: Optimizing Information Flow in Multi-Agent Systems via Test-Time Rectify-or-Reject Pruning

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: While Multi-Agent Systems (MAS) excel in complex reasoning, they suffer from the cascading.

2. Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: Multi-robot task planning requires decomposing natural-language instructions into executable actions for heterogeneous robot.

3. Training Generalizable Collaborative Agents via Strategic Risk Aversion

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Abstract: Many emerging agentic paradigms require agents to collaborate with one another (or people) to achieve.

4. SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + expert verification. Focus: cost. Abstract: The code, datasets, and evaluation protocols for SparkMe are.

5. PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + pairwise preferences. Focus: helpfulness. Abstract: By embedding privacy preferences into each agent, PrivAct enhances.

6. The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: accuracy. Abstract: As Large Language Models (LLMs) transition from standalone chat interfaces to foundational.

7. Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability

   Adds automatic metrics for broader protocol coverage within this hub. Signals: automatic metrics. Focus: accuracy. Abstract: In multi-agent IR pipelines for tasks such as search and ranking, LLM-based.

8. A Multi-Agent Framework for Medical AI: Leveraging Fine-Tuned GPT, LLaMA, and DeepSeek R1 for Evidence-Based and Bias-Aware Clinical Query Processing

   Adds automatic metrics for broader protocol coverage within this hub. Signals: automatic metrics. Focus: accuracy. Abstract: Large language models (LLMs) show promise for healthcare question answering, but clinical.

Known Limitations

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (8.3% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot