Multi Agent + Automatic Metrics (Last 120 Days)

Why This Matters For Eval Research

• 15.4% of papers report explicit human-feedback signals, led by expert verification.
• automatic metrics appears in 100% of papers in this hub.
• multi-agent setups appears in 100% of papers, indicating agentic evaluation demand.

Protocol Takeaways

• Most common quality-control signal is adjudication (7.7% of papers).
• Rater context is mostly domain experts, and annotation is commonly ranking annotation; use this to scope replication staffing.
• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.

Metric Interpretation

• accuracy is reported in 53.8% of hub papers (7/13); compare with a secondary metric before ranking methods.
• cost is reported in 15.4% of hub papers (2/13); compare with a secondary metric before ranking methods.

Researcher Checklist

• Gap: Papers with explicit human feedback

  Coverage is a replication risk (15.4% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (7.7% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (0% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (84.6% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (7.7% vs 35% target).

• Gap: Papers with known annotation unit

  Coverage is a replication risk (15.4% vs 35% target).

Strengths

• Agentic evaluation appears in 100% of papers.

Known Gaps

• Only 7.7% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (7.7% coverage).
• Annotation unit is under-specified (15.4% coverage).

Suggested Next Analyses

• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
• Track metric sensitivity by reporting both accuracy and cost.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. AgentDropoutV2: Optimizing Information Flow in Multi-Agent Systems via Test-Time Rectify-or-Reject Pruning

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: While Multi-Agent Systems (MAS) excel in complex reasoning, they suffer from the cascading.

2. Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: Multi-robot task planning requires decomposing natural-language instructions into executable actions for heterogeneous robot.

3. Training Generalizable Collaborative Agents via Strategic Risk Aversion

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Abstract: Many emerging agentic paradigms require agents to collaborate with one another (or people) to achieve.

4. SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + expert verification. Focus: cost. Abstract: The code, datasets, and evaluation protocols for SparkMe are.

5. PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + pairwise preferences. Focus: helpfulness. Abstract: By embedding privacy preferences into each agent, PrivAct enhances.

6. The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: accuracy. Abstract: As Large Language Models (LLMs) transition from standalone chat interfaces to foundational.

7. From Competition to Coordination: Market Making as a Scalable Framework for Safe and Aligned Multi-Agent LLM Systems

   Adds automatic metrics for broader protocol coverage within this hub. Signals: automatic metrics. Focus: accuracy. Abstract: As foundation models are increasingly deployed as interacting agents in multi-agent systems,.

8. Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability

   Adds automatic metrics for broader protocol coverage within this hub. Signals: automatic metrics. Focus: accuracy. Abstract: In multi-agent IR pipelines for tasks such as search and ranking, LLM-based.

Known Limitations

• Only 7.7% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (7.7% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot