Multi Agent Papers (Last 60 Days)

Why This Matters For Eval Research

• 34.6% of papers report explicit human-feedback signals, led by expert verification.
• automatic metrics appears in 41.4% of papers in this hub.
• Lawbench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is rater calibration (3.4% of papers).
• Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.
• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.

Benchmark Interpretation

• Lawbench appears in 3.8% of hub papers (1/29); use this cohort for benchmark-matched comparisons.
• LiveCodeBench appears in 3.8% of hub papers (1/29); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 23.1% of hub papers (6/29); compare with a secondary metric before ranking methods.
• cost is reported in 7.7% of hub papers (2/29); compare with a secondary metric before ranking methods.

Researcher Checklist

• Moderate: Papers with explicit human feedback

  Coverage is usable but incomplete (34.6% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (3.8% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (15.4% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (38.5% vs 35% target).

• Moderate: Papers with known rater population

  Coverage is usable but incomplete (26.9% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (23.1% vs 35% target).

Strengths

• Agentic evaluation appears in 100% of papers.

Known Gaps

• Only 3.8% of papers report quality controls; prioritize calibration/adjudication evidence.
• Annotation unit is under-specified (23.1% coverage).
• Benchmark coverage is thin (15.4% of papers mention benchmarks/datasets).

Suggested Next Analyses

• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
• Stratify by benchmark (Lawbench vs LiveCodeBench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and cost.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Toward Expert Investment Teams:A Multi-Agent LLM System with Fine-Grained Trading Tasks

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: Crucially, further analysis of intermediate agent outputs suggests that alignment between analytical outputs and downstream.

2. AgentDropoutV2: Optimizing Information Flow in Multi-Agent Systems via Test-Time Rectify-or-Reject Pruning

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: While Multi-Agent Systems (MAS) excel in complex reasoning, they suffer from the cascading.

3. Decentralized Ranking Aggregation: Gossip Algorithms for Borda and Copeland Consensus

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: The concept of ranking aggregation plays a central role in preference analysis, and numerous algorithms.

4. Team of Thoughts: Efficient Test-time Scaling of Agentic Systems through Orchestrated Tool Calling

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: expert verification. Focus: LiveCodeBench. Abstract: Existing Multi-Agent Systems (MAS) typically rely on static, homogeneous model configurations, limiting.

5. TherapyProbe: Generating Design Knowledge for Relational Safety in Mental Health Chatbots Through Adversarial Simulation

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: simulation environments + expert verification. Abstract: As mental health chatbots proliferate to address the global treatment gap,.

6. World-Model-Augmented Web Agents with Action Correction

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: VisualWebArena. Abstract: A world model, specialized in environmental state transitions, simulates action outcomes, which.

7. The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: accuracy. Abstract: As Large Language Models (LLMs) transition from standalone chat interfaces to foundational.

8. SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery

   Adds automatic metrics with expert verification for broader protocol coverage within this hub. Signals: automatic metrics + expert verification. Focus: cost. Abstract: The code, datasets, and evaluation protocols.

Known Limitations

• Only 3.8% of papers report quality controls; prioritize calibration/adjudication evidence.
• Annotation unit is under-specified (23.1% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot