Multi Agent + General (Last 45 Days)

Why This Matters For Eval Research

• 25% of papers report explicit human-feedback signals, led by pairwise preferences.
• automatic metrics appears in 41.7% of papers in this hub.
• Mind2Web is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
• Rater context is mostly domain experts, and annotation is commonly ranking annotation; use this to scope replication staffing.
• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.

Benchmark Interpretation

• Mind2Web appears in 8.3% of hub papers (1/12); use this cohort for benchmark-matched comparisons.
• OSWorld appears in 8.3% of hub papers (1/12); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 25% of hub papers (3/12); compare with a secondary metric before ranking methods.
• cost is reported in 8.3% of hub papers (1/12); compare with a secondary metric before ranking methods.

Researcher Checklist

• Gap: Papers with explicit human feedback

  Coverage is a replication risk (25% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (0% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (16.7% vs 35% target).

• Moderate: Papers naming evaluation metrics

  Coverage is usable but incomplete (33.3% vs 35% target).

• Moderate: Papers with known rater population

  Coverage is usable but incomplete (25% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (33.3% vs 35% target).

Strengths

• Agentic evaluation appears in 100% of papers.

Known Gaps

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Benchmark coverage is thin (16.7% of papers mention benchmarks/datasets).
• LLM-as-judge appears without enough inter-annotator agreement reporting.

Suggested Next Analyses

• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
• Stratify by benchmark (Mind2Web vs OSWorld) before comparing methods.
• Track metric sensitivity by reporting both accuracy and cost.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Toward Expert Investment Teams:A Multi-Agent LLM System with Fine-Grained Trading Tasks

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: Crucially, further analysis of intermediate agent outputs suggests that alignment between analytical outputs and downstream.

2. Decentralized Ranking Aggregation: Gossip Algorithms for Borda and Copeland Consensus

   Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: The concept of ranking aggregation plays a central role in preference analysis, and numerous algorithms.

3. Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: Multi-robot task planning requires decomposing natural-language instructions into executable actions for heterogeneous robot.

4. World-Model-Augmented Web Agents with Action Correction

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: LLM-as-judge. Focus: VisualWebArena. Abstract: A world model, specialized in environmental state transitions, simulates action outcomes, which a.

5. Multi-Agent Comedy Club: Investigating Community Discussion Effects on LLM Humor Generation

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: pairwise preferences. Abstract: Across 50 rounds (250 paired monologues) judged by five expert annotators using A/B preference.

6. The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: accuracy. Abstract: As Large Language Models (LLMs) transition from standalone chat interfaces to foundational.

7. Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents

   Adds simulation environments for broader protocol coverage within this hub. Signals: simulation environments. Focus: WebArena. Abstract: The paper introduces GUI-Owl-1.5, the latest native GUI agent model that features.

8. Cooperative-Competitive Team Play of Real-World Craft Robots

   Adds simulation environments for broader protocol coverage within this hub. Signals: simulation environments. Abstract: Multi-agent deep Reinforcement Learning (RL) has made significant progress in developing intelligent game-playing agents.

Known Limitations

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Benchmark coverage is thin (16.7% of papers mention benchmarks/datasets).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot