Skip to content
← Back to explorer

HFEPX Hub

Multi Agent + General (Last 120 Days)

Updated from current HFEPX corpus (Mar 1, 2026). 13 papers are grouped in this hub page.

Read Full Context

Updated from current HFEPX corpus (Mar 1, 2026). 13 papers are grouped in this hub page. Common evaluation modes: Automatic Metrics, Simulation Env. Most common rater population: Domain Experts. Common annotation unit: Ranking. Frequent quality control: Adjudication. Frequently cited benchmark: Mind2Web. Common metric signal: accuracy. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 16, 2026.

Papers: 13 Last published: Feb 16, 2026 Global RSS Tag RSS
Multi AgentGeneralLast 120d

Researcher Quick Triage

This hub is best used for protocol triage and replication planning from abstract-level evidence. Quality band: Developing .

High-Signal Coverage

100.0%

13 / 13 sampled papers are not low-signal flagged.

Replication-Ready Set

0

Benchmark + metric + eval mode explicitly present.

Judge/Human Comparability

0

Papers containing both `human_eval` and `llm_as_judge`.

  • 0 papers are replication-ready (benchmark + metric + explicit evaluation mode).
  • 0 papers support judge-vs-human agreement analysis.
  • 1 papers report explicit quality controls (calibration/adjudication/IAA).

Primary action: Use this page for scouting only; collect additional papers before attempting replication-critical comparisons.

Why This Matters (Expanded)

Why This Matters For Eval Research

  • 23.1% of papers report explicit human-feedback signals, led by pairwise preferences.
  • automatic metrics appears in 46.2% of papers in this hub.
  • Mind2Web is a recurring benchmark anchor for cross-paper comparisons in this page.
Protocol Notes (Expanded)

Protocol Takeaways

  • Most common quality-control signal is adjudication (7.7% of papers).
  • Rater context is mostly domain experts, and annotation is commonly ranking annotation; use this to scope replication staffing.
  • Pair this hub with a human_eval-heavy hub to validate judge-model calibration.

Benchmark Interpretation

  • Mind2Web appears in 7.7% of hub papers (1/13); use this cohort for benchmark-matched comparisons.
  • OSWorld appears in 7.7% of hub papers (1/13); use this cohort for benchmark-matched comparisons.

Metric Interpretation

  • accuracy is reported in 30.8% of hub papers (4/13); compare with a secondary metric before ranking methods.
  • cost is reported in 7.7% of hub papers (1/13); compare with a secondary metric before ranking methods.
Researcher Checklist (Expanded)

Researcher Checklist

  • Gap: Papers with explicit human feedback

    Coverage is a replication risk (23.1% vs 45% target).

  • Gap: Papers reporting quality controls

    Coverage is a replication risk (7.7% vs 30% target).

  • Gap: Papers naming benchmarks/datasets

    Coverage is a replication risk (15.4% vs 35% target).

  • Strong: Papers naming evaluation metrics

    Coverage is strong (38.5% vs 35% target).

  • Moderate: Papers with known rater population

    Coverage is usable but incomplete (23.1% vs 35% target).

  • Moderate: Papers with known annotation unit

    Coverage is usable but incomplete (30.8% vs 35% target).

Strengths

  • Agentic evaluation appears in 100% of papers.

Known Gaps

  • Only 7.7% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Rater population is under-specified (23.1% coverage).
  • Benchmark coverage is thin (15.4% of papers mention benchmarks/datasets).

Suggested Next Analyses

  • Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
  • Stratify by benchmark (Mind2Web vs OSWorld) before comparing methods.
  • Track metric sensitivity by reporting both accuracy and cost.
  • Add inter-annotator agreement checks when reproducing these protocols.
Recommended Queries (Expanded)

Recommended Queries

Start Here (Best First 6)

Ranked for protocol completeness (human signal, benchmark + metric anchors, quality controls, and judge/human overlap).

Protocol Matrix (Top 12)

Use this to quickly compare protocol ingredients instead of scanning long prose.

Paper HF Signal Eval Modes Benchmarks Metrics QC
Multi-Agent Comedy Club: Investigating Community Discussion Effects on LLM Humor Generation

Feb 16, 2026

Yes Not Reported Not Reported Not Reported Not Reported
World-Model-Augmented Web Agents with Action Correction

Feb 17, 2026

No
Not Reported
Llm As Judge , Simulation Env VisualWebArena , Mind2Web Not Reported Not Reported
Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents

Feb 15, 2026

No
Not Reported
Simulation Env WebArena , OSWorld Not Reported Not Reported
Toward Expert Investment Teams:A Multi-Agent LLM System with Fine-Grained Trading Tasks

Feb 26, 2026

Yes Not Reported Not Reported Not Reported Not Reported
Decentralized Ranking Aggregation: Gossip Algorithms for Borda and Copeland Consensus

Feb 26, 2026

Yes Not Reported Not Reported Not Reported Not Reported
From Competition to Coordination: Market Making as a Scalable Framework for Safe and Aligned Multi-Agent LLM Systems

Nov 18, 2025

No
Not Reported
Automatic Metrics Not Reported Accuracy Adjudication
The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

Feb 19, 2026

No
Not Reported
Llm As Judge , Automatic Metrics Not Reported Accuracy Not Reported
Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability

Feb 19, 2026

No
Not Reported
Automatic Metrics Not Reported Accuracy Not Reported
Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

Feb 25, 2026

No
Not Reported
Automatic Metrics Not Reported Accuracy , Success rate Not Reported
The Headless Firm: How AI Reshapes Enterprise Boundaries

Feb 24, 2026

No
Not Reported
Automatic Metrics Not Reported Throughput , Cost Not Reported
Cooperative-Competitive Team Play of Real-World Craft Robots

Feb 24, 2026

No
Not Reported
Simulation Env Not Reported Not Reported Not Reported
Architecting AgentOS: From Token-Level Context to Emergent System-Level Intelligence

Feb 24, 2026

No
Not Reported
Simulation Env Not Reported Not Reported Not Reported

Protocol Diff (Top Papers)

Fast side-by-side comparison for the highest-ranked papers in this hub.

Signal Multi-Agent Comedy Club: Investigating Community Di… World-Model-Augmented Web Agents with Action Correc… Mobile-Agent-v3.5: Multi-platform Fundamental GUI A…
Human Feedback Pairwise Preference, Rubric RatingNot reportedNot reported
Evaluation Modes Not reportedLlm As Judge, Simulation EnvSimulation Env
Benchmarks Not reportedVisualWebArena, Mind2WebWebArena, OSWorld
Metrics Not reportedNot reportedNot reported
Quality Controls Not reportedNot reportedNot reported
Rater Population Domain ExpertsDomain ExpertsUnknown
Annotation Unit PairwiseUnknownTrajectory
Suggested Reading Order (Extended)

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

  1. Toward Expert Investment Teams:A Multi-Agent LLM System with Fine-Grained Trading Tasks

    Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: Crucially, further analysis of intermediate agent outputs suggests that alignment between analytical outputs and downstream.

  2. Decentralized Ranking Aggregation: Gossip Algorithms for Borda and Copeland Consensus

    Start here for detailed protocol reporting and quality-control evidence. Signals: pairwise preferences. Abstract: The concept of ranking aggregation plays a central role in preference analysis, and numerous algorithms.

  3. Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

    Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: Multi-robot task planning requires decomposing natural-language instructions into executable actions for heterogeneous robot.

  4. World-Model-Augmented Web Agents with Action Correction

    Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: LLM-as-judge. Focus: VisualWebArena. Abstract: A world model, specialized in environmental state transitions, simulates action outcomes, which a.

  5. Multi-Agent Comedy Club: Investigating Community Discussion Effects on LLM Humor Generation

    Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: pairwise preferences. Abstract: Across 50 rounds (250 paired monologues) judged by five expert annotators using A/B preference.

  6. The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

    Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: accuracy. Abstract: As Large Language Models (LLMs) transition from standalone chat interfaces to foundational.

  7. Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents

    Adds simulation environments for broader protocol coverage within this hub. Signals: simulation environments. Focus: WebArena. Abstract: The paper introduces GUI-Owl-1.5, the latest native GUI agent model that features.

  8. From Competition to Coordination: Market Making as a Scalable Framework for Safe and Aligned Multi-Agent LLM Systems

    Adds automatic metrics for broader protocol coverage within this hub. Signals: automatic metrics. Focus: accuracy. Abstract: As foundation models are increasingly deployed as interacting agents in multi-agent systems,.

Known Limitations

Known Limitations

  • Only 7.7% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Rater population is under-specified (23.1% coverage).
  • Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
Research Utility Snapshot (Detailed)

Research Utility Snapshot

Human Feedback Mix

  • Pairwise Preference (3)
  • Expert Verification (1)
  • Rubric Rating (1)

Evaluation Modes

  • Automatic Metrics (6)
  • Simulation Env (4)
  • Llm As Judge (2)

Top Benchmarks

  • Mind2Web (1)
  • OSWorld (1)
  • VisualWebArena (1)
  • WebArena (1)

Top Metrics

  • Accuracy (4)
  • Cost (1)
  • Success rate (1)
  • Throughput (1)

Rater Population Mix

  • Domain Experts (3)

Quality Controls

  • Adjudication (1)
Coverage diagnostics (sample-based): human-feedback 23.1% · benchmarks 15.4% · metrics 38.5% · quality controls 7.7%.

Top Papers

Related Hubs

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.