Skip to content
OpenTrain AIFor AI Companies

HFEPX · Eval paper review

Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation

Minsoo Song, Chanwoo Kim, Sugyeong Eo, Chanjun Park

Published

Aug 31, 2026

Citations

0

Trust level

High

Usefulness score

67/100 (Medium)

Extraction confidence

75% (High)

Derived from extracted protocol signals and abstract evidence.

Rater population

Not reported

Signals refreshed

Aug 31, 2026

Open arXiv Find implementation Shortlist (0) RSS feed

Tags

General Llm As Judge Multi Agent Rubric Rating

Should you rely on this paper?

This paper has useful evaluation signal, but protocol completeness is partial; pair it with related papers before deciding implementation strategy.

Use this as a practical starting point for protocol research, then validate against the original paper.

Best use

Secondary protocol comparison source

Use if you need

A secondary eval reference to pair with stronger protocol papers.

What to verify

Validate the evaluation procedure and quality controls in the full paper before operational use.

Main weakness

No major weakness surfaced.

Human feedback signal
Detected
From extracted signals
Evaluation signal
Detected
Eval setup described
Usefulness for eval research
67/100
Moderate-confidence candidate

Useful as a secondary reference; validate protocol details against neighboring papers.

If you are doing eval pipeline work, start here

Human Eval Papers LLM-as-Judge Papers Pairwise Preference Papers Tool-Use Evaluation Search

Abstract

Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-agent agreement does not guarantee alignment with human judgments. In this paper, we compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks and design three ablations to isolate the impact of role prompting, multi-round interaction, and explicit score sharing. Evaluations across six LLMs show that the single-judge baseline achieves the strongest human alignment on average across six judge models, whereas MAD shows degradation in human alignment on both tasks. Our ablations demonstrate that this performance drop stems primarily from asymmetric role prompting rather than the interaction itself. Specifically, assigning a strict judge role introduces a systematic downward bias that the consensus process fails to correct. The central finding is that this bias reflects strict-stance dominance beyond averaging: the consensus score falls well beyond the arithmetic midpoint of the standalone strict and lenient conditions, rather than averaging them out. Removing role asymmetry (Symmetric MAD) largely recovers baseline performance, while masking peer scores widens inter-agent disagreement on average and worsens average human alignment. These findings demonstrate that multi-agent consensus can enforce artificial agreement at the expense of true human alignment, revealing a structural limitation in consensus-style, role-specialized MAD protocols for subjective scoring.

What we could verify

These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.

Human Feedback Types

strong

Rubric Rating

Directly usable for protocol triage.

"Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus."

Evaluation Modes

strong

Llm As Judge

Includes extracted eval setup.

"Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus."

Quality Controls

missing

Not reported

No explicit QC controls found.

"Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus."

Benchmarks / Datasets

strong

DROP

Useful for quick benchmark comparison.

"Our ablations demonstrate that this performance drop stems primarily from asymmetric role prompting rather than the interaction itself."

Reported Metrics

missing

Not extracted

No metric anchors detected.

"Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus."

Benchmarks and datasets

DROP

Reported metrics

No metric terms were extracted from the available abstract.

Human feedback details
Uses human feedback
Yes
Feedback types
Rubric Rating
Rater population
Not reported
Unit of annotation
Multi Dim Rubric
Expertise required
General
Evaluation details
Evaluation modes
Llm As Judge
Agentic eval
Multi Agent
Quality controls
Not reported
Evidence quality
High
Use this page as
Secondary protocol comparison source

Research brief

Metadata summary

Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus.

Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.

Key takeaways

  • Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus.
  • However, for subjective rubric-based scoring, inter-agent agreement does not guarantee alignment with human judgments.
  • In this paper, we compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks and design three ablations to isolate the impact of role prompting, multi-round interaction, and explicit score sharing.

Researcher actions

  • Compare this paper against nearby papers in the same arXiv category before using it for protocol decisions.
  • Check the full text for explicit evaluation design choices (raters, protocol, and metrics).
  • Use related-paper links to find stronger protocol-specific references.

Caveats

  • Generated from abstract + metadata only; no PDF parsing.
  • Signals below are heuristic and may miss details reported outside the abstract.

Recommended queries

Rubric rating evaluation Agreement / Kappa

Contribution summary

  • Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus.
  • However, for subjective rubric-based scoring, inter-agent agreement does not guarantee alignment with human judgments.
  • In this paper, we compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks and design three ablations to isolate the impact of role prompting, multi-round interaction, and explicit score sharing.

Why it matters for eval

  • Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus.
  • However, for subjective rubric-based scoring, inter-agent agreement does not guarantee alignment with human judgments.

Researcher checklist

  • Human feedback protocol is explicit

    Detected: Rubric Rating

  • Evaluation mode is explicit

    Detected: Llm As Judge

  • Quality control reporting appears

    No calibration/adjudication/IAA control explicitly detected.

  • Benchmark or dataset anchors are present

    Detected: DROP

  • Metric reporting is present

    No metric terms extracted.