MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Games

Jacob Eisenstein, Fantine Huot, Adam Fisch, Jonathan Berant, Mirella Lapata · Feb 27, 2026 · Citations: 0

Abstract

We present a scalable methodology for evaluating language models in multi-turn interactions, using a suite of collaborative games that require effective communication about private information. This enables an interactive scaling analysis, in which a fixed token budget is divided over a variable number of turns. We find that in many cases, language models are unable to use interactive collaboration to improve over the non-interactive baseline scenario in which one agent attempts to summarize its information and the other agent immediately acts -- despite substantial headroom. This suggests that state-of-the-art models still suffer from significant weaknesses in planning and executing multi-turn collaborative conversations. We analyze the linguistic features of these dialogues, assessing the roles of sycophancy, information density, and discourse coherence. While there is no single linguistic explanation for the collaborative weaknesses of contemporary language models, we note that humans achieve comparable task success at superior token efficiency by producing dialogues that are more coherent than those produced by most language models. The proactive management of private information is a defining feature of real-world communication, and we hope that MT-PingEval will drive further work towards improving this capability.

HFEPX Relevance Assessment

This paper appears adjacent to HFEPX scope (human-feedback/eval), but does not show strong direct protocol evidence in metadata/abstract.

Eval-Fit Score

5/100 • Low

Treat as adjacent context, not a core eval-method reference.

Human Feedback Signal

Not explicit in abstract metadata

Evaluation Signal

Detected

HFEPX Fit

Adjacent candidate

If you are doing eval pipeline work, start here:

Human Eval Hub LLM-as-Judge Hub Pairwise Preference Hub Tool-Use Eval Hub

Human Data Lens

Uses human feedback: No
Feedback types: None
Rater population: Unknown
Unit of annotation: Unknown
Expertise required: General
Extraction source: Runtime deterministic fallback

Evaluation Lens

Evaluation modes: Automatic Metrics
Agentic eval: None
Quality controls: Not reported
Confidence: 0.45
Flags: low_signal, possible_false_positive, runtime_fallback_extraction

Protocol And Measurement Signals

Benchmarks / Datasets

mt-pingeval

Reported Metrics

task successcoherence

Research Brief

Deterministic synthesis

Generated Mar 4, 2026, 12:25 AM · Grounded in abstract + metadata only

Key Takeaways

We present a scalable methodology for evaluating language models in multi-turn interactions, using a suite of collaborative games that require effective communication about private…
We find that in many cases, language models are unable to use interactive collaboration to improve over the non-interactive baseline scenario in which one agent attempts to…

Researcher Actions

Treat this as method context, then pivot to protocol-specific HFEPX hubs.
Cross-check benchmark overlap: mt-pingeval.
Validate metric comparability (task success, coherence).

Caveats

Generated from title, abstract, and extracted metadata only; full-paper implementation details are not parsed.
Low-signal flag detected: protocol relevance may be indirect.

Recommended Queries

human-eval protocol design pairwise preference data quality inter-rater agreement adjudication

Research Summary

Contribution Summary

We present a scalable methodology for evaluating language models in multi-turn interactions, using a suite of collaborative games that require effective communication about private information.
We find that in many cases, language models are unable to use interactive collaboration to improve over the non-interactive baseline scenario in which one agent attempts to summarize its information and the other agent immediately acts --…
While there is no single linguistic explanation for the collaborative weaknesses of contemporary language models, we note that humans achieve comparable task success at superior token efficiency by producing dialogues that are more coherent…

Why It Matters For Eval

We find that in many cases, language models are unable to use interactive collaboration to improve over the non-interactive baseline scenario in which one agent attempts to summarize its information and the other agent immediately acts --…
While there is no single linguistic explanation for the collaborative weaknesses of contemporary language models, we note that humans achieve comparable task success at superior token efficiency by producing dialogues that are more coherent…

Researcher Checklist

Gap: Human feedback protocol is explicit

No explicit human feedback protocol detected.
Pass: Evaluation mode is explicit

Detected: Automatic Metrics
Gap: Quality control reporting appears

No calibration/adjudication/IAA control explicitly detected.
Pass: Benchmark or dataset anchors are present

Detected: mt-pingeval
Pass: Metric reporting is present

Detected: task success, coherence

Category-Adjacent Papers (Broader Context)

These papers are nearby in arXiv category and useful for broader context, but not necessarily protocol-matched to this paper.

From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning Category Neighbor

Citations: 0 Relevance: 3.65
- Shared arXiv category (cs.CL, cs.LG)
- Shared terminology (evaluating)
InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models Category Neighbor

Citations: 0 Relevance: 3.65
- Shared arXiv category (cs.CL, cs.LG)
- Shared terminology (information)
Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking Category Neighbor

Citations: 0 Relevance: 3.65
- Shared arXiv category (cs.CL, cs.LG)
- Shared terminology (success)
Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems Category Neighbor

Citations: 0 Relevance: 3.30
- Shared arXiv category (cs.CL)
- Shared metric mentions
- Shared terminology (coherence, collaboration)
Assessing Deanonymization Risks with Stylometry-Assisted LLM Agent Category Neighbor

Citations: 0 Relevance: 3.20
- Shared arXiv category (cs.CL, cs.LG)
Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models Category Neighbor

Citations: 0 Relevance: 3.20
- Shared arXiv category (cs.CL, cs.LG)
Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning Category Neighbor

Citations: 0 Relevance: 3.20
- Shared arXiv category (cs.CL, cs.LG)
A Mixture-of-Experts Model for Multimodal Emotion Recognition in Conversations Category Neighbor

Citations: 0 Relevance: 2.50
- Shared arXiv category (cs.CL)
- Shared terminology (multi-turn, information)

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote