Human Feedback Types
missingNone explicit
No explicit feedback protocol extracted.
"Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents."
HFEPX · Eval paper review
QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents
Ye Yuan, Rui Song, Weien Li, Zeyu Li +11 more
Published
May 26, 2026
Citations
0
Trust level
Low
Usefulness score
10/100 (Low)
Extraction confidence
40% (Low)
Derived from extracted protocol signals and abstract evidence.
Rater population
Not reported
Signals refreshed
Aug 31, 2026
Should you rely on this paper?
This paper is adjacent to HFEPX scope and is best used for background context, not as a primary protocol reference.
Use this as background context only. Do not make protocol decisions from this page alone.
Low-signal caution for protocol decisions
Use this page for context, then validate protocol choices against stronger HFEPX references before implementation decisions.
- The available metadata is too thin to trust this as a primary source.
- The abstract does not clearly name benchmarks or metrics.
Best use
Background context only
Use if you need
A secondary eval reference to pair with stronger protocol papers.
What to verify
Read the full paper before copying any benchmark, metric, or protocol choices.
Main weakness
The available metadata is too thin to trust this as a primary source.
Human feedback signal
Not explicit
Not explicit in abstract metadata
Evaluation signal
Detected
Eval setup described
Usefulness for eval research
10/100
Adjacent candidate
Treat as adjacent context, not a core eval-method reference.
If you are doing eval pipeline work, start here
Abstract
Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents. However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent's language is actually grounded in what it perceived and did, or to identify the failure modes underlying its behavior. To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning. QUACK evaluates agents at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent's ground-truth trajectory from engine logs and checks every discussion claim against it, automatically flagging spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Evaluating three frontier VLMs in both homogeneous and cross-model adversarial settings, we find that even the strongest agent hallucinates 15.1% of its verifiable spatial claims and 11.5% of accusations are strictly unsupported. We release the full engine, evaluation framework, toolkit, and logs in https://github.com/AAAAA-Academia-Attractions/QUACK.
What we could verify
These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.
Evaluation Modes
partialAutomatic Metrics
Includes extracted eval setup.
"Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents."
Quality Controls
missingNot reported
No explicit QC controls found.
"Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents."
Benchmarks / Datasets
missingNot extracted
No benchmark anchors detected.
"Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents."
Reported Metrics
missingNot extracted
No metric anchors detected.
"Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents."
Benchmarks and datasets
No benchmark or dataset names were extracted from the available abstract.
Reported metrics
No metric terms were extracted from the available abstract.
Human feedback details
- Uses human feedback
- No
- Feedback types
- None
- Rater population
- Not reported
- Unit of annotation
- Trajectory (inferred)
- Expertise required
- General
Evaluation details
- Evaluation modes
- Automatic Metrics
- Agentic eval
- Long Horizon
- Quality controls
- Not reported
- Evidence quality
- Low
- Use this page as
- Background context only
Research brief
Metadata summary
Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents.
Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.
Key takeaways
- Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents.
- However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent's language is actually grounded in what it perceived and did, or to identify the failure modes underlying its behavior.
- To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning.
Researcher actions
- Compare this paper against nearby papers in the same arXiv category before using it for protocol decisions.
- Validate inferred eval signals (Simulation environment) against the full paper.
- Use related-paper links to find stronger protocol-specific references.
Caveats
- Generated from abstract + metadata only; no PDF parsing.
- Signals below are heuristic and may miss details reported outside the abstract.
Recommended queries
Contribution summary
- However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent's language is actually grounded in what it perceived and did, or to…
- To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning.
- Evaluating three frontier VLMs in both homogeneous and cross-model adversarial settings, we find that even the strongest agent hallucinates 15.1% of its verifiable spatial claims and 11.5% of accusations are strictly unsupported.
Why it matters for eval
- However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent's language is actually grounded in what it perceived and did, or to…
- To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning.
Researcher checklist
-
Human feedback protocol is explicit
No explicit human feedback protocol detected.
-
Evaluation mode is explicit
Detected: Automatic Metrics
-
Quality control reporting appears
No calibration/adjudication/IAA control explicitly detected.
-
Benchmark or dataset anchors are present
No benchmark/dataset anchor extracted from abstract.
-
Metric reporting is present
No metric terms extracted.