MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

Research Summary

Contribution Summary

• We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models.
• We release a benchmark of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora.
• Our experiments show that retrieval and generation models continue to struggle on conversations with UNanswerable, UNderspecified, and NONstandalone questions and UNclear responses.

Why It Matters For Eval

• We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models.
• We release a benchmark of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora.

Related Papers

• Personalized Graph-Empowered Large Language Model for Proactive Information Access Embedding Neighbor
• Emotion Collider: Dual Hyperbolic Mirror Manifolds for Sentiment Recovery via Anti Emotion Reflection Embedding Neighbor
• Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History Embedding Neighbor
• A Curious Class of Adpositional Multiword Expressions in Korean Embedding Neighbor
• Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning Embedding Neighbor
• Context Shapes LLMs Retrieval-Augmented Fact-Checking Effectiveness Embedding Neighbor
• Rethinking Disentanglement under Dependent Factors of Variation Embedding Neighbor
• DeVisE: Behavioral Testing of Medical Large Language Models Embedding Neighbor
• Yor-Sarc: A gold-standard dataset for sarcasm detection in a low-resource African language Embedding Neighbor
• The Headless Firm: How AI Reshapes Enterprise Boundaries Tag Overlap
• Measuring AI Ability to Complete Long Software Tasks Tag Overlap
• Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers Tag Overlap