Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval

Research Summary

Contribution Summary

• We present the Conversational Data Retrieval (CDR) benchmark, the first comprehensive test set for evaluating systems that retrieve conversation data for product insights.
• With 1.6k queries across five analytical tasks and 9.1k conversations, our benchmark provides a reliable standard for measuring conversational data retrieval performance.
• Our evaluation of 16 popular embedding models shows that even the best models reach only around NDCG@10 of 0.51, revealing a substantial gap between document and conversational data retrieval capabilities.

Why It Matters For Eval

• We present the Conversational Data Retrieval (CDR) benchmark, the first comprehensive test set for evaluating systems that retrieve conversation data for product insights.
• With 1.6k queries across five analytical tasks and 9.1k conversations, our benchmark provides a reliable standard for measuring conversational data retrieval performance.

Related Papers

• TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Models Embedding Neighbor
• SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models Embedding Neighbor
• Indic-TunedLens: Interpreting Multilingual Models in Indian Languages Embedding Neighbor
• Programming by Backprop: An Instruction is Worth 100 Examples When Finetuning LLMs Embedding Neighbor
• ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models Embedding Neighbor
• Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG Embedding Neighbor
• RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics Embedding Neighbor
• DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries Embedding Neighbor
• Uncovering Context Reliance in Unstructured Knowledge Editing Embedding Neighbor
• Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination Tag Overlap
• Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models Tag Overlap
• EARL: Entropy-Aware RL Alignment of LLMs for Reliable RTL Code Generation Tag Overlap