On the scaling relationship between cloze probabilities and language model next-token prediction

Research Summary

Contribution Summary

• Recent work has shown that larger language models have better predictive power for eye movement and reading time data.
• While even the best models under-allocate probability mass to human responses, larger models assign higher-quality estimates of next tokens and their likelihood of production in cloze data because they are less sensitive to lexical co-occur
• The results provide support for the claim that the greater memorization capacity of larger models helps them guess more semantically appropriate words, but makes them less sensitive to low-level information that is relevant for word recogni

Why It Matters For Eval

• While even the best models under-allocate probability mass to human responses, larger models assign higher-quality estimates of next tokens and their likelihood of production in cloze data because they are less sensitive to lexical co-occur

Related Papers

• FeatBench: Towards More Realistic Evaluation of Feature-level Code Generation Embedding Neighbor
• PVminer: A Domain-Specific Tool to Detect the Patient Voice in Patient Generated Data Embedding Neighbor
• SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery Embedding Neighbor
• A Systematic Review of Algorithmic Red Teaming Methodologies for Assurance and Security of AI Applications Embedding Neighbor
• ARGUS: Adaptive Rotation-Invariant Geometric Unsupervised System Embedding Neighbor
• World Simulation with Video Foundation Models for Physical AI Embedding Neighbor
• Natural Language Processing Models for Robust Document Categorization Embedding Neighbor
• CogniAlign: Survivability-Grounded Multi-Agent Moral Reasoning for Safe and Transparent AI Embedding Neighbor
• PerSoMed: A Large-Scale Balanced Dataset for Persian Social Media Text Classification Embedding Neighbor
• The Headless Firm: How AI Reshapes Enterprise Boundaries Tag Overlap
• Measuring AI Ability to Complete Long Software Tasks Tag Overlap
• Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers Tag Overlap