Uncovering Autoregressive LLM Knowledge of Thematic Fit in Event Representation

Research Summary

Contribution Summary

• The thematic fit estimation task measures semantic arguments' compatibility with a specific semantic role for a specific predicate.
• We investigate if LLMs have consistent, expressible knowledge of event arguments' thematic fit by experimenting with various prompt designs, manipulating input context, reasoning, and output forms.
• We set a new state-of-the-art on thematic fit benchmarks, but show that closed and open weight LLMs respond differently to our prompting strategies: Closed models achieve better scores overall and benefit from multi-step reasoning, but they

Why It Matters For Eval

• We set a new state-of-the-art on thematic fit benchmarks, but show that closed and open weight LLMs respond differently to our prompting strategies: Closed models achieve better scores overall and benefit from multi-step reasoning, but they

Related Papers

• BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models Embedding Neighbor
• Entropy-Based Data Selection for Language Models Embedding Neighbor
• MultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learning Embedding Neighbor
• MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation Embedding Neighbor
• Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs Embedding Neighbor
• From Raw Corpora to Domain Benchmarks: Automated Evaluation of LLM Domain Expertise Embedding Neighbor
• A Comparative Analysis of Social Network Topology in Reddit and Moltbook Embedding Neighbor
• Transforming Science Learning Materials in the Era of Artificial Intelligence Embedding Neighbor
• Disentangling Geometry, Performance, and Training in Language Models Embedding Neighbor
• BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learning Tag Overlap
• Provably Safe Generative Sampling with Constricting Barrier Functions Tag Overlap
• Creating a digital poet Tag Overlap