LLMs Process Lists With General Filter Heads

Research Summary

Contribution Summary

• We investigate the mechanisms underlying a range of list-processing tasks in LLMs, and we find that LLMs have learned to encode a compact, causal representation of a general filtering operation that mirrors the generic "filter" function of
• Using causal mediation analysis on a diverse set of list-processing tasks, we find that a small number of attention heads, which we dub filter heads, encode a compact representation of the filtering predicate in their query states at certai
• We demonstrate that this predicate representation is general and portable: it can be extracted and reapplied to execute the same filtering operation on different collections, presented in different formats, languages, or even in tasks.

Why It Matters For Eval

• Our results reveal that transformer LMs can develop human-interpretable implementations of abstract computational operations that generalize in ways that are surprisingly similar to strategies used in traditional functional programming patt

Related Papers

• Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence Embedding Neighbor
• A Survey on the Optimization of Large Language Model-based Agents Embedding Neighbor
• Modality-Guided Mixture of Graph Experts with Entropy-Triggered Routing for Multimodal Recommendation Embedding Neighbor
• PII-Bench: Evaluating Query-Aware Privacy Protection Systems Embedding Neighbor
• Vichara: Appellate Judgment Prediction and Explanation for the Indian Judicial System Embedding Neighbor
• DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning Embedding Neighbor
• Scaling Beyond Masked Diffusion Language Models Embedding Neighbor
• Uncovering Context Reliance in Unstructured Knowledge Editing Embedding Neighbor
• ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns Embedding Neighbor
• Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination Tag Overlap
• Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models Tag Overlap
• EARL: Entropy-Aware RL Alignment of LLMs for Reliable RTL Code Generation Tag Overlap