Subgroups of $U(d)$ Induce Natural RNN and Transformer Architectures

Research Summary

Contribution Summary

• This paper presents a direct framework for sequence models with hidden states on closed subgroups of U(d).
• We use a minimal axiomatic setup and derive recurrent and transformer templates from a shared skeleton in which subgroup choice acts as a drop-in replacement for state space, tangent projection, and update map.
• We then specialize to O(d) and evaluate orthogonal-state RNN and transformer models on Tiny Shakespeare and Penn Treebank under parameter-matched settings.

Related Papers

• Toward LLM-Supported Automated Assessment of Critical Thinking Subskills Embedding Neighbor
• Symmetry in language statistics shapes the geometry of model representations Embedding Neighbor
• Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs Embedding Neighbor
• A Content-Based Framework for Cybersecurity Refusal Decisions in Large Language Models Embedding Neighbor
• Compressing Language Models for Specialized Domains Embedding Neighbor
• Symphonym: Universal Phonetic Embeddings for Cross-Script Name Matching Embedding Neighbor
• Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training Embedding Neighbor
• Scaling Beyond Masked Diffusion Language Models Embedding Neighbor
• Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions Embedding Neighbor
• The Headless Firm: How AI Reshapes Enterprise Boundaries Tag Overlap
• Measuring AI Ability to Complete Long Software Tasks Tag Overlap
• Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers Tag Overlap