MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation

Research Summary

Contribution Summary

• We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code.
• Through targeted adaptation, this model family achieves state-of-the-art results on Catalan- and Spanish-specific tasks, while establishing robust performance across specialized biomedical and legal domains.
• To bridge the gap between research and production, we incorporate Matryoshka Representation Learning (MRL), enabling flexible vector sizing that significantly reduces inference and storage costs.

Related Papers

• Refusal Direction is Universal Across Safety-Aligned Languages Embedding Neighbor
• Utility-Preserving De-Identification for Math Tutoring: Investigating Numeric Ambiguity in the MathEd-PII Benchmark Dataset Embedding Neighbor
• Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguards Embedding Neighbor
• A Problem-Oriented Perspective and Anchor Verification for Code Optimization Embedding Neighbor
• BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop Embedding Neighbor
• When More Is Less: A Systematic Analysis of Spatial and Commonsense Information for Visual Spatial Reasoning Embedding Neighbor
• Slm-mux: Orchestrating small language models for reasoning Embedding Neighbor
• Counterfactual Simulation Training for Chain-of-Thought Faithfulness Embedding Neighbor
• Helpful to a Fault: Measuring Illicit Assistance in Multi-Turn, Multilingual LLM Agents Embedding Neighbor
• MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models Tag Overlap
• Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models Tag Overlap
• ConformalNL2LTL: Translating Natural Language Instructions into Temporal Logic Formulas with Conformal Correctness Guarantees Tag Overlap