OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report

Research Summary

Contribution Summary

• Language identification (LID) is an essential step in building high-quality multilingual datasets from web data.
• Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related languages and to distinguish valid natural language from noise, which contaminates language-specific subsets, especially for low-resource languages.
• In this work we extend the OpenLID classifier by adding more training data, merging problematic language variant clusters, and introducing a special label for marking noise.

Why It Matters For Eval

• We call this extended system OpenLID-v3 and evaluate it against GlotLID on multiple benchmarks.
• During development, we focus on three groups of closely related languages (Bosnian, Croatian, and Serbian; Romance varieties of Northern Italy and Southern France; and Scandinavian languages) and contribute new evaluation datasets where exi

Related Papers

• Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Models Embedding Neighbor
• Context-Aware Mapping of 2D Drawing Annotations to 3D CAD Features Using LLM-Assisted Reasoning for Manufacturing Automation Embedding Neighbor
• An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems Embedding Neighbor
• Motivation is Something You Need Embedding Neighbor
• Continuous Telemonitoring of Heart Failure using Personalised Speech Dynamics Embedding Neighbor
• Bridging Symbolic Control and Neural Reasoning in LLM Agents: Structured Cognitive Loop with a Governance Layer Embedding Neighbor
• Intent Laundering: AI Safety Datasets Are Not What They Seem Embedding Neighbor
• Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes Embedding Neighbor
• NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors Embedding Neighbor
• Physical Commonsense Reasoning for Lower-Resourced Languages and Dialects: a Study on Basque Tag Overlap
• IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models Tag Overlap
• Structured Prompt Language: Declarative Context Management for LLMs Tag Overlap