OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report
Mariia Fedorova, Nikolay Arefyev, Maja Buljan, Jindřich Helcl, Stephan Oepen +2 more
Abstract
Language identification (LID) is an essential step in building high-quality multilingual datasets from web data. Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related languages and to distinguish valid natural language from noise, which contaminates language-specific subsets, especially for low-resource languages. In this work we extend the OpenLID classifier by adding more traini...
Results & Benchmarks
| Task | Dataset | Metric | Value |
|---|---|---|---|
| Improving Precision Closely Related Language Identification | OpenLID-v3 | Precision | 99.85 |
| Improving Precision Closely Related Language Identification | GlotLID | NN F1 | 91.14 |
| Improving Precision Closely Related Language Identification | OpenLID-v3, 0.5 | Precision | 99.91 |
| Improving Precision Closely Related Language Identification | GlotLID, 0.5 | Precision | 99.89 |
Hardware Requirements
- Expect multi-day setup/compute for meaningful reproduction based on current guidance.
Best Implementation
Maintained implementation evidence is not confirmed for this paper yet.
Use the Implementation Status and Reproduction Path sections below for the current action plan.
Reproduction Path
Follow this baseline workflow to decide if this paper is worth immediate prototyping.
- 1
Use the paper-linked Hugging Face release as the starting artifact, then reconstruct training and evaluation settings from the paper.
- 2
Use the paper and benchmark evidence to scope a baseline reproduction plan.
- 3
Track assumptions and missing details in an experiment log before coding.
Additional Implementations
No additional verified repositories beyond the primary recommendation.