Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography

Research Summary

Contribution Summary

• We present a crowdsourced dataset for Piedmontese, an endangered Romance language of northwestern Italy.
• The dataset comprises 145 Italian-Piedmontese parallel sentences derived from Flores+, with translations produced by speakers writing in their natural orthographic style rather than adhering to standardized conventions, along with manual wo
• We use this resource to benchmark several large language models on tokenization parity, topic classification, and machine translation.

Why It Matters For Eval

• We use this resource to benchmark several large language models on tokenization parity, topic classification, and machine translation.

Related Papers

• Refusal Direction is Universal Across Safety-Aligned Languages Embedding Neighbor
• VIRAASAT: Traversing Novel Paths for Indian Cultural Reasoning Embedding Neighbor
• 1-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning Embedding Neighbor
• Toward an Agentic Infused Software Ecosystem Embedding Neighbor
• Reasoning Up the Instruction Ladder for Controllable Language Models Embedding Neighbor
• AdapTools: Adaptive Tool-based Indirect Prompt Injection Attacks on Agentic LLMs Embedding Neighbor
• NIMMGen: Learning Neural-Integrated Mechanistic Digital Twins with LLMs Embedding Neighbor
• Beyond Refusal: Probing the Limits of Agentic Self-Correction for Semantic Sensitive Information Embedding Neighbor
• ClearFairy: Capturing Creative Workflows through Decision Structuring, In-Situ Questioning, and Rationale Inference Embedding Neighbor
• ConformalNL2LTL: Translating Natural Language Instructions into Temporal Logic Formulas with Conformal Correctness Guarantees Tag Overlap
• Aladdin-FTI @ AMIYA Three Wishes for Arabic NLP: Fidelity, Diglossia, and Multidialectal Generation Tag Overlap
• WebFAQ 2.0: A Multilingual QA Dataset with Mined Hard Negatives for Dense Retrieval Tag Overlap