Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language

Nischal Karki, Bipesh Subedi, Prakash Poudyal, Rupak Raj Ghimire, Bal Krishna Bal · Feb 27, 2026 · Citations: 0

Abstract

Transformer-based models such as BERT have significantly advanced Natural Language Processing (NLP) across many languages. However, Nepali, a low-resource language written in Devanagari script, remains relatively underexplored. This study benchmarks multilingual, Indic, Hindi, and Nepali BERT variants to evaluate their effectiveness in Nepali topic classification. Ten pre-trained models, including mBERT, XLM-R, MuRIL, DevBERT, HindiBERT, IndicBERT, and NepBERTa, were fine-tuned and tested on the balanced Nepali dataset containing 25,006 sentences across five conceptual domains and the performance was evaluated using accuracy, weighted precision, recall, F1-score, and AUROC metrics. The results reveal that Indic models, particularly MuRIL-large, achieved the highest F1-score of 90.60%, outperforming multilingual and monolingual models. NepBERTa also performed competitively with an F1-score of 88.26%. Overall, these findings establish a robust baseline for future document-level classification and broader Nepali NLP applications.

HFEPX Relevance Assessment

This paper appears adjacent to HFEPX scope (human-feedback/eval), but does not show strong direct protocol evidence in metadata/abstract.

Eval-Fit Score

0/100 • Low

Treat as adjacent context, not a core eval-method reference.

Human Feedback Signal

Not explicit in abstract metadata

Evaluation Signal

Detected

HFEPX Fit

Adjacent candidate

If you are doing eval pipeline work, start here:

Human Eval Hub LLM-as-Judge Hub Pairwise Preference Hub Tool-Use Eval Hub

Human Data Lens

Uses human feedback: No
Feedback types: None
Rater population: Unknown
Unit of annotation: Unknown
Expertise required: Multilingual
Extraction source: Runtime deterministic fallback

Evaluation Lens

Evaluation modes: Automatic Metrics
Agentic eval: None
Quality controls: Not reported
Confidence: 0.35
Flags: low_signal, possible_false_positive, runtime_fallback_extraction

Protocol And Measurement Signals

Benchmarks / Datasets

No benchmark or dataset names were extracted from the available abstract.

Reported Metrics

accuracyf1precisionrecallauroc

Research Brief

Deterministic synthesis

This study benchmarks multilingual, Indic, Hindi, and Nepali BERT variants to evaluate their effectiveness in Nepali topic classification. HFEPX signals include Automatic Metrics with confidence 0.35. Updated from current HFEPX corpus.

Generated Mar 3, 2026, 7:09 AM · Grounded in abstract + metadata only

Key Takeaways

This study benchmarks multilingual, Indic, Hindi, and Nepali BERT variants to evaluate their effectiveness in Nepali topic classification.
Ten pre-trained models, including mBERT, XLM-R, MuRIL, DevBERT, HindiBERT, IndicBERT, and NepBERTa, were fine-tuned and tested on the balanced Nepali dataset containing 25,006…

Researcher Actions

Treat this as method context, then pivot to protocol-specific HFEPX hubs.
Identify benchmark choices from full text before operationalizing conclusions.
Validate metric comparability (accuracy, f1, precision).

Caveats

Generated from title, abstract, and extracted metadata only; full-paper implementation details are not parsed.
Low-signal flag detected: protocol relevance may be indirect.

Recommended Queries

human-eval protocol design pairwise preference data quality inter-rater agreement adjudication

Research Summary

Contribution Summary

This study benchmarks multilingual, Indic, Hindi, and Nepali BERT variants to evaluate their effectiveness in Nepali topic classification.
Ten pre-trained models, including mBERT, XLM-R, MuRIL, DevBERT, HindiBERT, IndicBERT, and NepBERTa, were fine-tuned and tested on the balanced Nepali dataset containing 25,006 sentences across five conceptual domains and the performance was…
The results reveal that Indic models, particularly MuRIL-large, achieved the highest F1-score of 90.60%, outperforming multilingual and monolingual models.

Why It Matters For Eval

This study benchmarks multilingual, Indic, Hindi, and Nepali BERT variants to evaluate their effectiveness in Nepali topic classification.

Researcher Checklist

Gap: Human feedback protocol is explicit

No explicit human feedback protocol detected.
Pass: Evaluation mode is explicit

Detected: Automatic Metrics
Gap: Quality control reporting appears

No calibration/adjudication/IAA control explicitly detected.
Gap: Benchmark or dataset anchors are present

No benchmark/dataset anchor extracted from abstract.
Pass: Metric reporting is present

Detected: accuracy, f1, precision, recall

Category-Adjacent Papers (Broader Context)

These papers are nearby in arXiv category and useful for broader context, but not necessarily protocol-matched to this paper.

InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models Category Neighbor

Citations: 0 Relevance: 5.25
- Shared arXiv category (cs.CL, cs.LG)
- Shared metric mentions
- Shared terminology (accuracy)
Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning Category Neighbor

Citations: 0 Relevance: 4.45
- Shared arXiv category (cs.CL, cs.LG)
- Shared metric mentions
- Shared terminology (accuracy)
Confusion-Aware Rubric Optimization for LLM-based Automated Grading Category Neighbor

Citations: 0 Relevance: 4.10
- Shared arXiv category (cs.CL)
- Shared metric mentions
- Shared terminology (accuracy, precision)
When Metrics Disagree: Automatic Similarity vs. LLM-as-a-Judge for Clinical Dialogue Evaluation Category Neighbor

Citations: 0 Relevance: 4.10
- Shared arXiv category (cs.CL)
- Shared metric mentions
- Shared terminology (accuracy, precision)
Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking Category Neighbor

Citations: 0 Relevance: 3.65
- Shared arXiv category (cs.CL, cs.LG)
- Shared terminology (benchmarking)
DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science Category Neighbor

Citations: 0 Relevance: 3.30
- Shared arXiv category (cs.CL)
- Shared metric mentions
- Shared terminology (accuracy, benchmarking)
IDP Accelerator: Agentic Document Intelligence from Extraction to Compliance Validation Category Neighbor

Citations: 0 Relevance: 3.30
- Shared arXiv category (cs.CL)
- Shared metric mentions
- Shared terminology (accuracy, classification)
AgentDropoutV2: Optimizing Information Flow in Multi-Agent Systems via Test-Time Rectify-or-Reject Pruning Category Neighbor

Citations: 0 Relevance: 2.85
- Shared arXiv category (cs.CL)
- Shared metric mentions
- Shared terminology (accuracy)

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote