Summary
MixLoRA is a parameter-efficient fine-tuning method that extends LoRA with a Mixture-of-Experts design for large language models, implemented within the MoE-PEFT framework. This page includes benchmark evidence for commonsense QA PEFT fine-tuning on ARC-e/ARC-c/BoolQ/OBQA (LLaMA2-7B). Reproduction guidance focuses on implementation viability and concrete risk controls.
Key Contributions
- MixLoRA is a parameter-efficient fine-tuning method that extends LoRA with a Mixture-of-Experts design for large language models, implemented within the MoE-PEFT framework.
- The MixLoRA implementation is designed to be used via the MoE-PEFT library and also provides integrations with HuggingFace Transformers for inference.
- The official MixLoRA repository includes scripts to reproduce the evaluation results by running MoE-PEFT v1.0.1 or later with predefined launch configurations.
Implementation Guidance
Use TUDB-Labs/MixLoRA first because deterministic ranking and extracted evidence align on implementation viability. Start with the repo setup path, then validate benchmark reproduction before adaptation.
Reproducibility Notes
- Reproduction quality may degrade if the original preprocessing pipeline and task mixing details from the paper are not matched closely.
- Differences in hyperparameters such as router loss coefficient, rank, or training schedule from those used in the paper can lead to noticeably different benchmark scores.
Results & Benchmarks
| Task | Dataset | Metric | Value |
|---|---|---|---|
| LoRA | ARC | Router Loss Coef. | 77.7 |
| LoRA | ARC(Avg) | Accuracy | 71.3 |
Best Implementation
State-of-the-art Parameter-Efficient MoE Fine-tuning Method
- Selected TUDB-Labs/MixLoRA as the strongest maintained implementation for new work.
- Includes CI workflow signals.
- Includes dependency/environment manifest signals.
- Repository activity is within the last 24 months.
Reproduction Path
- 1
Start with TUDB-Labs/MixLoRA and validate setup instructions in README.
- 2
Reproduce the baseline result with the provided defaults before modifying hyperparameters.
- 3
Log exact dependency versions and runtime environment for reproducibility.
Additional Implementations
No additional verified repositories beyond the primary recommendation.
Hugging Face Artifacts
No direct paper-linked artifacts were found. Showing strongest curated related artifacts.