Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
Chenghao Fan, Zhenyi Lu, Sichen Liu, Chengfeng Gu, Xiaoye Qu +2 more
Abstract
While Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning for Large Language Models (LLMs), its performance often falls short of Full Fine-Tuning (Full FT). Current methods optimize LoRA by initializing with static singular value decomposition (SVD) subsets, leading to suboptimal leveraging of pre-trained knowledge. Another path for improving LoRA is incorporating a Mixture-of-Experts (MoE) architectur...
Summary
This paper introduces GOAT, a Mixture-of-Experts LoRA framework that integrates SVD-structured priors and a theoretically derived scaling factor to better align LoRA optimization with full fine-tuning. The authors report strong results across a broad set of NLU, NLG, image classification, and commonsense reasoning benchmarks, and provide an official implementation (goat-peft) with scripts for both image and language tasks.
Key Contributions
- Proposes GOAT, a LoRA-MoE framework that uses SVD-structured priors and a theoretical scaling factor to align LoRA optimization with full fine-tuning.
- Replaces static SVD subset usage with an adaptive SVD-structured Mixture-of-Experts that selects relevant pretrained priors during LoRA fine-tuning.
- Shows that properly scaling LoRA-MoE parameters can substantially improve efficiency and performance without changing architectures or training algorithms.
- Reports state-of-the-art performance across 25 datasets spanning NLU, commonsense reasoning, image classification, and generation, largely closing the LoRA vs full FT gap.
- Provides an official goat-peft repository with training scripts for ViT-based image classification and multiple natural language generation tasks.
Implementation Guidance
Use the official facico/goat-peft repository as the starting point. Follow the README to set up dependencies (MIT-licensed, with requirement manifests) and first reproduce the default training runs for a supported task, such as ViT-based image classification or one of the NLG workloads. Once a baseline is matched, adapt configuration files to mirror the paper’s GOAT or GOAT-s settings and evaluation protocols. Given the absence of CI, carefully log exact package versions, hardware, and preprocessing choices to preserve reproducibility and simplify later comparisons or ablations.
Reproducibility Notes
- Official implementation is available in the goat-peft repository with dependency manifests declared.
- No CI workflows are present, so environment drift and dependency breakage are possible over time.
- Researchers should record exact library versions, hardware details, and preprocessing steps during reproduction.
Results & Benchmarks
| Task | Dataset | Metric | Value |
|---|---|---|---|
| Image classification | GLUE | SST-2 | 96.22 |
Best Implementation
[ICML2025] Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
- Selected facico/goat-peft as the strongest maintained implementation for new work.
- Includes dependency/environment manifest signals.
- Repository activity is within the last 24 months.
Reproduction Path
- 1
Start with facico/goat-peft and validate setup instructions in README.
- 2
Reproduce the baseline result with the provided defaults before modifying hyperparameters.
- 3
Log exact dependency versions and runtime environment for reproducibility.
Additional Implementations
Official
No additional official repositories detected.
Community
- Facico/GOAT-PEFTConfidence: low
[ICML2025] Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
Stars: 145Forks: 15Last push: Nov 2025License: MIT
Hugging Face Artifacts
No direct paper-linked artifacts were found. Showing strongest curated related artifacts.