LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

Best Implementation

• Selected pjlab-sys4nlp/llama-moe as the strongest maintained implementation for new work.
• Includes dependency/environment manifest signals.
• Repository activity is within the last 24 months.

Reproduction Path

1. 1

   Start with pjlab-sys4nlp/llama-moe and validate setup instructions in README.

2. 2

   Reproduce the baseline result with the provided defaults before modifying hyperparameters.

3. 3

   Log exact dependency versions and runtime environment for reproducibility.

Additional Implementations

No additional verified repositories beyond the primary recommendation.

Hugging Face Artifacts

No direct paper-linked artifacts were found. Showing strongest curated related artifacts.