What is the best open-source implementation of "Accelerating Transformer Pre-training with 2:4 Sparsity"?

The best maintained implementation is huyz2023/2by4-pretrain with 62 stars on GitHub. Confidence: high. Reproducibility: Limited.

How reproducible is "Accelerating Transformer Pre-training with 2:4 Sparsity"?

Estimated time to first reproduction: a few days. Risk flags: No CI workflows detected, Dependency manifest is missing. Start with huyz2023/2by4-pretrain and validate setup instructions in README.

Are there pretrained models available for "Accelerating Transformer Pre-training with 2:4 Sparsity"?

Yes, 3 Hugging Face models found. The top result is ostris/zimage_turbo_training_adapter with 51,727 downloads.

What framework is used to implement "Accelerating Transformer Pre-training with 2:4 Sparsity"?

The primary implementation uses pytorch.

Accelerating Transformer Pre-training with 2:4 Sparsity

Published: Apr 1, 2024

Best maintained implementation now

Evidence: Direct

Domain fit: AI-core

Verified repos: 2

Top repo stars: 62

Core AI workload signals detected from paper context and implementation/artifact evidence.

Framework: pytorch

Time to first repro: a few days

2 risk flags

arXiv PDF

Technical details

Canonical key: arxiv-2404.01847

Cache status: Stale (SWR served)

Generated at: Jun 18, 2026, 4:55 AM

Artifact coverage: direct

HF provider: ok (token)

PWC source used: Yes

LLM status: not_generated

LLM model: n/a

LLM generated: Unknown

LLM content type: n/a

HF policy: hf-relevance-v27

implementation starting point

Benchmarks: thin evidence

Time to repro: a few days

2 risk flags

pytorch

Results & Benchmarks

Freshness tier: cold

Direct + Inferred Evidence

Transformer

Dense

Val BLEU

26.56

Source: paper fulltext

Transformer

Half

Val BLEU

26.36

Source: paper fulltext

Transformer

STEP

Val BLEU

25.85

Source: paper fulltext

Benchmark evidence drill-down

3 findings

Audit each benchmark finding before selecting an implementation path. Evidence refs map to the disclosure section below.

Task	Dataset	Metric	Value	Source	Evidence refs
Transformer	Dense	Val BLEU	26.56	paper-derived	No explicit refs
Transformer	Half	Val BLEU	26.36	paper-derived	No explicit refs
Transformer	STEP	Val BLEU	25.85	paper-derived	No explicit refs

Accelerating Transformer Pre-training with 2:4 Sparsity presents a transformer method.

Use This Implementation Because…

Confidence: high

huyz2023/2by4-pretrain is the strongest maintained implementation based on ranking signals. License is declared (BSD-3-Clause).

Open huyz2023/2by4-pretrain

Reproduction Risks

No CI workflows detected
Dependency manifest is missing

Hardware Notes

Expect multi-day setup/compute for meaningful reproduction based on current guidance.

Evidence disclosure

Evidence graph: 4 refs, 4 links.

Utility signals: depth 100/100, grounding 95/100, status high.

Implementation Comparison

Top 2 paths

Compare maintenance quality, reproducibility coverage, and evidence confidence before choosing a reproduction baseline.

huyz2023/2by4-pretrain

best maintained

Maintenance: Stale

Confidence: High

Reproducibility: Limited

Official implementation from Papers with Code · Repository link is mentioned in the paper metadata

Stars: 62
Last push: Dec 8, 2024 (559d ago)

Risk flags

No push in 12+ months
No CI pipeline detected
No tagged releases

thu-ml/2by4-pretrain-acc-examples

historical official

Maintenance: Stale

Confidence: High

Reproducibility: Limited

Official implementation from Papers with Code · Repository link is mentioned in the paper metadata

Stars: 27
Last push: Dec 8, 2024 (559d ago)

Risk flags

No push in 12+ months
No CI pipeline detected
No tagged releases

Best implementation now

huyz2023/2by4-pretrain

Confidence: High

Reproducibility: Limited

Efficient 2:4 sparse training algorithms and implementations

Stars: 62

Forks: 1

Last push: Dec 8, 2024

License: BSD-3-Clause

Official implementation from Papers with Code

Repository link is mentioned in the paper metadata

Community adoption signal (62 stars)

License ✓

CI –

Deps –

Docker –

Selected huyz2023/2by4-pretrain as the strongest maintained implementation for new work.
Repository activity is within the last 24 months.
Official repository is preserved separately as historical context.

Historical official implementation

Preserved for provenance. Not recommended as the default path for new builds.

thu-ml/2by4-pretrain-acc-examples

Stars: 27

Last push: Dec 8, 2024

Reproduction readiness

Major Work

Time to first repro: days

Last checked: Jun 18, 2026

Hardware requirements

Expect multi-day setup/compute for meaningful reproduction based on current guidance.

No dependency manifest — manual reconstruction required

· huyz2023/2by4-pretrain has no requirements.txt, environment.yml, pyproject.toml, or Dockerfile.
· You will need to reverse-engineer dependencies from import statements in the source code.
· Last push was 559 days ago.

Open huyz2023/2by4-pretrain