Skip to content
OpenTrain AIFor AI Companies

Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repair

Yuelin Hu, Zhenbo Yu, Zhengxue Cheng, Wei Liu, Li SongPublished Apr 24, 2026
arXiv PDF
Researcher verdict
Context only
Use as context only
Benchmark evidence
Thin evidence
Verify before relying
Time to first repro
A few hours
Fast first run
Risk flags
1
Review before use

Abstract

Domain fit: Niche / domain-specific · No strong AI-core implementation/artifact signals were detected from current providers.

Many continual-learning methods modify gradients upstream (e.g., projection, penalty rescaling, replay mixing) while treating Adam as a neutral backend. We show this composition has a hidden failure mode. In a high-overlap, non-adaptive 8-domain continual LM, all shared-routing projection baselines collapse close to vanilla forgetting (12.5--12.8 vs. 13.2). A 0.5% replay buffer is the strongest shared alternative but still reaches 11.6, while fixed-strength decoupling falls below vanilla at 14.1. Only adaptive decoupled routing remains stable at 9.4, improving over vanilla by 3.8 units. On a 16-domain stream, its gain over the strongest shared-routing projection baseline grows to 4.5--4.8 units. The failure is largely invisible on clean benchmarks. We explain this effect through Adam's second-moment pathway: in the tested regime, projection induces a 1/(1-alpha) inflation of the old-direction effective learning rate, matching measurements within 8% across eight alpha values. The same conflict appears with penalty methods, replay mixing, and at 7B scale under LoRA. Our fix routes the modified gradient only to the first moment while preserving magnitude-faithful second-moment statistics, with overlap-aware adaptive strength. This simple change is the only tested configuration that consistently avoids collapse across methods, optimizers, and scale.

Results and benchmarks

Freshness tier: warm
Many continual-learning methods modify gradients upstream (e.g., projection, penalty rescaling, replay mixing) while treating Adam as a neutral backend.
Task Dataset Metric Value Source
LoRA CIFAR-100 Accuracy 20 paper-derived

Audit each benchmark finding before selecting an implementation path. Evidence refs map to the disclosure below.

Implementation

No direct implementation yet

Maintained implementation evidence is not confirmed for this paper yet.

Use the implementation status and reproduction sections for the current action plan.

Implementation evidence summary
Confidence: low

This is primarily a method paper. Reproduce it within a maintained framework baseline instead of chasing paper-specific repos.

Reproduction risks
  • No maintained paper-verified implementation is currently available

Reproduction readiness

Time to first repro: hours
Last checked: Aug 24, 2026

No repo

No verified implementation available

  • No maintained repository has been identified for this paper. Check adjacent implementations or HF artifacts below.

Framework baselines

Hugging Face artifacts

No trustworthy direct or curated related Hugging Face artifacts were found yet. Use targeted searches to quickly locate candidate models, datasets, and demos.

Tip: start with models, then check datasets and spaces if you need evaluation data or demos.

Research context

Tasks

LoRA

Methods

LoRA / Parameter-efficient tuning, Stochastic optimization

Domains

None detected

Evaluation and human feedback data

Open this paper in HFEPX to review benchmark signals, evaluation modes, and human-feedback protocol context.

Open in HFEPX
Explore similar papers

Jump to Paper2Code search queries derived from this paper's research context.