Large language models have become astonishingly capable learners, absorbing new skills from relatively small amounts of fine-tuning data. Yet this flexibility comes with a well-known curse: when an AI model is trained sequentially on new tasks, it tends to overwrite the knowledge it acquired earlier, a phenomenon researchers call catastrophic forgetting. Now, a pair of computer scientists at Pennsylvania State University has proposed an elegant fix that works by changing a surprisingly small detail of how models are fine-tuned — the number of low-rank parameters, or rank, allotted to each layer of the network. Their method, described in the journal Machine Learning, adaptively adjusts these ranks as new tasks arrive, allowing language models to keep learning without erasing what came before.
The research, conducted by Fuli Qiao and Mehrdad Mahdavi of the Department of Computer Science and Engineering at Penn State, builds on Low-Rank Adaptation, or LoRA, one of the most widely used parameter-efficient fine-tuning techniques in modern artificial intelligence. Instead of retraining the billions of weights inside a large language model, LoRA freezes the original weights and injects small pairs of low-rank matrices into each layer. Only these small matrices are updated during training, which slashes memory and computation costs by orders of magnitude. The catch, the authors note, is that conventional LoRA fixes the rank to the same value across every layer and every task, a one-size-fits-all choice that leaves a crucial question unexplored: how much capacity does each layer actually need for each new task?
Qiao and Mahdavi’s answer is a method that treats rank as a dynamic resource rather than a fixed hyperparameter. Their approach, which the team calls CL-Rank, relies on a subspace similarity metric to measure how orthogonal — that is, how non-overlapping — the low-rank subspaces occupied by different tasks are within a given layer. When a new task arrives, the algorithm evaluates the geometry of the learned subspaces and adaptively increases the layerwise rank for the new task where overlap threatens to interfere with previously learned representations. By steering new knowledge into directions of parameter space that are roughly orthogonal to old knowledge, the method minimizes interference while still giving each new task enough expressive capacity to generalize well.
The underlying intuition echoes a classical idea from neuroscience and machine learning known as the stability–plasticity dilemma. A learning system must be plastic enough to absorb new information but stable enough to retain old skills. Since the pioneering work on catastrophic interference in connectionist networks in the late 1980s and the advent of modern continual learning benchmarks, researchers have tried an arsenal of remedies: replaying stored examples, regularizing important weights, growing new network branches, or constraining gradient updates to null spaces of prior tasks. Many of these approaches are either memory-hungry, brittle to task order, or impractical for models with billions of parameters. What distinguishes the new work is that it tackles the problem entirely within the cheap, low-rank adapter framework, requiring no access to old data and no freezing of the base model.
Technically, the method tracks the principal subspaces spanned by the low-rank updates in each layer and computes a similarity score between the subspace of an incoming task and those of earlier tasks. If the new task’s gradient updates are projected into directions that overlap heavily with prior subspaces, the rank of the adapter in that layer is expanded, giving the optimizer room to find solutions that spare old representations. Layers whose subspaces remain naturally disjoint need no expansion, so the parameter budget is spent only where it matters. The paper’s supplementary analyses reveal that the learned rank distributions differ markedly across layers and modules — for example, the variation in ranks among the value projection modules of encoder layers is larger than among query projections, while encoder layers as a whole show more consistent rank allocations than decoder layers. This suggests that different parts of a transformer genuinely serve distinct roles, and that a uniform rank silently wastes capacity in some layers while starving others.
To test the idea, the researchers ran experiments on T5 and Llama-2-7b language models, using GPU servers with DeepSpeed for efficient training. For T5 experiments they employed four NVIDIA A6000 GPUs with a learning rate of 1e-3 and a batch size of 32, while the larger Llama-2-7b runs used four NVIDIA A100 GPUs at a learning rate of 1e-4. The evaluation spanned standard natural language processing continual learning benchmarks built from 15 datasets, including the classic text classification benchmark of Zhang and colleagues, the GLUE and SuperGLUE suites, and the IMDB movie review corpus, with six different task sequence orders to control for ordering effects. The team also pushed into harder territory with challenging mathematical reasoning benchmarks, GSM8K and MATH, each split into sequential tasks and tested in both orders to probe how task sequencing shapes forgetting.
The results show that the adaptive-rank method matches or beats strong baselines on three fronts at once: it mitigates forgetting of earlier tasks, improves accuracy on the tasks being learned, and preserves strong generalization to unseen data — all in a memory-efficient manner. On the math benchmarks, where sequential training on GSM8K and MATH typically causes steep declines in earlier-task accuracy, the method with orthogonal projection consistently outperformed the O-LoRA baseline in final average testing accuracy, forgetting less on the first task while generalizing better on the second. Notably, the study observes that fully fine-tuning a T5-large model on MATH yields only about 3.0 percent accuracy, and about 4.2 percent on GSM8K, consistent with prior work — underscoring how difficult these mathematical tasks are and how much room remains for continual learning approaches that squeeze more capability out of small adapters.
Why should this matter beyond the benchmark tables? Continual learning is arguably the missing ingredient between today’s static AI assistants and the adaptive, lifelong learning systems that both researchers and companies envision. Every time a deployed model needs a new skill — a new language, a new domain, a new tool — retraining it from scratch or even fully fine-tuning it is prohibitively expensive. Adapter-based continual learning offers a path to modular, on-demand skill acquisition, but only if adding skills does not corrupt existing ones. By showing that the humble LoRA rank, tuned per layer and per task, is a powerful lever for balancing stability and plasticity, the Penn State work gives practitioners a new, inexpensive knob to turn. Because the method requires no stored examples of past tasks, it also sidesteps the privacy and storage concerns that plague replay-based approaches.
The findings also carry a broader scientific message: the internal geography of large models is far from uniform, and adaptation schemes that respect that heterogeneity can outperform uniform ones. The observation that encoder and decoder layers, and even the query and value modules within them, prefer different rank allocations suggests that future parameter-efficient methods may benefit from treating adapters as structured, layer-aware components rather than interchangeable plug-ins. The authors have released their code publicly, and all datasets used in the experiments are openly available, making the results straightforward for other groups to verify and extend. The work was partially supported by the National Science Foundation under award EFMA-2318101, with experiments designed and conducted by Fuli Qiao under the supervision of Mehrdad Mahdavi.
As large language models continue to spread through science, medicine, and industry, the question of how to teach them new things without breaking old ones is shifting from academic curiosity to engineering necessity. This study does not solve continual learning outright — no single method does — but it demonstrates that a careful, geometry-aware allocation of low-rank capacity can deliver competitive anti-forgetting performance at a fraction of the cost of full retraining. In a field where progress often comes from scaling up, it is a reminder that sometimes the most powerful improvements come from scaling the right things in the right places, one layer at a time.
Subject of Research: Adaptive layerwise LoRA ranks for continual learning in large language models
Article Title: Learning Without Forgetting for Continual Learning in LLMs Through Adaptive LoRA Ranks
Article References: Qiao, F., & Mahdavi, M. (2026). Learning Without Forgetting for Continual Learning in LLMs Through Adaptive LoRA Ranks. Machine Learning, 115(9), Article 212. https://doi.org/10.1007/s10994-026-07145-4
Image Credits: AI Generated
DOI: 10.1007/s10994-026-07145-4
Keywords: continual learning, large language models, LoRA, catastrophic forgetting, parameter-efficient fine-tuning, adaptive ranks, subspace orthogonality, stability-plasticity, GLUE, GSM8K, MATH, T5
Cite Scienmag News
APA MLA Chicago
Denise Maddox. (September 12, 2026). Adaptive LoRA Ranks Help AI Models Learn New Tasks Without Forgetting Old Ones. Scienmag. https://scienmag.com/adaptive-lora-ranks-help-ai-models-learn-new-tasks-without-forgetting-old-ones/
Denise Maddox. “Adaptive LoRA Ranks Help AI Models Learn New Tasks Without Forgetting Old Ones.” Scienmag, 12 September 2026, https://scienmag.com/adaptive-lora-ranks-help-ai-models-learn-new-tasks-without-forgetting-old-ones/. Accessed 12 September 2026.
Denise Maddox. “Adaptive LoRA Ranks Help AI Models Learn New Tasks Without Forgetting Old Ones.” Scienmag. September 12, 2026. https://scienmag.com/adaptive-lora-ranks-help-ai-models-learn-new-tasks-without-forgetting-old-ones/
Copy citation Download RIS
Tags: Adaptive LoRA ranksadaptive ranksavoiding knowledge overwriting in AI modelscatastrophic forgettingcatastrophic forgetting mitigation in large language modelscontinual learningdynamic rank allocation in neural network layersfine-tuning neural networks without losing prior knowledgeGLUEGSM8Kincremental learning in language modelslarge language modelsLoRalow-rank adaptation (LoRA) methodologylow-rank parameter adjustmentMATHmemory-efficient AI trainingparameter-efficient fine-tuningparameter-efficient fine-tuning techniquesPenn State University AI researchsequential task learning in AIstability-plasticitysubspace orthogonalityT5
