Interactive note · NeurIPS 2026

A Theoretical Bridge Between Long-Tailed Recognition and Continual Learning

Mahdiyar Molahasani, Michael Greenspan, Ali EtemadNeurIPS 2026
[arXiv, BibTeX]

Theorem numbers on this page follow the NeurIPS 2026 version. The arXiv preprint is an earlier version, titled On the Relationship Between Continual Learning and Long-Tailed Recognition, and numbers some results differently until the camera-ready is posted.

Observation

Train on long-tailed data and the weights land within \(\mathcal{O}(1/\sqrt{\mathrm{IF}})\) of a model that only ever saw the Head.

Reframing

So the Tail is effectively a new task. Long-tailed recognition becomes a continual learning problem.

Guarantee

Under the theorem's local assumptions, a standard continual-learning objective upper-bounds the balanced loss. Optimizing it lowers that bound; experiments show improved balanced performance.

1 · The setting

Most classes are rare

Real datasets are rarely balanced. A few Head classes have plenty of examples and a long Tail of classes has very few. Benchmarks like CIFAR-100-LT build this in on purpose: class \(c\) keeps \(500 \cdot \mathrm{IF}^{-c/99}\) images, so the largest class is IF times the smallest.

The imbalance factor, IF, is the single number this page is about. Move it and watch where the data goes.

100
1 is balanced. CIFAR-100-LT is usually reported at 10, 50 and 100.
Images per class in CIFAR-100-LT, sorted from the largest class to the smallest. With cross-entropy averaged over every image, the gradient each class contributes is roughly proportional to its bar.
2 · Theorems 3.3 & 3.5

Training lands next to the Head

Below is a real model, trained in your browser. Six classes live in the plane: three Head classes with 300 points each and three Tail classes with 300/IF each. The model is softmax regression with cross-entropy and a small \(L^2\) penalty, which is the convex setting of Theorem 3.3.

We train it twice. Once on the Head alone, giving \(\theta^\star_H\). Then on the full long-tailed set, giving \(\theta^\star\). The question is how far apart they end up.

20
3 Tail classes × 15 points
Input spacestep 1500 of 1500
Head classesTail classesboundaries of \(\theta^\star_H\)
Weight spacea 2-D slice of 18 dims
\(\theta^\star_H\)balanced solutionpath to \(\theta^\star\)
Distance to the Head solutionone point per run
\(\lVert\theta^\star-\theta^\star_H\rVert\)\(C/\sqrt{\mathrm{IF}}\)
Left: colored regions are the long-tailed model's predictions; dashed lines are where the Head-only model draws its boundaries. As IF grows the Tail regions shrink and the two sets of boundaries merge. Middle: the Head loss \(\mathcal{L}_H\) over a plane through the starting point, \(\theta^\star_H\) and the balanced solution (IF = 1). Darker is lower. Right: during training, the purple dot tracks the current distance to \(\theta^\star_H\) and settles onto the run's final point. Every IF you visit adds a point. The dashed line uses the smallest constant \(C\) that bounds all runs so far.
Theorem 3.3 (convex) and 3.5 (deep networks)

Training on a long-tailed dataset leaves the weights in a neighborhood of the Head-only solution whose radius shrinks with the imbalance:

\[\lVert \theta^\star - \theta^\star_H \rVert = \mathcal{O}\!\left(\tfrac{1}{\sqrt{\mathrm{IF}}}\right), \qquad \operatorname{dist}(\theta^\star, S_H) = \mathcal{O}\!\left(\tfrac{1}{\sqrt{\mathrm{IF}}}\right).\]

This is what you see on the left. The long-tailed model behaves almost exactly like a model that never saw the Tail. Theorem 3.5 extends the result from the convex case to feedforward networks under the Kurdyka–Łojasiewicz condition, where \(S_H\) is the set of Head minimizers, and Theorem 3.7 extends it to any number of Head–Tail partitions.

Why the square root?

At \(\theta^\star\) the gradients balance: the Head's pull toward \(\theta^\star_H\) cancels the Tail's pull away from it. The Tail's share of the averaged loss shrinks with its sample count, so its pull is weak and \(\theta^\star\) can only move a short distance before the Head's restoring force catches up. Appendix C.1 turns this into the bound above.

3 · Proposition 3.9 & Theorem 3.12

Learn the Tail as a new task

If a long-tailed model is already a Head model, then learning the Tail is a second task that arrives after the first. That is a continual learning problem. Proposition 3.9 makes it exact: split the data into partitions sorted by size and treat each one as the next task in a stream.

The obvious approach is to fine-tune on the Tail. It fails in the familiar way, called catastrophic forgetting. Continual learning methods add a penalty that holds on to what was learned. In its simplest form, the parameter route of Eq. 6, the objective for stage two is

\[\mathcal{L}_{\mathrm{CL}}(\theta) = \mathcal{L}_T(\theta) + \beta \cdot \tfrac12 \lVert \theta - \theta^\star_H \rVert^2 .\]

Start from the Head-only model at IF = 20 and try both. The figure uses a held-out balanced test set, 200 points per class.

0.20
Input spacestep 800 of 800
Balanced loss vs. the continual-learning surrogatechecked at every frame
\(\mathcal{L}_{\mathrm{bal}}(\theta_k)\)\(\mathcal{L}_{\mathrm{CL}}(\theta_k) + \tfrac12\mathcal{L}_H(\theta^\star_H)\)
Stage two of CLTR on the toy problem, starting from \(\theta^\star_H\). With β = 0 the model forgets the Head. With a large β it never leaves the Head solution. In between, it learns the Tail and keeps the Head. Right: Theorem 3.12 says the dashed curve sits above the solid one once β passes a threshold, so lowering the dashed curve lowers a certified upper bound on the balanced loss. The bound alone does not force the balanced loss down at every step.
Theorem 3.12

Under local smoothness of \(\mathcal{L}_H\) around \(\theta^\star_H\), if \(\beta \ge \beta_{\min} = L^{\mathrm{sm}}_H / 2m_\Psi\) (or the output-distillation weight \(\alpha \ge \alpha_{\min}\)), then for every \(\theta\) in that neighborhood

\[\mathcal{L}_{\mathrm{bal}}(\theta) \;\le\; \mathcal{L}_{\mathrm{CL}}(\theta) + \tfrac12\,\mathcal{L}_H(\theta^\star_H).\]

The right-hand side differs from the CL objective only by a constant. So any optimizer that decreases \(\mathcal{L}_{\mathrm{CL}}\) decreases a certified upper bound on the balanced loss: inside the neighborhood, the balanced loss can never exceed it. An upper bound alone does not make the balanced loss fall at every step; in a special convex case, with the Head loss itself as the penalty, the paper shows the two objectives differ only by a constant and so share their minimizers (Appendix C.8). Off-the-shelf CL methods such as EWC, LwF, GPM, SGP, FOSTER, TPL and DualPrompt all fit the form of Eq. 6.

The threshold is local: it holds inside the neighborhood where the smoothness constant was measured. In the toy above, runs with a very small β wander far from \(\theta^\star_H\), leave that neighborhood, and the bound stops holding. The green tick on the slider marks where it starts holding on this problem. In the paper's ResNet-18 setting, standard CL methods clear \(\beta_{\min}\) by orders of magnitude (Appendix G.5).

4 · Proposition 3.10

How many stages?

A long tail can be split into more than two tasks. More partitions make each one closer to balanced: with \(N\) partitions the local imbalance is \(\mathrm{IF}^{1/N}\). But every extra task is another chance to forget. Proposition 3.10 prices both sides with a bias term and a forgetting term, where \(\eta\) is the forgetting coefficient of the CL method:

\[N^\star \in \arg\min_N \; 2B\,\frac{\mathrm{IF}^{1/(2N)} - 1}{\mathrm{IF}^{1/(2N)} + 1} \;+\; (N-1)\,\eta .\]
100
0.10
Lower means the CL method forgets less.
Bias + forgetting
Where the partitions fall
Left: the two terms of Proposition 3.10 with \(B = 1\), for \(N = 1\) to 8 partitions; the ring marks \(N^\star\). Right: the optimal boundaries give every partition the same local imbalance. For an exponential profile that means equal numbers of classes. In the experiments, weaker methods (LwF, EWC, GPM, SGP) use \(N = 2\) and stronger ones (FOSTER, TPL) use \(N = 4\).
5 · Experiments

What the experiments show

The geometry holds beyond the toy

Distance scales linearly with \(1/\sqrt{\mathrm{IF}}\)

R² ≥ 0.97

logistic regression, MNIST-LT

R² ≥ 0.98

ResNet-18, CIFAR-100-LT

The same measurement as §2, with and without weight decay. The deep, non-convex case follows the prediction of Theorem 3.5.

Order is part of the method

Head first, then Tail

CLTR (SGP) on CIFAR-100-LT at IF = 100. Reversing or shuffling the order loses 15 to 18 points, as the theory predicts.

Foundation models

Adapting CLIP to long-tailed data

CLIP ViT-B/32 on CIFAR-100-LT, top-1 accuracy. CLTR with DualPrompt beats full fine-tuning by 5.8, 7.1 and 5.0 points without any extra semantic supervision.

Method IF 100 IF 50 IF 10
Zero-shot CLIP 64.5 64.5 64.5
Linear probe 60.0 67.2 75.6
Prompt learning 60.4 63.1 66.9
Naive sequential 48.0 51.2 53.1
Full fine-tuning 73.4 74.8 81.0
CLTR (DualPrompt) 79.2 81.9 86.0
Standard benchmarks

With a single model and no extra supervision, CLTR is competitive with the strongest single-model LTR methods across CIFAR-100-LT, CIFAR-10-LT and ImageNet-LT, and gives the best reported numbers on CIFAR-10-LT (84.7 and 87.6 at IF 100 and 50, with TPL) and ties the best on ImageNet-LT (53.9). The paper also covers long-tailed class-incremental learning and the naturally skewed Caltech256.

Cite

BibTeX

@inproceedings{molahasani2026bridge,
  title     = {A Theoretical Bridge Between Long-Tailed
               Recognition and Continual Learning},
  author    = {Molahasani, Mahdiyar and Greenspan, Michael
               and Etemad, Ali},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}

The models in §2 and §3 are trained live in your browser: 6-class softmax regression on 2-D data, full-batch gradient descent with momentum. The numbers in §5 are from the paper.

← All writing