Shared Gradient Discovery and Superposition: Learning Dynamics of Generalization in LLMs

Andrei Mircea 1,2, Ildus Sadrtdinov 3, Irina Rish 1,2, Ekaterina Lobacheva 1,2
1 Mila - Quebec AI Institute, 2 University of Montreal, 3 Constructor University
Paper Code

Abstract

We propose shared gradient discovery and superposition as a mechanism underlying generalization in LLMs, where shared gradients lead to inherently generalizing shared solutions. To validate our hypothesis, we study circuit emergence as one form of learning such generalizing solutions. We find that our hypothesis can indeed explain and shed new light on circuit emergence and generalization.

Poster (ICLR 2026 Sci4DL Workshop)

Paper poster

BibTeX

@inproceedings{
      mircea2026shared,
      title={Shared Gradient Discovery and Superposition: Learning Dynamics of Generalization in {LLM}s},
      author={Andrei Mircea and Ildus Sadrtdinov and Irina Rish and Ekaterina Lobacheva},
      booktitle={Workshop on Scientific Methods for Understanding Deep Learning},
      year={2026},
      url={https://openreview.net/forum?id=c1pEF21dsY}
}