Quantifying concentration phenomena of mean-field transformers in the low-temperature regime

Conference item

Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models. In this paper, we study the evolution of tokens in deep encoder-only transformers at inference time which is described in the large-token limit by a mean-field continuity equation. Leveraging ideas from the convergence analysis of interacting multiparticle systems, with particles corresponding to tokens, we prove that the token distribution rapidly concentrates onto the push-forward of the initial distribution under a projection map induced by the key, query, and value matrices, and remains metastable for moderate times. Specifically, we show that the Wasserstein distance of the two distributions scales like log(β+1)/β exp(Ct) + exp(−ct) in terms of the temperature parameter β−1 → 0 and inference time t ≥ 0. For the proof, we establish Lyapunov-type estimates for the zero-temperature equation, identify its limit as t → ∞, and employ a stability estimate in Wasserstein space together with a quantitative Laplace principle to couple the two equations. Our result implies that for time scales of order log β the token distribution concentrates at the identified limiting distribution. Numerical experiments confirm this and, beyond that, complement our theory by showing that for finite β and large t the dynamics enter a different terminal phase, dominated by the spectrum of the value matrix.

Authors: Alcalde, A; Bungert, L; Riedl, K; Roith, T

• Publication status: Accepted
• Peer review status: Peer reviewed
• Publisher: NeurIPS
• Acceptance date: 2026-09-24
• Event title: 40th Conference on Neural Information Processing Systems (NeurIPS 2026)
• Event location: Sydney, Australi
• Event website: https://neurips.cc/Conferences/2026
• Event start date: 2026-12-06
• Event end date: 2026-12-12
• Language: English