Pause tokens strictly increase the expressivity of constant-depth transformers

Conference item

Pause tokens, simple filler symbols such as “...”, consistently improve Transformer performance on both language and mathematical tasks, yet their theoretical effect remains unexplained. We provide the first formal separation result, proving that adding pause tokens to constant-depth, logarithmic-width Transformers strictly increases their computational expressivity. With bounded-precision activations, Transformers without pause tokens compute only a strict subset of AC⁰ functions, while adding a polynomial number of pause tokens allows them to express the entire class. For logarithmic-precision Transformers, we show that adding pause tokens achieves expressivity equivalent to TC⁰ , matching known upper bounds. Empirically, we demonstrate that two-layer causally masked Transformers can learn parity when supplied with pause tokens, a function that they appear unable to learn without them. Our results provide a rigorous theoretical explanation for prior empirical findings, clarify how pause tokens interact with width, depth, and numeric precision, and position them as a distinct mechanism, complementary to chain-of-thought prompting, for enhancing Transformer reasoning.

Authors: London, C; Kanade, V

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: NeurIPS
• Article number: 116941
• Publication date: 2025-12-05
• Acceptance date: 2025-09-17
• Event title: 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
• Event location: San Diego, CA, USA
• Event website: https://neurips.cc/Conferences/2025
• Event start date: 2025-12-02
• Event end date: 2025-12-07
• Language: English