A mathematical perspective on transformers

Abstract

Transformers play a central role in the inner workings of large language models.We develop a mathematical framework for analyzing transformers based on their interpretation as interacting particle systems, with a particular emphasis on long-time clustering behavior.Our study explores the underlying theory and offers new perspectives for mathematicians as well as computer scientists.

Topics & Concepts