Wikinventia — Atlas of discoveries and inventions · Digital Age

Transformer architecture — Vaswani et al. / Google Brain

2017 AD · Transmission: Global
AIMethodNorth American

In 2017 a Google team published "Attention Is All You Need," presenting the Transformer architecture: a sequence-processing model based exclusively on attention mechanisms, without the recurrent networks that had dominated language processing until then. The advantage was twofold: better performance on translation and other language tasks, and full parallelization during training — allowing scaling to sizes previously impossible. The Transformer is the underlying architecture of all modern large language models: GPT, BERT, LLaMA, Gemini, Claude. It is the AI equivalent of what von Neumann architecture was for general computing.

InstitutionGoogle Brain / Google Research
Historical regionMountain View, California, United States
Primary sourceVaswani, A. et al. — "Attention Is All You Need" (NeurIPS, 2017)
Secondary sourceWolfram, S. — "What Is ChatGPT Doing... and Why Does It Work?" (2023)
Original languageEnglish
View this entry in the interactive atlas → View in graph →