In 2014, a graduate student named Dzmitry Bahdanau was working on machine translation when he devised a simple mathematical operation called "attention." It allowed a neural network to look anywhere in a source sentence to decide which word to generate next, solving the bottleneck that had limited translation quality for long sentences. The approach scaled, and by 2017 Google's transformer architecture made attention the central mechanism of AI, leading to large language models like ChatGPT and Claude.
The power of attention is easy to see in ambiguous sentences. Consider "The animal didn't cross the street because it was too tired" versus "...because it was too wide." To translate correctly, the model must figure out what "it" refers to. Attention lets the network weigh the most relevant words in the sentence, mimicking the way a human translator glances back and forth between source and target text.
Attention's success has made it the backbone of modern AI, but it also raises a deeper question: is attention the essence of intelligence? The Quanta Magazine article explores this idea, noting that the mechanism that began as a translation trick now underlies systems that converse, diagnose, and solve math problems. Whether attention alone explains intelligence remains an open question, but its role in the current AI revolution is undeniable.