Attention Mechanism
2014A learned, differentiable weighting over a set of representations via compatibility scores, letting a decoder attend differentially to encoder states instead of compressing an entire input into one fixed vector (Bahdanau, Cho & Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate," arXiv:1409.0473, ICLR 2015). Introduced inside an RNN encoder-decoder for neural machine translation; Transformer's self-attention (Vaswani et al. 2017) is a genuine architectural generalization -- removing recurrence and convolution entirely -- captured by a downstream edge, not the same fact restated.
Discovery Velocity
Normalized OpenAlex paper velocity, one point per year.
Originators
- Bahdanau, D.
- Cho, K.
- Bengio, Y.
Landmark Paper
Checked 2026-08-09 — interim signal only, see docs/BASIC_ROADMAP.md Phase 10
Connections
- is precursor to Transformerbasis: empirical
OpenAlex co-citation check (Phase 3 Step 4 dry-run, 2026-08-09): candidate term "attention mechanism" "encoder-decoder" vs. applied-side verification term "self-attention" (distinct from Transformer's own production backfill term "multi-head attention", which returned near-zero as a bridge proxy -- diagnosed as a term-scoping artifact, not a weak relationship) -- 14.0% overlap (28/200 sampled). Real, meaningfully-above-zero signal supporting an empirical basis. Relationship type is_precursor_to chosen over is_application_of per BASIC_SCOPING.md SS7.2: Transformer's self-attention (2017) generalizes 2014-style attention (removes recurrence and convolution entirely) rather than directly implementing it unmodified.
- evolved into Decomposable Attentionbasis: reasoned
Parikh et al.'s Decomposable Attention (ref [27]) generalizes Bahdanau-style attention by removing the recurrent network it was embedded in, using only feed-forward attention over token alignments -- cited by Vaswani et al. as one of the few pre-Transformer models that is attention-only, without recurrence.