← Back to Basic Science

H3 (Hungry Hungry Hippos)

2023
Computer Science (theoretical)Machine Learning TheoryFrameworkfoundational

End-to-end language model architecture combining S4 with a linear-attention-style mechanism, generalizing linear attention to more general recurrences (Dao, Fu, Saab, Rudra & Ré, "Hungry Hungry Hippos: Towards Language Modeling with State Space Models," ICLR 2023). Mamba's own block design is presented as directly simplifying the H3 block.

Originators

  • Dao, T.
  • Fu, D.Y.
  • Saab, K.K.
  • Rudra, A.
  • Ré, C.

Landmark Paper

W4313442864 ↗
Not retracted (OpenAlex)

Checked 2026-09-19 — interim signal only, see docs/BASIC_ROADMAP.md Phase 10

Connections

  • is precursor to Mamba
    basis: reasoned

    Figure 3's own caption: "Our simplified block design combines the H3 block, which is the basis of most SSM architectures, with the ubiquitous MLP block of modern neural networks... Compared to the H3 block, Mamba replaces the first multiplicative gate with an activation function." Mamba's block is a direct, named simplification of H3's.