Frontier models still run on attention; state-space layers join only as hybrids — resolves 2027-07-14

Mainstream large models use the transformer architecture built on attention, which lets a model look back over all earlier text at each step, while state-space models such as Mamba read text step by step with a fixed-size memory and promise faster responses, and the debate is whether they will replace the transformer.

The architecture route decides what models cost to run and which kinds of computation chips and software are optimized for.

One camp says state-space models or linear attention will replace the transformer outright, while SecondSource says the two will coexist, with hybrids entering the mainstream on efficiency.

SecondSource’s reading: transformers and hybrid state-space models coexist. Hybrids, which interleave state-space layers with attention layers in one model to save computation, are entering the mainstream as an efficiency moat, but outright replacement has not happened.

If, at the July 2027 check, frontier labs’ main models still use attention as the backbone and state-space layers appear only as interleaved hybrid layers, the call is right.

arXiv (Albert Gu & Tri Dao)
“Mamba enjoys fast inference (5× higher throughput…”

Reviewed July 2027: if frontier flagship models still use attention as the backbone, with SSMs appearing only as interleaved hybrid layers, the call stands.

What would overturn it

Any frontier lab shipping a flagship built on pure SSM or linear attention.