Research & models Aug 24, 2026

Today · published August 24

A long-standing standard optimisation drops the cache hit rate from a theoretical 96.0% to a measured 28.8% under today's kind of traffic.

The same measurement holds one section you can act on directly. DP-attention splits the attention computation (the step that works out which words in a sentence relate to which) along data-parallel lines, and in single-turn fixed-length settings it usually pays for itself. Under long-context multi-turn traffic it turns negative. On MiniMax M3, a model from the Chinese AI lab MiniMax, at concurrency 40, throughput falls to 0.60x of not using it, and time to first token (the wait between pressing send and seeing the first word) degrades by more than 3x. At concurrency 32 the measured cache hit rate is 28.8% against a theoretical 96.0%. ⚠️ Those two figures come from different concurrency points (40 and 32), not from two columns of one measurement. The mechanism is stated plainly: each compute unit owns only a private quarter of the cache pool, so a 300,000-token conversation that lands on a different unit next time is recomputed from scratch (SemiAnalysis, 08-24).