KV compression makes HBM faster, not bigger: we bet Rubin Ultra stays at 288 GB — resolves 2027-03-31

As KV compression shrinks the working memory a model keeps during a conversation, the question is whether AI chips still need ever more high-bandwidth memory (HBM).

Memory makers, chipmakers and the cloud companies buying GPUs all use this call to decide whether to put money into memory capacity or memory speed.

One camp says compression will cut memory demand; SecondSource reads demand as not disappearing but moving from capacity to speed and to memory outside the GPU.

SecondSource’s reading: KV compression (shrinking the working data a model holds during a conversation) has not made memory demand go away. It has moved demand from the capacity of HBM (high-bandwidth memory stacked next to the GPU) to its speed, and to memory outside the GPU.

That is why NVIDIA’s next-generation GPUs will add HBM speed before they add capacity.

The concrete bet: the per-GPU HBM capacity NVIDIA officially publishes for Rubin Ultra will be no higher than 288 GB, the official figure for today’s Rubin.

The original material behind this call has not yet been linked here in a form readers can open and check.

Settled by NVIDIA's official per-GPU HBM capacity for Rubin Ultra, as published on its technical blog, a product page or GTC keynote slides; 30-day grace period.

What would overturn it

If Rubin Ultra's per-GPU HBM capacity is above 288 GB, sections three and four of the piece, which argue that HBM is getting faster rather than taller, must be rewritten.