The memory tutorial track at Hot Chips this year was mostly one argument made from several directions, and the numbers are worth having in one place.
\- Accelerator FLOPS have grown about 3× every two years. HBM bandwidth has grown less than 2× over the same period. (Micron tutorial)
\- Memory's share of an AI chip's component cost went from 52% in Q1 2024 to 63% in Q4 2025. (Samsung, citing Epoch AI)
\- HBM takes roughly 3× the die area of DDR per bit, DRAM wafer capacity has been flat for over a decade, and a new fab has a 2+ year lead time. Spot DRAM is up \~7× per GB, and most of the memory vendors' revenue growth is price, not bits. (Jim Handy, Objective Analysis)
\- HBM4 doubles the interface to 2,048 I/O for \~2.8 TB/s nominal per stack, against 128 GB/s for HBM1. SK hynix quoted >2 TB/s in practice and \~40% better power efficiency than HBM3E. (SK hynix and Micron tutorials)
At 63% of BOM, halving the bytes moved is worth more than a node shrink, which is why the responses on stage were so different from each other:
\- \\Samsung, custom HBM base die.\\ Move the base die to 4nm logic, replace the 8 × 4 mm HBM PHY with a compact die-to-die link, offload the memory controller off the GPU, and eventually stack DRAM directly on the processor ("zHBM") for a claimed 70% lower I/O power and \~100 W saved per package.
\- \\Oxmiq, High Bandwidth Flash.\\ NAND in an HBM-style stack: 8 to 16× the capacity per dollar at \~3 TB/s per cube. Their rack sim at equal cost and power came out at \~14× aggregate capacity for \~0.6× per-GPU bandwidth. Wins on MoE expert weights and cold KV, loses on dense models. No production inference engine has a backend for it yet.
\- \\Samsung, LPDDR5X-PIM.\\ MAC trees in all 16 banks, 614 GB/s internal against 76.8 GB/s external. The clever part is "Address Align Mode," which maps MAC instructions onto ordinary DRAM addresses so an unmodified controller still works. Measured 3.01× tokens/sec on Llama 3.1 8B on real silicon.
My read is that PIM is the only one of the three whose gating problem is already solved, since it needs no host redesign, while HBF's gate is entirely software and custom HBM's is a business-model change for DRAM vendors. Interested in whether people closer to this see it differently.
Disclosure: I wrote these up in more detail in a free primer on this year's conference. The memory chapter is at https://allaboutchips.com/#memory, and every figure there is sourced to the specific talk and slide.