High-bandwidth flash (HBF) is a new memory tier intended for AI inference systems. The architecture intends to stack NAND flash in 3D with the goal of implementing mostly static model weights while reserving DRAM-class memory for dynamic data like activations and intermediate results.
The motivation is the same pressure that created HBM: processor throughput has outpaced the practical movement of data. In large inference systems, the weights can be far larger than the memory available directly beside the GPU or accelerator. Today those weights may move from SSD or network-attached storage into HBM DRAM before they are consumed. HBF attempts to collapse part of that path by placing a high-capacity, high-I/O NAND stack in the GPU/accelerator package.
Static weights versus dynamic data
AI compute can rely on dynamically changing data in the AI model, which corresponds to training, or static information encoded in the model, which corresponds to inference. The trained model weights that are used in inference are mostly static during execution. Meanwhile, activations and intermediate tensors are dynamic and must be written, overwritten and accessed with low latency. HBF is attractive for inference because data written to NAND is persistent and it provides much more local capacity than DRAM-based HBM.
The limit on the amount of RAM that can be reliably stacked into HBM is described as a significant system-level constraint by Jan Jongboom, senior director of engineering at Qualcomm and co-founder/CTO at Edge Impulse.
“If I want to run state-of-the-art LLMs, I am RAM-constrained there,” Jongboom said. “It's just how [many] parameters fit into the RAM of the GPU … that's almost always the constraint that we have. And RAM is expensive.”
HBM (3D-stacked DRAM) has provided significant capacity needed to intake and process activations, better known as the model inputs, which are dynamically changing and thus RAM is an ideal storage memory for these data. The other set of data needed for AI compute is the model weights, and these data do not change during an inference run.
Publicly described HBF targets closely match HBM bandwidth while providing much higher capacity than HBM at a similar cost profile. One representative capacity target is 512 GB per 16-die stack with an HBM4-like footprint, power profile and stack height. First-generation read bandwidth targets have also been described around 1.6 TB/s (according to Sandisk), which places HBF in a different category than conventional SSD-attached NAND.
The practical architecture is likely a mixed memory system. HBF may hold large weight sets near the accelerator, while HBM continues to handle dynamic compute state. That split also changes software assumptions. The runtime, compiler and memory controller need to understand which tensors are static enough to live in HBF and which tensors require DRAM behavior.
Jongboom noted the importance of removing the RAM constraint in favor of an alternative mechanism for storing model weights.
“If you can get unconstrained by the RAM, this will speed things up and you can do things more efficiently without having to transfer data back and forth, you've won on all fronts,” Jongboom said.
Package and interconnect complexity
HBF stacks dies of NAND flash in a similar way as the 3D-stacked DRAMs in HBM. Each NAND die contains hundreds of vertical memory layers, which must be interconnected and addressable. In a multi-die HBF stack, there could be thousands of total internal memory layers with difficult routing from upper dies down to the interposer or package substrate. How this routing will look without the use of thousands of TSVs is still an open question: will the interconnects route through the NAND stack layers or around them?
Design teams should expect HBF adoption to depend on several ecosystem items:
- A stable interface standard that accelerator vendors can implement.
- Known package footprints, stack heights and power delivery requirements.
- Thermal models that include both accelerator power and dense NAND stacks.
- Firmware and controller behavior that exposes predictable read bandwidth.
- EDA/package flows that can validate routing, power integrity and signal integrity at the full assembly level.
The need for standardization is already visible, and HBM innovation was already outrunning the standardization efforts. HBF adoption requires coordination around technical specifications, package behavior and interface definition. That work is necessary because HBF uses a different interface and cannot simply drop into an existing design to replace an HBM stack.
At the system-level, integrating storage into the GPU/accelerator package could simplify the rest of the design. For example, if the model data stays in-package, the system may reduce traffic across PCIe, CXL, Ethernet or proprietary accelerator fabrics. That can impact power budgeting, retimer IC count, backplane bandwidth and the overall storage architecture in a server and in a rack. It also changes failure analysis because performance bottlenecks move from raw storage throughput toward package-level memory.
A look at the structure of high bandwidth flash. Source: SanDisk
What will drive HBF adoption?
In summary, HBF and HBM solve different parts of the memory problem. HBM gives the compute processor a large capacity of dynamic memory, essentially expanding things like prompt length and context window. HBF gives the accelerator much larger local access to data that does not need frequent updates, essentially allowing larger parameter models or multiple smaller parameter models stored near the AI inference processor.
From this difference, the adoption criteria for HBF is clearer. HBF’s strongest use case is large-model inference where weights are read repeatedly, updates are infrequent and local capacity has a direct effect on throughput and energy per query.
Now, companies are looking at optimizing compute used in AI workloads, either by using small specialized models, faster legacy models or deploying local small-parameter models on internal IT infrastructure. This argues against the use of larger parameter spaces and thus against the use of HBF, instead relying on in-compute memory (ICM) to store and access model weights. In reality, growth will likely be in both paradigms and HDF will see plenty of adoption as it becomes commercialized.
