Bridging the memory gap for large models
SK Hynix and SanDisk have introduced High-Bandwidth Flash (HBF) as a new memory solution aimed at easing bottlenecks in AI systems. As neural networks expand to include trillions of parameters, the limitations of High Bandwidth Memory (HBM) become more apparent. HBM provides the necessary speed, but its high cost and physical constraints limit how much data accelerators can access at once. This has become a major challenge as models grow larger and more data-intensive.
HBF addresses this by introducing a high-density memory tier that sits between HBM and standard PCIe-attached solid-state drives. It uses advanced 3D and '4D' NAND flash technology, arranged in eight- or sixteen-high stacked die configurations. Each stack can hold up to 512GB of data. Unlike traditional storage, HBF is designed to act as near-memory, allowing AI accelerators to offload large parameter sets without the severe latency and performance losses of PCIe connections. This makes it particularly useful for AI inference workloads, especially those using Mixture of Experts (MoE) topologies or per-layer embedding offloading.
UCIe interconnects drive performance
To deliver near-RAM-level latencies and massive data transfer speeds, HBF relies on the Universal Chiplet Interconnect Express (UCIe) standard. While still relatively new, UCIe is quickly becoming the preferred choice for die-to-die communication. It is supported by a broad group of industry leaders, including AMD, Intel, TSMC, and Arm. These companies have traditionally used proprietary interconnect methods, which have limited the compatibility and flexibility of chiplet-based designs. UCIe changes that by enabling chiplets from different manufacturers to work together, allowing for more modular and adaptable system designs.
By integrating UCIe chiplet packaging, HBF die stacks can be placed directly on the same silicon interposer or high-density organic substrate as the host GPU or custom ASIC. This close integration minimizes latency and maximizes throughput, which is critical for AI accelerators that need to process large amounts of data efficiently. This approach also makes it possible to mix and match chiplets from various suppliers, promoting innovation and competition in the chip design space.
High bandwidth tiers and efficiency
The HBF specification defines three scalable bandwidth tiers, labeled Grades 1 through 3, capable of delivering up to 3 terabytes per second of host interface bandwidth. This level of performance is essential for handling the massive data throughput required by modern AI workloads. This makes it a highly efficient option for data centers and other enterprise environments where energy consumption is a major concern.
Despite its impressive capabilities, HBF is not expected to make its way to consumer hardware any time soon. The technology relies on advanced 3D heterogeneous packaging, silicon interposers, and dedicated UCIe layers—features that significantly increase production costs. These expenses are only justified in the high-margin environment of enterprise data centers, where the scale and performance benefits of HBF can be fully realized. For PC hardware enthusiasts, the likelihood of seeing HBF in desktop systems or high-end gaming GPUs is slim, as the costs would push these already-expensive components even further beyond most budgets. Instead, HBF is being developed specifically for large language model context expansions and multi-agent AI systems, ensuring its place remains firmly in server racks and specialized AI accelerator boards.