Part 2: zHBM: Next-Generation Memory for the Data Movement Era

Part 1 explored how HBM has evolved to supply data to AI chips. Today, as inference, reasoning, long-context processing, and agentic AI expand, AI systems must access and process increasingly large amounts of data. More generated tokens and larger KV caches place greater demands on memory.

As a result, AI system performance increasingly depends not only on compute capability, but also on how quickly and efficiently data moves through the system. Higher bandwidth must be achieved while controlling the power and heat associated with data transfer.

Samsung zHBM, currently under development, is a next-generation AI memory architecture designed around this challenge. Moving beyond conventional HBM architectures in which memory sits beside the AI chip, zHBM directly stacks memory on the processor.

Through direct 3D integration and a Distributed I/O architecture, zHBM creates shorter, denser connections designed to deliver high bandwidth with greater energy efficiency.

Expanding bandwidth through high-density vertical interconnects

A core technology behind zHBM is higher interconnect density between memory and compute.

Combining Wafer-on-Wafer (WoW) stacking with Hybrid Copper Bonding (HCB) enables direct copper-to-copper connections at fine pitches. Reducing the size and pitch of Through Silicon Vias (TSVs) also allows more vertical connections within a given area.

Together with a Distributed I/O architecture, these high-density interconnects increase data-transfer parallelism. More data can move simultaneously, enabling high memory bandwidth without relying solely on increasingly high per-link data rates. 

Lowering data-transfer energy through shorter paths

Direct stacking also shortens the distance data needs to travel. Shorter interconnects can reduce driver requirements and lower the burden on interface circuitry.

With zHBM, Samsung is exploring a short, wide interconnect architecture that reduces reliance on complex interface circuits and lowers the energy required to transfer data.

This becomes increasingly important as bandwidth scales. zHBM is designed to deliver more data while improving the energy efficiency of that movement.

Managing the thermal challenges of direct stacking

Bringing memory and compute closer together also creates new thermal challenges. When memory is stacked directly above an AI chip and heat is dissipated upward, heat generated by the processor may need to pass through the memory stack.

Samsung is therefore exploring stack-height optimization that balances memory density with thermal performance.

Stable zHBM operation requires the stack structure, interconnect density, and thermal path to be designed together. Thermal management is therefore a core part of system-level integration.

Co-designing memory with the AI chip

Because zHBM is more tightly integrated with the AI chip, memory and compute need to be considered together from the early stages of development.

Each AI chip has different requirements for performance, power, thermals, and physical dimensions. Memory bandwidth, capacity, interconnect placement, and stack configuration may therefore vary by system architecture.

Optimizing these requirements becomes more difficult when memory and compute are designed independently.

For zHBM, co-design and co-optimization are therefore critical. Data interfaces, power delivery, stacking, and thermal management need to be considered together from the start to create a solution optimized for each partner’s architecture.

Open collaboration built on integrated technology capabilities

This approach requires coordination across multiple technology domains.

Samsung brings together capabilities in memory, logic and foundry technologies, and advanced packaging. Memory technology defines how data is stored and supplied. Logic and foundry technologies support interface implementation. Advanced packaging enables the fine-pitch vertical connections and stacking structures required for 3D integration.

The ability to evaluate these technologies together provides a foundation for developing zHBM around system-level requirements.

Building on these capabilities, Samsung aims to work with customers to develop zHBM solutions tailored to the performance, power, thermal, and architectural requirements of their AI chips. The goal is to jointly optimize memory-to-compute connectivity within the actual AI system.

As AI workloads become more complex, memory and compute will need to work more closely together. Through zHBM and co-design with customers and ecosystem partners, Samsung is working to improve data-movement efficiency and advance memory solutions for the next generation of AI systems.