Running a language model is largely a memory problem.

The first job, called prefill, processes every token of the prompt at once. That amounts to a large batch of matrix multiplication, which is what accelerators are designed for, so this phase is usually limited by how much arithmetic the chip can do.

The second job, decode, can’t be parallelized in the same way, because each new token depends on the ones before it. To produce a token, the chip reads the model’s weights and the stored state of the conversation from memory, and then it does the same again for the next token, and the one after that.

That is a small amount per token, but conversations can be long. One user at 8K tokens needs about 1.25 GB, at 32K about 5 GB, and at 128K about 20 GB.

A server also handles many conversations at the same time, and each has its own cache. The paper gives an example that grows from about 10 GB for a single user to 320 GB for 32 users, far more than the 70 GB taken up by the model’s weights.

HBM places stacks of memory chips next to the processor. It holds much more, 192 GB per card, but delivers 18 TB/s.

In the kitchenHBM is a large pantry down the hall with a single door.

Raptor’s 3D-DRAM sits directly underneath the processor and provides 100 TB/s and 32 GB per card. It is neither the fastest nor the largest of the three, but it comes much closer to offering both.

In the kitchen3D-DRAM is a fridge built in under the countertop, with a drawer at every workstation. It holds more than the countertop and less than the pantry, and everything in it is one reach away.

Raptor stacks the two instead. Its logic die, made on TSMC’s N4P process, is bonded face to face with a DRAM die, so the memory sits directly beneath the circuits that use it.

The connection between memory and logic now covers the whole area of the die instead of a strip along one edge. A dense array of microscopic solder bumps, called µbumpsMicroscopic solder bumps that connect two chips stacked face to face., joins the two dies, and each bump carries one bit per clock cycle.

In the kitchenThe ingredients now sit in a fridge right under the countertop. Every station has its own drawer, so the chef no longer has to walk all the way to the pantry.

Each MCM contains 4 chipletsA small chip designed to be combined with others in one package. and is designed for about 422 W. Around them are 8 conventional LPDDR5X-9600 memory chips, which add 128 GB of slower memory for data that doesn’t fit in the stacked DRAM.

A chiplet is the stack from the previous section: a logic die bonded to a DRAM die. The DRAM die is divided into 840 banksA DRAM chip is divided into many banks: independent grids of storage cells that can be read in parallel.. Most of them are grouped into 256 independent channelsA dedicated path from a group of banks to one compute engine., and the rest are held back as spares.

The logic die is divided into 4 gangs. A gang groups neighboring slices so that they can work together on a larger part of the model without involving the rest of the chip.

Each gang contains 4 slices, which the paper treats as the basic unit of the design. A slice has a 4×4 array of tensor engines for matrix arithmetic, a SIMD core for other operations, and 16 DRAM channels of its own directly underneath. The channels are independent, so maintenance work on one of them doesn’t hold up the others.

The DRAM die, however, has only 840 banks.

The design also keeps 72 of them in reserve as spares, for reasons covered in the third challenge. That leaves 768, which works out to exactly 3 banks for each of the 256 channels.

Three banks return 96 bytes per read rather than 128. A simple layout would need two reads for every chunk, fetching 192 bytes to use 128 and throwing away a third of what it reads. The paper calls this overfetch.

One fix is to stagger the data so that pieces of a chunk straddle two columns, read both into a 192-byte buffer, and shift the pieces into place, tracking a different alignment for each address. Nothing is wasted, but the shifting circuitry is complicated and makes it harder to run the memory controller at high speed.

Raptor deals with the problem in the way the software lays out data, a technique the paper calls stream blocking. Each chunk is split into a 96-byte part, stored across the 3 banks at the same column, and a 32-byte remainder. Remainders from neighboring chunks are packed together in a separate region.

One read fetches a column of remainders into a small 96-byte cache, which holds the leftovers for the next three chunks. Each chunk then needs just one more read for its main part. The pattern never changes, and every byte that is read gets used.

The figure shows a stream of 128-byte chunks crossing the connections, with the newest at the top. Orange cells mark connections whose value changed since the previous chunk, and the counter keeps a running total.

Conventional memory reduces switching with a technique called data bus inversionData bus inversion: sending data inverted (0s↔1s) when that means fewer wires switch, plus a flag saying so., but that relies on an extra signal pin and on data arriving in bursts over several clock cycles. Raptor’s stacked interface has neither, so the same job is done in the memory controller instead. The paper calls this stream flipping.

It works because of stream blocking: chunks are read back in the same order they were written. So the controller decides when it writes. It compares each chunk with the previous one on that channel, stores it inverted if that means fewer connections switch, and records the choice in a single flag bit kept in a small side region of the DRAM. When the stream is read back, the connections see the low-switching version, and the flag says which chunks to flip back. Because the flag lives in memory, no extra pin is needed.

The figure shows one chain from the paper’s example, with 24 banks and 2 spares forming 8 channels of 3. When a bank fails, every channel after it moves over by one position.

After a second fault the channels move again, and all 8 still have their full width. The reassignment is done by simple multiplexers on the logic die, next to the tensor engines, so no long wires are needed to route around the faulty banks.

A third fault leaves too few working banks. With two spares, a chain can tolerate up to two faults, wherever along the chain they happen to be.

Because the DRAM is pressed against the logic, the interface between them can reach 105 °C, and charge leaks faster when it’s hot. Above 85 °C, the paper says, the DRAM needs a refresh every 4 ms to hold its data, 8× as often as HBM’s nominal interval.

The paper measures the cost of moving from a 16 ms interval to 4 ms, which the slider uses for cooler temperatures. Its performance results assume an even shorter 2 ms interval.

Refreshing that often would usually be expensive, but Raptor’s banks are small. Each one has 1,364 rows, 16–32× fewer than a conventional DRAM bank, so a full refresh pass finishes quickly.

Each bank also protects its data with error-correcting codes. The last 8 columns hold a pair of interleaved [144, 140] Reed–Solomon codewords, stored alongside the stream-flipping flag bits. On a read, the controller fetches the codes and flags first, then the data, correcting errors before the chunk reaches the tensor engine. Background scrubbing catches errors before they pile up.

Measurements on the chip confirm this. Refreshing every 4 ms costs only 1.37% of the bandwidth, and even the 2 ms interval used in the performance results costs 2.7%.

Frequent refresh also protects against rowhammer, an attack that flips bits by activating one row over and over. With this chip’s timing, an attacker needs about 8.8 ms of continuous activations to reach the threshold. At a 4 ms refresh interval, every row is refreshed before its neighbors can get there, so the refresh schedule alone stops the attack.

Heat leaves the stack upward, through the logic die, the lid and the heatsink. Since the DRAM is underneath the logic, it runs about 3.5 °C cooler than the logic. In HBM the arrangement is reversed, with the memory sitting on top of hot logic. Most of the resistance to heat flow, about 80%, is in the cooling solution; the whole die stack, DRAM included, adds only about 1.5%.

The team also varied the DRAM clock between 500 MHz and 1 GHz. A faster clock raised bandwidth and lowered latency, as expected.

For Kimi K2, which has a trillion parameters, the gap is much wider: 1,584 SRAM cards (a figure the paper gives directly), compared with 54 Raptor cards and 10 HBM cards. The menu in the figure lists the other models.

HBM needs even fewer cards, but each one reads memory at 18 TB/s, compared with 100 TB/s for Raptor. Capacity determines how many cards a model needs, and bandwidth determines how quickly each of them works.

The number of cards matters because the cards working on one model constantly exchange partial results, in group operations called collectivesA group communication step where many cards exchange partial results.. Raptor’s connections form a hierarchy: an on-chip network inside each chiplet, die-to-die links inside each module, and PCIe Gen 7 or Ethernet between modules and cards.

In one specific setting, with a 4K context and a network with 0.5 µs latency and 1 TB/s of bandwidth, the improvement is 4.38× over HBM and 3.15× over SRAM.

Raptor doesn’t come out ahead in every case. Small speech models fit on a single card, so capacity stops mattering and raw bandwidth decides the outcome; for those, SRAM is fastest. All of these figures also come from a model rather than from measurements of a working deployment.

Because HBM holds more per card, the smallest HBM setups use fewer cards than Raptor. The paper also checks a fairer split, giving every design the same number of cards. HBM then runs larger batches, but at small and medium batch sizes 3D-DRAM stays ahead.