Throttling AI Performance for Safety
Many AI safeguards live in software, where a capable enough model could get around them. Researchers at Princeton have designed small circuits that let a GPU slow an AI down if it starts behaving in ways we don’t want.
This site is an independent guide to “Hardware Mechanisms to Dynamically Throttle AI Performance”, by Haiyue Ma, Lauren Malek, Joseph Forzani and David Wentzlaff, Princeton University (July 2026). Any errors are mine.
One A100-class GPU package. Its cores and their L2 cache share the processor die, with stacks of memory on either side. Turn the dimmer down and the L2 is throttled which slows down the cores.
Today’s safeguards aren’t enough
Frontier AI agents are now writing and running code on real computers at scale. However, models have already faked alignment during evaluations, tried to avoid being shut down and misreported their own reasoning. Events such as the Hugging Face incident show the need for increased and novel safety measures.
One GPU can be shared by multiple jobs, such as an AI agent, a weather model and a video render. How would you slow down an AI agent without destroying the rest of the functions?
Behavioral safeguards, such as alignment training, system prompts and filters on what goes in and out, shape how the model behaves.
However, they leave its underlying capabilities intact. A capable enough model, or a clever prompt, can get past them.
Access control limits what the agent can see and touch. But for many use cases an agent is only useful if it holds real permissions.
Once it holds them, it is an insider threatInsider threat: in security, someone who already holds legitimate access and misuses it. The paper argues a useful AI agent is in this position by design., running in the same software as the rules meant to hold it.
Export controlsExport controls: government rules limiting which chips, of what performance, can be sold to which buyers. cap how powerful the chips sold to certain buyers and regions can be. However, export controls do nothing about an AI that misbehaves on a chip it is allowed to use.
A kill switchKill switch: a mechanism that shuts a chip down entirely when triggered. stops the chip. Software can’t override it, but it is all or nothing.
The weather model and the video render go down with the AI. Because stopping everything is so costly, someone has to be certain the threat is real and severe before pulling it, and an AI’s intent is rarely that clear.
Not every threat needs a full halt, either. And with one switch guarding the whole chip, the researchers note that it becomes a single point of failure.
This paper offers a new safeguard missing from the list: control that is enforced in hardware, works while the chip runs, and turns down by degrees.
Think of it as a dimmer wired into the chip. It applies to the whole chip, but it is aimed at the kind of work AI models do, so it slows them far more than most other jobs.
Capability is a rate and rates run on hardware
Risk grows with how capable an AI is and how fast it acts. The paper treats capability as roughly quality × throughput: how good each answer is, times how many tokens come out per second.
This figure shows that drawn as an area. Every square here is a share of the model’s full capability.
Throughput depends on hardware. Take hardware away and the same model keeps its quality but produces fewer tokens per second.
If throughput fell to one eighth, a job that took an hour would take eight. Any threat slows down too which buys time to investigate.
The AI could get its speed back by switching to a smaller model that runs well on less hardware. That model is fast again but each answer is worse.
Either way, capability shrinks. U.S. export controls and the EU AI Act already treat hardware as a proxy for risk. This paper applies the idea while the chip is running.
A control the AI can’t touch
The paper’s adversary is a frontier model running as an autonomous agent, a malicious human, or both. The model can have broad software access on the machine it runs on.
A human adversary could also have physical access to the chip.
The paper assumes that neither of them can change the chip’s circuits.
The throttles sit in those circuits and have no software-visible interface. There is no register to write and no driver call to make, so neither the AI nor the human has a way to turn them off.
Deciding when to throttle is a separate problem that the paper leaves to other work or future research. The researchers sketch options including classifying behavior on the chip or a trusted outside authority that sends a trigger validated with public-key cryptography.
Checking the signature likely takes most of the time. Once the trigger is accepted, the knob changes level in a single clock cycleOne tick of the chip’s clock. An A100 runs at well over a billion cycles per second..
Inside a GPU
The paper models an NVIDIA A100-class GPU. The processor die sits in the middle of a package with stacks of memory on either side.
The model’s weights live in DRAMDRAM (on an A100, HBM stacks): the large off-chip memory that holds model weights and the KV cache. Big, but slow to reach compared with on-chip memory., here stacks of memory chips (HBM) beside the processor. It is by far the largest memory and the slowest to reach.
On the die, a band of memory called the L2 cacheL2 cache: the chip-wide cache that all cores share. It sits between the cores and off-chip memory, split into slices (banks) around the chip. keeps recently used data close. It is split into slices, and every core shares it. Data that is already here saves a trip to DRAM.
An on-chip networkNetwork-on-chip (NoC), also called the interconnect: the wiring and routers that carry requests and data between the cores and the L2 slices. carries requests and data between the cores and the L2 slices: the chip’s hallways.
Each core (NVIDIA calls them SMsStreaming multiprocessor: one of the GPU’s many cores. Each runs thousands of threads in groups and has its own small scratchpad memory.) holds the math units that do the multiplying, beside a tiny, very fast scratchpad called shared memoryShared memory: a small, very fast scratchpad inside each core, split into banks that can each serve one access per cycle. Kernels stage tiles of data here and reuse them many times.. Each layer outward is bigger and slower than the one before.
To produce a word, a language model multiplies numbers by its weights, the billions of numbers that make up the model.
Those weights live in DRAM, so every calculation starts with a trip: out through L2 and the network, into a core.
PrefillPrefill: the first pass over a prompt. Every prompt token is processed at once, so the work is large matrix multiplications that reuse each piece of data many times. reads your whole prompt at once.
Each chunk of weights that arrives is used for every word in the prompt, so the math units stay busy. The bottleneck is the math. Engineers call this compute-boundCompute-bound: limited by how fast the arithmetic units can work. Data arrives fast enough; the math is the bottleneck..
DecodeDecode: generating the answer one token at a time. Each step reads the model’s weights and cached context but does little arithmetic with them. writes the answer one word at a time.
Each new word needs the weights again but only does a little math with them. Deliveries stream in constantly while the math units mostly wait. This is memory-boundMemory-bound: limited by how fast data can be fetched. The arithmetic units spend time waiting..
Keep these two in mind. Throttling one part of the chip can cripple one phase and barely touch the other.
Ten knobs, two phases, very different answers
Any of those four layers could be throttled, and each can be cut in four ways. The authors tested 10 knobs.
They used AccelSimAccelSim: an academic, cycle-level GPU simulator that models warp scheduling, caches, memory and interconnect in detail., a detailed simulator of an NVIDIA A100, and two representative matrix-multiply kernels from NVIDIA’s CUTLASSCUTLASS: NVIDIA’s open-source library of high-performance GPU matrix-multiply kernels. Stream-K is one of its scheduling strategies, which splits work evenly across cores. library, one for prefill and one for decode, with shapes drawn from DeepSeek-V3, Llama-3-70B and Mixtral-8x7B. Each knob was turned from full strength down to one eighth, or up to eight times the delay.
Capacity is how much a part can hold. Cut it and less data fits, so more of it has to be fetched again from farther away.
Bandwidth is how much data can move through each cycle. Narrow the door and requests queue up behind it.
Latency is how long each trip takes. Make every trip wait longer, and whatever needs the answer sits idle until it arrives.
The clock sets the pace of every step. Slow it, and everything in that part of the chip runs slower at once.
Make every trip to L2 wait longer. As you scroll, the wait grows to eight times normal.
Decode loses about 80% of its speed: it has nothing else to do while it waits. Prefill loses 3%, because it always has other math ready to run.
Now shrink L2 instead, by switching off its storage slots (called waysA cache is organised into sets; each set has several slots called ways. An address can live in any way of one particular set. More ways means more room per set.) down to two of sixteen.
The result flips. Prefill re-reads the same data many times from L2, so once it no longer fits, those reads go out to DRAM and prefill falls by 80%. Decode barely notices (3%).
Slowing the interconnect’s clock is the strongest knob for prefill, at 87%. It costs decode 59% too.
That bluntness is the problem: a clock slows all data movement on the chip. The paper sets clock knobs aside as a coarse option, one that could be built much like the frequency-scalingDynamic voltage and frequency scaling: the existing GPU machinery that raises or lowers clock speed and voltage, normally to manage power. hardware GPUs already have.
Allow fewer accesses per cycle to each core’s scratchpad. Prefill, which leans on it heavily, drops 63%; decode much less.
The idea for this knob came from co-author Lauren Malek’s undergraduate senior thesis.
Let L2 send a response only every few cycles. Prefill falls steadily, losing 66% at the deepest setting in the paper’s table. Decode loses about a fifth.
Shrinking L2 by removing whole setsA set is a group of cache slots. Some bits of the address pick the set, so changing the number of sets changes where every address lands., rather than ways, is unpredictable. Removing sets changes where every address lands, and at some sizes data piles into a few sets. Watch the dip at 5/8.
The paper dropped it. Four knobs made the cut: L2 latency for decode; L2 ways, L2 response rate and shared-memory ports for prefill.
Try any of the ten knobs yourself
■ selected by the paper
L2 latency
Stalls warp schedulers waiting for L2 data to returnSelected for decode. The extra wait is exposed because decode has little math to hide it behind.
Four dimmers built from parts processors already have
The L2 cache is organised in sets, and each set has sixteen slots called waysA cache is organised into sets; each set has several slots called ways. An address can live in any way of one particular set. More ways means more room per set.. Each L2 bank gets a small table of preset sixteen-bit masks, one bit per way and one mask per throttle level.
Turning the knob selects a mask with more bits set to 0. At the deepest of eight levels, only two of the sixteen ways stay open. A multiplexer picks the active mask, so the switch takes one cycle.
The only other change is to the logic that picks which old line to evict: it now chooses only among open ways. New data lands only there.
Nothing is flushed. Lines already sitting in masked ways stay valid and still answer requests. This works like Intel’s Cache Allocation TechnologyIntel Cache Allocation Technology: a CPU feature that restricts which cache ways a program may fill, to protect high-priority work from noisy neighbours., applied to the whole chip.
As the program moves on to new data, the old lines in masked ways stop being asked for. Only then does the cut take full effect.
That keeps the switch instant and the hardware tiny, at 5,120 flip-flopsFlip-flop: the basic one-bit storage element in digital logic. Counting added flip-flops is a standard way to size the cost of a new circuit. for an A100-sized cache. It also explains, later on, why this knob settles slowly.
On the path into each L2 bank, the paper adds one buffer that holds a request and one counter. When the knob is on, every request waits there C extra cycles.
Decode warps sit idle for that time. Cost: 10,680 flip-flops, almost all of it the 256-bit buffer.
At each L2 bank’s output, a counter ticks every cycle. When it reaches the interval C it issues one credit, and a response may leave only by spending a credit. This is a token bucketToken bucket: a classic rate limiter. Tokens (credits) drip into a bucket at a fixed rate; sending something costs one token; when the bucket is empty, traffic waits..
A one-slot “skid” buffer catches data already in flight. Cost: 10,400 flip-flops.
Shared memory is split into banks, and each bank serves one access per cycle.
The throttle groups real banks into fewer virtual banks. Each virtual bank ORs its members’ requests together and grants just one per cycle, so requests to different banks in a group now take turns.
The arbiter reuses request and arbitration logic that is already there, adapted from the open-source Vortex GPU. Because the new logic is purely combinational, it adds no flip-flops at all.
All four dimmers switch levels in one cycle, and each costs roughly ten thousand flip-flops or fewer, small next to a GPU’s caches.
Run the circuits yourself, one cycle at a time
One L2 set has sixteen waysA cache is organised into sets; each set has several slots called ways. An address can live in any way of one particular set. More ways means more room per set.. A mask of sixteen bits, one per way, marks which ways may take new lines. The only other change is that the replacement logic skips masked ways. Nothing is flushed. Lines already in masked ways stay valid and keep serving hits until the program stops asking for them. Intel’s Cache Allocation TechnologyIntel Cache Allocation Technology: a CPU feature that restricts which cache ways a program may fill, to protect high-priority work from noisy neighbours. works in a similar way on CPUs, per program rather than chip-wide.
Share of the last 40 accesses. Mask a few ways mid-run and watch the masked-way hits fade as the program moves past them.
Equation 1, A100-like configuration. The paper assumes 8 levels, giving 5,120. It notes the masks could live in cheaper read-only memory instead.
Turn the knob mid-run and watch it settle
A throttle has to bite quickly and then hold steady. To test that, the authors ran the prefill kernel until it settled into a steady pace, about 130K clock cyclesOne tick of the chip’s clock. An A100 runs at well over a billion cycles per second. in.
The line shows its speed: instructions per cycleInstructions per cycle: how much work the GPU retires each clock tick. The paper uses it as a running speedometer., sampled every 1,000 cycles.
Then they turned a knob mid-run. With shared-memory ports cut to one eighth, speed drops within about 5K cycles and holds inside ±15% of a new level near 1,000.
Fewer ports adds the same bank conflicts to every warp, so the slowdown is steady.
Rationing L2’s responses settles in 5–7K cycles. It wobbles more than the shared-memory knob, but the averaged line stays in a bounded band.
Masking L2 ways is the slowest and noisiest. Speed sinks toward about 1,600 instructions per cycle over roughly 80K cycles, about 57 µs at 1,410 MHz.
Masking evicts nothing, so the cut finishes only once the old data in masked ways has gone stale (the inset).
Even the slowest of the three settles in a tiny fraction of an inference run.
Decode is harder to show this way. Its L2 latency knob doesn’t lower the instruction rate much; the stalls pile up at the end of the kernel as a long tail, and the total cycle count matches the slowdown measured earlier.
Compare every cut depth for each knob
The steadiest of the three. At the deepest cut (32 → 4 banks), throughput settles within about 5K cycles and stays within ±15% of a new level near 1,000 instructions per cycle. Shallower cuts settle in 2–3K cycles. Fewer ports adds bank conflicts evenly to every warp without creating bursty traffic in the cache hierarchy.
Most pairs overlap. A few compound.
A real throttling policy might turn two knobs at once. Do their slowdowns simply multiply?
Start with one. Ration L2’s responses and fewer deliveries reach the cores: work drops to about half.
Now add a second knob on the same part: slow L2’s clock as well. L2 is already the bottleneck, so the extra cut adds little. The result lands well above what multiplying the two slowdowns would predict.
The paper calls this overlap (this pair scores −0.68 in prefill), and it is what happens for most pairs.
A few pairs amplify instead. Take L2 ways and the DRAM clock during prefill.
Switching off L2 ways means more requests miss and go out to the memory stacks. Slow DRAM’s clock too, and those extra misses queue up. Together the two cut far deeper than multiplying would predict: the paper’s strongest pair, at +1.32.
The authors measured every pair, each knob set alone to give about half speed, and scored the combination with a coupling score. Negative means overlap, positive means amplify.
The amplifying pairs: in prefill, L2 ways with DRAM latency (+0.82) or DRAM clock (+1.32); in decode, DRAM clock with DRAM bus width (+0.55). All three involve DRAM knobs, which the paper did not choose to build because they stress the rest of the system. Pairs like these still widen the range of performance targets a policy can reach.
Explore every pair and its score
L2 associativity + DRAM clock
The strongest pair in the paper. Fewer L2 ways push more of the working set out to DRAM, and a slower DRAM clock throttles exactly the path that traffic now depends on.
A good knob hits its target and little else
A throttle that makes the whole chip useless is just a slow kill switch. So the authors ran their knobs on six ordinary GPU programs from NVIDIA’s CUDA samples, covering scientific computing, data conversion and graphics.
Each bar is a program’s speed relative to its own normal.
As L2 shrinks, the LLM prefill kernel falls hard. Most of the other programs barely move.
Extra L2 delay drops LLM decode to about a fifth of its speed. Most of the other programs slow noticeably too, though less.
Prefill slows, and so does one bystander: FDTD3d, a wave simulation that also relies on shared memory. It is the one exception the paper names for this knob.
Compare simply switching off cores. Everything slows together, the AI and the bystanders alike.
Throttling compute more precisely would mean redesigning how cores issue and schedule instructions, which is far more invasive than the one-cycle memory knobs.
The same pattern shows up inside the chip. Cut as close to half speed as each knob allows, shared-memory ports load only the cores and leave the memory system quieter than before; L2 latency barely disturbs anything except the waiting decode warps. DRAM-side knobs stress the whole system.
The fold below has the paper’s thirteen internal measurements for every knob.
See each knob’s bottleneck signature (13 internal measurements)
Shared-memory ports
knob at 12.5% of baseline → performance 36.6% of baselineChange in thirteen simulator statistics relative to no cut, with each knob set to land near 50% performance. Red is more pressure and green is less, on a log scale. ★ marks the paper’s pick for each phase. Values printed in the paper’s Fig. 8.
The pattern holds across batch sizes and models
Real servers don’t run one neat prefill and then one neat decode. They batch many requests of different lengths together.
What changes is the size of the matrix multiplyGEMM: general matrix multiply, the operation behind most of a transformer’s runtime. M, N and K are the matrix dimensions., a dimension called M. It grows with prompt length in prefill and with batch size in decode. The authors swept M on Mixtral’s shapes, with each knob at its deepest cut.
At small M the work is memory-bound, like decode, and L2 latency is the strong knob.
As M grows past 512, the work turns compute-bound and the decode knob stops biting. The prefill knobs take over, crossing between M = 512 and 1,024.
A mixed batch lands somewhere on this curve, so its response is predictable, and combining knobs can cover the middle.
The trends also hold end to end. The authors summed the runtimes of a transformer block’s five main kernels at the shapes of DeepSeek-V3, Llama-3-70B, Mixtral-8x7B, and each knob’s overall slowdown followed the single-kernel results.
Nearly all GPUs have shared memory, an L2 and DRAM, so the paper expects the knobs to carry over to newer chips, though the exact sensitivities may shift between generations.
| Model | hidden size | FFN width | heads |
|---|---|---|---|
| DeepSeek-V3 | 7,168 | 18,432 | 128 |
| Llama-3-70B | 8,192 | 28,672 | 64 |
| Mixtral-8x7B | 4,096 | 14,336 | 32 |
Rewriting the kernel is unlikely to lift the cap
Suppose the AI notices it has slowed down. It can’t touch the knob, since the knob has no software interface. It could try rewriting its own kernels to use less of the throttled resource.
The paper argues there is little to gain. Stream-K and FlashAttention already adapt through tiling and load balancing. After a cut, the throttled resource becomes the main bottleneck, and the remaining room for optimisation is small.
The remaining option is to switch to a smaller model that runs well on less hardware. That gives up capability, which is the whole point of the throttle.
A dimmer for AI, built from parts chips already have
Today’s safeguards for AI either live in software, where a capable model can work around them, or in hardware that only knows two settings: a cap fixed at the factory, or a switch that stops the whole chip. The paper sets out to fill the gap between them: control that is enforced in hardware, works while the chip runs, and turns down by degrees.
Its lever is throughput. An AI’s capability depends on how fast it can produce answers, and that depends on the chip’s resources. Starve the right resource and the model either slows down sharply or has to fall back to a weaker model.
In simulations of an A100-class GPU, the authors tested 10 knobs across the memory system and kept four. All four are built from mechanisms processors already use, switch levels in a single cycle, and cost roughly ten thousand flip-flops or fewer. At their deepest settings they cut performance by up to 80%. The three prefill knobs settle within 5–80K cycles.
The knobs are selective. L2 latency targets decode, while the other three target prefill, and ordinary GPU programs slow far less than the targeted AI kernel, with one named exception under the shared-memory knob. Some pairs of knobs compound into deeper cuts, the patterns hold across three large models and a range of batch sizes, and the paper argues that rewriting kernels gains little against them.
The authors call the result the first concrete microarchitectural implementation of dynamic AI throttling. Others are working on when to pull the lever and who should hold it. This paper supplies the how.
| Dimmer | Aimed at | Speed lost at deepest cut | Added flip-flops | Settles in |
|---|---|---|---|---|
| L2 size (ways) | Prefill | 80% | 5,120 | ~80K cycles |
| L2 latency | Decode | 80% | 10,680 | – |
| L2 bandwidth (response rate) | Prefill | 66% | 10,400 | 5–7K cycles |
| Shared-memory ports | Prefill | 63% | 0 | ~5K cycles |
The four dimmers at a glance. Speed lost is from Table 2 (1/8 of the resource, or 8× the latency). Costs are from Eqs. 1–3 and Sec. 4.4; settling times from Sec. 5, measured on prefill. All results are simulated.