Depending on what you're building, you may or may not have had to care much about accelerator (AI chip) supply constraints. But if you're building and operating software at scale, especially agentic software, you may be starting to feel a new infra bottleneck. Every engineering leader I talk with is starting to contemplate it.

The tech industry now has a CPU shortage on our hands, and we've all got a shift to make as a result. Server orders are quoting ~6 months (it used to be a week or two), and prices are up ~10-20% since March. Intel's CFO said on an earnings call that demand has blindsided them. A big part of the cause of this is agents, but really if you build and run software of any kind, you rely heavily on CPUs. Teams are starting to need to plan for CPU capacity, likely for the first time in the cloud-native era. I'll explain what's behind this and how your team might want to think about it.

Why we all rely on CPUs

You probably haven't thought a ton about how much you rely on CPUs, but think about it for a minute: your API services and web servers run on CPUs. Anything doing durable execution (queues, workers, workflow engines, cron) runs on CPUs. Your databases need CPUs with a lot of memory attached. CI needs a ton of CPU, especially now that agents are writing most code, which means we're all running way more builds and test suites than we were a year ago.

If you're running agents, there's even more CPU usage. There's of course model inference running on an accelerator, but an agent is also everything around model inference: a harness looping, pulling context together, calling tools, parsing what it gets back, keeping a session alive for longer and longer as models get better. When an agent wants to actually do something, like run code it wrote or operate a browser, you give it a sandbox or a container. If you're running many sessions concurrently, sandboxes can sneakily become one of the bigger things in your fleet.

This is likely why Intel said on their Q1 earnings call that AI-forward datacenters have gone from roughly 1 CPU for every 8 GPUs to 1 for every 4, and that agentic workloads could eventually even push it to 1:1. AMD cited something similar too. So more accelerators means more CPUs, and this is on top of all the non-AI software running in the world.

How supply works

To help understand why CPU supply is short, it's worth having a high level mental model of how it gets made.

A fab (fabrication plant) prints chips onto silicon wafers. A leading-edge fab costs >$20B and takes 3-5 years to build. Chips come in 2 broad families. First there's logic chips (compute): CPUs, accelerators, and chips for phones. Most of the world's logic chips are manufactured by TSMC in Taiwan, followed by Intel, and then Samsung. Then there's memory chips (storage): DRAM, which is regular RAM, and HBM (high-bandwidth memory), which is DRAM stacked 8-12 layers high and bonded onto an accelerator so it can feed it fast enough. Memory comes from 3 companies (SK Hynix, Samsung, Micron) out of their own fabs. A server is basically a computer in a rack: a CPU and sticks of DRAM, plus GPUs (with HBM) if it's an AI server. And then of course there's power. A big AI site needs hundreds of megawatts up to a gigawatt running on grids where new connections supposedly take 3-7 years to come online.

In the past few years, AI-fueled demand has skyrocketed, and these few companies suddenly needed multiple years and 10s of billions of dollars to actually add enough capacity. We ended up with 3 separate bottlenecks in factory capacity that AI is exacerbating. At TSMC, GPUs are competing with CPUs (and with Apple, Qualcomm, and Broadcom) for production lines. And at SK Hynix, Samsung, and Micron, HBM is competing with regular DRAM for wafers.

What we've ended up with is CPUs getting squeezed from both sides. AMD doesn't own fabs, so its CPUs need to come out of TSMC's constrained allocation. Intel does own fabs, but it's been working through yield problems and is now pulling some of its capacity from PC chips in order to make more server chips. And CPUs need DRAM, which has gotten more expensive because memory production has shifted toward HBM. Analysts are expecting CPU supply to add more comfortable headroom before memory does, but their expectation is that it's still going to be multiple quarters away.

What this means for software teams

Most of us have never capacity-planned CPUs. We planned databases, we maybe planned accelerators if we needed them, and we autoscaled on-demand into CPU capacity as much as our budgets allowed us to. But general purpose compute is now something many teams will need to commit to ahead of time, which means you should probably start to forecast and plan around it. If you're operating at scale, there are some things to spend your energy on.

The first is constraints on where and how you can use your capacity. One example is isolation constraints. You probably can't use hardware shared with other tenants for certain workloads, and creating isolation means you get less efficiency. Plus, you might need to run some workloads in certain geos or regions. And of course, some workloads need specific machine types. This is just an optimization problem.

Next, delivery is actually just the first step. In reality, it takes time to bring new clusters online. This could be days, weeks, or months depending on how complex your cluster strategy is. Making bring-up more efficient can be really impactful.

And if you've never had to think about it before, there's probably some inefficiencies you can find in how large you're running your fleet. Average Kubernetes CPU utilization across the industry is supposedly only ~10%, mostly because you plan for headroom around estimated peaks and rarely revisit those estimates.

Lastly, there's plenty you can do to actually make your software stack more efficient, across both your services layer and your agentic layer. I'm excited for my team to share learnings about making our API and agentic stack more efficient in the future.

If you're planning a roadmap for a team that operates software at scale and you haven't felt this already, you probably will soon, so you likely should bake in some time to deal with the CPU shortage.