Why Every AI Engineer Needs to Understand Hardware

I teach this on the first day of my AI systems course at Columbia. Students ask whether they really need to learn hardware to work in AI. Here is the answer.

In 1982, Alan Kay told a room of engineers that "people who are really serious about software should make their own hardware" [1]. For a long time that sounded like something only a chip company needed to worry about. You wrote your model in PyTorch, you called fit, and the next generation of chips showed up on schedule to make last year's code faster. Hardware was somebody else's job.

That deal is over. Not because the chips stopped improving, but because they stopped improving in the one way your model actually needs.

The most effective AI engineers I work with, at IBM and at Columbia, are not the ones who know the most model architectures. They are the ones who can look at a slow or expensive model and tell you which layer of the stack is holding it back. That skill starts with understanding the machine.

The gap that quietly changed everything

For twenty years, raw compute on server chips grew at about 3 times every two years. Over the same period, the memory bandwidth that feeds those chips grew at only 1.6 times every two years, and the links between chips grew slower still, at about 1.4 times [2]. Over two decades those rates compound into a very wide gap. Peak compute rose roughly 60,000 times, while memory bandwidth rose about 100 times and interconnect bandwidth about 30 times [2].

Fig. 1: Peak compute has raced ahead while the bandwidth that supplies it has crawled. Model size and training compute grew faster still. Source: Gholami et al., "AI and Memory Wall," IEEE Micro, 2024.

Amir Gholami and his coauthors named this the memory wall [2]. Chips improved at arithmetic much faster than they improved at reading the numbers to do arithmetic on. And modern AI does a lot of reading. A large model has billions of weights that must be loaded for every step, and a long context builds a key-value cache that must be read again for every new token. Those reads, not the multiplications, are increasingly what you wait on.

You can see it in the hardware sheet if you know where to look. An NVIDIA A100 does 312 trillion floating-point operations per second in its tensor cores, but its memory delivers about 2 terabytes per second. An H100 roughly tripled the compute to about 989 trillion dense operations, while its memory bandwidth only went up to 3.35 terabytes per second [3]. Each new generation asks your code to do more math per byte it moves, just to keep the expensive silicon busy.

Your model is probably waiting on memory, not compute

There is a simple mental model for this, and every AI engineer should know it. It is called the roofline, and Sam Williams and his colleagues at Berkeley introduced it in 2009 [4].

The graph has two axes. The horizontal axis is arithmetic intensity, which is how many operations you do for each byte you read from memory. The vertical axis is how fast you can actually run. There are two ceilings. One is flat, set by the chip's peak compute. The other is a diagonal, set by memory bandwidth. Your workload sits under those two ceilings depending on its intensity. If it does a lot of math per byte, it sits on the right and is limited by the compute ceiling. If it does little math per byte, it sits on the left and is limited by the bandwidth line, well below the peak the chip advertises.

Fig. 2: Attainable speed is the lower of two ceilings. Training and prefill do enough math per byte to reach peak compute. Token-by-token generation does not, so it is capped by memory bandwidth. Sources: Williams et al., CACM, 2009; Yuan et al., 2024.

This is not an abstract worry. It explains the single most common surprise in LLM serving. Training and the prefill phase, where the model reads your whole prompt at once, do thousands of operations per byte and comfortably hit the compute ceiling. But generation, where the model writes one token at a time, reuses each big weight matrix against a single skinny vector. In a careful profile of Llama-2-7B, those generation steps came out at roughly 1 operation per byte, clearly memory-bound, while prefill sat above 1,000 [5]. So the same chip runs your prompt fast and your answer slow, and the reason is not the model, it is the arithmetic intensity of the two phases.

If you have ever wondered why a chatbot streams its reply at a steady pace no matter how much you paid for the GPU, this is why. The speed limit during generation is memory bandwidth, not compute.

Moving the data is the part that costs you

Once you accept that data movement is the bottleneck, a lot of "obvious" optimizations flip.

In plain eager-mode PyTorch, each operation runs as its own separate step on the GPU. Every step reads its inputs from memory, does a small amount of work, and writes the result back out, and then the next step reads that result in again. Horace He gives a clear example: computing the cosine of a tensor twice, the naive way, reads and writes memory four times. Fuse the two operations so the data stays in fast on-chip memory between them, and you touch memory twice. Same math, half the memory traffic, and for a memory-bound operation that is a 2 times speedup for free [6].

Multiply that across the hundreds of small operations in a real model and it adds up fast. When the PyTorch team measured torch.compile, which fuses operations automatically among other tricks, across 163 open-source models on an A100, it ran them about 43 percent faster on average, with no change to the model itself [7]. The performance was sitting there the whole time. It was lost to data movement that nobody asked for.

There is a deeper point here that took me years to fully internalize. The same A100 that does 312 trillion matrix operations per second does only about 19.5 trillion of the ordinary non-matrix operations, the reshapes and normalizations and activations [6]. Those "cheap" pointwise steps are cheap in math and expensive in memory traffic, so they quietly dominate the clock. If you only think about floating-point operations, you will optimize the wrong thing.

What the hardware lets you trade

Understanding the machine is not only about diagnosing slowness. It is about knowing which changes are available to you. Two of them matter more than the rest.

The first is precision. A number does not have to be stored in 32 bits. If the model can tolerate it, you can use 16, or 8, or even 4. Every halving of the bits halves the memory you move and lets the hardware pack more numbers through each multiply. This is why the chip vendors keep adding new low-precision formats: the V100 leaned on 16-bit, the A100 added the bfloat16 format, the H100 introduced 8-bit floating point through what NVIDIA calls its Transformer Engine, and Blackwell added a 4-bit format [8]. NVIDIA reports that 8-bit floating point halves storage and doubles throughput compared to 16-bit, and that training in it can converge about as well as the wider format [8].

Fig. 3: The hardware added each lower-precision format as it became clear models could tolerate it. Fewer bits mean less memory moved and more math per cycle. Source: NVIDIA architecture documentation, 2022 to 2025.

I have written about where this ends up in practice, in Why Quantization Won, and Where Pruning Survived. The short version is that precision is the highest-impact change most teams have not pushed far enough, and it works only because the number format and the silicon were designed together.

The second lever is the algorithm itself, redesigned to respect the memory hierarchy. FlashAttention is the example I point students to first. Attention, done the textbook way, writes a large intermediate matrix out to memory and reads it back. Tri Dao and his coauthors rewrote it to keep the working data in fast on-chip memory and never write that large matrix out at all. Same exact result, far fewer memory reads and writes [9]. It delivered a 3 times speedup on GPT-2 and, more importantly, made context lengths practical that simply were not before, reaching sequences of 16,000 and 64,000 tokens on tasks where standard attention ran out of room [9]. That work did not come from a better theory of attention. It came from someone who understood exactly how a GPU's memory is laid out.

Sometimes the whole design is wrong for the job

Everything so far stays inside the conventional design, where compute and memory are separate and data moves between them for every operation. But if data movement is the real cost, the strongest option is to stop moving the data.

That is the idea behind computing in memory, and it is a large part of what my own group at IBM Research works on. Instead of loading weights from memory into a processor, you store the weights in a grid of memory devices and let the multiplication happen inside the array, as a physical effect of the currents. The weights never move.

Fig. 4: The conventional design pays for every trip across the memory bus. In-memory computing does the multiply-accumulate inside the memory array, so the weights never move. Sources: Nature Electronics, 2023; Science, 2023.

The payoff is energy. Our 64-core analog in-memory chip, built on phase-change memory, reaches about 9.76 trillion operations per second per watt for the matrix-vector math that dominates inference [10]. IBM's NorthPole chip, a digital cousin that keeps all its memory on-chip beside the compute, delivered 25 times more frames per joule than a graphics processor built on a comparable manufacturing process, on a standard image benchmark [11]. Those are not tuning gains. They come from removing the constant data movement that the conventional design requires.

You do not have to build one of these chips. But you should know they exist, because they tell you which of today's costs are fundamental and which are only a cost of the conventional architecture, not a hard limit.

What this means if you build with AI

I am not arguing that every AI engineer should become a chip designer. I am arguing for something smaller and more useful: enough hardware literacy to ask the right questions and understand the answers.

In practice that means being able to reason about a handful of things. Where does my data live, and how far does it travel to get multiplied. Is this workload bumping into the compute ceiling or the bandwidth ceiling. What precision can this model tolerate before accuracy moves. And when I profile the thing, where is the time actually going, as opposed to where I assume it is going.

None of those are questions you can answer from the math of the model alone. The students in my courses who struggle are rarely the ones with weak theory. They are the ones who never look below the abstraction, who treat the GPU as a box that is simply fast or slow. The ones who pull ahead learn to read a roofline and a profiler, and then they can reason about where the time actually goes. I have made the fuller cost argument in The Real AI Cost Problem: the biggest inference savings almost never come from one clever trick, they come from the algorithm, the compiler, the runtime, and the silicon being designed to work with each other.

My take, from the systems side

The old approach, write the model and let the next chip make it fast, worked for a specific decade and that decade is over. The valuable work now happens at the boundaries between the layers, and few people can work across those boundaries because most training teaches one layer at a time. If you are early in your career, this is the most durable thing you can learn. Frameworks get replaced every couple of years. The way memory hierarchies and parallelism work does not.

Why this is worth your time

There is a reason to care about all of this that goes beyond making your own model faster.

Sara Hooker made the argument sharply in an essay called "The Hardware Lottery" [12]. Her point is that research ideas do not win only because they are good. They win because they happen to fit the hardware and software of their moment. Neural networks were dismissed as impractical for decades, until graphics processors made them cheap to train. The idea did not change. The hardware did. And as chips get more specialized for the methods that are already popular, straying from the well-worn path gets more expensive, which means the hardware you have quietly shapes the ideas you are able to try.

That is the real stakes. The cost of running a model that performs at the level of the 2022 flagship fell from about 20 dollars per million tokens to about 7 cents in two years, a decline of more than 280 times [13], and most of that came from hardware and the software wrapped around it, not from smarter models. Meanwhile the electricity going into data centers is on track to more than double this decade, toward roughly 945 terawatt-hours by 2030 [14]. Every one of those trends is a hardware-software story. If you understand the machine, you get some say in how it goes. If you do not, you are limited to whatever the hardware already makes cheap, and you have to hope that the ideas you can afford to run are the same as the ideas that are actually best. They are not always the same. That is the whole point.

---

References

[1] Andy Hertzfeld, "Creative Think," Folklore.org, an account of Alan Kay's talk, July 20, 1982.

[2] Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, Kurt Keutzer, "AI and Memory Wall," IEEE Micro, vol. 44, no. 3, May/June 2024.

[3] NVIDIA, "H100 Tensor Core GPU" and "A100 Tensor Core GPU" product specifications, accessed 2026. Tensor-core throughput figures are dense; memory bandwidth is HBM.

[4] Samuel Williams, Andrew Waterman, David Patterson, "Roofline: An Insightful Visual Performance Model for Multicore Architectures," Communications of the ACM, vol. 52, no. 4, April 2009.

[5] Zhihang Yuan et al., "LLM Inference Unveiled: Survey and Roofline Model Insights," arXiv:2402.16363, 2024. Decode operations near 1 op/byte (memory-bound) versus prefill above 1,000 (compute-bound), profiled on Llama-2-7B.

[6] Horace He, "Making Deep Learning Go Brrrr From First Principles," 2022.

[7] PyTorch Foundation, "PyTorch 2.x: faster, more pythonic and as dynamic as ever." torch.compile measured across 163 models on an A100, 43 percent faster on average in training.

[8] NVIDIA, "NVIDIA Hopper Architecture In-Depth," 2022, and "Floating-Point 8: An Introduction to Efficient, Lower-Precision AI Training," 2025.

[9] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," NeurIPS 2022.

[10] Manuel Le Gallo, Riduan Khaddam-Aljameh, et al., "A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference," Nature Electronics, vol. 6, 2023.

[11] Dharmendra S. Modha, Filipp Akopyan, Alexander Andreopoulos, et al., "Neural inference at the frontier of energy, space, and time," Science, vol. 382, October 2023.

[12] Sara Hooker, "The Hardware Lottery," Communications of the ACM, vol. 64, no. 12, December 2021.

[13] Stanford HAI, "The 2025 AI Index Report." Cost to query a model at GPT-3.5 level fell from about $20.00 to $0.07 per million tokens between late 2022 and late 2024.

[14] International Energy Agency, "Energy and AI: Energy demand from AI," 2025. Data-center electricity projected to more than double to roughly 945 TWh by 2030.