Jeff Dean

The Hardware-Software Loop Behind AI's Cost Curve

Jeff Dean is the engineer behind much of Google's core plumbing — MapReduce, Bigtable, Spanner — who then co-founded Google Brain and helped build TensorFlow and the TPU chips that train today's largest models. In August 2026 he leaves Google after 27 years to co-found Discovery Loop, a company betting that AI can run the scientific method itself — proposing, running, and evaluating experiments in parallel. The takeaway for a CTO: his long habit of co-designing hardware and software around cheap, low-precision arithmetic and sparse, mixture-of-experts models is where the real cost curve of AI lives, and it's a lever you can pull today.

The story starts in Hawaii, in the summer of 1968. Jeff Dean is born into a family that never sits still. His father studies tropical disease; his mother is a medical anthropologist, and their work moves the family from place to place while he's growing up. He's forever the new kid — learning a new town, a new school, a new set of rules. It's a childhood that rewards one specific skill: walking into a strange system and figuring out how it actually works.

Before he even finishes graduate school, he gets a preview of the thing he'll spend his life on. In 1990 and 1991, he writes software for the World Health Organization's Global Programme on AIDS — statistical tools to model and forecast the spread of HIV and AIDS. Code in the service of science. Hold onto that. It comes back around thirty-five years later.

He does his undergraduate degree at the University of Minnesota, graduating in 1990, summa cum laude, in computer science and economics. His honors thesis is the tell. He takes backpropagation — the algorithm that trains a neural network — and splits it across a thirty-two-processor machine, trying two different ways to divide the work. As he tells Dwarkesh Patel in February 2025, he was sure those thirty-two processors would be enough to train really impressive neural networks. They weren't. It turned out, he says, that the field needed about a million times more compute before the idea would work on real problems. He is early by roughly two decades, and he can see that clearly now.

Then comes a detour that turns out not to be a detour. He earns a PhD at the University of Washington in 1996, working under Craig Chambers on compilers — specifically, whole-program optimization, which is the art of making a program faster by reasoning about all of it at once. He goes to Digital Equipment Corporation's Western Research Lab, where he works on profiling tools and processor performance, and where he starts collaborating with a quiet, brilliant engineer named Sanjay Ghemawat. That partnership becomes one of the most productive in the history of software.

In 1999, Dean gets what he later calls the itch — the itch to join a startup. He lands at Google. He's one of the company's earliest engineers, its thirtieth employee, joining a couple dozen people working above what is now a T-Mobile store in Palo Alto. Speaking at the University of Washington commencement in June 2026, he tells the graduates he "got the itch to join a startup in 1999," and he describes that moment plainly.

What he and Ghemawat build over the next decade is the foundation almost everything else at Google sits on. Protocol Buffers, a compact way to move structured data between machines. MapReduce, published in 2004 — a way to process enormous datasets across thousands of cheap computers without the programmer having to think about how the machines coordinate, fail, or recover. Bigtable, in 2006, a storage system built to hold billions of rows. Later, Spanner, a database that stays consistent across data centers on different continents. If you've used Hadoop, you've used an open-source echo of MapReduce. The through-line here is worth naming: Dean's gift is hiding the hardest parts of distributed computing so that a normal engineer can command a fleet of machines and never feel the complexity underneath.

He becomes a kind of folk hero inside the company. Engineers write a running joke called "Jeff Dean facts," modeled on the Chuck Norris jokes — one of them claims that when the index servers went down in early 2002, Jeff Dean answered user queries by hand for two hours, and quality actually went up. It's a joke, obviously. But Bloomberg, writing about his departure, calls him the "most Google person" there is, and the affection is real.

Now the turn.

By 2011, Dean has spent more than a decade building the plumbing of the web. And he keeps bumping into the same idea from his undergraduate thesis — neural networks — which have spent twenty years in the wilderness because nobody had enough compute to make them work. That's changing. He runs into the researcher Andrew Ng, and they decide to do something that sounds slightly mad at the time: train genuinely large neural networks on Google's infrastructure. The team that forms is Google Brain. They build a system called DistBelief, which spreads a single neural network across thousands of processor cores — the same model-parallel and data-parallel ideas from his 1990 thesis, finally running on hardware big enough to matter. In 2012 they publish what everyone calls the cat paper: a network that watches millions of unlabeled YouTube frames and, with no one telling it what a cat is, grows a neuron that responds to cats. Nobody labeled anything. The structure emerged from the data.

That result changes the trajectory of the company, and arguably the industry. DistBelief becomes TensorFlow, which Google open-sources in November 2015, and which for years is the default way the world builds deep learning. In April 2018, Dean takes over as head of Google AI. In April 2023, when Google Brain and DeepMind merge into a single unit under Demis Hassabis, Dean becomes Google's chief scientist. He's also the person who proposes the name Gemini — because, as he puts it, it's like twins coming together, two research traditions joining into one.

It isn't a frictionless story, and an honest profile shouldn't pretend it is. In December 2020, the researcher Timnit Gebru leaves Google in a dispute over a paper on the risks of large language models — a rupture that Dean, in an email to staff, acknowledged had surfaced large and important issues about research culture. And a 2021 Google result on using AI to design chip layouts, published in Nature, drew sustained criticism over its methods and a lawsuit from a fired researcher. Dean's career is not a clean highlight reel. It's the real thing: consequential work, with contested edges.

Which brings us to the decision that makes this an episode. On August 5th, 2026, after 27 years, Jeff Dean leaves Google. He doesn't leave alone. He takes three of his closest collaborators — Sanjay Ghemawat, his partner of more than two decades; Oriol Vinyals, a research vice president at DeepMind and a technical lead on Gemini; and Quoc Le, a co-founder of Google Brain. Together they start a company called Discovery Loop.

The reshuffle around them is dramatic. The same day, Hassabis steps back from running DeepMind day to day to become its chair and Alphabet's chief scientist, and Koray Kavukcuoglu steps up to oversee Gemini. Alphabet's stock falls about four percent on the news. Sundar Pichai, in his message to employees, notes that the Gemini app has passed 950 million monthly users and that the smaller Gemma models have crossed 900 million downloads — and then says, of Dean, that after an incredible 27-year run, he's at a moment where he wants to try something new.

Here's the part that tells you who Dean is. Less than a day after leaving, he appears on stage at Stanford and jokes that he was "only unemployed for one second." He is 58 years old, he has nothing left to prove, and he is starting over — no office, no staff on day one, an empty page.

So what is he actually building? This is where we go under the hood.

To understand Discovery Loop, you have to understand the two technical bets that run through Dean's entire career, because he's about to make them again.

The first bet is about hardware. Somewhere around 2013, Dean and his colleagues do a back-of-the-envelope calculation and get scared by the answer. If neural networks start doing real work at Google scale, the general-purpose processors in the data centers won't keep up — not on cost, not on power. The bet they make is that you shouldn't run this new kind of math on old general-purpose chips. You should build a chip around the math. That chip is the TPU, the Tensor Processing Unit, first deployed inside Google in 2015.

The key insight is precision. A neural network is, at its core, a staggering number of multiply-and-add operations on matrices. And it turns out you don't need the full precision of a normal floating-point number to get good answers. As Dean tells Dwarkesh Patel in February 2025, when they started the first TPU, they weren't even sure they could quantize a model down to eight-bit integers for serving — but they had early evidence it might work, so, in his words, they decided to build the whole chip around that. That's the move. If each number is smaller and cheaper, you can pack far more arithmetic into the same silicon, the same watts, the same square millimeters. Over the years the field pushes precision even lower, down to four-bit integers for inference.

If that washes over you, hold onto one idea: the cost of AI is not really about doing fancy math. It's about how cheaply you can do an ocean of simple math, and how little you can move data around while you do it. As Noam Shazeer puts it in that same February 2025 conversation, arithmetic has become very, very cheap, and what's expensive now is moving the data. Dean's whole hardware philosophy is co-design — the algorithms and the chips shaped to fit each other, rather than one being forced onto the other.

The second bet is about sparsity, and it's the one CTOs should sit up for.

A normal, dense neural network activates every one of its parameters for every single word it processes. If the model has half a trillion parameters, all half a trillion light up for the word "the." That's enormous capacity, paid for on every token. The alternative is an architecture called mixture of experts. In 2017, Dean is a co-author on a paper with a wonderful title — "Outrageously Large Neural Networks" — that makes the idea practical. Instead of one giant dense network, you build many smaller expert sub-networks and a small router that, for each word, picks just a couple of experts to do the work. The model can be huge in total capacity, but only a small slice is active at any moment. You get the knowledge of a giant model at a fraction of the compute per token. Google's Gemini models use this approach.

It is not free. The router has to learn which experts to trust, and it can collapse — sending everything to a few favorite experts while the rest starve. The experts are scattered across many chips, so the router creates a storm of communication as tokens get shipped to wherever their chosen expert lives. Getting mixture-of-experts to train stably, and to serve efficiently, is genuinely hard systems work. Which is exactly the kind of problem Dean has been solving for thirty years.

And this points at the vision he's clearly chasing. In February 2025, he describes wanting a more organic structure — a model where the experts aren't all the same size, where some paths branch off for mathematical reasoning and never merge back, and, crucially, where different pieces can be developed independently. Imagine, he says, a hundred teams around the world, each improving the part of the model they care about — one team on a family of languages, another on a particular kind of code — and hooking their piece into the larger model without retraining the whole thing. He calls it a form of continual learning. A model grown in pieces, over time, more like an ecosystem than a single monolithic build.

Now put those two bets together and you can see the shape of Discovery Loop. The company is a public benefit corporation based in Palo Alto, and its mission is to automate the loop of the scientific method itself: propose an experiment, implement and run it, evaluate the result, learn, and go again — not one at a time, but thousands of these loops running in parallel. The plan, in stages: first, point it at machine learning research itself; then use it to improve their own systems; then generalize to any problem where you can actually score the outcome — chip design, biology, drug discovery, materials science.

The founders are candid about where the hard part is. As Quoc Le tells Wired in August 2026, he's excited that automating machine learning might let them discover a genuinely different architecture than the transformer. But Oriol Vinyals names the real gap in the same reporting: current models aren't strong at coming up with new ideas to try. Generating hypotheses worth testing — that's the frontier. Some observers frame the underlying bet as a shift from today's memoryless models toward systems that accumulate experience, hold hypotheses, and update their beliefs against real results over time. The investor Vinod Khosla, one of the backers, sums up the thesis bluntly: for years humans have used AI to do research; the premise here is that AI is the researcher.

Google is a founding investor and the cloud partner, and it's supplying the compute for at least the first year. Radical Ventures and Khosla Ventures are backing it, with Radical's Jordan Jacobs joining the board. The founders aren't disclosing the round's size or valuation. And in a detail that tells you everything about Dean's theory of the work, he becomes the chief executive almost by default, and describes the advantage of a small company in one word: focus. A small team, he argues, plus massive cloud compute, can concentrate on a single goal without the coordination costs of a giant organization. Small team, massive compute. That's the whole thesis, and it's the same belief he shared with those graduates in June 2026 — that small groups of people can build things with enormous impact.

So let's pull back. What should a CTO actually take from all of this?

Start with the part that's already true, not speculative. The cost curve of AI is set by precision, memory bandwidth, and interconnect — not by raw model size alone. Dean's career is a thirty-year argument that you win by co-designing the math and the machine. In practice, for you, that means low-precision inference — eight-bit, even four-bit — is not a corner-cutting hack; it's the mainstream lever for serving models affordably. If your inference bill is a line item that scares you, the questions to ask are about quantization, memory bandwidth, and how much your data is moving, not just how many parameters you're running.

Second: sparsity is where the economics bend. Mixture-of-experts is how you get more capability without paying for it on every token. But it moves the difficulty into serving — you have to hold all those experts in memory even though you only use a few at a time, and the routing adds real systems complexity. When you evaluate a model or a vendor, the active-parameter count matters as much as the total, and the serving story matters as much as the benchmark. This is a build-versus-buy conversation that MoE quietly changes.

Third, and more uncertain: the Discovery Loop bet is really a bet about organizational design. It says a handful of focused people with enough compute can outrun large teams on well-defined problems. If that's even partly right, it reshapes how you staff research, what you keep in-house, and what you rent from the cloud. But be honest about the state of it. As of its founding in August 2026, Discovery Loop has a mission, four extraordinary founders, and almost nothing built yet. Its own people say the models can't reliably generate novel ideas. The open question — the one the company is structured to answer — is whether an automated loop produces something the field calls a real discovery, or just a faster optimization of things we already know how to do. Don't budget on the strong version of that claim. Do watch for the first result that can't be waved away.

Fourth, a second-order effect worth tracking. If AI-driven research works anywhere first, it works in machine learning itself — models helping design better models. If that loop starts turning, the cadence of improvement in the tools you depend on could shift from human-paced to search-paced. You'd feel it as vendors shipping architectural changes faster than your team can absorb them. Plan for a world where your foundation models are a moving target, and build your systems so you can swap the engine without rebuilding the car.

And finally, the talent signal, because you hire and you retain. Four of the most decorated researchers in the field walked out of the best-resourced AI lab on earth — not for lack of money, but for focus and autonomy over their compute. That's the thing that pulled them. If your best engineers are drowning in coordination and can't get their hands on the resources to chase a real idea, no compensation package fixes that. Dean has said, over and over, in different words across the decades, that small groups of people can build things with enormous impact. He built a thirty-year career proving it inside a giant company. At 58, he's betting his reputation that it's more true outside of one. Whether he's right is the experiment. But the man has a long record of being early — and then being right about a decade later than everyone else was ready for.