Jim Keller on the Enduring Relevance of Moore's Law and the Future of Computation

Jim Keller, a renowned computer architect, challenges the pervasive notion that Moore's Law is dead, arguing instead that it continues to drive exponential performance improvements in computing. He defines Moore's Law not just as the doubling of transistor count every two years, but more broadly as a 2x increase in computer performance every 2-3 years, a cadence he has observed throughout his 40-year career. Keller attributes this sustained progress to a "cascade of diminishing return curves" across thousands of innovations in equipment, optics, chemistry, physics, and material science, rather than a single, linear advancement. He emphasizes that while individual innovation curves may plateau, new inventions and approaches continuously emerge, collectively sustaining the exponential growth.

Keller delves into the physical limits of shrinking transistors, noting that modern transistors are around 1000x1000x1000 atoms, while quantum effects become significant at 2-10 atoms, suggesting a potential for millions of times smaller devices. He highlights the ongoing roadmap for shrinking, including advancements like nanowires, which promise significant further reductions in size and improvements in control. This continuous shrinking, he argues, necessitates a fundamental shift in computer design and architecture, where designers must anticipate and leverage the increasing transistor density rather than being overwhelmed by its complexity. The challenge lies not just in manufacturing smaller components but in developing effective methods to control and utilize them cheaply and reliably.

Addressing the challenges of design in an era of increasing complexity, Keller points out two constants: human intelligence doesn't significantly increase, and team sizes have practical limits. This forces the adoption of robust abstraction layers, breaking down complex designs into manageable pieces, and the use of faster computers to design even faster ones, albeit with the caveat that N-squared algorithms require refactoring. He also discusses the evolution of computation itself, from simple equations to linear algebra, matrix operations, and now to the "deeper kind of computation" seen in AI, where data is viewed as a topology problem. He distinguishes AI training from simple search, describing it as a process of "endless projections" that tease out attributes, leading to mathematical abstractions that are not always intuitively understood.

Keller connects the continued validity of Moore's Law to the future of AI and other computational advancements, suggesting that expecting continued hardware improvement is a more viable strategy than solely relying on algorithmic breakthroughs. He introduces Bell's Law, which posits that every 10x increase in computation generates a new kind of computing, leading to transformations from mainframes to mobile devices and the impending "smart world" enabled by 5G. While acknowledging the unpredictable societal impacts of these technologies, he expresses a sense of luck and participation in a vast, collective endeavor, rather than sole responsibility. He concludes by noting that computation is now enabling mathematical explorations so sophisticated that the underlying patterns and results are not always fully describable or understood, marking a profound shift from traditional physics-based function discovery.

"the simple statement was from Gordon Moore was double the number of transistors every two years something like that and then my operational model is we increase the performance of computers by 2x every 2 or 3 years"

"somewhere where I became aware of it I was also informed that Moore's law was gonna die in 10 to 15 years... and at some point I decided not to worry about that particular product on occasion for the rest of my life"

"under the sheets there's literally thousands of innovations and almost all those innovations have their own diminishing return curves"

"a modern transistor is something like a thousand by a thousand by thousand atoms right and you get quantum effects down around two to two to ten atoms so you can imagine a transistor as small as 10 by 10 by 10 so that's a million times smaller"

"I'm expecting affecting more transistors every two or three years by a number large enough that how you think about design how you think about architecture has to change"

"there's two constants right one of those people don't get smarter... and then teams can't grow that much"

"the power of abstraction layers is really high we used to build computers out of transistors now we have a team that turns transistors and logic cells and our team that turns them into functional you know it's another one it turns in computers"

"the difference in quantity is the difference in kind you know the difference between ant and ant hill right or neuron and brain"

"shrinking the transistor is literally thousands of innovations right so there's so this they're all answers and it's there's a whole bunch of s-curves just kind of running their course and and being reinvented and new things"

"computation lets you do math McMath ematic allah computations that are sophisticated enough that nobody understands how the answers came out"