Yann LeCun on AI Benchmarks, Specialized Human Intelligence, and the Misconception of AGI

This podcast clip features Yann LeCun discussing the critical role of benchmarks in evaluating AI progress, while expressing skepticism about claims of achieving Artificial General Intelligence (AGI) without rigorous, community-accepted testing. He emphasizes that even 'toy problems' like the BaBBI tasks can be immensely useful for testing specific machine reasoning abilities, contrasting this with the tendency for investors to be misled by exaggerated claims about AI systems mimicking the human brain. LeCun highlights the shift in AI research from traditional supervised learning on static, statistically independent datasets to interactive environments, such as robotics simulations and games, where an agent's actions dynamically influence the subsequent data it perceives and interacts with, thus breaking the assumptions of independent samples.

A central argument LeCun makes is his strong dislike for the term AGI, contending that human intelligence is far from 'general' and is, in fact, highly specialized. He illustrates this point with a thought experiment involving the permutation of optical nerve fibers, demonstrating that the human visual cortex's hardware is intrinsically built to exploit the 'locality' of the real world. This specialization means that our brains can only compute a minuscule fraction of all possible boolean functions, underscoring the profound limitations of even human cognitive capabilities when viewed against the vast landscape of all possible information processing.

LeCun suggests that the perception of human intelligence as general stems from our inability to conceive of tasks or phenomena outside our own comprehension. He draws an analogy to physics, comparing our limited perception of a system's state (e.g., pressure, temperature) to the vast, unperceivable 'entropy' or 'heat' representing the complete information of every molecule's position and momentum. This implies that while we feel general within our apprehended world, there's an infinite realm of structures and information we are simply not wired to perceive.

The broader implications of this discussion touch upon the philosophical understanding of intelligence itself and the future direction of AI research. Instead of chasing a potentially ill-defined AGI, LeCun advocates for focusing on building 'damn impressive intelligence' that excels in specific, complex domains, particularly those involving interaction and learning within dynamic environments. The conversation underscores the importance of grounding AI development in measurable progress and acknowledging the inherent specialization of both biological and artificial intelligence, rather than being swayed by abstract, often misleading, notions of 'generality.'

"ask them what the error rate they get on em 'no store imagenet"

"benchmarks and the practical testing the practical application is where you really get to test the ideas"

"toy problems can be very useful"

"new ideas the ideas that push the field forward may not yet have a benchmark or it may be very difficult to establish a benchmark"

"what if the answer you give determines the next sample you see which is the case for example in robotics"

"I don't like the term a GI because it implies that human intelligence is general and human intelligence is nothing like general it's very very specialized"

"our entire the hardware is built in many ways to support the locality of the real world"

"we are ridiculously specialized"

"we think we're general because we're general of all the things that we can apprehend"

"there is an infinite amount of things we're not wired to perceive any"