George Hotz's Three Problems of Autonomous Driving: Static, Dynamic, and Counterfactual AI Challenges

George Hotz critically analyzes the state of autonomous driving, suggesting that many leading companies like Cruise, Waymo, Aurora, and Zuke's share a common, potentially limiting, codebase derived from the DARPA Urban Challenge. He argues that true innovation requires a departure from this shared direction, possibly through breakthroughs in global HD mapping or advanced simulation. Hotz introduces a novel framework that divides autonomous driving into three distinct problems: static, dynamic, and counterfactual, asserting that the industry often overlooks the third and most complex challenge.

The static driving problem, according to Hotz, assumes the autonomous vehicle is the sole entity on the road. He contends this problem is 100% solvable through precise mapping and localization, especially with technologies like lidar and HD maps. While acknowledging it's not "easy" for all real-world conditions (e.g., varying weather, lane markings, road deterioration), he emphasizes that the methodology for solving it is "obvious." The dynamic driving problem, in contrast, involves real-time detection and prediction of other moving objects, such as a car stopped at a red light, whose presence and behavior are not fixed in a map and require predictive modeling of their actions.

The most challenging and often neglected aspect is the counterfactual problem, which addresses the autonomous vehicle's influence on the behavior of other agents, primarily humans. Hotz connects this concept to Judea Pearl's work on counterfactuals, highlighting its profound complexity. He admits to not having a complete solution for this problem but proposes reinforcement learning on the world, with a focus on exploitation rather than exploration, as the most promising path forward, given that the other agents are humans.

Hotz's framework underscores the escalating complexity of autonomous driving, moving from predictable, isolated scenarios to highly interactive and socially intelligent environments. His critique implies that current industry approaches might be stuck in solving only the first two problems, neglecting the crucial human-interaction aspect. The emphasis on reinforcement learning for the counterfactual problem suggests that achieving full autonomy requires not just advanced perception and prediction, but also a sophisticated understanding and strategic interaction with human behavior, pushing the boundaries of AI beyond mere technical execution into the realm of social intelligence and ethical decision-making.

"the way you leapfrog right is you come up with an idea or you take a direction perhaps secretly that the other people aren't taking"

"we divide driving into three problems and I actually haven't solved the third yet but I haven't idea how to do it"

"the static driving problem is assuming you are the only car on the road right right and this problem can be solved with 100% with mapping and localization"

"The static driving problem is not easy for the world the static driving problem is easy for one route and one route in one weather condition with one state of lane markings and like no deterioration no cracks in the road"

"you can get ten centimeters without liner you can get one centimeter with light our main concern about the one or ten centimeter I'm concerned every once in a while you're just way off"

"The second one is dynamic the second one is like say there's a an obvious examples like a car stopped at a red light right you can't have that car in your map"

"that car is not a fixed object that car may move and you have to predict with that car will dim alright so this is the dynamic problem"

"the counterfactual the only way I have to do it yet they don't give you thing that I want to do once we have all these cars is I want to do reinforcement learning on the world"

"the only real way to get at the counterfactual is to do reinforcement learning because the other agents are humans"