Andrej Karpathy on AGI & AI timelines
Karpathy thinks AI is worth solving first because it can then solve everything else, but he expects no sudden god-in-a-box: AI continues a decades-long intelligence explosion and diffuses gradually, with agents taking about a decade to really work. He expects digital work to be transformed first and the physical world later, and his main worry has shifted from a catastrophic flip to a gradual loss of human understanding and control.
14 points · 5 appearances · 2022–2026
How the view evolved
Optimus is a hedge in case internet data isn't enough → Eventually AI must ask the universe questions via experiments
In 2022 he called Optimus a hedge in case internet data alone isn't enough for AGI; by 2026 he was firmer that AI will run out of purely digital material and must move to sensors, actuators and real-world experiments.
AGI's bad outcomes may be a flipped minus sign away → Most likely outcome: gradual loss of understanding and control
In 2022 his worry was that bad AGI outcomes might be a tiny flipped sign away; by 2025 he called the most likely bad outcome a gradual loss of understanding and control as AI layers in everywhere.
Up next
2022–2026 · changed view
“Neural networks have done language modeling for a long time. So really what's new or interesting or exciting is just realizing that when you scale it up with a powerful enough neural net, Transformer, you have all these emergent properties, where basically what happens is if you have a large enough data set of text, you are in the task of predicting the next word, you are multitasking a huge amount of different kinds of problems. You are multitasking understanding of, you know, chemistry, physics, human nature. Lots of things are sort of clustered in that objective. It's a very simple objective, but actually you have to understand a lot about the world to make that prediction.”
2022–2026 · changed view
“Yeah, so I had a blog post on Software 2.0, I think several years ago now. And the reason I wrote that post is because I kept— I kind of saw something remarkable happening in like software development and how a lot of code was being transitioned to be written not in sort of like C++ and so on, but it's written in the weights of a neural net. Basically just saying that neural nets are taking over software, the realm of software, and taking more and more tasks. And at the time I think not many people understood this deeply enough, that this is a big deal, it's a big transition. Neural networks were seen as one of multiple classification algorithms you might use for your data set problem on Kaggle. Like, this is not that; this is a change in how we program computers. And I saw neural nets as, this is going to take over, the way we program computers is going to change, is not going to be people writing a software in C++ or something like that and directly programming the software. It's going to be accumulating training sets and data sets and crafting these objectives by which we train these neural nets, and at some point there's going to be a compilation process from the data sets and the objective and the architecture specification into the binary, which is really just the neural net, you know, weights and the forward pass of the neural net, and then you can deploy that binary.”
2022–2026 · changed view
“One more thing that I would say is that many times, when people think about education, they think more about what I would say is a softer component of diffusing knowledge. I have something very hard and technical in mind. In my mind, education is the very difficult technical process of building ramps to knowledge. In my mind, nanochat is a ramp to knowledge because it's very simple. It's the super simplified full-stack thing. If you give this artifact to someone and they look through it, they're learning a ton of stuff. It's giving you a lot of what I call eurekas per second, which is understanding per second. That's what I want, lots of eurekas per second. So to me, this is a technical problem of how do we build these ramps to knowledge. So I almost think of Eureka as maybe not that different from some of the frontier labs or some of the work that's going on there.”
2022–2026
“Yeah, the data engine is what I call the almost biological feeling like process by which you perfect the training sets for these neural networks. So because most of the programming now is in the level of these data sets, and make sure they're large, diverse and clean. Oh, basically you have a data set that you think is good, you train your neural net, you deploy it, and then you observe how well it's performing, and you're trying to always increase the quality of your data set. So you're trying to catch scenarios basically there are basically rare, and it is in these scenarios that the neural nets will typically struggle in, because they weren't told what to do in those rare cases in the data set. But now you can close the loop, because if you can now collect all those at scale, you can then feed them back into the reconstruction process I described and reconstruct the truth in those cases and add it to the data set. And so the whole thing ends up being like a staircase of improvement of perfecting your training set. And you have to go through deployments so that you can mine the parts that are not yet represented well in the data set. So your data set is basically imperfect, it needs to be diverse, it has pockets there are missing, and you need to pad out the pockets. You can sort of think of it that way.”