Andrej Karpathy on learning & education
Karpathy treats education as a hard technical problem: building ramps to knowledge that deliver the most understanding per second, by finding the first-order ideas and explaining them plainly. For learners he stresses building things yourself, putting in the hours and measuring against your past self. As AI takes over more teaching, he sees human understanding as the part that can't be outsourced and the reason education matters so humans aren't sidelined.
13 points · 6 appearances · 2022–2026
How the view evolved
AI tutoring needs an auditable course to keep AI on track → Teachers should add only what agents can't do
In 2025 he said AI can't simply teach and must be kept on a leash by a human-made, auditable course; by 2026 he said the teacher contributes only the few bits agents can't, and agents can probably do the rest better.
Up next
2022–2026 · changed view
“Neural networks have done language modeling for a long time. So really what's new or interesting or exciting is just realizing that when you scale it up with a powerful enough neural net, Transformer, you have all these emergent properties, where basically what happens is if you have a large enough data set of text, you are in the task of predicting the next word, you are multitasking a huge amount of different kinds of problems. You are multitasking understanding of, you know, chemistry, physics, human nature. Lots of things are sort of clustered in that objective. It's a very simple objective, but actually you have to understand a lot about the world to make that prediction.”
2022–2026 · changed view
“Yeah, so I had a blog post on Software 2.0, I think several years ago now. And the reason I wrote that post is because I kept— I kind of saw something remarkable happening in like software development and how a lot of code was being transitioned to be written not in sort of like C++ and so on, but it's written in the weights of a neural net. Basically just saying that neural nets are taking over software, the realm of software, and taking more and more tasks. And at the time I think not many people understood this deeply enough, that this is a big deal, it's a big transition. Neural networks were seen as one of multiple classification algorithms you might use for your data set problem on Kaggle. Like, this is not that; this is a change in how we program computers. And I saw neural nets as, this is going to take over, the way we program computers is going to change, is not going to be people writing a software in C++ or something like that and directly programming the software. It's going to be accumulating training sets and data sets and crafting these objectives by which we train these neural nets, and at some point there's going to be a compilation process from the data sets and the objective and the architecture specification into the binary, which is really just the neural net, you know, weights and the forward pass of the neural net, and then you can deploy that binary.”
2022–2026 · changed view
“The consistent thing I've been always interested in, for me at least, is AI, and that's probably where I'm spending my rest of my life on, because I just care about a lot. And I actually care about like many other problems as well, like say aging, which I basically view as disease, and I care about that as well, but I don't think it's a good idea to go after it specifically. I don't actually think that humans will be able to come up with the answer. I think the correct thing to do is to ignore those problems and you solve AI and then use that to solve everything else. And I think there's a chance that this will work; I think it's a very high chance. And that's kind of like the way I'm betting.”
2022–2026
“Yeah, the data engine is what I call the almost biological feeling like process by which you perfect the training sets for these neural networks. So because most of the programming now is in the level of these data sets, and make sure they're large, diverse and clean. Oh, basically you have a data set that you think is good, you train your neural net, you deploy it, and then you observe how well it's performing, and you're trying to always increase the quality of your data set. So you're trying to catch scenarios basically there are basically rare, and it is in these scenarios that the neural nets will typically struggle in, because they weren't told what to do in those rare cases in the data set. But now you can close the loop, because if you can now collect all those at scale, you can then feed them back into the reconstruction process I described and reconstruct the truth in those cases and add it to the data set. And so the whole thing ends up being like a staircase of improvement of perfecting your training set. And you have to go through deployments so that you can mine the parts that are not yet represented well in the data set. So your data set is basically imperfect, it needs to be diverse, it has pockets there are missing, and you need to pad out the pockets. You can sort of think of it that way.”