LLMs are frequently recommended for learning. They can act as a tutor and let you ask specific questions about the topic and quickly get back a response. After using them pretty extensively to help me learn mathematics, I’ve never had a ton of success. By removing a lot of the difficulty in understanding dense textbooks/lectures, I can never remember what I learned.

I can only speak for mathematics, but I imagine the problems I have with LLMs are pretty universal for any topic. I spend a lot of time reading textbooks, watching lectures, and solving problems. The process is sometimes unbearably slow, which can be demotivating. I have tried in the past to go faster (mostly by doing fewer problems), but it is just unproductive. Very quickly nothing makes sense anymore and I just have to redo it slowly anyway.

When I’m really confused about something and staring at the offending statement in the book is not helping, I basically have three options. I can skip it and read ahead and hope it makes sense later in retrospect, I can find another source, or I can ask an LLM.

Using AI is definitely the option with the least friction. I find that using AI as a tutor makes everything feel so easy. There have been many times I have been so confused on a definition or a theorem and after consulting AI everything feels so crystal clear and intuitive. In those moments, I feel like it’s the greatest thing ever.

The problem comes when I try to remember it later. I think that since it removes all the difficulty in learning the topic, nothing gets internalized. Anytime you have a follow up question you can quickly get an answer to whatever level of detail you want. Only when I try to apply what I learned to solving a problem or recalling it later I realize how fleeting my understanding of it is. Even after reading back what the LLM had said before, often I cannot rebuild the mental state I was in where it all felt clear. Eventually I have to do the hard work of wrestling with the definitions and the logic in the proofs anyway, and using the LLM just was a waste of time in the process.

I have tried this so many times, and I have had this experience so often. It can still be helpful for very basic, clarifying questions where I can’t quite make out what the author means by a certain phrase, but for conceptual understanding I find it to be deceptively useless. It has a devious way of making you think you understand something when in fact you are nowhere close.