I think a lot of blogging is reactive. You read other people’s blogs and you’re like, no, that’s totally wrong. A part of what we want to do with this scenario is say something concrete and detailed enough that people will say no, that’s totally wrong, and write their own thing.

— Scott Alexander

I recently read the AI 2027 predictions¹ . I think they’re way off. I was visualizing my self at Christmastime 2027, sipping egg nog and gloating about how right I was, but then I realized it doesn’t count if I don’t register my prediction publicly, so here it is.

This blog post is mostly about me trying to register my predictions than trying to convince anyone, but I’ve also included my justifications below, as well as what I think went wrong with the AI 2027 predictions (assuming anything did go wrong).

My predictions for AI by the end of 2027

Nothing particularly scary happens (beyond the kind of hype-driven scariness already present in 2025).
AI is still not meaningfully self-improving.
People still use the term “superintelligence” to describe something that will happen in the future, not something that is already happening
AI research is not fully automated by AI, and they certainly won’t be so advanced at AI research that humans can’t even follow along.
AI will not have meaningful control over the day-to-day operation of companies, AI companies or otherwise.
AI does not start out-performing a majority of AI researchers or coders.
AI will not substantially speed up software development projects. For example, the AI 2027 prediction that 2025-quality games will be made in a single month by July 2027 is false.
- I still believe my use of AI is less than a 25% improvement to my own productivity as a programmer, whether using full agentic AI or just chatbots. I still believe people who think AI is better than this are basically just mistaken.
AI have not taken a large number of jobs except in a few specific fields. (I am open to more hype-driven job difficulties faced by programmers, but not actual-capabilities-driven job loss for programmers.)
Large language models are still very stupid and make basic mistakes a 5-year-old would never make, as is true in 2025. Yet they are increasingly praised in the media for doing well on the SAT, math olympiad, etc., as in 2025.
- LLMs are broadly acknowledged to be plateauing, and there is a broader discussion about what kind of AI will have to replace it.
- LLMs still use text-based chain-of-thought, not neuralese.
- Most breakthroughs in AI are not a result of directly increasing the general intelligence/”IQ” of the model, e.g. advances in memory, reasoning or agency. AI can stay on task much longer than before without supervision, especially for well-specified, simple tasks. Especially since AI coding platforms will have gotten better at tool use and allowing AI to manually test the thing they’re working on. By the end of 2027, AI can beat a wide variety of video games is hasn’t played before.
AI still can’t tell novel funny jokes, write clever prose, generate great business ideas, invent new in-demand products, or generate important scientific breakthroughs, except by accident.
There is more public discussion on e.g. Hacker News about AI code rot and the downsides of using AI. People have been burned by relying too much on AI. But I think non-coders running businesses will still by hyped about AI in 2027.
Useful humanoid robots do not emerge (other than as demos and so on).
AI still can’t drive a damned car well enough that if I bought a car I wouldn’t have to.
AI are not writing good books or well-respected articles, but they have gotten better at mediocrity, and slop articles and comments are becoming a real problem. It’s really hard to figure out what percentage of e.g. Reddit comments are currently AI generated, so I can’t put a number on this other than to say it becomes noticeably more of a problem.
AI girlfriends/boyfriends/companions do explode in popularity (the market at least doubles compared to now). It becomes common for particularly lonely young people to spend time interacting with them. This is driven as much by the loneliness epidemic as by AI progress, and the meaningful improvements that lead to this are in AI voice natural-ness and memory, not intelligence.
- Similarly, LLM-driven NPCs in some AAA games start to show up, but are still not common.
None of the sci-fi Hollywood stuff in the AI 2027 predictions come true. AI safety teams do not bring back an older model for safety reasons. There is no government oversight committee that discusses pausing/slowing AI that actually has the power to do that. The president seriously discussing nationalizing AI, CCP spies stealing model weights, plans for attacks on Chinese data centers, humans on the verge of losing control of superhuman AI — none of this happens.
We will still have no idea how to safely align AI.

Justification

Why am I so pessimistic?

The primary reason is that I think LLMs will plateau.² I think there are two “complexity classes” of human thinking, and LLMs are only good at one of them.³ Let’s call them deep and shallow thinking.

Shallow vs. Deep Thinking

Shallow thinking is the kind of automatic, easy thinking you do when you have a regular conversation, and you’re just blurting out whatever comes to mind⁴. I’m also including effortfully manipulating information within consciousness as shallow thinking. For example, doing arithmetic, however complex, is shallow thinking. Shallow thinking is any thinking that requires simple recall and basic linear/logical information processing.

Deep thinking is what happens when your unconscious mind makes new, useful connections.⁵ Humans cannot do it consciously. Doing it unconsciously feels like listening to a “muse”. Clever jokes come from deep thinking, as do brilliant ideas. Nobody has identified as step-by-step process to generate funny jokes, because such a process would probably be exponential in nature and take forever. Instead, the unconscious mind does this in some kind of parallel way.

LLMs are terrible at deep thinking. All of the recent gains in AI capabilities have come from making them better and better at shallow thinking.

Noticing Where LLMs Fail

A seasoned mechanic might know exactly what’s wrong with your engine based on the clicking noise it makes. If you ask him how he knows, he’ll shrug. But he’s right. Probably. My own intuition about LLMs comes from working with them a lot on real-world coding problems. Not toy problems. Not benchmarks. Not English essays. You can BS your English teacher, but you can’t BS reality. I get to see the situations in which LLMs trip and slam their faces against reality. One minute, they seem smart — even brilliant — then the next, they’re like a shrimp that knows Python. From working with them, I’ve gotten a sense for what kinds of problems LLMs fail at. (Yep, my predictions are based on subjective intuition. I can’t prove anything. I’m just guessing.)

LLMs obviously do well at problems which closely match the training data. It’s definitely easier to solve a problem if you’ve already memorized the answer! It’s not always easy to tell if a problem is novel or not, but for novel problems, it seems to me they struggle most with deep thinking problems. As they have scaled up, they’ve gotten much better at shallow thinking (including arithmetic), but not much better at deep thinking.

Benchmarks sound like a way to see how well LLMs do when up against reality, but I don’t think they really are. Solving SAT problems or Math Olympiad problems only involves deep thinking if you haven’t seen millions of math problems of a broadly similar nature. Given that the SATs and the Math Olympiad are designed to be solvable by at least some high school students in a few hours, they probably don’t include problems that would require an LLM to do any deep thinking. Recall + shallow thinking would be enough. This is why LLMs have improved more on benchmarks than in reality.

My position is essentially the “LLMs are glorified autocomplete” hypothesis, except asserting that 90% of human cognition is also glorified autocomplete, and you can get pretty far with that kind of thinking actually. But it’s not the kind of thinking that leads to clever jokes, good business ideas, scientific breakthroughs, etc.

Why the Architecture of LLMs Makes Them Bad at Deep Thinking: They’re Too Wide

GPT-3 is 96 layers deep (where each layer is only a few “operations”), but 49,152 “neurons” wide at the widest. This is an insanely wide, very shallow network. This is for good reasons: wide networks are easier to run efficiently on GPUs, and apparently deep networks are hard to train.

An obvious problem with this architecture is you can’t really solve problems that require more than 96 steps (per token). Completing the next token of “23 + 15 =” might be solvable, but completing the next token of “The stopping time for the number 27 in the Collatz conjecture is: ” will not be (unless the answer is memorized).

A less obvious problem with this architecture is that the model is so big that instead of forming a coherent, hierarchical world model, it can just cram a bunch of training information into the model, outputting right answers by regurgitating memorized patterns. They do form generalizations and abstractions, but not to the extent humans do. We humans can’t just memorize the entire internet, or all of our sense data. Instead, we have to form a deep understanding of the world by boiling reality down to its most important pieces, and throwing the rest away. A very wide LLM doesn’t have to do that as much.

The rules of logic are implemented once in the human brain. Instead, LLMs might have hundreds of half-representations of the rules of logic throughtout their gargantuan minds. LLMs might have the implicit notion that correlation is not equal to causation inside whatever corner of its mind handles psychology research, then have a different representation of this same concept of correlation not being equal to causation in the part of its mind responsible for A/B testing software. It doesn’t have to learn the fully general concept of correlation vs. causation. This lack of centralization makes forming unique connections between disperate concepts tough, because the links between them that should be there aren’t. With LLMs, we were hoping to get one super-genius, but we ended up with 10,000 morons stapled together.

LLMs Are Also Too Linear

A further problem with LLMs is the lack of recurrence. It’s a linear network. Information goes through one layer, then the next, then the next. This makes exploring deep combinatorial spaces hard. When humans do deep thinking, our unconscious minds have to explore many paths through idea space, perhaps many in parallel. We have to be able to realize a path isn’t bearing fruit and backtrack, trying different paths.

LLMs, on the other hand, are feed-forward networks. Once an LLM decides on a path, it’s committed. It can’t go back to the previous layer. We run the entire model once to generate a token. Then, when it outputs a token, that token is locked in, and the whole model runs again to generate the subsequent token, with its intermediate states (“working memory”) completely wiped. This is not a good architecture for deep thinking.

Chain of thought is a hack that helps to add some artificial reflexivity to an otherwise non-recurrent model. Instead of predicting the next token of the answer, models predict the next token of a sequence of reasoning steps to get the answer. This makes it easier for the model to say, “actually no, that’s not right, let me try a different approach,” if it’s current committed path is a bad one. (I have a pet theory that this is what leads LLMs to overuse em dashes. They’re a good way for one run of an LLM that disagrees with the previous run to pivot away from what the previous run was trying to say — or not!)

I’m sure we’ve all seen LLMs repeatedly change approaches as they flounder on a difficult problem. Chain of thought is better than nothing, but it’s a far cry from how humans reason.

Imagine you had to solve a deep problem, but you were forced to pause your thinking every ten seconds. After every ten seconds of thinking, you had to write down one word and then have your memory of the problem wiped, conscious and unconscious. At the beginning of the next ten seconds, all you’d have access to is the words you’d written so far. Every deep exploration into combinatorial space would be cut short and have to begin anew. All your implicit memory about which paths seemed promising and which went nowhere would be wiped, as would any mental abstractions you’d formed along the way. If your mind operated like this, solving deep problems would be a nightmare. Encoding all your unconscious, parallel thinking into a single word is a hopeless endeavor. But this is how LLMs work!

Some people have suggested allowing LLMs to store a larger amount of non-linguistic information between runs (“neuralese recurrence”). The authors of AI 2027 predict this will happen in March 2027, by the way. This would help, but unless your “neuralese” memory is approximately the size of the entire LLM’s intermediate state, information is still being lost each run. Better to simply allow the model to have a persistent state across runs, though I’m not sure it’s right to call it an LLM anymore at that point. Improving the ability of LLMs to do deep thinking is not a simple matter of scaling up or tweaking the architecture, or more out-of-model hacks.

If I’m right about the architecture of LLMs being ill-suited to deep thinking, then it won’t be 2 years before we have superintelligent AI. It won’t happen at all until we switch architectures. And that won’t be simple. Recurrent models are hard to train and more expensive to run because they’re hard to parallelize. (And from what I understand, recurrent models themselves currently have similar information-bottleneck issues that would need to be addressed, perhaps along with new hardware that doesn’t separate compute from memory.)

Maybe I’m wrong. I hope not! Either way, it doesn’t change what we should do about AI safety at all. Finding out that the bad guy is going to shoot you in the head in 15 minutes instead of 5 doesn’t change your behaviour much. AI is eventually going to be a major problem, even if it won’t be in 2027.

What’s wrong with AI 2027

In this blog post I mostly wanted to share my own predictions and my reasons behind them, not talk about the AI 2027 predictions. But I wanted to include a section discussing some of the problems I have with them, because I think the problems with AI 2027 speak to common problems people have when reasoning about and predicting the future — problems that are actually related to the two kinds of thinking I outlined above.

The AI 2027 predictions⁶ are based on a few different key forecasts. I think their forecasting of compute increases for AI companies are basically reasonable. Where I think the forecasting goes wrong is in the Timelines Forecast (how long until we get superhuman coders?) and the Takeoff Forecast (how long from superhuman coders until subsequent milestones like artificial superintelligence?).

I will confess I didn’t read the entire AI 2027 document, since it’s 193 pages of data/theory/evidence, and I don’t have time for that. More than that, I am immediately skeptical of a document that requires 193 pages to support its main conclusions. Frankly, 193 pages is something to apologize for, not brag about. It is very easy to bury sloppy methodology inside a giant document, and I think that basically happened here. If you can’t explain your argument in a few paragraphs or maybe a few pages, you’re probably relying on a giant chain of reasoning, each link of which introduces further uncertainty until the whole thing is worthless. For brevity I’ll only go into detail about the Takeoff Forecast.

The Takeoff Forecast is Based on Guesswork

It was all very fancy and involved some Serious Mathematics and Statistics, so it took me a while to even parse the arguments. But it seems to boil down to this: the authors each guestimate how likely they think something is, then do some math to combine their guestimates. They do this repeatedly for each step of progress, forming a chain of predictions.

Let’s take one of the links in the chain: going from superhuman coders (SC) to superhuman AI researchers (SAR). They operationalize these terms in the document in a specific and robust way.⁷

They estimate it will take 0.3 years to go from SC to SAR. First, they estimate it would take a human coder 3-4 years to do so, then they predict that using SC would speed up AI research and development by 5x. Thereby, they arrive at an estimate of 0.3 years.

Similarly, they say going from SAR to the next step (superintelligent AI researcher, SIAR) would take human coders 19 years, but with the 25x speedup SARs bring, it will only take a few months.

This reasoning is fine. The critical part is the underlying assumptions: Why would it take a human 19 years to get from SAR to SAIR, as opposed to say, 1 month or 1000 years? And why would SAR give a 25x speedup vs. a 2.5x speedup or a 2500x speedup? These numbers are critical to the whole forecast, so let’s find out where they came from.

In the case of SC -> SAR, they break it down into four possibilities:

“The first SC is already an SAR or very close to one.” 15%. In this case, they guess SC -> SAR takes 0 years.
“Training the missing SAR skills isn’t more compute intensive than the SC.” 25%. In this case, they guess SC -> SAR takes 2 years.
“Training the missing SAR skills would be more compute-intensive than the SC absent further algorithmic progress.” 30%. In this case, they guess SC -> SAR takes 5 years.
“Crossing the gap from SC to SAR is a scientific rather than an engineering problem.” 30%. In this case, they guess SC -> SAR takes 5 years.

They do math on these percentages and timelines to get an overal estimate of 15% 0 years, otherwise 4 years. As long as the percentages and years are correct in the above list of possibilities, then this overall estimate will be correct too. The crucial part are those percentages and years estimates. Where do these come from?

As an example, for possibility #3, this is their reasoning:

We’d guess that this is the sort of limitation that would take years to overcome — but not decades; just look at the past decade of progress e.g. from AlphaGo to EfficientZero.” Remember, we are assuming SC is reached in Mar 2027. We think that most possible barriers that would block SAR from being feasible in 2027 would also block SC from being feasible in 2027.

Umm, what happened to all the precise mathematics we had so far? This is just vibes-based guesswork of the same kind I did when I made my own predictions! I thought we were “estimating” and “forecasting”, not just going with gut feelings!

They say there is a 30% chance that going from SC to SAR “would be more compute-intensive than the SC”. Why? Why not 0.001% or 99.999%.

I suspect it’s because when you know practically nothing about a subject that would let you form a high quality prediction, it would feel weird to make such a specific, confident guess. So instead you pick percentages that are roughly equivalent to the number of options you’re choosing from, in the same way someone who doesn’t know how lotteries and probability work might think there’s a 50% chance they’ll win the lottery. The fact the percentages here are so close to 25% (four options = 25% each) should reveal just how low confidence and arbitrary they are.

There are different kinds of 50% guesses. There are high-confidence 50% guesses (flipping a coin), and low-confidence 50% guesses (is God real?) I suspect the guesses they are giving here are in the latter category.

In the case where “crossing the gap from SC to SAR is a scientific rather than an engineering problem”, why should it take 5 years, and not one month or 1000 years? They justify it only with guesswork.

It took us a while to dig into this one link in the chain, but it turns out that under all that fancy math, it’s just intuition-based guesswork.

I Don’t Take These Predictions Seriously

The authors do admit to “wide error margins”, but their distributions are still centered around ~2027, which I think is an absurd estimate. Saying they’re not sure, they’re highly uncertain, giving a probability distribution instead of just a median estimate; none of that changes the fact that they’re predicting doomsday in ~2027.

I am sure that combining the estimates of several forecasters is more likely to be correct than simply taking one of their guesses randomly (duh). But averaging guesses also seems like terribly inaccurate methodology, not unlikely to be off by many orders of magnitude, especially since the guesses are likely correlated and possibly wrong in the same direction. Basically, this methodology does not convert the “guesswork” into “forecasting”. It’s still guesswork.

I felt similarly about all the other links in the chain when I looked into them. The entire chain of reasoning in AI 2027 is a shining tower of statistics and logic based on a foundation of sand. Maybe SC -> SAR takes zero minutes. Maybe it takes centuries or millenia. I do not believe the authors know any better than I do, and I don’t think all of the fancy analysis and methodology adds anything when it’s built on an extremely low-quality, low-confidence guess.

I especially don’t like that it isn’t clear to people reading the AI 2027 predictions that they’re just guesses, not the detailed forecasts they appear to be. (Except perhaps to the bold few like me who decide to crack open the 193 pages of justification and see what it’s based on!).

The Presentation was Misleading

Nothing wrong with guesswork, of course, if it’s all you’ve got! But I would have felt a lot better if the front page of the document had said “AI 2027: Our best guess about what AI progress might look like, formulated by using math to combine our arbitrary intuitions about what might happen.”

But instead it claims to be based on “trend extrapolations, wargames, expert feedback, experience at OpenAI, and previous forecasting successes”, and links to 193 pages of data/theory/evidence. That makes you think when you open up the 193 pages of research, you’ll see specific predictions based on data like, “AI will likely proceed from SC to SAR in 0.3 years because SAR’s architecture requires us to build a neuralize memory adapter for agents, and we believe this will take 0.3 years because blah blah blah…” But there’s nothing like that. It’s all just based on vibes.

They never outright stated it wasn’t based on vibes, of course, and if you dig into the document, that’s what you find out. They openly admit they “are highly uncertain”.

Yet, imagine I told you I was “highly uncertain” about when I was going to die, but estimated I was going to die on Thursday, January 5th, 2073 at three seconds past 4:18 pm. And then I showed you a lognormal graph which peaks sharply in 2073. And then imagine the news reported on my statement as “rigorously researched”, and many very intelligent people held the statement in high regard. You might get the impression I was a lot more certain than I am.

The whole AI 2027 document just seems so fancy and robust. That’s what I don’t like. It gives a much more robust appearance than this blog post, does it not? But is it any better? I claim no. We shall see.

Deep Thinking vs. Shallow Thinking For Making Predictions

I assert if they’d put more effort into the intuition and less into the post-processing and justification of that intuition, they’d have arrived at more accurate estimates. AI 2027 cites Daniel Kokotajlo’s previous 2021 estimates for AI up to 2026. His “lower-effort” predictions from 2021 were quite accurate. Why?

I think it’s because he was doing deep, intuitive thinking when he formed those predictions. Ironically, the higher-effort AI 2027 prediction looks a lot less reasonable to me because it leans too heavily on shallow thinking (logic, reasoning, and extrapolation).

Listening to several of the authors discuss the AI 2027 predictions after they were published leads me to believe they don’t intuitively believe their own estimates. I won’t try to read their minds, but I think it’s worth mentioning: Even if you can’t spot any flaws in an argument, you shouldn’t believe it’s convincing if it hasn’t actually convinced you.⁸

Shallow thinking can be very useful, but it’s a recipe for disaster when broadly predicting the future. Shallow thinking can only take into account a small amount of information at a time, and the present and future are very complex. Once you’re doing your reasoning in consciousness, you’re limited on the amount of information you can process, and it’s easy to form a fragile chain of logic, any piece of which could make the whole chain useless.

The alternative is deep thinking, which will either be amazing or terrible depending simply on how deep and accurate of an intuition you have for the subject. Logic/shallow thinking is then best used as a fact-checking and communication step.

Was AI 2027 a Valuable Exercise?

Many others have said that while they don’t really buy the actual predictions made by AI 2027, they think writing a concrete AI timeline scenario was a valauble exercise, and that it’s good to “stir up fear about AI so that people will get off of their couches and act”⁹

I am undecided.

Having people believe there is a credible doomsday scenario is possibly good for AI alignment, true. More eyeballs on the AI alignment problem is a good thing, perhaps even if you have to outright lie to get it. If all you have to do is handwave, then even better.

But I do worry about what happens in 2028, when everyone realizes none of the doomsday stuff predicted in 2025 actually came true, or even came close. Then the AI alignment project as a whole may risk being taken as seriously as the 2012 apocalypse theory was in 2013. The last thing you want is to be seen as crackpots.

Conclusion

Predicting the future is hard. I may be totally wrong. If it turns out I’m right and AI 2027 is wrong, I think the takeaway will be that it’s a good idea to be skeptical of fancy, academic reasoning.

Science doesn’t work because of probability distributions and eigenvalues, it works because you’re going out and gathering evidence to find out what’s true about reality. All the statistics mumbo-jumbo is actually in the way of finding out the truth, and you should do as little of it as actually necessary to get value out of your data. In this case, there is no data.¹⁰ A fact which is made unclear by the elaborate, official-seeming presentation and compelling fictional scenario.

AI 2027 predictions by Daniel Kikotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, Romeo Dean. Published in April 2025. ↩
Actually, I think LLMs are already plateauing, but focus on out-of-model (agency) or reasoning progress has covered this up by giving AI extra capabilities without substantially improving the model’s actual within-model intelligence. ↩
Scott Aaronson points out that if P=NP, the “world would be a profoundly different place than we usually assume it to be. There would be no special value in “creative leaps,” no fundamental gap between solving a problem and recognizing the solution once it’s found. Everyone who could appreciate a symphony would be Mozart; everyone who could follow a step-by-step argument would be Gauss; everyone who could recognize a good investment strategy would be Warren Buffett.” This is basically the distinction I am referring to here. ↩
Therapy would be one example of conversations that involve more deep thinking. Long pauses are an indication lots of deep thinking is happening. ↩
This is not quite the distinction between System 1 and System 2 thinking, from what I understand. Shallow thinking encompasses all System 2 (consicous) thinking and some System 1 thinking, and deep thinking is just the kind of System 1 thinking you do when focusing. The point is that shallow thinking is algorithmically/computationally less demanding. I don’t actually know if shallow and deep thinking are fundamentally different, but it doesn’t matter for this discussion. It only matters that they’re practically different. ↩
By the way, Daniel Kokotajlo has adjusted his estimates slightly from 2027 to 2028 since publishing in response to some criticism. I am responding to the original estimates, though the original estimates had large error margins anyway, and I don’t think shifting from 2027 to 2028 meaningfully changes any of my criticisms. ↩
SC: An AI system that can do the job of the best human coder on tasks involved in AI research but faster, and cheaply enough to run lots of copies.” Specifically: “An AI system for which the company could run with 5% of their compute budget 30x as many agents as they have human researchers, each which is on average accomplishing coding tasks involved in AI research… at 30x the speed… of the company’s top coder.” SAR: “An AI system that can do the job of the best human AI researcher but faster, and cheaply enough to run lots of copies…” Specifically: “An AI system that can do the job of the best human AI researcher but 30x faster and with 30x more agents, as defined above in the superhuman coder milestone…” I think these definitions are basically fine even though they assume coders are basically fungible, only differing in the speed it takes to accomplish a task. ↩
For an example of someone’s intuition conflicting with their conscious analysis, read my new short story Suloki and the Magic Stones ↩
https://garymarcus.substack.com/p/the-ai-2027-scenario-how-realistic ↩
Or at least, no good data, at least not for the Timeline/Takeoff sections. ↩

Posts tagged "ai"