It’s a real sign of our times that so many can not differentiate between a plagiarism fueled talking machine and a thinking machine.
Well, in fairness, if you ask Chatgpt a question it says “…thinking…”
You can see how confusion might occur.
Most of them are just person shaped.
Might be because AI isn’t cognitive or actually intelligent. I imagine a washing machine wouldn’t do well either.
I know. We should totally invoke the 25th amendment before- wait. It said AI. Oh, my bad.
sanitation — ‘classic psychology test’ covers a lot of ground. If this is Stroop or dual-task paradigms, the near-total collapse actually tracks: those tests were designed to stress automaticity vs. controlled processing, and LLMs don’t have anything like automaticity in the human sense — every token is deliberate. So ‘collapse’ might be the wrong word; it’s more like the architecture was never built for that cognitive mode. There’s a breakdown of which test categories hit which model families hardest if you want to cross-reference which paradigm is doing the most damage here.
Given that the LLMs could follow the short lists of words well but not the longer lists, and that they were processing images, not text, I think it’s more likely that their context just filled up and they forgot the original instructions (or they were assigned a lower weight in the computation).
Thanks for the explanation. I just repost the most popular content from reddit.
These models tested are so old they’re from the era where they couldn’t pass a math test or count letters in words
So… last week then?
I get that you hate AI but there’s no reason to lie about its capabilities.
That’s not lying. There’s nothing linguistic about numerical computation.
You know the “DeepMind and OpenAi models” is the hint that the LLM model is not the one doing the math. The LLM provides a hypothesis and the DeepMind model provides grounding or feedback on whether the hypothesis even makes sense or works.
It is totally irrelevant that the model calls tools to do the math. That is still a success.
It’s relevant to what the parent was saying about LLMs. The success of the LLM in using mathematical tools does not contradict what they were saying. To then accuse them of lying because of a misunderstanding is… bad form.
All of these features are not something the models themselves can do, but are grafted on.
I could easily write a Home Assistant automation pattern matching for nearly every way someone could say “how many Rs are in strawberry”, depluralize a plural letter, and run it against “wc” in a bash terminal.
That doesn’t mean it’s smarter. It’s that I’ve added something specific to it.
MCP and the like is just that too, gluing on functions or the ability to hopefully invoke a function. That’s why so many hilariously mundane ones exist.
At the core, it’s still a large language model: a statistical model of frequency of word and word chunk (token) patterns.
Sometimes one model can invoke another via that tooling but it’s still a grafting on. It isn’t a singular thing or system, but disjointed pieces so completely detached from how brains work.
This isn’t AI hate, it’s reality. I love the field of artificial intelligence and machine learning. It’s cool as hell. But an LLM is fundamentally incapable of being anything more than an LLM with glued on pieces that invoke functionality.
OpenAI saw people mock the inability to count so they wrote a specialized tool to count letters and glued it on.
The world is full of endless edge cases. The inability to simply resolve them without gluing on every single one means it just isn’t doing anything new.
They regularly win olympiad mathematics up from not standing a chance and just created a novel solution to the erdos conjecture, them counting the r’s in strawberry is inconsequential but also something they can do even if you just use the raw api or a local model.
Using computers to search for a counter example to a conjecture isn’t exactly new ground and I suspect they did so with the aide of some harness tweaks like some numerical LSP. Like cool, it pushed the envelope but like what the parent said, they grafted on the ability to do a specific task.
That doesn’t change the fact that llm’s are capable of acing math olympiads. So what if it uses tools? You probably would too. I doubt anybody there did it without a calculator.
Aren’t you the least bit curious what tools they gave the LLM and how the LLM used those tools? It’s like back in math class you are asked to solve a quadratic formula but you forgot how. So you use the calculator to try different numbers and the calculator is telling you if you are getting closer. Sure I got the right answer, but it’s hardly a testament to my math skills.
Afaik that is handled through tool use in modern models (ie they didn’t learn to do maths, they learnt to use a calculator), assuming that’s true and I haven’t missed some advance, their conclusions are likely still relevant
Edit: though the article does seem to discard the chain of thought techniques a little readily, feels like they could come close to fitting the role of executive control, but perhaps that’s just the article lacking detail from the original work.
My high school math teachers would be so disappointed in them.
If I could wire a calculator into my brain I would have cheated on all the maths tests tbf
so? you know if people were cherry picking articles and research without understanding what they are, about anything else what would you think of them?
One positive of AI is that the ownership class is getting a lesson in just how complex, flexible, reliable, and capable “unskilled” workers are. You can watch them realize in real time that a model capable of running a dinner-rush drive-thru would be a trillion dollar quantum leap.
Anthropic sonnet 3.5 is an old mid tier model. GPT 4o is also multiple generations ago.
Newer models handle this much better. Not claiming sentience or anything.
