Artificial intelligence continues to grow more powerful, but a new study suggests that even the most advanced systems still fail at basic problem-solving tasks, especially those that require human-like reasoning. The findings highlight key differences in how humans and AI think, showing that while AI can follow patterns, it often misses the deeper understanding humans naturally apply to unfamiliar problems.
Researchers say this shortfall could have real-world consequences as AI is increasingly used in sensitive areas like law, healthcare, and education.
The findings were published in February in the Transactions on Machine Learning Research journal. The research focused on how well large language models (LLMs)—advanced computer programs trained on large amounts of text—could solve analogy-based puzzles. These puzzles test the ability to spot patterns and apply them in new situations.
GPT-4 struggles with complex tasks
Humans performed well across all tests. However, the AI models, including the well-known GPT-4, struggled as the tasks became more complex. Their performance dropped significantly when the puzzles required deeper understanding rather than pattern matching.
“They’re good at identifying and matching patterns, but not at generalizing from those patterns,” said Martha Lewis, assistant professor of neurosymbolic AI at the University of Amsterdam and a study co-author. But they struggle to go beyond that. They can’t easily generalize or understand the rules behind those patterns.
Test based on letter-string analogies
One part of the test used letter-string analogies. “Letter string analogies have the form of ‘if abcd goes to abce, what does ijkl go to?’ Most humans will answer ‘ijkm’, and [AI] tends to give this response too,” Lewis said.
“But another problem might be ‘if abbcd goes to abcd, what does ijkkl go to? Humans will tend to answer ‘ijkl’ – the pattern is to remove the repeated element. But GPT-4 tends to get problems [like these] wrong.”
The study also revealed that AI responses changed depending on the order in which questions were asked—a sign that these models are easily influenced by how tests are structured.
Instead of solving problems directly, AI systems often paraphrase or repeat content and phrases in new ways without addressing the core challenge.
AI lacks “zero-shot” learning
Researchers noted that AI lacks “zero-shot” learning—the ability to solve new problems without prior examples. Humans do this regularly. AI systems, in contrast, depend heavily on the size and style of their training data.
This difference has enormous implications. AI is currently used in the legal field for tasks like reviewing past cases and making sentencing suggestions. If the system cannot recognize how an earlier legal case applies to a slightly different new case, the result could be flawed legal advice or unfair outcomes.
“It’s less about what’s in the data, and more about how data is used,” Lewis said. AI systems should be evaluated not only for their accuracy but also for their ability to handle changes and unfamiliar situations. As AI becomes part of everyday decision-making, experts warn that its inability to reason like humans must not be overlooked.
See all the latest news from Greece and the world at Greekreporter.com. Contact our newsroom to report an update or send your story, photos and videos. Follow GR on Google News and subscribe here to our daily email!
