Logic puzzles AI

[Nouha] Dziri’s team showed that LLMs that have only been trained to predict the next word in a sequence —which is most of them —are fundamentally limited in their ability to solve compositional reasoning tasks. 

Other researchers have shown that transformers, the neural network architecture used by most LLMs, have hard mathematical bounds when it comes to solving such problems. 

Scientists have had some successes pushing transformers past these limits, but those increasingly look like short-term fixes. 

If so, it means there are fundamental computational caps on the abilities of these forms of artificial intelligence —which may mean it’s time to consider other approaches.

“The work is really motivated to help the community make this decision about whether transformers are really the architecture we want to embrace for universal learning,” said Andrew Wilson, a machine learning expert at New York University who was not involved with this study.