Impossible languages

Language models [LLMs], like humans, prefer to learn some linguistic patterns over others. 

Their preferences bear some resemblance to human preferences, but they’re not necessarily identical, and it’s still possible that aspects of Chomsky’s theories play a role in how humans learn. 

Human brains and neural networks are each so complicated that understanding how they differ —especially when it comes to a task as subtle as language learning —can seem hopeless. 

The paper title “Mission: Impossible Language Models” is fitting in more than one way.

But like action heroes, researchers have a habit of accepting seemingly impossible missions and finding creative ways to make progress. Kallini and her co-authors pinpointed a simple principle called “information locality” that explains why their models found some of the impossible languages harder than others. That principle might also be relevant to human language acquisition. 

Their results have already prompted several concrete proposals for follow-up studies.

“That’s what I really like about the paper,” said Ryan Nefdt, a philosopher of cognitive science at the University of Cape Town in South Africa. “It opens up so many different avenues and questions.”