| In 1948, the legendary computer scientist Claude Shannon proposed a radical new way to write a sentence: Pull a book off your shelf, choose a random word on a random page, and repeat. That doesn't sound very promising, and sure enough, Shannon's experiments produced strings of gibberish. But something curious happened when he tweaked his procedure slightly to account for how often different words appear together. The sentences still made no sense, but they looked much more like ordinary English text. Shannon's observation was the first hint that simply mimicking statistical patterns in text, with no regard for meaning, could yield surprisingly fluent prose. It also marked the beginning of a fruitful but often fraught relationship between the study of language, called linguistics, and the young field of computer science.
Today, more sophisticated versions of Shannon's statistical methods are at the heart of the large language models, or LLMs, that power chatbots such as ChatGPT. But the road from Shannon to modern LLMs was not straightforward. For decades, linguistics research was dominated by a rival theoretical framework called generative linguistics, devised by the pioneering linguist Noam Chomsky, which rejected statistical methods in favor of clear-cut rules.
Generative linguistics (which is unrelated to today's "generative AI" systems) had its own connections to computer science. Chomsky's early work was a major influence on the theory of programming languages. But there was a stark difference between the generative and statistical approaches to linguistics. Chomsky and his followers argued that statistical methods could never reveal the fundamental laws that govern how language works. They wouldn't be able to differentiate between ungrammatical sentences and ones that are grammatically sound but nonsensical, like "Colorless green ideas sleep furiously."
The statistical approach to linguistics grew more popular in the 1990s and 2000s, as powerful computers and techniques borrowed from machine learning enabled researchers to trawl through vast troves of text in search of subtle patterns. But even researchers who had embraced statistical methods weren't prepared for the advent of LLMs. These new models were so good at certain language processing tasks that they rendered many competing methods obsolete overnight. Earlier this year, John Pavlus interviewed 19 researchers for a vivid oral history of how LLMs impacted computational linguistics. But where some researchers saw a crisis, others saw new opportunities. They've used LLMs to explore old questions about language learning and even ones about the origin of meaning itself.
What's New and Noteworthy
Already in 2019, language models fared strikingly well on reading comprehension tests, sometimes doing even better than humans. But how reliable were those tests? Pavlus also reported on that early debate among computational linguists over how to evaluate understanding in language models. Since then, newer and more powerful models have aced increasingly sophisticated tests. Steve Nadis reported on one recent study, which found that a state-of-the-art LLM could correctly parse the grammatical structure of complex English sentences and infer the rules of made-up languages. Subtle questions about evaluation remain relevant today, but it's increasingly difficult to devise tests that can stump the best LLMs.
The latest studies of LLMs' linguistic abilities challenge Chomsky's claims that statistical methods alone can never grasp the underlying rules of language. Other researchers have tested the predictions of generative linguistics by focusing on the learning process itself. Chomsky has argued that humans are hard-wired to learn certain kinds of linguistic rules over others, while LLMs can't distinguish between real languages based on these favored rules and "impossible languages" based on other rules. But in January, I wrote about recent experiments showing that LLMs have more difficulty learning impossible languages than ordinary English. Those results suggested an intriguing new way to study human language learning indirectly — by tailoring the design of LLMs to make them worse at tasks that humans also struggle with.
The rise of powerful language models has also enabled researchers to explore more philosophical questions that once seemed beyond the reach of empirical tests. One of these questions, called the symbol grounding problem, asks whether contact with the outside world is necessary to give words meaning. Last year, Pavlus spoke to the language model researcher Ellie Pavlick about how LLMs, which learn from text alone, can illuminate the grounding problem from a new angle. And in April, Quanta staff writer Joseph Howlett spoke to the mathematician Tai-Danae Bradley about how a branch of math called category theory might help explain why LLMs can get so good at generating fluent text without any external grounding. | 
