Module 01

Generative AI: promises and limits

How ChatGPT and its peers work, what "generating" text means, and why hallucinations are not a bug.

8 min3 resources

In November 2022, OpenAI released ChatGPT and the world was never the same — or so the dominant narrative tells us. In reality, large language models (LLMs) had existed for years, and the underlying techniques (transformers, introduced by Google in 2017) were not new. What was new was accessibility: for the first time, anyone could converse with a language model through a simple interface.

But how does an LLM actually work? In simplified terms: it is a model that, given a sequence of words, predicts the most probable next word. It does this because it has been trained on enormous amounts of text — books, articles, websites, forums, source code — and has "internalised" the statistical patterns of human language. When it generates a response, it does not "reason": it calculates, word by word, the most probable continuation of the conversation.

This mechanism produces results that seem magical — coherent texts, fluent translations, effective summaries — but it has intrinsic limitations. The most famous are "hallucinations": the model produces false but plausible information with the same confidence as true facts. This is not a bug: it is a direct consequence of how it works. The model does not distinguish between true and false — it produces what is statistically probable in context.

Emily Bender and Timnit Gebru coined the expression "stochastic parrots" to describe these models. Like a parrot that repeats phrases without understanding them, an LLM produces text that seems meaningful without having any understanding of meaning. This does not mean it is not useful — it means we must use it consciously, knowing what it can and cannot do.

Generative AI is not limited to text: it produces images (DALL-E, Midjourney, Stable Diffusion), music, video, code. In all these domains, the principle is the same: a model trained on enormous amounts of data generates new output that is statistically coherent with that data. And in all these domains, the same questions emerge: who owns the training data? Who benefits from the output? Who is harmed?

An often-overlooked aspect is the relationship between generative AI and creative work. These models are trained on the work of millions of artists, writers and programmers — often without their consent and without compensation. When an AI generates an image "in the style of" an artist, it is extracting value from that creative work. The question is not only technical, it is political: who has the right to use and monetise other people's work?

Key takeaways

  • LLMs predict the most probable next word, they do not "reason"
  • Hallucinations are not a bug: they are a consequence of the model's statistical operation
  • "Stochastic parrots": models produce plausible text without understanding meaning
  • Generative AI is trained on the creative work of millions of people, often without consent

Reflection prompt

Try asking an AI chatbot for information you know well (e.g. the history of your neighbourhood, a specific fact from your field of work). Is the result accurate? If it contains errors, how plausible are they? What happens if someone unfamiliar with the topic reads that answer?

Further reading