Skip to content
Tekoälli

Try it

Watch how a language model actually picks its words

Below is a working language model, a small n-gram model to be precise. It solves the same basic task as the large ones, predicting a likely continuation, but by a different technique: counting which words have followed which before. Switch the training set and you can see for yourself how much the amount of data decides.

Training data

Mary Shelley's novel from 1818, about 75,000 words. The model knows 7,151 words and 36,416 word pairs.

Pick a word and the machine will show which words most likely follow it.

This is an n-gram model: it remembers the last two words, and if it has not seen that particular pair it drops back to one word of memory. A large language model solves the same task by a different technique, a neural network, and takes thousands of words into account at once.

What is actually happening here

  1. 01

    The machine counted word pairs

    It went through the text and noted which word followed which, and how many times. That is all it knows. It does not know what any of the words mean.

  2. 02

    You choose, the machine counts

    The machine looks at the last two words and finds their continuations in its table. If it has not seen that particular pair, it settles for one word. The percentage says how often the continuation appeared in the data.

  3. 03

    Switch the data and see the difference

    The same algorithm, two different training sets. With the large one the sentence sounds like English. With the small one it falls apart and starts reciting the source text from memory. The algorithm did not change, only the amount of data.

Read next