Chapter 10. Language models: Generating text
In Chapters 5 and 8, we used computing power to analyse text. First, in Chapter 5, we calculated some statistics, such as the number of words and reading complexity, and then, in Chapter 8, we looked at the content of the text. In this chapter, we will go a step further: we will generate text – using the computer to actually produce some new writing.
Text generation is used in various different programs. One example you may be familiar with is "predictive text" on your mobile phone. When you type "hope you have a good" in a chat app, then it often suggests words such as "day" as the next word, saving you to type it in yourself.
Another, more interesting example is found in large language models (LLMs). These models are used as the basis of AI chat systems, and they are very good at producing English language. You can, for example, ask them to write a short story, and they will do so amazingly well.
In this chapter’s project, we will write a system that can generate text. We will not nearly reach the abilities of modern LLMs, but we will implement a simple technique that is the basis of LLMs. This algorithm is able to produce words that look deceptively like meaningful English sentences, and this will give us a basic understanding of the internal workings of LLMs.
10.1. Bigrams
So how can we predict what the next word in a sentence may be? The obvious idea is to look at the preceding words, and then try to figure out what next word might fit to continue a sentence.
At its most basic, we can look at only the one word immediately before the next one. This will not be terribly good at producing sensible text, but it’s a start. So let us try to do that first.
How can we know what word might work to follow another word? Again, the basic idea here is simple: We look at a good amount of existing text, and look what words follow other words there.
Let us investigate an example by looking at an English sentence:
I read the news today, oh, boy.
In this sentence, we can see that the word "I" can be followed by the word "read", the word "read" can be followed by the word "the", and so on.
We can now write down all the pairs of words that we can see may follow each other:
(I, read) (read, the) (the, news) (news, today) (today, oh) (oh, boy)
These pairs of consecutive words from a text are called bigrams. The list of pairs above are all the bigrams from our sentence.
A bigram is a pair of two adjacent elements from a list, such as two consecutive words in a text.
Now, doing this with just a short sentence is not very useful, but doing the same with a very large body of text – say, a whole library – gives us very interesting information. It will tell us all the words that may follow a given other word, and it will also hold information about the likelihood of one word appearing after another. Pairs of words that appear frequently together will appear more often in our list of bigrams.
To experiment with this, let us write a program that can produce a list of bigrams from a text.
Exercise 10.1 Open the starter project text-generation-v1 from the Chapter10 folder of the book projects. You will see that it contains just one helper method to read in a text file (read_file), and a sample text. In this project, create a new function called make_bigrams that takes one parameter (the text to use for the bigrams).
10.1.1. The zip function
Python has a standard function called zip, which can combine elements from two lists into a list of pairs. It will pair the first elements from each list, and the second elements, and the third elements, and so on. Let us look at this with an example.
Figure 10.1 shows two lists of strings. We can take these two lists and merge them with the zip function:
The zip function will pair up the elements of the two lists as shown in Figure 10.2. A pair is simply another name for a tuple of length 2, so what the zip function actually creates is a sequence of tuples with two elements each.
zip functionThe zip function does not return a collection itself, but a special object called an iterator. This object represents a sequence of elements that can then be turned into a collection of our choice. We can turn it into a list by writing:
We could also have chosen to turn it into another type of collection. The resulting list in this case is shown in Figure 10.3.
The zip function merges two lists into a single list of tuples, pairing the elements of each list.
Exercise 10.2 Write some test code that creates two lists and then merges them into one list of pairs, as discussed above. Print out your resulting list.
Exercise 10.3 What happens when the two original lists are of different length? Try this out and describe your observation.
So the question now is: How can we use the zip function to create our bigrams from our text?
The solution lies in the idea to take the list of words in the text, and then to match this with a list that is shifted by one word. We create this shift by just leaving out the first word in the second copy of the list. (Remember you can create a copy of a list without the first element by using the slice list[1:].) This is illustrated in Figure 10.4.
If we merge these lists as we did previously, we will get our list of bigrams.
Exercise 10.4 In your program, implement the make_bigrams function. This function should take a text (a single string) as a parameter, and it should return a list of bigrams of that text.
Exercise 10.5 Test your function with a single sentence. Check whether the output is what you expected.
Exercise 10.6 Test your function with a long text, such as a whole book. A statement that reads in a book is provided in your starter project. You can enable it for this test.
Exercise 10.7 After your tests, you can remove the printing of the list of bigrams, but keep the list stored in a variable for further use.
The text that we use to create our bigrams is the training text. We can use different texts to train our system, leading to different output.
10.2. Predicting the next word
Now that we can generate a list of bigrams for any given text, we can use this to predict the next word. The idea is now straightforward: First, we look at the previous word. From the list of bigrams, we select just those where the first word of the bigram matches our previous word. For example, if our previous word was "It", then the list of matching bigrams might start with
("It", "sets")
("It", "is")
("It", "seems")
("It", "is")
("It", "is")
("It", "has")
("It", "was")
("It", "is")
...
From this, we can make a list with all the second words, and then randomly select a word from that list. Since words that more frequently follow our previous word occur more often in that list, they will have a higher change of being selected as our next word. In other words: the frequency of a word appearing as the second word determines its probability to be selected. (For predictive text, we would select the most frequent word. For creative writing, we will randomly select one of the list.)
Exercise 10.8 Make a new function with two parameters, called generate_next_word (prev_word, bigrams). The first parameter will receive the previous word, and the second the list of bigrams from our training text.
Exercise 10.9 In this function, create a new empty list called following, which will hold all the words that can follow the previous word (the second words from the matching bigrams). Then write a for-loop that iterates over the list of bigrams. Remember that the list contains pairs, so the loop should have two loop variables:
Inside this loop, check whether the first word is the same as our previous word. If it is, add the second word of the bigram to the following list. For a first test, print the 'following' list. Test it with several words as previous words. Does it look plausible?
Exercise 10.10 Extend your code so that your generate_next_word function randomly selects and returns a single word from the following list. Your project already contains an import statement for the random library. This library includes a function called choice, which takes a collection as a parameter and returns one randomly selected element. Thus, you can return a random word from your following list using this statement:
Exercise 10.11 It is possible that the following list is empty, even after searching all the bigrams. In that case, the choice function will fail with an error: it cannot choose an element if there are no elements.
Extend you code so that it checks for this case, and returns any random second word from your bigrams if the following list is empty.
Exercise 10.12 Test your function first by calling it once with a given word and printing out the generated next word. Test multiple times with the same test word. Does it always return the same next word, or different ones? Explain what you observe.
Exercise 10.13 Put your call to the generate_next_word function in a loop, so that it generates 50 words. At first, use the word "It" as the start word, and make sure that you use the generated word as the previous word in the next loop iteration. Print out each word as you run through the loop.
Exercise 10.14 Initially, your words may be printed on separate lines, one word per line. This is because the print function automatically ends each output with a line break. You can change this by specifying what should be printed at the end of the output by using the end parameter:
In this example, we have specified that a space should be printed after each word instead of a line break, so all words will appear in the same line. Change your code to do the same.
The choice(list) function from the random library randomly selects an element from the given list. Import random to use this function.
A copy of the project with all this functionality implemented is in the book projects as text-generation-v2. Feel free to compare your own code to the one in that version.
So what have we achieved so far?
We can now read in a training text that generates a list of bigrams. The bigram-list is our "model" – this is what is used to generate text later on. We "train our model" by reading in text. The kind of text documents we train our model on determines the kind of language our system produces. If you train your model using a different text, it will produce different language. (Try it out!)
Of course, we could read in not just a single book, but a whole stack of books. The more text we use to train our model, the closer our system will follow normal English speech patterns.
When we look at the output (for example, in Figure 10.5), we can see that the result so far is somewhat entertaining, but not very impressive. Very superficially, it looks like English, but there is no recognisable grammar, and this output is not fooling anyone. We will improve this in the next section.
Another observation worth pointing out: This time, we have not stripped out punctuation when we split the text into words. As a result, punctuation is still included in the bigrams, and thus in the output. The words "it" and "it," are just considered different words, and handled separately, each with their own following words. This makes the resulting text look more natural.
10.3. Improving the model
The next idea to improve our model is again quite simple: instead of looking at only one previous word to generate the next one, we look at two previous words. In our model, we do not use bigrams, but instead we use trigrams – tuples of length 3 with all the three-word sequences form our original text.
Generating the trigrams is quite straightforward, since the built-in zip function can also merge together three lists (or any number of lists). In our program, generating the trigrams looks like this:
Change your entire program to work with trigrams now, using the following list of exercises.
Exercise 10.15 Change your make_bigrams function to make_trigrams. Adapt the parameter list, and make sure that you generate and return a list of trigrams. (Make sure to update the function comment.)
Exercise 10.16 Change your generate_next_word function so that it generates the next word from two previous words and your list of trigrams. Return the last word from a trigram where the two first words match the previous words.
Exercise 10.17 Adapt your test code in your "My code" section to set two start words, and then generate the next 50 words from there. It works well if you use fairly generic start words, such as "It" "was".
If you have done the exercises so far – how is your output now?
You will see that this simple improvement of our "model" – going from bigrams to trigrams – significantly improves the quality of the text it produces. Now the output looks almost like grammatically correct English! That is quite astonishing for such a simple algorithm.
Of course, the sentences are still gibberish – they don’t mean anything, and you cannot read any sense into them. We discuss below how real LLMs manage to do better than this.
But first of all, let us try to improve our training data.
10.4. Improving the data
There is a very simple change that we can make to improve the quality of our system: use more data.
In real LLMs, the models are trained on millions or billions of documents. Many of them are fed any text you can access on the Internet, and much more besides.
We will not use that amount of text, but we have two files available that contain just a bit more than a single book. The Strype file system folder includes a file called /books/fairy-tales.txt, which holds the full text of 65 fairy tales by the Brothers Grimm, and another file called /books/sherlock-holmes.txt, which has the text of 27 Sherlock Holmes stories (three books and 24 short stories). Try your program with these texts as the training data.
Exercise 10.18 Change the file you read in to build your trigrams. Use the file /books/fairy-tales.txt. Run your program. How is the output changed from before?
Exercise 10.19 Do the same again with the file /books/sherlock-holmes.txt as the training data. How does the output differ now?
A version of the project with these changes is in the book projects as text-generation-v3.
If you did these experiments yourself, you will have noticed two things: Firstly, using longer texts to "train our model" improves the quality of the output. The more text we feed in, the more closely the output resembles actual language. We have not, in fact, used very much data. Even the "sherlock-holmes" file contains only about 330,000 words. This is miniscule in the context of large language models, but it already starts to deliver quite surprising results.
And secondly, the content of the text we feed into the model determines the language and style of the text that is produced.
These two observations lead to two very significant issues: Firstly, with the aim to be able to produce any kind of output, large AI companies have been scanning all writing they could get access to. This includes, for example, millions of published books. For the performance of the system, this is great: you can tell your AI system to write a short story in the style of Stephen King, and it will very competently do this. It has read in all existing Stephen King stories, and so it can copy the style amazingly well.
The problem is that these stories are under copyright, the AI companies have not legally licensed them, and Stephen King has not given his permission.
So, is this legal?
Currently, this is an undecided questions. The AI companies argue that it falls under the "Fair use" clause in copyright law, since the source works are not reproduced directly. Many authors argue that this is copyright theft, since their own original hard work is being used – and being commercially exploited for profit – by other companies without their permission.
The second problem is the sheer amount of data involved, and the amount of processing power and electricity required to process it. Reading and transforming the "sherlock-holmes" file is trivial, and a standard laptop can easily do it. But processing the amount of data involved in real LLMs – more text than the content of all the libraries in the world combined – requires so much processing power that the environmental impact is immense. Power is not only needed to train the model (this is only done infrequently), but for every single question that an AI system is asked to answer. Even the most trivial query requires huge processing power and consumes large amounts of energy.
To cope with this, AI companies are building huge data centres in many areas of the world, full of processors which only serve AI queries from users. A single data centre can consume as much energy as a city of a million people. For the environment, this is a disaster; it creates significant new problems for the goal of reducing the environmental damage which we cause.
Exercise 10.20 Research "fair use" in copyright law. What is it? What does it allow, and what does it not allow?
Exercise 10.21 Do you think AI companies should be allowed to use copyrighted books without the author’s permission? Name some arguments for both sides of this argument. What about films or TV programs? What about using the likeness of a famous person?
Exercise 10.22 Try to find out where the nearest data centre is to the place where you live. Try to find some information about the amount of energy a data centre consumes. If you can, try to find this information for the data centre nearest to you.
Exercise 10.23 Try training the system with some of your own writing. Find as many pieces of text you have written, copy them together into a single plain text document (there is no need to maintain formatting), and then see what your system produces from your own words. (Again, refer to the Strype tip in Section 5.5 for information how to access your own files.)
10.5. Large Language Models
We have seen that our model does not come close to the abilities of current AI systems, and there are many reasons for this.
Firstly, there is room to improve our current system. One idea is again simple: use longer chains of words. We have used bigrams and trigrams. More generic systems use n-grams – sequences of larger and variable length.
An n-gram is a sequence of n consecutive tokens from a list, such as n words from a text.
Using more contextual words does indeed improve the performance of these systems. And the project that we investigated in this chapter does indeed give a glimpse into the history of text processing.
Until the mid-2010s, n-gram-based systems were considered the state of the art in text processing. They were successfully used in simple auto-complete systems and speech recognition.
After that, these systems evolved significantly, and modern LLMs have a number of important differences that make them more effective:
-
While n-grams use a fixed size window of the last n-1 words, LLMs use flexible and much larger windows – often thousands or millions of tokens. Doing this, they can make long-range connections, important for staying on a topic, making arguments, and so on.
-
N-gram systems typically work with lists of words. In LLMs, each word is represented not by its actual letters, but by a large, multidimensional vector that encodes many aspects of the word. This includes semantic and contextual information, which helps to make many more connections. For example, it may know that "cat" and "dog" are somewhat related words, even though their letters are entirely different. This helps the model to generalise and transfer knowledge between related concepts.
-
LLMs are trained not by simply counting or storing the words and probabilities, but using neural networks, which expose much deeper connections between words. They also implicitly learn about grammatical structure and language patterns.
The resulting systems are known as generative pre-trained transformers (GPT). The full details of these are more complex than what we can discuss here. But ultimately, they are generalisations of the simple system we have built here. Or in other words: An n-gram model is a special, extremely limited case of what a GPT transformer can represent.
N-gram systems, however, are not obsolete. They continue to be useful in predictive text, spell checking, and in document retrieval systems.
While our project here does not allow us to understand fully how LLMs work, it does give an insight into the core of the idea that underlies LLM-based AI systems. Perhaps most importantly: It is still the case that LLMs have no real understanding of the content and meaning of the text they produce, and are ultimately based on clever manipulation and prediction of language. These systems may state an entirely invented "fact" with the same confidence as an obvious truth.
AI companies are working hard to improve the reliability of LLMs. This tension, however, between our desire of trust and these systems, which ultimately just manipulate language, will remain at the core of the issue of reliability in AI systems for some time.
-
A bigram is a pair of two adjacent elements from a list, such as two consecutive words in a text.
-
The
zipfunction merges two lists into a single list of tuples, pairing the elements of each list. -
The
choice(list)function from the random library randomly selects an element from the given list. Importrandomto use this function. -
An n-gram is a sequence of n consecutive tokens from a list, such as n words from a text.