Creative Python Programming with Strype

Pedagogy is the term for the art of teaching. The authors of this book have many years of experience in teaching programming, and this experience has shaped the approach of this text. This book follows a carefully designed pedagogy based on worked examples, scaffolding, a spiral approach and working from the concrete to the abstract.

Worked examples are used throughout, and all concepts are introduced and practiced via concrete projects. Emphasis is placed not only on the end product – the finished program – but also on the process of developing it.

Many textbooks follow a common pattern: A problem is presented, a solution is shown, and then the constructs used in that solution are explained. This is a terrible approach! It creates the illusion that a good programmer goes in one step from reading the problem description to writing down a fully working solution. This is not how programming works!

A result of this approach often is that learners who struggle with finding the right solution, or with making it work quickly, think they are failing. They react by developing a self-image of "I am not a good programmer" and "programming is just not for me".

The fact that we all work in small steps, try things out, run into problem, backtrack, try something different, and slowly work our way to a solution seems to be one of the best kept secrets in programming!

This is a shame.

In this book, we show not only the product, but the process of programming. With each project, we go through several phases of extension and refinement. Learners experience the process, and can follow along. We use a process of scaffolding to provide meaningful context while keeping learners focussed on the topic at hand, and fading of the scaffold and guidance over time.

At the same time, the examples are selected to be open-ended and extendable, so that learners are encouraged to take each example further in a direction of their own interest. Each chapter includes suggestions to do more.

We also use a spiral approach. When we first encounter a construct, we do not try to explain or understand everything about it. We learn what is necessary at that point, and then refine our understanding later. We come back to the concept in a different context, and then again later still – and in each iteration we deepen our understanding.

This is a more natural and useful approach to understanding programming. Firstly, many programming constructs are circularly dependent on each other, and there is no linear sequence in which everything can be explained only by backwards reference. But more importantly, when books try to explain everything about any given concept at once (think, for example: types) learners cannot usually distinguish what is needed at that moment ("what do I need to know right now?") from what is there only for completeness’s sake. The important is buried in the detail, and learning is made harder.

The spiral approach serves to break this linear form, iterate over concepts several times, and lead to a deeper understanding.

The other obvious difference of this book is the use of the Strype environment.

Strype is a browser-based educational Python development environment designed specifically for novice learners. The use of Strype, and its integration in this book, are absolutely fundamental to our approach.

As we stated above, our firm goal is to make the first experience of programming engaging, fun and creative. One way to do so, for example, is the early use of graphics and animation. Yet animated graphics are much too complicated for novices to use in most Python environments.

Strype is designed to make this approach simple, painless and possible. We can create a graphic in a single line of code, and animate it in just a few lines more. The effect of program statements becomes immediately visible on screen in a way that no "Hello world" program can ever manage, and learners can observe their program execute.

The technology and the pedagogy used in this book were developed in tandem, with environment design strongly guided by our pedagogical principles. The result is a package of software, examples and explanation that is seamlessly integrated, and enables learners to achieve more satisfying results than they might with other systems.

Overall, we strongly believe that this helps the most important aspect of all: motivation. We hope to spark our readers' curiosity, to leave you wanting to read on, to continue to learn, because you have caught the programming bug just like we have.

We hope that you will see not only how programming works, but also why you might be interested in it, and what it can do for you.

Getting started with programming in Strype is easy: Point your web browser to strype.org, click on the "Start coding now" button, and you will see the Strype editor. Here, you can enter, edit and run Strype programs. Figure 1.1 shows a screenshot of the editor.

When you first open Strype, you will see a short program to get you started.

We will start our exploration of graphics programming in Strype with a very simple program that demonstrates some of the basic concepts.

Open the "yellow-fish" project from the "Chapter01" folder in the book projects. Execute the program. You should see a result similar to the one shown in Figure 1.3.

When we compare the two lines we currently have in our program to create actors, we can see that they look different:

The different is not only the additional coordinates that we have specified with our second actor, but also the arrow symbol and name (fish) that we have used in the first line. This is called an assignment, and we will investigate this next.

The purpose of this is to keep a reference to our actor object, so that we can continue to work with it afterwards. When we create an actor in a line on its own (as in our second line above), the actor object is created, it is automatically placed into the world, and that is all that happens. After this statement, we do not have access to the actor anymore, and we cannot directly work with it further.

This is why we often use an assignment to assign the actor to a variable, which allows us to continue working with this actor. Let us investigate what this means.

The next obvious step towards our goal to turn this into a game is to give us control: We would like to control the fish with our keyboard, so that it can become our player character. That’s what we will do next.

Our plan is to use the left and right arrow keys to control the fish: If the left arrow key is pressed, we want the fish to turn left, and if the right arrow key is pressed, it should turn right.

Luckily, Python has a statement to express exactly this: an if-statement.

Modify your program so that the loop looks like this:

In this code, we have added an if-statement into the loop, and using this if-statement, we make the fish turn only if the left arrow key is pressed down.

When you insert a frame for an if-statement into your code, it looks like this:

We can see that this frame has two slots: a text slot for the condition and a frame slot for the statements that should be executed if this condition is true. This frame slot is also called the body of the if-statement.

In our loop, we use a function call as the condition: key_pressed. This function takes as its parameter the name of the key we want to check, and returns true if this key is currently pressed down. If it is true, then it executes its body to make the fish turn.

If the condition is false, then the body is not executed, and the fish does not turn.

All keys on the keyboard have names that we can use with the key_pressed function. The name of the character keys is just their character (so the a-key is called "a", and so on) and other key names include "left", "right", "up" and "down" for the arrow keys, and "enter", "tab" and "backspace" for some of the others.

Using this, we can now add a second if-statement to make the fish turn right when the right arrow key is pressed.

You can find this version of the program in the book projects as yellow-fish-v3.

In this chapter, we have jumped into the start of our first Python program: a simple animated game. So far, we only have the beginning of the game implemented: a moving graphical character that we can control with our keyboard.

We will continue developing this game further in Chapter 3, and if you are really keen to continue with this project, you could jump ahead and continue reading there.

In this book, however, we will first take a moment and look more closely at the language constructs we have encountered so far. We have seen quite a few: function calls, parameters, variables, assignments, objects, methods, if-statements and while statements.

That is a long list!

We have had a first contact with each of these constructs, and we can roughly understand what they do, but there is more to learn about each of them. So we will now – in Chapter 2 – have a closer look at each of these constructs to deepen our understanding, before we get back to completing our yellow-fish game.

We have seen function calls a couple of times now. For example, we have seen the call

and the call

In each case, we call, or invoke, an existing function to get it to do some task for us (setting the background colour, in this case, or setting the pace of the loop.)

In general, a function call always has the format

The function name specifies the name of the function we wish to call, and the parameters provide some more information to execute the call. Both functions above have one parameter, but a function can have more than one parameter, or none at all, indicating that no further information is required. If a function does not have any parameters, then we still write the parentheses. We just write nothing in between. For example

calls a function called "stop" that has no parameters. It is the presence of the round brackets (also known as parentheses) that identifies this statement as a function call.

To practice with function calls, we will now use another project: fireworks.

If you have completed the exercise above, you will have seen that this project includes some code, but does very little when we run it. In fact, you have to look very carefully to notice any effect at all: It sets the background of the graphics world to a dark blue colour (which is not so different from the default black), and does nothing else.

This project lets us create a fireworks display. It offers us methods to place some firework rockets into the world, and then ignite them to set them off. At the moment, nothing much happens because the project does not place any fireworks into the world.

Let us look at the program code that is executed when we run this project. It is the code in the "My code" section of your project:

We can see two method calls and one comment. The method calls, in this order, prepare the project to be ready (including setting of the background colour) and ignite the fireworks. In the middle, we see a comment that tells us where in our code we should place our firework rockets.

A comment always starts with the symbol #, and continues with some text. The comment is ignored by the Python system; it is here only for a human reader to give us a hint. We can insert a comment into our program by adding a comment frame.

We would now like to start creating a fireworks display by placing some rockets into our world, and then igniting them.

If all went well, you should now see a single fireworks rocket being fired into the night sky. The program, as it is now, prepares some internals to get the fireworks ready, places one rocket into the world, and then ignites it.

This is nice enough so far, and we could now go on to make our fireworks more interesting by adding more rockets. But we are running into a fundamental question: How do we actually know what methods are available? What are their names, what do they do, and how do we call them?

The first data type we have encountered above was str, which is short for "string". We have seen that the values for this type are snippets of text, written in quotes: "This is a string".

If you paid attention when you did the exercises above, you may have noticed that the comment for the place_rocket function specified that the type of the last three parameters is float. "float" is short for "floating point number", and it means that the value for this parameter should be a number that may include a decimal point. 3.14 and 0.0001 and 42.0 are all floating point numbers. It is allowed to provide numbers without a decimal point: 128, for example, will just be treated the same way as 128.0.

There is a third data type that is worth mentioning at this point, because we will encounter it very soon: int. "int" is short for "integer", which means a whole number. If a parameter type is int, then we have to supply a number, but this number cannot have a decimal point. Only whole numbers are allowed, such as 1001, -42, or 0.

After doing these exercises, you might like to compare your project to the "fireworks-finished" example from the "Chapter02" folder, which is an implementation of these exercises.

Often, when we work with bits of data, it is not enough to just pass a value to a function. We often want to store a piece of data for use later on, or for doing further computations with it. This is when we need variables.

You have seen the first variable briefly at the start of Chapter 1, when we looked at the default program in the Strype editor. Let us briefly go back to this program and look at it more closely.

You will see that the program consists of just two lines of code:

In this code segment, we can now recognise what the second line means: This is a function call to a function named print with one parameter. The print function is built into Python, and it prints its parameter to the Console (the text output area in the bottom right of the Strype window).

The first line is an assignment statement. We can add an assignment statement by inserting an assignment frame. A new assignment frame has two slots with an arrow symbol in the middle:

On the left side of the assignment, we can write the name of a variable. We can make up this name, and a variable with this name will be created. On the right side we write a value that we want to store in this variable. This can be any type of data that Python knows, including the three types we have seen above: strings, integers and floating point numbers.

In the example above we have stored a string into this variable ("Hello from Strype"), and we have given the variable a name that indicates that it stores a string (myString).

Variable names must not have any spaces in them. If you need to use multiple words (for example, big fish), there are two programming conventions to do this:

In this book, we generally use the first format.

In the previous section, we have seen that we can store values, such as strings or numbers, into variables. Next, let us look at a special kind of data type to store and work with: objects. To investigate this topic, we will use another project called knock-knock.

This project makes use of an Actor object, which we first encountered in Chapter 1. The creation of this Actor object looks like this:

We can see that it looks very similar to a function call: It has a name, followed by a parameter list in parentheses. Because these look so similar, Python uses a convention: function names start with a lowercase character, while object types start with an uppercase character. This bit of Python code creates an object of type Actor, using the parameters provided.

In our code, we can see that we then assign this object to a variable, so that we can use it later:

This shows us that variables can also be used to store objects as values.

Before we go on and examine the rest of the program, let us ask a question: How do we know what object types and what functions exist for us to use?

Looking at our knock-knock program, we can see that it uses objects of type Actor, and functions called set_background and pause. What other objects and functions are available for us, and how do we find out?

When working in Strype, we will often make use of objects and functions from the Strype graphics library. Over time, we will have to become familiar with most of the functions available in it. So how can we see what it offers?

The answer to this lies in the library documentation. This documentation lists all object types and functions, with all their parameters and information how to use them. In Strype, you can access this documentation by selecting "Library documentation…​" from the Strype menu.

Learning to read this documentation will be very useful for our future programming tasks. It enables us to find out what we can do. Luckily, it is not very difficult.

The main points to understand are these:

You will quickly get used to reading this type of documentation. Having it available will help us both understand the remaining parts of the knock-knock program, as well as enable us to find out what else we could do.

We have now seen that functions and classes can be defined in two different places: They can come from a library (such as the set_background function we have used from the strype.graphics library) or they can be defined in our own project, as we have seen in the fireworks example.

In Chapter 1, we have briefly mentioned the difference between functions and methods. They are similar in that they both perform specific actions. Functions, however, perform a stand-alone action (such as set_background("red") or ignite()), while methods are actions that belong to a class and are performed by a specific object. We have seen for example, that we write

to make our fish object move. move is a method of the Actor class, as we have seen in the library documentation. There, methods are listed under the class, while functions are listed on their own at the end of the module.

Looking at the Cat methods, we can see a number of methods starting with the prefix is_, for example is_hungry, is_tired, and so on. These methods return a value of either True or False, and we can use them to introduce conditional actions: make our cat do something only is a given condition is true.

In Chapter 1, we have already seen that we can use if-statements to call a method only only under certain conditions. In this case, we can, for example, make the cat eat only if it is hungry:

Try this yourself.

If-statements can also have an else-case. An else-case is added to the if-statement by moving the frame cursor to the end of the body of the if-statement (the last line within the if-statement), and then pressing the e key. The result is an extended if-statement that looks like this:

The if-statement now has two bodies: one for the if-case, and one for the else-case. The first one is executed when the condition is true, and the second one is executed if the condition is false. Thus, one or the other of the bodies will always be executed, but never both.

You can find a version of the project that does some of these things in the book projects as fatcat-v2.

If you have done all the exercises in this chapter, you will have a good enough understanding of the Python constructs we have used thus far. We are now ready to get back to our game project and continue developing it into something more interesting.

In this chapter, we have investigated and experimented with some of the most important Python constructs. We have seen more detail about reading function definitions and calling those function. We have seen that functions can take parameters (which can include optional parameters), and that the types of the formal and actual parameters must match.

We have also discussed how to use Actor objects, and how to read the library documentation to find out more detail about what we can do with them. The documentation will be important when we work with our projects; it shows us the methods we have available.

And finally, we had a closer look at if-statements to execute specific statements only if a certain condition is true. If-statements will be useful in just about every project we write from now on.

To make this game even a little bit interesting, we need to have a task, a goal, and some danger. For our yellow fish, we have already seen how we can make the fish the player character by letting the player control the movement of the fish. The task will be to eat: We will introduce some shrimp to our game, which the fish can eat. The goal of the game then is to eat all the shrimp on screen.

To make this a bit more interesting, we will also add a shark: the shark likes to eat fish, so we need to stay away from it, otherwise we will be eaten by the shark. The aim of the game then is to manage to eat all shrimp without being first eaten by the shark.

Our first task is to add some food to the game: a shrimp in our case. We can start by adding one single shrimp. This should now be quite easy: We can add a line of code to add the shrimp immediately after the line that adds the fish, with a very similar structure. We just use a different name for the variable, different coordinates and a different image:

Of course, if you have changed your player character and image in this project to a different actor, you can choose a different kind of actor for the food as well. If your player is, for example, a turtle, then the food could be a lettuce. Or if your player character is a spaceship, then the second character could be an astronaut. (In that case, it is not "food", and the player is not "eating" the second character, but rather picking them up from space.) You get the idea: whatever your game scenario is, choose an appropriate image for your second character. Whenever we write "shrimp" in the remainder of this chapter, use your own character instead.

In case you’d like to stick with our fish/shrimp theme, you can find a shrimp image in the Chapter03/images folder for you to copy and use.

Let’s say we’d like to have 15 shrimp in our game. We could now go ahead and duplicate the line that creates the shrimp another 14 times, each time changing the world coordinates to different values. This would work, but it would be tedious.

Computers are good at doing things repeatedly, but that does not mean that we should need to write the same statement repeatedly. We can just tell the computer to execute a statement 15 times. (This will become even more important later on when we will execute statements not 15 times, but thousands of times. In that case, it is really not practical to write all the statements out one by one.)

The type of Python statement to do something repeatedly is called a loop. Python has several different kinds of loop statement, and the one we will use here is called a for-loop. It looks like this:

The for-loop executes the statements within it repeatedly. (There can be more than one statement in the body.) How often it executes depends on the header of the loop. The loop we see above counts from 0 to 15, and for each count executes the loop body (the statement with in the loop) once.

Let us discuss this in a bit more detail. The empty for-loop frame looks like this:

In the place of the expected identifier, we can write the name of a variable we want to use for counting. We can make up the name for the variable ourselves, and a variable with that name will be created. In our example above, we have used count as the variable name, but we could have named the variable anything we like. (In programming, the variable name i is also often used as a loop counter.)

In the 'list' slot, a list of numbers is expected. Here, we use the range function: this function creates a list of numbers from a lower bound to an upper bound. In our case we have used 0 and 15, so we will get a list of numbers from 0 to 15. One important detail to be aware of is that the range includes the lower bound, but excludes the upper bound. So strictly speaking, the numbers we get are 0, 1, 2, …​, 14. This suits us well, because it means that we are getting 15 different numbers (0 to 14), and the loop will run 15 times.

When the loop executes, it first assigns the first number (0) to the loop variable count and then executes the loop body once. Then it assigns the second number (1) to count and executes the body again. Then again for the third number, and so forth.

Once it has done this for the last number, the loop body will have been executed 15 times. For each run of the loop body, the count variable has a different value, and we can use this value to our advantage to solve our next problem.

In the last exercise above, you will have noticed a problem: The code created 15 shrimp, but they were all placed at the same location. Because all shrimp were drawn exactly on top of each other, it made it look as if there was only one.

We can fix this by placing each shrimp at a different location. How do we do that if all 15 shrimp are created using the same statement?

The answer is not to used fixed values for the x- and y-coordinates, but to use variables instead.

As our first try, let us use the loop counter variable as the coordinates for both the x- and y-position:

Using this code, the x/y coordinates will be 0,0 for the first shrimp, 1,1 for the second, and so on. This way, the shrimp are not all drawn at the same location.

In the exercises above, we have seen that we can use the * symbol for multiplication. Python has built-in operators for many frequently used mathematical operations; you can see a list of all available operators in Appendix D.

For the purposes of our game, being able to place the shrimp at different locations is good, but placing them in such a regular pattern seems odd in our context. So let us make the next improvement: Let’s place the shrimp at random locations.

Often, we want an element of randomness in our programs. In our case, we would like to place our shrimp at random locations, but we may also want other characters to move randomly, or have interesting things happen at random times. Randomness makes our game less predictable, and thus more fun to play.

In computer programs, all random behaviour is based on random numbers. Python can create random numbers, and we can then write code to translate this into any kind of random behaviour we need. We will see an example here for the random placement of the shrimp, and another example later in this chapter, when we want to make our shark move randomly.

The next task we wish to implement is the eating of the shrimp: When the fish swims across the shrimp, the shrimp should be removed from the world.

What we need here is technically called collision detection: We want to know when one actor (the fish) touches another actor (a shrimp).

The Strype graphics library provides a method to help with this: the is_touching() method from the Actor class. Figure 3.1 shows the documentation of this method (you can also find this in the library documentation in Strype).

We have previously discussed how to read the header of the method: We know to use the method name to invoke this method, and to provide a matching number of parameters. Let’s first take a closer look at the parameter of this method.

We can see that we can provide either an actor (using a variable that holds an actor) or a "tag" as a parameter. A tag is a label that we can provide for actor objects to identify them later. In our case, we wish to check for collision not just with a single actor, but with any of the shrimps. To achieve this, we will tag all shrimp objects as "shrimp" and then check for that tag.

Figure 3.2 shows the header of the Actor constructor, and we can see there that it has an optional fourth parameter: the tag we have just mentioned.

We can use this to attach a tag to our shrimp objects to identify them later:

The tag could be of any type, but it is most common to use a simple string to mark a set of actors so that we can easily recognise them later. In the method call to check whether we are touching a shrimp actor, we can then write:

and this call will return true only if we are touching an actor that was tagged with the string "shrimp". This would ensure, for example, that we are not eating other fish, should we decide later to introduce other fish into our game. Note also that this is a method – called on an object – not a stand-alone function, so we call it on the fish object. This makes sense, since it is the fish which wants to check whether it is touching another actor.

We should now also pay closer attention to the return value of the method. We already know that the documentation typically includes information about the general function of the method, and about its parameters. Here, we can see that the documentation also includes information telling us what the method returns to us.

We have already seen earlier that some methods and functions return a result to the caller of the function. It is time to see how we can find out about this from the documentation.

Figure 3.1 shows two bits of information about the return value, in its two last lines. It first tells us what it returns, and then it tells us the type of the return value. The first line tells us that the method will return the value True if this actor touches a specified other actor, and False if it does not. The type in this case is a data type we have not discussed before: bool.

The boolean type (in Python abbeviated to bool) is a type that can only hold two possible values: True or False.

In Chapter 2, we have already seen the types str, int and float. To this list, we should now add our boolean type:

When a method returns a value, we typically do something with that value. One option is to store it in a variable:

For method calls returning boolean values, we have already seen before that we can also use them directly in if-statements. Figure 3.3 illustrates this: The condition in an if-statement needs an expression that returns a boolean value (true or false), and our method call returns just such a value.

If you have done the exercises in Chapter 2, you have, in fact, made use of this before when you worked with the Fat Cat project. There, too, we used return values from method calls as conditions in our if-statements.

If you have watched closely when playing with your project, you may have noticed that often the shrimp disappear just before the fish seems to touch them. This has to do with how graphics are managed in many computer graphic systems (including Strype).

In Strype, even though images appear to have free form, all graphics drawn to the screen are, in fact, rectangular. Figure 3.4 illustrates this: it shows the bounding box of each image. The bounding box is the actual boundary of the graphic being painted. The image appears to be of arbitrary shape by using transparent pixels: the parts of the image where we do not want to show anything are transparent, making them essentially invisible. For the computer, they are, however, still part of the image, even though we cannot see them.

For the purposes of collision detection, two actors are "touching" when their images overlap. Since the invisible pixels are part of the image, this means that they are effectively touching when their bounding boxes overlap. The effect is that they are often technically "touching" even though the visible images do not touch each other.

Now, let’s make the game more interesting by adding a predator that hunts the fish: a shark. An image for this is in the Chapter03/images folder, but as always, you are free to use your own actor type and image for this. For example, if your player actor is a spaceship, then the "predator" might be an asteroid which you have to avoid hitting.

The beginning of this is easy: at the beginning of the game, where we create the fish and shrimp, we now add a line to create a shark:

Next, we add code into our main loop (after all the code dealing with our fish) to make the shark move:

It might be a good idea to make the shark move a bit faster than the fish to increase the challenge.

Of course, at the moment the shark moves only in a straight line, and it gets stuck at the right edge of the screen. To fix this, we want to make the shark turn when it reaches the edge of the screen. The Actor class has a method to check for this: the is_at_edge() method returns True if the actor is at the edge of the world.

Next, we should make the shark eat the fish when they meet. So let’s add some collision detection checking whether the shark touches the fish. This is now not too difficult, because it closely mirrors the code for the fish eating the shrimp:

We need to remember, of course, to tag the fish with the "fish" tag when we create it, so that it will be identified.

We now have the beginnings of a simple game: We have a keyboard-controlled character that has a task and has to avoid being caught. (This version of the program is in the book projects as yellow-fish-v5.)

If you have programmed along with yout own images and characters, your story might, of course, be different: you might have a game where you control a spaceship that picks up astronauts, or you might be controlling a white blood cell that removes bacteria and tries to avoid a virus. You can see that very similar game playing code can be used to present different stories.

There are many ways in which we can now improve this game and make it more interesting, and we will suggest some in the next chapter.

In this chapter, we have made good progress in developing our yellow fish game. In fact, we are getting close to having a playable game now. We have used some more Actor objects, and we have seen how we can use a for-loop to create several actors without writing separate statements for each one. We have also investigated collision detection of actors, using the is_touching() method.

And finally, we have seen that we can import functions from the random module to generate random numbers. These numbers allow us to implement random behaviour in our programs.

When we observe the development of our project through the steps we have taken so far, we notice that it gets longer and longer. This is natural: the more functionality we add, the more code we will have, and the longer our program text will be.

So far, this is not too bad. The program still spans only about a page (Figure 4.1), and with some effort and practice we can manage to read and understand it. Over time, however, this will get worse. Our program is still very simple, but as we add more improvements, its size will quickly grow. We will soon deal with programs that are not one page long, but dozens of pages. In professional systems, programs often span thousands or millions of lines of code.

The more we work with programs, the more we will need to read programs. Sometimes beginning programmers think programming is all about writing code. That is certainly important, but it is by far not everything: reading code will be just as important – if not more important – as writing code.

As your programs become more interesting and larger, you will take more and more time to work on them. You will work with code that you have written months earlier, and you have to read it again to remind yourself what it does.

If you continue programming, you will start working with other programmers. You will need to read code that others have written, and you will need to write code for others to read. When you progress even further, you may contribute to an open source project, or you may find some publicly available code on the internet that you’d like to use and extend. In each case, your work will start by reading large amounts of code that others have written before you can add your own.

In each of these cases, it is important that the code you are working with is readable. Being readable means that it is written in a way that helps a human reader to make sense of it and understand it. Not all code is easily readable, and there are big differences in how different programmers write code. Writing your code with readability in mind is one of the most important principles if you care about your program.

The most basic thing you can do to aid readability is to choose good names for your variables. From the beginning, we have named our variables so that they describe what they are used for. We used, for example "fish" for the variable holding the fish, and "shark" for the variable holding the shark.

This seems obvious in retrospect, but it is by no means automatic. We could have named our variables "a", "b" and "c" instead of "fish", shrimp" and "shark", and the program would have worked just the same. In fact, we have seen many programs where programmers did just this: they used one-letter variable names, just to save a bit of typing, and the program is full of variables called things such as "n", "p1", or "c".

A programmer can get away with using lazy names for variables as long as the program is really small, and no other programmer needs to read it. But as soon as we want to work with others, or work with our own program for a longer time, we should take variable naming seriously. If you name your variables in a way that they describe well what is stored in them, the whole program will be much easier to read and understand. You should get into the habit of always thinking carefully about naming your variables.

So what has this discussion to do with our use of functions?

The answer is that using functions gives us a chance to attach a name to a section of code to describe what it does. For example, we have taken all the lines of code that have to do with moving the fish, and put them into a function called "move_fish". The name of the function describes to the reader what happens in this function, and at the place in the code where it is actually needed, it now says "move_fish()" instead of listing a long sequence of statements. This is of great help to a reader of this program, as it gives them a very useful hint what the program does here, without cluttering the main loop.

The ability to attach a meaningful name to the block of code – in the form of a function name – is the first advantage of our restructure.

The second advantage of using functions in our program is abstraction. Abstraction is the technique of solving a sub-problem first, and then being able to ignore the details of the problem once it has been solved.

For example, if we now read our main loop, we can see that in each loop iteration it moves the fish, moves the shark, and maintains the pace of the loop. This is really easy to understand, and as long as we are happy with the behaviour of the fish movement, and we trust that it works as intended, we do not need to worry about the details of how it works any longer.

We have essentially treated the programming of the fish behaviour as a distinct sub-problem, which we solved by writing a function. When we need to invoke the fish behaviour, we can now call it just using its name, without needing to worry about the details of how it works. We treat "fish_movement" as if it were a single task, and we free our minds to concentrate on other things. We "abstract from" the details of the task, and treat it as a single, easy thing.

As our programs become longer and the tasks we wish to implement more complex, this becomes really important. Towards the end of this book, we will write programs solving problem that are too big to hold all details in your head at the same time. We will solve those problems by dividing them into sub-problems, which are smaller and can be solved more easily. These sub-problems are often solved in functions. And once we have written enough functions solving enough sub-problems, the solving of the overall problem becomes easy and straight forward.

Abstraction is thus a technique that wraps a potentially complex task into one unit and gives it a name. It hides the complexity, and makes it easy for us to use it without needing to think about the complicated details.

Even more importantly: it allows us to share the work between different people. For example, one programmer, somewhere, once implemented the randint function from the random library, or the is_touching method from the graphics library. But since we are using abstraction, we do not need to understand or worry about how these functions work internally. We just use them as a single instruction that does what we need. We aim to achieve the same with our own functions.

To use a function without the need to study its body, we just need to understand how to call it and what it does. The one additional thing that is needed is a good function comment that gives a reader enough information to understand what a function does without the need to read the body. So far, we have neglected the comment in our own functions. It is time to fix this.

The idea of abstraction is supported in Strype using the folding functionality of the function definition frame. Once the implementation of a function is finished, you can fold in the implementation to simplify the view of your program. Since you only need to know the header to call the function, this will be all you need to see for much of the time. Only when you later want to modify the function do you need to fold it out again.

Yet another advantage of using functions for part of our code is reuse: Once we have created a function for a subtask, we can call this function multiple times in our program, without the need to write out the whole code again. We will see more examples of this in later projects.

The technical term for what we have done when we move part of our code into functions is refactoring. Refactoring is an activity of improving the structure of our code without adding any new functionality. Typically, after a step of refactoring, the program behaves in exactly the same way as before, but the internal structure is better. The structure matters, because it typically makes it easier to understand, extend or modify the program

The program as it looks after the improvements described here is available in the book projects as yellow-fish-v6.

So far, we have only made structural improvements to our game, but not added more functionality. In the remainder of this chapter, we discuss some ideas of functionality you could add to your game. These do not need to be done in this order, or indeed at all. You are very much invited to invent your own improvements and try to add those as well. You might like to treat the following sections as suggestions rather than as strict recipes you have to follow. If you want to compare your code to ours, you can look at yellow-fish-v7, which is the last of the yellow-fish projects in the examples folder and contains implementation of some of the functionality suggested here,

Currently, our game is silent. One obvious addition is to add sound effects. This is fairly straight forward:

When we paste the sound literal into our program, we have various options how we choose to use this data. One option is to assign the sound to a variable, and then call the play method of the sound object to play the sound:

This is particularly useful if we wish to use the sound effect at different places in our program. If the sound is used only in one location in our program, we can also invoke the play method directly on the sound object, without assigning it to a variable. We do this by entering a function call frame and then pasting the sound directly into it:

At the moment, the game just runs on when the fish is caught. Implement a proper end to your game.

Winning the game is still quite easy. This is partly because the shark swims in a very predictable way: when it is not at the edge of the world, it just swims straight. We can make it a bit more interesting if we make the shark swim more randomly.

We can use randomness here in two ways: we can make the shark turn at random times, and turn by a random amount. Let’s start with the second part of this: turning by a random amount. This is quite straight forward. We can use the shark’s turn method, and then use a call to the random function for the actual parameter of the degree to turn. For example

will make the shark turn a random angle, between -45 and 45 degrees.

When you test this version, you will notice that it makes the shark appear too nervous. The constant turning makes it flick back and forth – it does not look good. So our next step is to make it turn less often. Let’s say, at every step we want a 10% chance of turning, and in 90% of the steps we just continue straight. How can we do this?

One way to express a 10% chance is to generate a random number between 0 and 99, and then look at the number. There is an exactly 10% chance that this number is less than 10.

In Python, we can use the < operator to test whether one number is less than another one. For example number < 20 returns True if the value in the variable number is less than 20. (Again, see Appendix D for a full list of operators.)

This means that the expression randint(0, 99) < 10 will be True exactly 10% if the time. By using this expression as the condition in an if-statement, we can make the turn happen in 10% of the steps:

Another possible addition is to keep a score. We could, for example, award 20 points for every shrimp eaten. Doing this involves creating a variable for the score, initialising it to zero, and then incrementing it by 20 every time a shrimp is eaten.

We have not yet discussed using variables in this way, and we will do so only in the next chapter. However, if you feel adventurous, or you know a bit of Python already, try this yourself with the following exercises. If you are uncertain, leave these exercises until after you have studied the next chapter

Many more extensions to this game are possible. The following exercises suggest some ideas. We will not discuss these here, but leave it up to you to work these out. Some of them are advances exercises that require you to look ahead and find out some things for yourself.

We will leave this project behind for now, but try to have fun with it by making it your own with your own ideas.

In this chapter we have suggested some more functionality which you can add to your game. But most importantly, we have discussed how to structure your program as it gets longer. We have discussed modularisation, which is the structuring of your program into distinct logical units, and we have used functions to create those units. Each function is given a name, and we have discussed the important aspect of how you should name functions and variables.

And finally, we had a first look at adding sound output to your programs, to create more engaging games.

The incredible speed of computers can be used to achieve many very different things. In the previous chapters, we have seen how it can be used to create games: the computer can draw an image consisting of many, many pixels, then draw it again slightly differently, and then again, so fast that it appears to us as smooth motion. Animation is just an optical illusion: it is a series of still images drawn so quickly that we cannot recognise the individual images.

The example we have programmed in the previous chapters is a very simple one. You will likely have seen more professional computer games, possibly including three-dimensional representations of imaginary worlds and complex interactions of game characters. Those examples show you even more clearly how fast computers are: they compute complicated models of a three-dimensional scene and display these on screen so quickly that we see it as smooth animation.

Ultimately, this effect is based just on the fact that the computer can hold vast amounts of data (thousands of game objects, millions of pixels) and can process them very quickly.

The same is true for any other of the many things that computers can do. You see examples of the amount of data and the computer’s speed when a search engine searches hundreds of millions of webpages in fractions of a second, or when an online map can show you details of any place in the world almost instantaneously.

One specific area which makes use of the memory size and speed of computers is data science. We will now have a closer look at this topic, and especially at a specific sub-field: text analysis.

Data science is a field that is concerned with gaining insights from large amounts of data. It usually combines techniques from statistics, computer science and visualisation to extract meaning out of data sets that are too big to analyse without computers.

The actual data can be many different things: data science can be used to analyse healthcare data to predict disease risks, or it can be used to analyse traffic and transport data to improve planning of public transport systems. Equally, it could be used to analyse financial data to detect online fraud, or environmental data to make climate or weather predictions.

A sub-field of data science is text analysis. This is one of the earliest examples of data science techniques, and it uses text documents as the data to be analysed. This might be used, for example, by historians to classify large amounts of ancient scanned documents, or it could be used by political scientists to analyse public opinion by scanning online comments.

In this chapter, we will start looking at some examples of simple text analysis to get a feel for how this works. Doing this, we will practice some important Python programming concepts, gain a first insight into data science and – in a later chapter – even get a first insight into the basics of how LLMs work.

The Python string type (usually abbreviated to str in the documentation) stores text strings. Text strings are just sequences of characters of any length. Thus, you can store your name in a string, or a whole book. In one extreme, a string might be empty (a text string of length zero), or it may hold the entire text of a Harry Potter book.

To get started with text analysis, we essentially have to get proficient with working with strings.

Now that we know how to access and process characters in a string, we can start with some simple analysis. As a first step, we would like to print out information in the following format:

You can make a start towards this with the following exercises.

In the last exercise, you may have encountered a small issue: We would like to print out double-quotes around the actual text. Since we have used double-quotes to identify the string literal itself, we cannot simply write a double quotes inside the string. If we tried to use the following string:

our program would fail, because the quotes intended to be within the string end the string itself, and we end up with invalid Python code. There are two possible solutions to this. The first one is to use a different string character. Since Python allows us to enclose strings with either single or double quotes, we can use single quotes to delimit the string if we wish to have double quotes within the string:

The second solution involves using a special character, the backslash: \. The \ character allows us to insert special characters into a string. Writing \" inserts a double-quote character into a string without ending the string:

Using this second technique is useful if our string includes both single and double quotes.

We have said above that an advantage of using computers is that they can do tasks very quickly, and with amounts of data that we would not want to process by hand. So let’s try this out: Before we go on to count words and sentences, let us first see whether our program can cope with larger text sizes.

Strype has a read-only file system that provides some data files to use with our projects, including the text of some books. One of them Pride and Prejudice by Jane Austen. The file is a plain text file, accessible at /books/pride-and-prejudice.txt. (This is the full text of the book.)

The project you have been working with includes a helper function called read_file(filename), which you can use to read a file from the file system. This function returns the content of the file as a string. So we can now easily read in this whole book from our file system, and then pass it to our analyse_text function:

We can see that we can quite easily process large amounts of text with our program. We would certainly not want to count the numbers of characters in War And Peace by hand, and getting a computer to do this for us helps.

But why would we want to know the amount of characters in the first place? In reality, the plain number of characters itself is not often that interesting. But it gets more interesting once we start counting words and sentences and analysing them. Then we get into the area of linguistic analysis, which can be used, for example, to draw conclusions about the author of a written work.

A famous case concerns the works of William Shakespeare. A passionate debate has raged on and off at least since the 19th century about the authorship of Shakespeare’s work. While theories that another author has written all of Shakespeare’s works are almost certainly untrue, and are usually dismissed by serious scholars, debates about individual works are quite valid. There are several written works where authorship is unclear: either works attributed to Shakespeare where authorship is not proven, or unattributed works that may have been written by him.

It gets even more complicated: In the 17th and 18th century, it was not uncommon for theatres to commission for existing plays to be padded out by paying other writers to add scenes or acts to other authors' plays to make them longer. Many scholars believe that Shakespeare’s plays contain parts that may not have been written by him.

Linguistic analysis has been used as one tool to provide some evidence in this debate. It is based on the observation that this analysis can detect a personal style in the use of words, which can then be used – in a way similar to a linguistic fingerprint – to identify the author. It alone cannot give conclusive answers, but it can add interesting pieces of evidence for scholars. See, for example, Shakespeare by the Numbers^[1] from the Shakespeare Oxford Fellowship.

Another example where linguistic analysis is used is to determine the reading complexity of written texts. Various algorithms exist to determine a "reading ease score" for a given text which expresses how hard or easy it is to comprehend. This may be used to choose appropriate books to study at school for specific age groups, or to make sure that important instructions provided with medication are not too complicated to be understood by the majority of the population.

One thing all forms of linguistic analysis have in common is that counting characters is not enough. At a minimum, we have to be able to identify and count words and sentences. So let us go on and investigate how to do this now, and then calculate a "reading ease" score for our documents.

The last thing we will do in this chapter is to calculate the text complexity using a reading ease score. Now that we can calculate all sorts of statistics of our texts, this is quite straight forward.

Several different formulas are in use by professionals to do this. We will use one of the most commonly used methods, the Flesch reading ease score. (If you want to find out details about this, do a web search.)

The Flesch reading ease score is calculatred with the following formula:

This formula will give us a score between 0 and 100. This score can then be used to classify the difficulty of the text, using the following table:

Looking at the formula, we can see that we need only three values as input, all of which we have already computed: The total number of words, the number of sentences, and the number of syllables in the text.

This means that we can now write a function to calculate the reading ease score, and call it with these values:

The body of this method here is incomplete – we leave it as an exercise for you to fill it in.

As always, an implementation of the project as discussed here is available in the book projects; it is called text-analysis-v5.

In this chapter, we have concentrated on two areas. Firstly, we have worked with strings. Strings represent text, and when we can work with strings, we can analyse and manipulate text. Text analysis is a first example of data science, of which we will see a bit more in a later chapter.

In working with strings, we have seen how we can access individual characters and words, and how we can format strings to create output that looks exactly as we would like it to look.

Secondly, we have again seen examples of improving the structure of our code. We have used functions to create a program that is modular and readable. This is important to create code of good quality, as it makes it easier for others reading our code to understand it.

At the same time, we have also seen how to use variables to store values, count and make calculations. This will be of good use in many of our programs in future.

We will investigate and practice working with a list using the temperature project from the Chapter06 folder of the book projects. This project provides a dataset of the average daily temperature (in degrees Celsius) in London, UK, for each day in 2025.

Let us start by looking at this data.

With the previous exercise, you will have seen that data contains a long list of numbers, which are printed separated by commas, and enclosed in square brackets. This tell us that data is a list: Lists in Python are written in square brackets, and their elements are separated by commas. Each value in this list is the average temperature of one day, starting at 1^st January.

In Python, the square brackets [ ] create a list, and we can write the list elements between the brackets. So the expression [1, 2, 3] created a list with three elements in it, and printing out the list prints the elements to the console.

If we are unsure about the type of one of our variables, we can inspect it in our code. The type() function tells us the type of data held in a variable. For example,

will print out the type of the variable numbers.

Now that we have a list of numbers, rather than a list of strings, we can start looking at values in a more meaningful way.

So far, all the operations we needed were available in ready-made functions. We used, for example, the min(), max() and sort() operations for our purposes, and it will often be the case that much of what we need is already available in the standard functions provided. Even if the exact function we need is not available, the existing functions often help doing a large part of the work. Assume, for example, that we want to calculate the average (mean) temperature for the year. There is no ready-made average function in Python, but we can quite easily compute the result using the sum() and len() functions. (Strictly speaking, this is only half true: there is an average function available in a standard module that we could import, but let us ignore this for now and work with the sum() and len() functions.)

Sometimes, however, we have tasks where the pre-made functions do not directly help us. In those cases, we typically have to process the list ourselves. This often involves a loop iterating over the list elements, and one or more variables to compute the result.

We will investigate this first by implementing two of the exising functions ourselves: sum() and max().

In the chapter this far, we have deferred two problems for later: the first was translating a day number to a more readable date format. We can print out the day number of a day, but we don’t really know what date this represents.

The second problem was that we had to know the day numbers of the start and end date to specify months for slicing. Again, using a different format for accessing months would be preferable. Let’s start by looking at the second problem.

We start our solution by creating a list that holds the numbers of days for each month. (This is slightly simplified, because we are ignoring leap years. But it works for our data, since 2025 was not a leap year.)

What we actually need, however, is not the length of every month, but a list with the day number of the start of every month. We can create such a list by going through our list of month-lengths and adding up the day numbers. Figure 6.2 illustrates this idea.

In our new list, start_day[0], for example, is that start of January, start_day[2] is the start of March, and start_day[11] is the start of December.

To create this list, we start with a list that just contains 0 as the start index for January. Then we create the rest of the list by adding the number of days of each month to the last value in our new list, as shown in Figure 6.2. The code to achieve this looks like this:

You may have to think carefully about this idea to fully understand it.

This list is now very useful for creating slices of data for a specific month. Every entry in this list, together with its next value, now tells us exactly the start and end of the slice for a month. Note that we have created this list with 13 values (not 12), so that December also has a following value to mark its end.

We can now write a function that gives us a slice for a given month:

This function can be called with a number for a month, and it will return the sliced list with the temperature values for that month. For example, get_month_data(0) will return the data for January.

An implementation of the exercises in this section is in the project temperature-v4.

There are many possible extensions to this project. We could, for example, modify our function to accept names of months, rather than a month number, and then use another list of month names to find the index of a given month by searching for the name. We can also use the same start_day list that we have used here to write a function that maps a day number to a date as a day/month pair. These are somewhat advanced exercises, and we leave it to the curious reader to work on them.

Instead of discussing more possibilities of processing our list of numbers, we will – in the next chapter – look at how we can use lists for other types of data. We will do so by looking at some graphical programs again.

In this chapter we had a good look at lists and some of their operations. We worked mostly with a list of numbers – floats, to be precise – and we have seen that we can find specific elements, sort or slice the list, and write loops that process every element.

Many convenient list functions are provided in Python by default, and it will be very useful to become familiar with them. In the next chapter, we will see other situations in which these are very useful.

The Space Rescue project (Figure 7.1) is essentially a copy of the Yellow Fish game with different graphics. Start by investigating it.

You will have seen when you tried out the program that it is essentially the same as yellow fish. We have changed the colour of the background, the graphics and some of the sounds, but the code is almost identical. The only two changes to the functionality are:

(introduce constants…​)

(explain variables here…​)

For the remainder of this chapter, we will work with sound files, and we will see that sounds, too, can be viewed as lists – in this case, lists of samples.

A sample is a reading of the sound input (the microphone), and it is stored as a number. When these samples are read and stored quickly enough, we get a sound recording, and we can play it back later. Thus, a stored sound is a list of samples, which itself is a list of numbers.

Start by opening a project with a sound in it. We have provided such a project in the Chapter07 folder; it is called sounds.

In our current implementation, playing the sound effects was hard-coded in our program. Perhaps you wrote your code to play a single effect, disabling the calls that played the previous effects. Or perhaps you tried to play all effects one after the other – in this case, you will have discovered that the program does not actually wait for the sound to finish before continuing, and the sounds likely played overlapping. That is not a great effect.

It would be more useful to let the user choose an effect, in any order they like, from a list of available effects.

Implement such a choice now. Your task it to show a menu on the screen that looks something like the one shown in Figure 7.3.

For the main structure, use a while loop with a boolean variable exit that controls how long the loop continues. Then, in the loop, you can react to key presses, playing sound effects when needed, or setting this variable to True to exit:

A version of the program with these exercises implemented is available in sounds-v3.

There are, however, many more experiments that we can do with our sounds. Some things you can try are suggested in the following exercises, but you are alo encouraged to think about your own effects and trying them out.

In this chapter, we have seen some more uses of lists in Python.

First, we have seen that the graphics world uses lists to manage the actors in it, and that we can obtain lists of actors from it (either a list of all actors, or lists of selected actors).

Then we have seen that sounds can also be represented as lists: lists of samples. Samples are numerical values, and by manipulating the list of samples, we can change a sound. Along the way, we have encountered several useful operations on lists, such as joining and repeating lists, and slicing with step values.

We can achieve our content analysis in three steps:

We will take this entire chapter to complete this task; here, we start at the beginning: counting how often each word is used in a text. This is also known as the frequency of the word.

To start this task, we have prepared a starter project named thematic-analysis, which you can find in the Chapter08 folder in the book projects. This project includes only two helper functions which we will use later, but no code to help us with the initial task.

In previous projects, we have seen how useful it is to use functions in our programs to do specific subtasks. So this time, we will start by writing a function to count the frequency of a given words in a text.

If you have reached this stage in the project, you will have seen that we can now count the frequency of a given word in a text. You may have noticed, however, that some problems remain that can result in incorrect counting. If not, you should be able to see the problem with the following exercises.

Why does the test work correctly for the word "say", but not for the word "go"? You have probably worked out that this is because of the uppercase character in one of the "Go" occurrences in the text.

For our purposes, we do not care about the upper- or lowercase nature of the words – we would like to count them in any case. We can easily achieve this by first converting our entire text to lowercase before starting our counting:

The lower() method is a built-in method of the string class that returns a new string with all characters converted to lowercase. Note that it does not modify the original string. Thus, we have to assign the result back to our string if we want to keep it.

With these exercises, you will have discovered two things: First, there are many useful string methods available which can save us a lot of work of writing code ourselves. Knowing what methods are available is important to becoming a good programmer. You do not need to memorise all methods, but you should have a link to the documentation page readily available when you work, so that you can look them up whenever you need them.

And secondly, we can see that a count method is available that is somewhat similar to the count_word function we are currently writing. The reason we are writing our own is two-fold: The built-in count method does not find words, but substrings. Thus, "car", for example, would also be found if the actual word is "scarlet". Also important is our goal to modify and extend this function to count not one word, but all words (and there is no ready-made method for this). So our counting function is a stepping stone for more powerful functionality.

Again, you may have noticed that your function does not count correctly. This time, it is the punctuation characters that are throwing a spanner in the works. When we split the text, we separate the original string at space characters, but all other characters are left in the words. Thus, our list of words contains the word "say," (including the comma), instead of the word "say".

To fix this, we will need to remove the punctuation characters from each word.

The project with all the changes made so far is available in the book projects as thematic-analysis-v2. If you are unsure about any of your own code, you can compare your project to this one.

Let us pause for a moment and take stock of what we have achieved so far. We can now count the number of occurrences of a given word in any given text. We do this ignoring the case of the letters. That is a good start. But a major problem remains: We would like to count all the words in a text, not only one given word. We know from previous chapters that we can hold a count in a variable, but the challenge here is that we need a count for each different word in the text.

It is not feasible to create a separate variable for each word. There may be thousands of different words, and we do not know in advance how many variables we would need. A list is also not directly suitable because we need to hold two pieces of information: the word, and its count, together.

The solution to this problem lies in a new data structure: a dictionary. In the next section, we discuss what a dictionary is, how we use it, and how it can help us to solve this problem.

A dictionary (or dict, as Python calls this type internally) is a data structure that stores key/value pairs.

Every entry in the dictionary associates a key with a value, and we can look up the value by providing the key. An example is a list of phone numbers for your friends (Figure 8.1).

We see that we can think of a dictionary as a table with two columns. The first column holds the keys (here: the names), and the second column holds the values (the phone numbers).

We can then look up the value by providing the key. Let us look at an example how this works. Assume we have the dictionary shown in Figure 8.1 stored in a variable called phonebook. We can then write

This will print Adam’s phone number. We use the key like an index in the dictionary, and we receive the associated value. This makes looking up a particular entry in the dictionary very easy.

We can also add more entries to our dictionary. For example

will add a new entry in the dictionary (a new row in the table) with the phone number for Zola. Again, we see that we use the intended key like an index in square brackets, and we just assign the value to create the entry.

Dictionaries are useful in many situations, and we are familiar with the concept from our daily lives. One important aspect is that they do not only work with strings. Both the key and the entry may be any type we choose.

For example, your contacts app on your phone holds a dictionary that uses the name of a person as a key and the address details as the value. (Here we can see that the value can be more than a single string: it can itself be a complex data type with several types of information). You use it by looking up the name, and it shows you the associated detail (the value).

Another example is a password file on a computer (associating usernames with passwords). Perhaps the most familiar example is an English language dictionary (from which the Python data structure gets its name): The key here is a word, and the value is its definition. We can find a word in the dictionary to look up its definition.

The last thing we need to know is how to create a dictionary. Python dictionaries are written with curly brackets: { }. Writing the curly brackets on their own creates an empty dictionary:

We can also create a dictionary pre-filled with some data using a colon to separate keys and values, and a comma to separate the entries:

Before we go on to use a dictionary in our word counting function, let us do some simple exercises to become more familiar with dictionaries.

Let us now get back to the word counting problem in our thematic-analysis project. We were at the point where we could count the frequency of a single word, and we need to figure out how to count the frequencies of all words. As we have stated above, a dictionary can help with this.

The idea is that we use a dictionary – let’s call it counts – with an entry for each word. We can use the word itself as a key, and use the value for the count (the number of times we have found the word).

For example, a word count of the words in the sentence

would result in the dictionary shown in Figure 8.2.

In other words: we will create a dictionary with one entry for every unique word in our text, where the value holds the number of times each word was used.

It is useful to remind ourselves that this dictionary – if we do a word count for a whole book – will hold thousands of entries. This, however, is not a problem: computers are good at dealing with large amounts of data.

The algorithm we will use can be informally described as follows:

You can modify your existing program to implement the algorithm described above by working step by step through the following exercises.

A version of the project that implements these changes is available fo you to study in the project folder as thematic-analysis-v3.

We have seen that printing out the dictionary using a simple print call works, but the output is not very readable. The list of word counts is very long, and the formatting is not very readable. Let us improve this by implementing a custom printing function for our word count table.

A first attempt at printing out the entries in our dictionary might involve printing each entry on a separate line:

With the above exercise, we can see that this version prints out the words in the dictionary, but not the counts. In other words: it prints the keys, but not the values of the dictionary.

How does this happen? Let us step back and take another look at the methods from the dictionary class that allow us to access its entries. There are three methods that give us lists of the dictionary contents:

The first two of these show us that we can get a list containing just the keys or the values. This means that we could print out either the keys or the values, line by line, using these two loops:

Try this out. What we observed with the last exercise above is the following: If we just iterate over dict, without stating whether we want to get the keys or the values, we will get the keys by default.

Let us look at the third method: items(). This method states that it returns a tuple with the key and value. A tuple is another of Python’s collection types. It can hold multiple arbitrary data items to group them together. We will discuss tuples in more detail in the next chapter; for now it is sufficient to know that it consists of multiple data items and is written using round brackets:

When we think about our dictionary, we can view a dictionary as a list of tuples. The first two entries of the dictionary shown in Figure 8.2, for example, can be regarded as two tuples:

Let us now try to use the items() method in our loop:

Try this out. You will see that this loop prints each entry of the dictionary as a tuple.

This is, so far, the most useful method for us: it allows us to get the full data (key and value), and print them out line by line. But we can get even more control using an additional feature of the for-loop.

If we use a Python for-loop to iterate over a collection, and the item type of the collection is a tuple, then the loop can contain multiple loop variables:

The rules for this type of for-loop are quite straight forward:

Using this variant of the for-loop allows us to get access to each key and value individually, and we are free to format the output ourselves. In the code example above, we have done this using an f-string.

Our output is now more nicely formatted. It is, however, still very long, and identifying frequently used words is still tricky. Change this by doing the following exercises.

The list is now much shorter, but it still contains a lot of very common words, such as "a", "to" and "the" – words that do not tell us much about what the text is concerned with. Let’s do a first attempt at identifying interesting words by dropping all short words.

We can see that we are getting a bit closer. If you run this version of your program on one of the book texts provided, you can see that a list of words slowly emerges that may characterise the book. But the rules we have used to select the words (filtering by a word count that is more than a fixed value, and by length of the word) are not really good enough.

What we need to do instead is not look at the absolute frequency of a word (the number of times it appeared), but look for words that appear more often than they would in a typical English text. The word "the", for example, may appear very often, but it does so in all English writing, so this does not make it an especially interesting word.

This will be our goal in the next section: We will use existing statistics about expected frequencies of words in the English language to identify "over-used" words in our text: words that are used more often in this text than they are used in average English texts.

In our project, we have provided a helper function called get_expected_frequencies(). This function returns a dictionary that lists the expected frequencies for the most common words in an average English language text. (It uses a dictionary of about 64,000 words.)

The dictionary has a word as its key, and the expected frequency as its value (Figure 8.3). The frequency is the ratio of occurrences of the word relative to the length of the text. It can be computed by dividing the word count by the total number of words in a text:

Using this formula, we can easily calculate the frequency of each word in the text we wish to analyse and then compare it to the expected frequency. Those words that are used far more often than expected are the "interesting words" – the words that might give us a clue to what the text is about.

Let us go ahead and implement this idea. We will identify the frequent words, which we define as words with a frequency that is at least 10 times the expected frequency and which were used at least 10 times.

With the code so far, we can print out the dictionary of frequent words. We have, however, no direct control over the exact number of words we will receive. This will depend on the length of the actual text and the words within it.

Let us make one last improvement to specify exactly how many words we wish to see. To do this, we will sort the frequent words by their word count, and then select the top words from this list.

We can do this by making good use of Python’s built-in sorted function:

We can add this statement to our get_overused_words function to create a list of the frequent words, sorted by their word count.

This is quite an advanced use of this function, so let us investigate that line in some detail.

After doing this, we can use a slice to take the first few elements. For example

will give us the 12 most frequent words. We can then return this list from our function.

In making these changes, it is likely that you ran into a number of errors. This is to be expected. We have made many changes, and we will always make errors along the way. It is not the goal to write our code error-free at the first attempt, but to get to know Python well enough that we can identify and fix any errors that occur. Remember, you can refer to Appendix C: Dealing with Errors to get some more information about errors and error messages.

Once you manage to complete these tasks, you will see that you can now get a vague impression what a book is about. You can see that War and Peace is about a prince, a princess, an emperor and Napoleon, or that Three Men in a Boat is about a boat, a picture, locks and rowing.

A version of this project that includes all the changes discussed here is available as thematic-analysis-v4; you may like to compare your own version to this one.

In this chapter, we have learned how to use dictionaries to achieve more advanced functionality.

Dictionaries are perhaps a little more unusual than lists, but they are equally useful. To become a good programmer, it is important to become familiar with dictionaries, and fluent in using them. They should be one tool in your standard toolbox, so that you can use them whenever they can help to solve a problem. You will see that they come in handy more often than you initially expect.

Python has four main built-in collection classes that every programmer should know: list, set, dict and tuple. (There are other collection classes, but these four are the most important ones.) All of these serve to collect several items of data into a single grouping, which we can then store in a single variable. They differ in their characteristics and their uses .

All of these classes have methods that allow us to perform useful operations on them.

Let us have a quick look at each of these four classes before we describe them in more detail.

A list is an ordered collection of flexible length. Elements are accessed by index.

A dictionary (also known as dict) holds entries of key/value pairs. We can look up the value by providing the key.

A set holds an unordered collection of its elements. Each element can occur only once.

A tuple is a fixed, unchangeable grouping of several items of data into single item.

Below, we describe each of these classes and their characteristics and uses in more detail.

We have used the list class a few times in this book, and you hopefully have quite a good understanding already of what a list is and how it works.

We can represent a list graphically as a sequence of elements:

In Python, lists are written using square brackets:

An empty list is created by using the square brackets on their own:

We can access individual list elements using their index:

List elements can be replaced. The following code will replace the second sighting with a different one:

Assume we are programming a bird watching app, and we would like to record sightings of birds on a given day. The diagram above shows how each individual sighting can be appended to a list. With each new sighting, we could append a new entry at the end of the list (using its append() method), and the list would keep a record of all the sightings of the day.

Since the length of the list is flexible, we can record any number of sightings, and the order of the sightings would be maintained (we could later retrieve the information in which order we saw each bird).

The list class also allows us to insert elements at arbitrary positions in the middle of the list, delete individual elements, or access elements by index.

We can iterate over the list to receive the elements one by one, in the order in which they are stored.

While lists allow elements of different types to be mixed – we could, for example, have a list where some elements are strings and some are integers – this is rarely useful. Most of the time, we use lists where all elements are of the same type.

A list has the following characteristics:

To find details of the following methods, look them up online. Each of these methods may have required and optional parameters.

A dictionary is a mapping from a key to a value. We can think of it as a table with two columns. Each entry in the dictionary is a new row. We can then look up entries by providing the key and receiving the associated value.

Internally, Python calls the dictionary class "dict". Dictionaries are written using curly brackets and colon-separated key: value pairs.

The statement above creates a dictionary with four entries, where each entry is a key:value pair.

An empty dictionary is created by using the curly brackets on their own:

To access elements, we provide a key in square brackets, and the dictionary will return the associated value. For example:

This will print the number 8 (assuming we have a dictionary as shown above.)

In our bird watching app, we might not be interested in the exact order in which we sighted each bird, but only in the number of each type of bird we have seen. In this case, a dictionary is an ideal collection to use. The example above illustrates how we can create a dictionary that uses the name of the bird species as a key and uses the value to store the number of sightings for that kind of bird.

When we later wish to add a sighting to our data, we need to carefully distinguish the case where we already have an entry for the bird species from the one where the bird does not yet exist. If the species is not in the dictionary, we will add a new entry. If it exists already, we need to read the current value, increment it by one, and write it back:

We can also iterate over a dictionary in a very similar manner as we do with a list:

Be aware, though, that this pattern – where we use only a single loop variable, will just give us just the keys of the dictionary. To retrieve both the keys and the values, we can write a loop with two loop variables iterating over the dictionary’s items:

A dictionary has the following characteristics:

The most important set methods are listed here. There are more methods available; you can find details online.

A set is a collection we have not encountered in this book before. It is, however, quite easy to understand: a set is an unordered collection that stores each element at most once. We can represent it graphically like this:

Sets in Python are written using curly brackets:

Note that with this example, the sightings set would hold only three elements. We inserted a "robin" twice – this is not an error, but since the robin was already included before, the second adding of a robin has no effect.

Note also that the set uses the same symbol (curly brackets) as the dictionary. When writing literals, we can distinguish them by their elements: if the elements are pairs, we are creating a dictionary, if the elements are single values, then we create a set.

We cannot, however, create an empty set by just writing a pair of curly brackets ({ }), because this creates a dictionary. To create an empty set, we use the set() function instead:

We cannot access the elements individually (there is no index provided). When we iterate over a set, we will receive all the elements, but they will be provided in an unspecified order. The order in which they are inserted is not maintained, and it may change at any time.

Thinking about our bird watching app again, we might use a set if we are only interested in the kinds of birds we spotted that day. If we do not care about the number of birds we saw, or in which order, then a set provides a much simpler and faster data structure to store this information.

We would still add an element each time we see a bird, and the set would automatically ignore duplicate entries. The size of the set is flexible as well, so an arbitrary number of species can be stored.

Elements in a set cannot be replaced. We can add and remove elements, but we cannot directly replace an element with another, as we can in a list.

As with lists, sets containing elements of different types are possible, but rarely useful. When mixing different types of elements, we also have to be aware of details of equality. For example, the values True and 0 are considered equal in Python, so in a set they would be considered duplicates. A set that contains the value False will therefore ignore the addition of a 0. This can be confusing, and sets with different element types are best avoided.

A set has the following characteristics:

The most important set methods are listed here. There are more methods available; you can find details online.

A tuple is a grouping of number of different data items into a single, fixed group. The following example shows two tuples which we may use for our bird watching app. In these tuples, we combine three pieces of data: the common name of the bird, the scientific name, and the maximum size (in cm).

Tuples are used to combine items of data that logically belong together and should be handled (created, stored, processed) as a single unit. The individual data items may be of different types. Once a tuple has been created, it is unchangeable: neither can the values in it be modified, nor can it be extended. It will remain as it is.

Tuples are written using round brackets (parentheses):

In theory, we can create an empty tuple by writing just a pair of parentheses: ( ), but in practice this is not really useful. Since tuples cannot change later, this particular tuple does not hold any interesting information.

To access elements from the tuple, we can either use indices (in the same way as we do in a list):

or we can iterate over the tuple using a for-loop.

Examples of tuples may include the representation of playing cards: ("diamonds", "7"), information about a person: ("Sam", "Smith", 21, 3, 2011, "f"), or information about a font style used in a text document: ("Courier", "Bold", 12, RED). In any situation in which different pieces of information belong together, they can be grouped in a tuple.

In the context of our bird watching app, let us assume that birds are actually captured, measured, recorded, and then released. Then we might want to record each sighting individually, including the recorded measurements. For this purpose, we might use a tuple to store the details:

When we store the sightings, we might then use a set or a list of these tuples, instead of storing just a list of strings.

Sometimes we want to store two data items that belong together – this is often called a pair. Examples might be coordinates in a cartesian coordinate system (x, y), or the description of a musical note (frequency, amplitude).

While we may talk about a "pair" as a logical data structure, in Python it is not a separate type. A pair is simply shortcut name for a tuple with two elements.

A tuple has the following characteristics:

The tuple class has two built-in methods.

The last – and most powerful – aspect of collections is this: collections can be combined.

Collection classes themselves are types of data, and collections can hold elements of any type, so a collection can hold other collections.

In the last section above, we have already mentioned the possibility of a list of tuples. Equally, we could have a list of lists, or a dictionary where each value is a set. The depth of these nestings can be even deeper: There is nothing stopping us from creating a list of sets of dictionaries, in which the values are tuples.

This may sound complicated at first, but is in fact quite common, and not that hard to understand. Think, for example, of a contacts app on your phone or computer. At the top level, it might hold a dictionary with the name of a person as the key. The value – the information stored under the key – however, is not a simple type, but a tuple holding various pieces of data (phone number, address details, email address, picture, etc.). If we look at this even more carefully, we might realise that a person may have multiple email addresses, or multiple phone numbers. We do not know how many there are, so this should probably be a list. So we now have a dictionary holding tuples, in which some items are lists.

Constructs like these are common in computing, and we will see an example of this in action in the next chapter.

Python has four main collection types: list, set, dictionary and tuple.

It is important for a good programmer to become familiar with these types. Each has different characteristics – some can grow, some cannot, some are ordered, other are not, etc. – and we need to be able to choose an appropriate collection for a task.

Collections are often combined – collections can hold instances of other collections – and this is useful for storing structured data.

So how can we predict what the next word in a sentence may be? The obvious idea is to look at the preceding words, and then try to figure out what next word might fit to continue a sentence.

At its most basic, we can look at only the one word immediately before the next one. This will not be terribly good at producing sensible text, but it’s a start. So let us try to do that first.

How can we know what word might work to follow another word? Again, the basic idea here is simple: We look at a good amount of existing text, and look what words follow other words there.

Let us investigate an example by looking at an English sentence:

In this sentence, we can see that the word "I" can be followed by the word "read", the word "read" can be followed by the word "the", and so on.

We can now write down all the pairs of words that we can see may follow each other:

These pairs of consecutive words from a text are called bigrams. The list of pairs above are all the bigrams from our sentence.

Now, doing this with just a short sentence is not very useful, but doing the same with a very large body of text – say, a whole library – gives us very interesting information. It will tell us all the words that may follow a given other word, and it will also hold information about the likelihood of one word appearing after another. Pairs of words that appear frequently together will appear more often in our list of bigrams.

To experiment with this, let us write a program that can produce a list of bigrams from a text.

Now that we can generate a list of bigrams for any given text, we can use this to predict the next word. The idea is now straightforward: First, we look at the previous word. From the list of bigrams, we select just those where the first word of the bigram matches our previous word. For example, if our previous word was "It", then the list of matching bigrams might start with

From this, we can make a list with all the second words, and then randomly select a word from that list. Since words that more frequently follow our previous word occur more often in that list, they will have a higher change of being selected as our next word. In other words: the frequency of a word appearing as the second word determines its probability to be selected. (For predictive text, we would select the most frequent word. For creative writing, we will randomly select one of the list.)

A copy of the project with all this functionality implemented is in the book projects as text-generation-v2. Feel free to compare your own code to the one in that version.

So what have we achieved so far?

We can now read in a training text that generates a list of bigrams. The bigram-list is our "model" – this is what is used to generate text later on. We "train our model" by reading in text. The kind of text documents we train our model on determines the kind of language our system produces. If you train your model using a different text, it will produce different language. (Try it out!)

Of course, we could read in not just a single book, but a whole stack of books. The more text we use to train our model, the closer our system will follow normal English speech patterns.

When we look at the output (for example, in Figure 10.5), we can see that the result so far is somewhat entertaining, but not very impressive. Very superficially, it looks like English, but there is no recognisable grammar, and this output is not fooling anyone. We will improve this in the next section.

Another observation worth pointing out: This time, we have not stripped out punctuation when we split the text into words. As a result, punctuation is still included in the bigrams, and thus in the output. The words "it" and "it," are just considered different words, and handled separately, each with their own following words. This makes the resulting text look more natural.

The next idea to improve our model is again quite simple: instead of looking at only one previous word to generate the next one, we look at two previous words. In our model, we do not use bigrams, but instead we use trigrams – tuples of length 3 with all the three-word sequences form our original text.

Generating the trigrams is quite straightforward, since the built-in zip function can also merge together three lists (or any number of lists). In our program, generating the trigrams looks like this:

Change your entire program to work with trigrams now, using the following list of exercises.

If you have done the exercises so far – how is your output now?

You will see that this simple improvement of our "model" – going from bigrams to trigrams – significantly improves the quality of the text it produces. Now the output looks almost like grammatically correct English! That is quite astonishing for such a simple algorithm.

Of course, the sentences are still gibberish – they don’t mean anything, and you cannot read any sense into them. We discuss below how real LLMs manage to do better than this.

But first of all, let us try to improve our training data.

There is a very simple change that we can make to improve the quality of our system: use more data.

In real LLMs, the models are trained on millions or billions of documents. Many of them are fed any text you can access on the Internet, and much more besides.

We will not use that amount of text, but we have two files available that contain just a bit more than a single book. The Strype file system folder includes a file called /books/fairy-tales.txt, which holds the full text of 65 fairy tales by the Brothers Grimm, and another file called /books/sherlock-holmes.txt, which has the text of 27 Sherlock Holmes stories (three books and 24 short stories). Try your program with these texts as the training data.

A version of the project with these changes is in the book projects as text-generation-v3.

If you did these experiments yourself, you will have noticed two things: Firstly, using longer texts to "train our model" improves the quality of the output. The more text we feed in, the more closely the output resembles actual language. We have not, in fact, used very much data. Even the "sherlock-holmes" file contains only about 330,000 words. This is miniscule in the context of large language models, but it already starts to deliver quite surprising results.

And secondly, the content of the text we feed into the model determines the language and style of the text that is produced.

These two observations lead to two very significant issues: Firstly, with the aim to be able to produce any kind of output, large AI companies have been scanning all writing they could get access to. This includes, for example, millions of published books. For the performance of the system, this is great: you can tell your AI system to write a short story in the style of Stephen King, and it will very competently do this. It has read in all existing Stephen King stories, and so it can copy the style amazingly well.

The problem is that these stories are under copyright, the AI companies have not legally licensed them, and Stephen King has not given his permission.

So, is this legal?

Currently, this is an undecided questions. The AI companies argue that it falls under the "Fair use" clause in copyright law, since the source works are not reproduced directly. Many authors argue that this is copyright theft, since their own original hard work is being used – and being commercially exploited for profit – by other companies without their permission.

The second problem is the sheer amount of data involved, and the amount of processing power and electricity required to process it. Reading and transforming the "sherlock-holmes" file is trivial, and a standard laptop can easily do it. But processing the amount of data involved in real LLMs – more text than the content of all the libraries in the world combined – requires so much processing power that the environmental impact is immense. Power is not only needed to train the model (this is only done infrequently), but for every single question that an AI system is asked to answer. Even the most trivial query requires huge processing power and consumes large amounts of energy.

To cope with this, AI companies are building huge data centres in many areas of the world, full of processors which only serve AI queries from users. A single data centre can consume as much energy as a city of a million people. For the environment, this is a disaster; it creates significant new problems for the goal of reducing the environmental damage which we cause.

We have seen that our model does not come close to the abilities of current AI systems, and there are many reasons for this.

Firstly, there is room to improve our current system. One idea is again simple: use longer chains of words. We have used bigrams and trigrams. More generic systems use n-grams – sequences of larger and variable length.

Using more contextual words does indeed improve the performance of these systems. And the project that we investigated in this chapter does indeed give a glimpse into the history of text processing.

Until the mid-2010s, n-gram-based systems were considered the state of the art in text processing. They were successfully used in simple auto-complete systems and speech recognition.

After that, these systems evolved significantly, and modern LLMs have a number of important differences that make them more effective:

The resulting systems are known as generative pre-trained transformers (GPT). The full details of these are more complex than what we can discuss here. But ultimately, they are generalisations of the simple system we have built here. Or in other words: An n-gram model is a special, extremely limited case of what a GPT transformer can represent.

N-gram systems, however, are not obsolete. They continue to be useful in predictive text, spell checking, and in document retrieval systems.

While our project here does not allow us to understand fully how LLMs work, it does give an insight into the core of the idea that underlies LLM-based AI systems. Perhaps most importantly: It is still the case that LLMs have no real understanding of the content and meaning of the text they produce, and are ultimately based on clever manipulation and prediction of language. These systems may state an entirely invented "fact" with the same confidence as an obvious truth.

AI companies are working hard to improve the reliability of LLMs. This tension, however, between our desire of trust and these systems, which ultimately just manipulate language, will remain at the core of the issue of reliability in AI systems for some time.

The Strype editor allows the creation and manipulation of Python programs using the mouse or keyboard. All editing can be achieved with keyboard commands.

In Strype, program statements are represented by frames – coloured boxes which may be nested within each other (shown under the "My code" heading in Figure A.1). Each Python statement is represented by a frame.

Some frames have a distinctive background colour. Simple frames (such as assignments or function calls) have the same colour as the code background, and no visible outline. They are still frames, and can be manipulated like all other frames.

Frames have slots – areas that must be filled in to complete the frame. Text slots are used for expressions and accept textual input. Frame slots hold nested frames. In an if-statement, for example, the condition is held in a text slot, while the body is held in a frame slot.

Strype has a frame cursor (the blue bar in Figure A.1) that indicates the current editing position.

A panel in the top right of the main interface (the frame palette) shows all available frames which may be inserted at the current cursor position (see Figure A.1).

A frame is inserted at the current position by either clicking the frame in the frame palette or by pressing the keyboard shortcut shown there. For example, pressing ``i'' will insert an if-statement, or space bar will insert a function call.

The frame cursor is moved using the cursor keys, which can also be used to select one of more frames.

You can wrap existing frames in a surrounding control frame (such as a loop or conditional), by selecting the intended body frame(s), and then inserting the control frame.

For example, try selecting a few frames, and then press i to insert an if-frame. This will place the selected frames inside an if-statement.

Right-clicking a frame with the mouse displays its context menu (Figure A.2). The context menu offers some useful functions for frames.