Creative Python Programming with Strype

We can achieve our content analysis in three steps:

We will take this entire chapter to complete this task; here, we start at the beginning: counting how often each word is used in a text. This is also known as the frequency of the word.

To start this task, we have prepared a starter project named thematic-analysis, which you can find in the Chapter08 folder in the book projects. This project includes only two helper functions which we will use later, but no code to help us with the initial task.

In previous projects, we have seen how useful it is to use functions in our programs to do specific subtasks. So this time, we will start by writing a function to count the frequency of a given words in a text.

If you have reached this stage in the project, you will have seen that we can now count the frequency of a given word in a text. You may have noticed, however, that some problems remain that can result in incorrect counting. If not, you should be able to see the problem with the following exercises.

Why does the test work correctly for the word "say", but not for the word "go"? You have probably worked out that this is because of the uppercase character in one of the "Go" occurrences in the text.

For our purposes, we do not care about the upper- or lowercase nature of the words – we would like to count them in any case. We can easily achieve this by first converting our entire text to lowercase before starting our counting:

The lower() method is a built-in method of the string class that returns a new string with all characters converted to lowercase. Note that it does not modify the original string. Thus, we have to assign the result back to our string if we want to keep it.

With these exercises, you will have discovered two things: First, there are many useful string methods available which can save us a lot of work of writing code ourselves. Knowing what methods are available is important to becoming a good programmer. You do not need to memorise all methods, but you should have a link to the documentation page readily available when you work, so that you can look them up whenever you need them.

And secondly, we can see that a count method is available that is somewhat similar to the count_word function we are currently writing. The reason we are writing our own is two-fold: The built-in count method does not find words, but substrings. Thus, "car", for example, would also be found if the actual word is "scarlet". Also important is our goal to modify and extend this function to count not one word, but all words (and there is no ready-made method for this). So our counting function is a stepping stone for more powerful functionality.

Again, you may have noticed that your function does not count correctly. This time, it is the punctuation characters that are throwing a spanner in the works. When we split the text, we separate the original string at space characters, but all other characters are left in the words. Thus, our list of words contains the word "say," (including the comma), instead of the word "say".

To fix this, we will need to remove the punctuation characters from each word.

The project with all the changes made so far is available in the book projects as thematic-analysis-v2. If you are unsure about any of your own code, you can compare your project to this one.

Let us pause for a moment and take stock of what we have achieved so far. We can now count the number of occurrences of a given word in any given text. We do this ignoring the case of the letters. That is a good start. But a major problem remains: We would like to count all the words in a text, not only one given word. We know from previous chapters that we can hold a count in a variable, but the challenge here is that we need a count for each different word in the text.

It is not feasible to create a separate variable for each word. There may be thousands of different words, and we do not know in advance how many variables we would need. A list is also not directly suitable because we need to hold two pieces of information: the word, and its count, together.

The solution to this problem lies in a new data structure: a dictionary. In the next section, we discuss what a dictionary is, how we use it, and how it can help us to solve this problem.

A dictionary (or dict, as Python calls this type internally) is a data structure that stores key/value pairs.

Every entry in the dictionary associates a key with a value, and we can look up the value by providing the key. An example is a list of phone numbers for your friends (Figure 8.1).

We see that we can think of a dictionary as a table with two columns. The first column holds the keys (here: the names), and the second column holds the values (the phone numbers).

We can then look up the value by providing the key. Let us look at an example how this works. Assume we have the dictionary shown in Figure 8.1 stored in a variable called phonebook. We can then write

This will print Adam’s phone number. We use the key like an index in the dictionary, and we receive the associated value. This makes looking up a particular entry in the dictionary very easy.

We can also add more entries to our dictionary. For example

will add a new entry in the dictionary (a new row in the table) with the phone number for Zola. Again, we see that we use the intended key like an index in square brackets, and we just assign the value to create the entry.

Dictionaries are useful in many situations, and we are familiar with the concept from our daily lives. One important aspect is that they do not only work with strings. Both the key and the entry may be any type we choose.

For example, your contacts app on your phone holds a dictionary that uses the name of a person as a key and the address details as the value. (Here we can see that the value can be more than a single string: it can itself be a complex data type with several types of information). You use it by looking up the name, and it shows you the associated detail (the value).

Another example is a password file on a computer (associating usernames with passwords). Perhaps the most familiar example is an English language dictionary (from which the Python data structure gets its name): The key here is a word, and the value is its definition. We can find a word in the dictionary to look up its definition.

The last thing we need to know is how to create a dictionary. Python dictionaries are written with curly brackets: { }. Writing the curly brackets on their own creates an empty dictionary:

We can also create a dictionary pre-filled with some data using a colon to separate keys and values, and a comma to separate the entries:

Before we go on to use a dictionary in our word counting function, let us do some simple exercises to become more familiar with dictionaries.

Let us now get back to the word counting problem in our thematic-analysis project. We were at the point where we could count the frequency of a single word, and we need to figure out how to count the frequencies of all words. As we have stated above, a dictionary can help with this.

The idea is that we use a dictionary – let’s call it counts – with an entry for each word. We can use the word itself as a key, and use the value for the count (the number of times we have found the word).

For example, a word count of the words in the sentence

would result in the dictionary shown in Figure 8.2.

In other words: we will create a dictionary with one entry for every unique word in our text, where the value holds the number of times each word was used.

It is useful to remind ourselves that this dictionary – if we do a word count for a whole book – will hold thousands of entries. This, however, is not a problem: computers are good at dealing with large amounts of data.

The algorithm we will use can be informally described as follows:

You can modify your existing program to implement the algorithm described above by working step by step through the following exercises.

A version of the project that implements these changes is available fo you to study in the project folder as thematic-analysis-v3.

We have seen that printing out the dictionary using a simple print call works, but the output is not very readable. The list of word counts is very long, and the formatting is not very readable. Let us improve this by implementing a custom printing function for our word count table.

A first attempt at printing out the entries in our dictionary might involve printing each entry on a separate line:

With the above exercise, we can see that this version prints out the words in the dictionary, but not the counts. In other words: it prints the keys, but not the values of the dictionary.

How does this happen? Let us step back and take another look at the methods from the dictionary class that allow us to access its entries. There are three methods that give us lists of the dictionary contents:

The first two of these show us that we can get a list containing just the keys or the values. This means that we could print out either the keys or the values, line by line, using these two loops:

Try this out. What we observed with the last exercise above is the following: If we just iterate over dict, without stating whether we want to get the keys or the values, we will get the keys by default.

Let us look at the third method: items(). This method states that it returns a tuple with the key and value. A tuple is another of Python’s collection types. It can hold multiple arbitrary data items to group them together. We will discuss tuples in more detail in the next chapter; for now it is sufficient to know that it consists of multiple data items and is written using round brackets:

When we think about our dictionary, we can view a dictionary as a list of tuples. The first two entries of the dictionary shown in Figure 8.2, for example, can be regarded as two tuples:

Let us now try to use the items() method in our loop:

Try this out. You will see that this loop prints each entry of the dictionary as a tuple.

This is, so far, the most useful method for us: it allows us to get the full data (key and value), and print them out line by line. But we can get even more control using an additional feature of the for-loop.

If we use a Python for-loop to iterate over a collection, and the item type of the collection is a tuple, then the loop can contain multiple loop variables:

The rules for this type of for-loop are quite straight forward:

Using this variant of the for-loop allows us to get access to each key and value individually, and we are free to format the output ourselves. In the code example above, we have done this using an f-string.

Our output is now more nicely formatted. It is, however, still very long, and identifying frequently used words is still tricky. Change this by doing the following exercises.

The list is now much shorter, but it still contains a lot of very common words, such as "a", "to" and "the" – words that do not tell us much about what the text is concerned with. Let’s do a first attempt at identifying interesting words by dropping all short words.

We can see that we are getting a bit closer. If you run this version of your program on one of the book texts provided, you can see that a list of words slowly emerges that may characterise the book. But the rules we have used to select the words (filtering by a word count that is more than a fixed value, and by length of the word) are not really good enough.

What we need to do instead is not look at the absolute frequency of a word (the number of times it appeared), but look for words that appear more often than they would in a typical English text. The word "the", for example, may appear very often, but it does so in all English writing, so this does not make it an especially interesting word.

This will be our goal in the next section: We will use existing statistics about expected frequencies of words in the English language to identify "over-used" words in our text: words that are used more often in this text than they are used in average English texts.

In our project, we have provided a helper function called get_expected_frequencies(). This function returns a dictionary that lists the expected frequencies for the most common words in an average English language text. (It uses a dictionary of about 64,000 words.)

The dictionary has a word as its key, and the expected frequency as its value (Figure 8.3). The frequency is the ratio of occurrences of the word relative to the length of the text. It can be computed by dividing the word count by the total number of words in a text:

Using this formula, we can easily calculate the frequency of each word in the text we wish to analyse and then compare it to the expected frequency. Those words that are used far more often than expected are the "interesting words" – the words that might give us a clue to what the text is about.

Let us go ahead and implement this idea. We will identify the frequent words, which we define as words with a frequency that is at least 10 times the expected frequency and which were used at least 10 times.

With the code so far, we can print out the dictionary of frequent words. We have, however, no direct control over the exact number of words we will receive. This will depend on the length of the actual text and the words within it.

Let us make one last improvement to specify exactly how many words we wish to see. To do this, we will sort the frequent words by their word count, and then select the top words from this list.

We can do this by making good use of Python’s built-in sorted function:

We can add this statement to our get_overused_words function to create a list of the frequent words, sorted by their word count.

This is quite an advanced use of this function, so let us investigate that line in some detail.

After doing this, we can use a slice to take the first few elements. For example

will give us the 12 most frequent words. We can then return this list from our function.

In making these changes, it is likely that you ran into a number of errors. This is to be expected. We have made many changes, and we will always make errors along the way. It is not the goal to write our code error-free at the first attempt, but to get to know Python well enough that we can identify and fix any errors that occur. Remember, you can refer to Appendix C: Dealing with Errors to get some more information about errors and error messages.

Once you manage to complete these tasks, you will see that you can now get a vague impression what a book is about. You can see that War and Peace is about a prince, a princess, an emperor and Napoleon, or that Three Men in a Boat is about a boat, a picture, locks and rowing.

A version of this project that includes all the changes discussed here is available as thematic-analysis-v4; you may like to compare your own version to this one.

In this chapter, we have learned how to use dictionaries to achieve more advanced functionality.

Dictionaries are perhaps a little more unusual than lists, but they are equally useful. To become a good programmer, it is important to become familiar with dictionaries, and fluent in using them. They should be one tool in your standard toolbox, so that you can use them whenever they can help to solve a problem. You will see that they come in handy more often than you initially expect.