A1: Document Similarity

How would you write a program that decides whether two documents are about the same thing, without understanding a word of either one? Search engines, plagiarism detectors, and recommendation systems all need to answer some version of this question, usually on far too many documents for a human to read them all.

One surprisingly effective approach ignores grammar, word order, and meaning entirely, and just looks at which words appear, and how often. This is called the bag of words model: a document is represented as a “bag” (an unordered collection, with repeats) of its words. Two documents that use similar words in similar proportions are probably about similar things, even if you shuffled every sentence into nonsense.

A “bag” is just a dictionary that maps each word to how many times it occurs in a document.

Python IDE and interpreter

If you used PyCharm and the Python interpreter from CS 1, you can continue using that same setup. It would also be fine to use a different editor, e.g. VS Code.

What to do

Think of each word-count dictionary as a vector, with one coordinate per word. Two documents that use words in similar proportions have vectors pointing in similar directions, whatever their lengths, so the cosine of the angle between the vectors measures how similar the documents are: 1 for vectors pointing the same way, and 0 for documents that share no words. The dot product $A \cdot B$ is the sum, over the words in both documents, of the product of their counts, and the magnitude $\lVert A \rVert$ is the square root of the sum of the squares of $A$‘s counts:

$\text{cosine\_similarity}(A, B) = \frac{A \cdot B}{\lVert A \rVert \, \lVert B \rVert}$

Write the following functions:

• build_wordbag(filename): reads a text file and returns a dictionary mapping each word to the number of times it appears. You’ll need to lowercase words and strip punctuation so that “Cat” and “cat,” are counted as the same word. Decide for yourself whether to filter out very short or common words (like “a” or “the”), and be ready to explain your choice.

  This is a good place to use split (to break a line into words) and strip (to remove punctuation from each word). There’s also a string method for lowercasing, lower(), which you’ll need for the “Cat” vs. “cat,” comparison above.

• dot_product(bag1, bag2): returns $A \cdot B$ for two word-count dictionaries.

  Hint: it might seem like we need to consider words that appear in bag1 but not bag2, and vice versa. But we don’t, since if a word appears in only one of the dictionaries, that word contributes 0 to the similarity measure. So it is sufficient to loop over words that appear as keys in bag1, and just check if those words also appear in bag2.

• magnitude(bag): returns $\lVert A \rVert$ for a word-count dictionary.

• cosine_similarity(bag1, bag2): returns the cosine similarity of two word-count dictionaries, using the two functions above.

• most_similar(filename, candidate_filenames): given one document and a list of other documents, return whichever candidate is most similar to it (by cosine similarity of their word bags).

Test cosine_similarity on small, hand-built dictionaries before trying it on real files, so that you can predict the answer yourself and check your function against it:

print(cosine_similarity({"cat": 2, "dog": 1}, {"cat": 1, "dog": 1, "bird": 3}))
print(cosine_similarity({"cat": 1}, {"dog": 1}))              # should print 0.0 -- no shared words
print(cosine_similarity({"cat": 3, "dog": 1}, {"cat": 6, "dog": 2}))  # should print 1.0 -- same proportions

Then try it on real documents. constitution.txt is one ready-made example; find a couple of others to compare against it — for instance, a news article, a different historical document, or a second copy of the Constitution with a few words changed — and see whether the similarity scores match your intuition.

Comparing many documents

Once you can compare two documents, you can compare many documents at once, which is where the “clustering” part of document similarity comes in.

Gather at least four or five short text documents — a mix of similar and dissimilar ones works best for testing (for instance, two documents on the same topic and two or three on unrelated topics). For each document, use most_similar to find its best match among the others, and print the results. Do the pairings match what you’d expect just from reading the documents yourself?

Extension (not required)

Group the documents into clusters, where every document in a cluster is more similar to something else in the cluster than to anything outside it. (There are principled ways to do this — you’ll see some in later courses — but even a simple greedy approach is an interesting exercise here.)

Feedback

At the top of your main Python file, add a comment that answers these questions:

• About how long did the assignment take you?
• What was hard?
• What was easy?
• Was anything in the assignment unclear?

We use your answers to improve the assignment.

Grading and requirements

• build_wordbag counts words correctly, lowercasing them and stripping punctuation so that “Cat” and “cat,” count as the same word. A comment explains your choice about whether to filter out short or common words.
• dot_product, magnitude, and cosine_similarity compute the formulas above, and cosine_similarity is written using the other two.
• most_similar returns the candidate whose word bag has the highest cosine similarity to the given document’s.
• Test code checks cosine_similarity on small, hand-built dictionaries, including the three above, before using it on real files.
• Test code compares at least four or five documents with most_similar and prints the best match for each, with a comment saying whether the pairings match what you expected from reading the documents.

What to submit

Zip your Python file together with any text files you used, and upload the zip file to Canvas as your Assignment 1 submission.