A11: Web Crawling

A search engine has to find the pages of the web before it can rank them. It does so with a web crawler: a program that downloads a page, collects its links, and then downloads the pages that those links lead to, and so on. In this assignment, you’ll crawl a small web site with breadth-first search, store its links in an adjacency matrix, and rank its pages with PageRank.

Prerequisites: Breadth-first search, adjacency matrices, PageRank

Learning Objective: See the web as a graph. You’ll search a graph that isn’t stored anywhere, but is discovered one page at a time, and then compute with the adjacency matrix of the part you found.

Background

The web is a directed graph: each page is a vertex, with an edge to each page it links to. No one has the whole graph. A crawler finds a page’s neighbors only by downloading the page and reading its links, so the web is an implicit graph, as described in the notes on graph search. A breadth-first crawl from a starting page reaches the pages one link away, then the pages two links away, and so on, and it uses its dictionary of reached pages to avoid downloading any page twice.

Once the pages are found, the notes on BFS and PageRank describe how to rank them with PageRank, by following a crowd of visitors who click on links at random. In this assignment, you’ll compute PageRank with NumPy, Python’s most widely used library for computing with arrays of numbers.

Setup

The web site

Crawling a real web site would send many requests to someone else’s server, and the site could change while you work. Instead, you’ll crawl a small site about an imaginary town, Millbrook, served from your own computer.

Download webcrawl.zip and unzip it. It contains a folder millbrook, holding the 25 pages of the site, and the provided file crawl_tools.py. Write your code in a file crawl.py, in the same folder as crawl_tools.py.

To serve the site, open a terminal, change to the millbrook folder, and start Python’s built-in web server:

cd webcrawl/millbrook
python3 -m http.server 8000

(On Windows, use py instead of python3.) The server prints a line for each page it sends, and keeps running until you stop it with Control-C. While it is running, open http://localhost:8000/index.html in a web browser to see the site’s home page. The name localhost always means your own computer, so no request leaves it. Leave the server running in its own terminal while you work, and run your code in a second terminal or in your editor.

NumPy

NumPy isn’t part of Python itself. Install it with pip, as you installed PyQt6 for A8:

• Mac: in the Terminal, run python3 -m pip install numpy.
• Windows: in the Command Prompt, run py -m pip install numpy.

The notes there, on installing into the Python that your editor uses and on the “externally-managed-environment” error, apply to NumPy too. To check that NumPy is installed, run python3 -c "import numpy". If it prints nothing, NumPy is ready.

Provided code

crawl_tools.py provides one function, get_links(url), which downloads the page at url and returns a list of the pages it links to:

from crawl_tools import get_links

print(get_links("http://localhost:8000/fishing.html"))

prints

['http://localhost:8000/river.html', 'http://localhost:8000/index.html']

Each link is a full URL, even though the page itself writes the first link as river.html. Each link appears once, and links that aren’t to web pages, such as email addresses, are left out. If the page can’t be downloaded, for example because it doesn’t exist, get_links prints a message and returns None.

Before each download, get_links waits 0.1 seconds. The wait imitates the delay of a real network, where each download takes a noticeable amount of time; it also keeps your crawler from flooding a real server with requests, if you ever point it at one. Crawling the whole Millbrook site takes a few seconds.

Part 1: Crawl

Write a function crawl(start, max_pages) that crawls the site with breadth-first search, starting from the URL start, and returns two dictionaries:

• distance, which maps each page reached to its distance from start: the smallest number of clicks needed to get there.
• links, which maps each page that was successfully downloaded to the list of its links that lead to pages on the site.

Start the crawl at http://localhost:8000/index.html. (The URL http://localhost:8000/ leads to the same page, but it is a different string, so a crawler would treat it as a different page.) Your crawl should follow these rules:

• Follow only the links that lead to pages on the site: those whose URLs start with http://localhost:8000/. One page links to a page on another web site, and your crawler must not follow that link.
• If get_links returns None for a page, skip that page: don’t add it to links, and don’t look for its neighbors.
• Stop after downloading max_pages pages, even if the queue isn’t empty. On the Millbrook site, a limit of 100 is far more than enough; the limit is a safeguard against a crawl that runs much longer than you expect.

Print each page that the crawl reached, with its distance.

Expected results. Your crawl should download 24 pages successfully, and get_links should print one message, about old_newsletter.html, which is linked from the events page but doesn’t exist. The distance dictionary should have 25 pages, including old_newsletter.html, and the links dictionary 24. By distance from the home page:

Distance	Pages
0	1: index.html
1	8: history, town_hall, bakery, library, park, events, map, about
2	13, including old_newsletter
3	2: legend, fishing
4	1: ghost

If your crawl reaches a page on another web site, check that it follows only links that start with http://localhost:8000/. If it takes much longer than a few seconds, check that it never downloads a page twice.

Part 2: The adjacency matrix

A short guide to NumPy

A NumPy array is like a list of numbers, or a list of lists of numbers, but its operations are written much more compactly, and run much faster. These are all you’ll need:

import numpy as np

A = np.zeros((3, 3))         # a 3-by-3 array of zeros, like [[0, 0, 0], [0, 0, 0], [0, 0, 0]]
A[0, 2] = 1                  # the entry in row 0, column 2; with lists, this would be A[0][2]
A[1]                         # row 1, as an array of 3 numbers
A.sum(axis=1)                # the sum of each row, as an array of 3 numbers
A.sum(axis=0)                # the sum of each column
x = np.full(3, 1 / 3)        # an array of 3 numbers, each equal to 1/3
A.T                          # the transpose of A: entry [i, j] is A[j, i]
A.T @ x                      # the product of the matrix A.T and the vector x
x / 2, x + 1, x - y          # arithmetic on arrays works entry by entry
np.abs(x - y).max()          # the largest difference between corresponding entries of x and y
np.argsort(x)                # the indices of x's entries, in order from smallest to largest entry

The product A.T @ x is the operation $A^T x$ described in the notes on BFS and PageRank: entry $j$ of the result is row $j$ of $A^T$ dotted with $x$, which is $\sum_i A_{ij} x_i$. NumPy computes it much faster than matrix_times_vector in the notes, but the result is the same.

Building the matrix

Write a function build_matrix(links) that numbers the downloaded pages and builds the site’s adjacency matrix. It should return two values: a list pages of the downloaded pages, so that page i is pages[i], and a NumPy array A, whose entry A[i, j] is 1 if page i links to page j, and 0 otherwise. Include only the pages in links, and skip any link to a page that isn’t one of them. (A dictionary that maps each page to its number will help.)

Expected results. The matrix should be 24 by 24, with 80 entries equal to 1, and none on the diagonal, since no page links to itself. Its column sums give each page’s number of incoming links: the home page should have 20, more than any other page, and the bakery should come next, with 6. Its row sums give each page’s number of outgoing links: exactly one row, for apple_pie.html, should add up to 0.

Part 3: PageRank

Write a function pagerank(A, jump) that computes the PageRank of each page from the adjacency matrix A, as in the notes, but using NumPy:

1. Build the transition matrix, by dividing each row of A by its sum, the page’s number of outgoing links. A page with no outgoing links gets a row in which every entry is $1/n$, as if it linked to every page.
2. Start with the crowd spread evenly, with $1/n$ of it on each page.
3. At each click, compute the product of the transpose of the transition matrix and the crowd, T.T @ x in NumPy, where T.T is the transpose of T. Then, to account for the visitors who jump to a random page, multiply the result by 1 - jump and add jump / n to each entry. In the notes, jump is 0.15.
4. Stop when no entry changes by more than $10^{-10}$ from one click to the next, or after 1000 clicks, whichever comes first.

Return the array of fractions. Run pagerank with jump equal to 0.15, and print the pages in order from the highest rank to the lowest, with each page’s rank and its distance from the home page.

Expected results. The fractions should add up to 1, and should stop changing after about 116 clicks. Rounded to three decimal places, the ranks should be as follows. (Pages whose ranks round to the same value may appear in either order.)

Rank	Page	PageRank	Distance	Incoming links
1	index.html	0.171	0	20
2	map.html	0.065	1	5
3	legend.html	0.062	3	2
4	bakery.html	0.062	1	6
5	ghost.html	0.060	4	1
6	history.html	0.051	1	5
7	park.html	0.051	1	4
8	library.html	0.049	1	4
9	town_hall.html	0.049	1	5
10	festival.html	0.046	2	4
11	events.html	0.039	1	3
12	river.html	0.039	2	4
13	about.html	0.034	1	2
14	diner.html	0.028	2	2
15	book_club.html	0.026	2	2
16	bridge.html	0.024	2	2
17	mayor.html	0.022	2	2
18	contact.html	0.022	2	1
19	apple_pie.html	0.020	2	1
20	council.html	0.017	2	1
21	founders.html	0.016	2	1
22	mill.html	0.016	2	1
23	farmers_market.html	0.015	2	1
24	fishing.html	0.015	3	1

If your fractions don’t add up to 1, check the rows for pages with no outgoing links. If the home page doesn’t rank first, check that your transition matrix divides each row by its sum, and that you multiply the crowd by the transpose, as T.T @ x.

Part 4: Experiments

Answer these questions in a comment at the end of crawl.py:

• The site has 25 pages, but your crawl downloaded only 24. Which page is missing, and why didn’t your crawl find it? How would someone visiting the site in a browser find it?
• The pages legend.html and ghost.html have only two incoming links and one incoming link, but they rank among the top five pages. Look at their links, and explain why.
• Run pagerank again with jump equal to 0, so that no one ever jumps to a random page. Where does the crowd end up? Does it ever settle down? Print the fractions after 1000 and after 1001 clicks, and explain what you see. (The fractions should still add up to 1, and the loop should run all 1000 clicks without stopping early.)
• The page apple_pie.html has no outgoing links. What does PageRank assume about the visitors who reach it? What would happen to the total of the fractions if it didn’t?
• Is a page’s rank related to its distance from the home page? Give an example from your results of a page that is close to the home page but ranks low, or one that is far from the home page but ranks high, and explain it.

What to do

1. Download and unzip the starter files, install NumPy, and start the web server.
2. Write crawl, build_matrix, and pagerank in crawl.py, as described above.
3. Run the crawl from the home page, build the matrix, and print the pages ranked by PageRank, with jump equal to 0.15 and then 0.
4. Answer the questions in Part 4 in a comment at the end of your file.

Feedback

At the top of your main Python file, add a comment that answers these questions:

• About how long did the assignment take you?
• What was hard?
• What was easy?
• Was anything in the assignment unclear?

We use your answers to improve the assignment.

Grading and requirements

• crawl performs a breadth-first search from the starting page, using a queue and a dictionary of reached pages, so that it downloads each page at most once and records each page’s correct distance from the start.
• crawl follows only links to pages on the site, skips pages that can’t be downloaded, and stops after max_pages downloads.
• build_matrix returns the list of pages and a NumPy adjacency matrix in which A[i, j] is 1 exactly when page i links to page j.
• pagerank builds the transition matrix correctly, including the rows for pages with no outgoing links, and uses @ to move the crowd forward at each click, stopping when the fractions stop changing.
• The output shows each page with its distance, and the pages ranked by PageRank with jump equal to 0.15 and 0.
• The answers to the questions in Part 4 explain the results in terms of the site’s links.

What to submit

Upload crawl.py to Canvas as your Assignment 11 submission. You don’t need to submit the Millbrook site.

Challenge problems (no extra points, not required)

• Crawl a real site. Point your crawler at a real web site, with max_pages set to 20 or less. Before you do, read the site’s robots.txt file (for example, https://example.com/robots.txt), which says which pages the site’s owners allow crawlers to download, and respect it. Keep the delay in get_links. Which pages rank highest, and why?
• Timing the product. Time the matrix_times_vector function from the notes, which takes one dot product for each row, against NumPy’s @, multiplying a 300-by-300 matrix of random numbers (np.random.rand(300, 300)) by a vector of 300 random numbers (np.random.rand(300)). How many times faster is NumPy?