Analysis of data structures

You’ve now seen two implementations of two ADTs: a stack built on a growing array, and a queue built on a linked list. Here we compare them directly, so you can reason about why each implementation was built the way it was — and so you’re ready to measure it yourself.

The first part of this page states the costs of the two data structures. The rest of the page shows where such costs come from: how to analyze the running time of a piece of code, loop by loop, and how to summarize the result with big-O, big-Ω, and big-Θ notation. The analysis of loops and of searching is a review of material from CS 1; the amortized analysis of a growing array near the end is new.

Describing cost: big-O

We describe an operation’s cost using $O(\cdot)$ (“big-O”) notation, which describes how the number of steps grows as the size of the data, $n$, grows: $O(1)$ (constant — doesn’t grow with $n$), $O(\log n)$, $O(n)$ (grows proportionally with $n$), and so on. The costs in this first part are worst-case costs unless they say otherwise, and they use big-O loosely. The second half of the page makes both ideas precise: best and worst cases, and big-O as an upper bound.

The growing array

• data[i] (indexing): $O(1)$ — constant time. A growing array is one contiguous block of memory; the address of element $i$ is computed directly from $i$, with no searching needed.
• data.append(value): $O(1)$ amortized. Most of the time, there’s spare room at the end of the underlying array, and appending is just writing into the next slot — $O(1)$. Occasionally, the array is full, and Python allocates a new, larger array and copies every element over — an $O(n)$ operation. But Python grows the array geometrically (multiplying its capacity by a constant factor each time it resizes, rather than adding a fixed number of slots), so those expensive resizes happen less and less often as the list grows. Spread out (“amortized”) over many appends, the average cost per append works out to $O(1)$. Amortized analysis, near the end of this page, shows why.
• data.pop() (from the end): $O(1)$ — no resizing, no shifting; just remove the last element.
• data.pop(0) or data.insert(0, value) (from the front): $O(n)$ — every other element has to shift over by one position to close (or open) the gap.

The linked list

• Adding or removing at the front or back (given a reference to that end, as LinkedQueue keeps in self.front and self.back): $O(1)$ — just re-point a couple of references, no shifting.
• Indexing, data[i]: $O(n)$ in the worst case — there’s no way to jump directly to node $i$; you have to follow $i$ links from the front. (The best case, $i = 0$, is $O(1)$.)
• Adding or removing in the middle, given a reference to the right spot: $O(1)$ to actually splice in or out — but $O(n)$ to find that spot in the first place, since finding it means walking the list from one end.

Putting the comparison side by side

Operation (worst case)	Growing array	Linked list
Add / remove at the end	$O(1)$ amortized	$O(1)$
Add / remove at the front	$O(n)$	$O(1)$
Access by index	$O(1)$	$O(n)$

This is why a growing array makes a great stack (all the action is at one end — the end where arrays are fast) but a poor queue (queues need fast access at both ends), and why a linked list makes a great queue but a poor choice if you need fast random access by index.

Neither implementation is better in general — they’re better at different things. Picking a data structure means matching its strengths to the operations your problem actually needs.

Analyzing running time

The costs above were stated, not derived. The rest of this page shows how to derive them, and costs like them, from code. Whatever the problem, we are concerned with two aspects of an algorithm:

• Correctness: showing that the algorithm produces the correct answer.
• Efficiency: understanding how much of some resource the algorithm uses. The resource we analyze most often is time.

The input size n, and the steps of an analysis

Running times are written as functions of $n$, where $n$ stands for whatever characteristic of the input determines the running time. It might be the length of a list passed to the function, the value of a numeric parameter, or the number of nodes in a linked list. Writing $n$ lets us talk about rates of growth in general terms, but before discussing a running time, always know what $n$ refers to.

Given an algorithm or a piece of code, you should be able to:

1. Identify the quantity $n$ that the running time depends on.
2. Decide which case to analyze, the best case or the worst case, and describe the inputs of size $n$ that produce it.
3. Compute the running time of that case in terms of constants and $n$, section by section.
4. Only then, describe that running time using big-O, big-Ω, or big-Θ notation, and say which case the bound is for.

Steps 2 and 4 answer different questions. Step 2 chooses which function of $n$ we are talking about. Step 4 describes how fast that function grows. Keeping them apart avoids a common mistake: thinking that big-O means “worst case.” It does not, as the section on cases and bounds near the end explains. The rest of this page follows these steps in order.

Run-time analysis of linear search

The method for finding a running time starts from code. Here is linear search, which checks the items of a list one at a time, starting at the front:

def linear_search(the_list, key):
    index = 0
    while index < len(the_list):
        if key == the_list[index]:
            return index
        else:
            index += 1

    return None

Let the_list have length $n$. The method is to divide the code into sections, where every line in a section runs the same number of times, give each section a constant for the time it takes to run once, count how many times each section runs, and add up the products. How many times the loop runs depends on where the key is, so start with the slowest possibility, the worst case: the key is not in the list at all. (The next section defines the worst case precisely.)

• First, Python creates the local variable index. That is fast, and it happens only once, but we account for it anyway: call its time $c_1$, a constant that does not depend in any way on $n$.
• Next, the while loop. In the worst case, the key is not in the list at all, and the body of the loop runs $n$ times. It is reasonable to assume that each iteration, the loop test plus the body, takes a constant amount of time (or at most a constant); call it $c_2$. The loop costs $c_2 n$ in total.
• Finally, in the worst case the loop test is made one last time, when index equals len(the_list), and comes up False. That is the $(n+1)$st test, since the first $n$ tests were for index equal to $0, 1, \ldots, n-1$. Then the function returns None. The last test and the return together take some constant time; call it $c_3$ (again, independent of $n$).

Adding up the sections, the worst-case running time of linear search is

$c_1 + c_2 n + c_3 = c_2 n + (c_1 + c_3).$

Best case, worst case, and average case

The running time of an algorithm usually depends on the size of its input: searching a list of a million items takes longer than searching a list of ten. But it also depends on which input of that size the algorithm gets. Linear search, above, checks the items of a list one at a time. If the key is at index 0, it finishes after one check. If the key is not in the list, it has to check every item before it can report that. Two inputs of the same size can take different amounts of time.

So for each size $n$, there is not one running time but many: one for each input of size $n$. To get a single function of $n$ to analyze, we choose which input of each size to consider:

• The worst-case running time is, for each $n$, the longest running time of any input of size $n$.
• The best-case running time is, for each $n$, the shortest running time of any input of size $n$.
• The average-case running time is, for each $n$, the average running time over all inputs of size $n$. Computing it requires knowing how likely each input is, and we won’t use it on this page.

The figure below shows the idea for linear search. Each dot is one input: a list of length $n$ with the key at some index, or with the key missing. For each $n$, the dots form a column. The worst-case running time is the top dot of each column, and the best-case running time is the bottom dot.

Two points about these definitions are easy to get wrong:

• The best case is not a small input. The best case is the best input of each size $n$. The best case of linear search is a list of any length whose first item is the key. It is not a list of length 1.
• Each case is a function of $n$. In the figure, the worst case of linear search grows with $n$, and the best case does not. Both are functions that we can analyze, and later we will describe each of them with big-O or big-Θ notation.

The worst case is usually of most interest, because it is a guarantee: “this algorithm is fast enough, as long as the list is not too long” is useful information. “This algorithm might be fast, if you are lucky and give it the right input” is much less useful. The best case is still worth knowing: it tells you what the algorithm does on favorable inputs, and for some algorithms, such as insertion sort below, favorable inputs are common in practice.

Sometimes the best case and the worst case are the same. To find the smallest item in an unsorted list of length $n$, the obvious (and best) algorithm looks at every item, keeping track of the smallest seen so far. Even if the smallest item happens to be first, you can’t know that it’s the smallest until you’ve looked at all the others. Every input of size $n$ takes the same time.

And some running times do not depend on $n$ at all: returning a randomly selected item from a list, or the smallest item of a sorted list, takes the same time however long the list is.

Best-case running time of linear search

The best case uses the same sections with different counts. When the key is at index 0, section $c_1$ runs once, section $c_2$ runs once (the first iteration finds the key and returns), and section $c_3$ does not run at all. The best-case running time is

$c_1 + c_2,$

a constant that does not depend on $n$. These two running times are the two lines in the figure in the previous section.

The values of $c_1$, $c_2$, and $c_3$ depend on the speed of the computer and the efficiency of the Python implementation. The form of each expression does not: in the worst case, a constant times $n$ plus a constant; in the best case, a constant.

Linear search vs. binary search

One of the most basic problems in computer science is finding the index of an item in a list. Suppose the list is sorted, so that binary search is an option: a list of country names, sorted alphabetically.

countries = ["Afghanistan", "Albania", "Algeria", "Andorra", "Angola",
             "Anguilla", "Argentina", "Armenia", "Aruba", ..., "Zimbabwe"]

(The ... stands for the rest of the country names; assume that every country is in the list.) To find the index of a country, such as "Yemen" or "Afghanistan", there are at least three approaches:

• Random search. Pick a random index. Check whether that location contains the item you are looking for. If not, repeat.
• Linear search. Loop over the items in the list, starting at index 0, then index 1, index 2, and so on, until you either find the item or run out of list.
• Binary search. This approach requires the list to be sorted. Keep track of the indices bounding the sublist still under consideration. If the sublist is ever empty, the item is not in the list. Otherwise, check the item at the midpoint of the sublist. If it is the item, return its index. If not, compare: if the midpoint item is greater than the item sought, discard the second half of the sublist; if smaller, discard the first half. Repeat with the half that remains.

Best and worst cases for the three searches

Which method finds the item most quickly? It depends on the input. Consider the best and worst inputs for each approach.

• Random search. In the best case, the first index picked holds the item, and the time taken is the time to examine a single item. In the worst case, random indices are picked forever, and the item is never found. (The probability of this outcome approaches 0 as time goes on, but probabilistic analysis of running time is beyond this page.) And if the item is not in the list at all, random search never discovers that fact.
• Linear search. In the best case, the item is first in the list, as in the best case for random search. In the worst case, the item is at the end of the list or not present at all. If the list contains $n$ items, all $n$ must be examined.
• Binary search. In the best case, the item is at the midpoint of the whole list, and it is found in the first place we look. In the worst case, the algorithm halves the list over and over until the remaining sublist is empty. A list of length $n$ can be halved about $\log_2 n + 1$ times before the sublist is empty, and this number is much less than $n$ for large $n$.

All three have the same best case: one look. They differ in the worst case, and the rest of this section makes the worst cases of linear and binary search precise.

Review of logarithms

Binary search is an example of repeated halving. We divide the list in half. The item cannot be in one of the halves, so we discard that half entirely; the item might be in the other half, so we divide that half in half, and so on, until we either find the item or the sublist is empty. The worst-case running time therefore depends on how many times a list can be halved before the sublist is empty. The answer is the base-2 logarithm of the length of the list.

Halve a number $n$, getting $n/2$. Halve it again, getting $n/4$, and again, getting $n/8$. Continue until the number is less than 1. A sublist cannot contain half an item, so a sublist size less than 1 must be 0, and to reach size 0, the sublist must have had size 1 the time before. So consider this question:

How many times do we halve $n$ until we get down to 1?

To answer it, try the opposite operation: start with 1 and keep doubling until reaching $n$. Doubling undoes halving, so the number of doublings needed to get from 1 up to $n$ is the same as the number of halvings needed to get from $n$ down to 1.

Doublings	Result
0	1
1	2
2	4
3	8
4	16
5	32
6	64

Each doubling multiplies by another factor of 2, so after $x$ doublings the result is $2^x$. The number of doublings needed to reach $n$ is therefore the value of $x$ for which $2^x = n$. That number is $\log_2 n$: the exponent to which 2 must be raised to get $n$.

$2^x = n$ if and only if $\log_2 n = x$.

Starting at 1 and doubling $\log_2 n$ times gives $n$; so starting at $n$ and halving $\log_2 n$ times gives 1. This argument assumes that $n$ is a power of 2. When it isn’t, the count is rounded, a detail the notation introduced later lets us ignore. It also counts halvings down to 1, not to 0, which is what the worst case of binary search (item not found) needs; but once the sublist has size 1, the next halving gives size 0, so the number of halvings to reach 0 is one more: $\log_2 n + 1$.

When computer scientists write logarithms, they almost always mean base-2 logarithms.

Run-time analysis of binary search

Binary search is analyzed the same way. Here it is written with a loop: left and right are the indices at the two ends of the sublist still under consideration, and each iteration of the loop discards half of that sublist.

def binary_search(the_list, key):
    left = 0
    right = len(the_list) - 1
    while left <= right:
        midpoint = (left + right) // 2
        if the_list[midpoint] == key:
            return midpoint             # found it
        elif key < the_list[midpoint]:
            right = midpoint - 1        # discard the second half
        else:
            left = midpoint + 1         # discard the first half
    return None                         # empty sublist: not found

• Setting up left and right happens once; call its time $c_5$.
• One iteration of the loop tests whether the sublist is empty, computes the midpoint, compares the key with the item at the midpoint, and moves left or right. Each of these steps takes constant time; call the total for one iteration $c_4$. In the worst case, the key is not in the list, and the loop runs once for each sublist size from $n$ down through 1: $\log_2 n + 1$ times, as the halving figure above shows.
• Finally, the loop test fails, when the sublist is empty, and the function returns None. Call the time for these $c_6$.

So the worst-case running time of binary search is

$c_4 (\log_2 n + 1) + c_5 + c_6 = c_4 \log_2 n + (c_4 + c_5 + c_6).$

In the best case, the key is at the midpoint of the whole list. The setup runs once, the loop runs once and returns, and the final section does not run, for a best-case running time of $c_5 + c_4$, a constant.

(Binary search can also be written recursively, as it was in CS 1, and the running time is the same.)

Run-time analysis of find_node on a linked list

The same method applies to operations on data structures. Here is find_node, a method of a circular, doubly linked list with a sentinel. It walks the list from the sentinel and returns the first node holding value, or None. Let $n$ be the number of items in the list.

This is linear search on a linked list, and the analysis has the same shape. In the best case, the first node holds the value: the loop runs once and returns, for a running time of $c_1 + c_2$. In the worst case, no node holds the value: the loop runs once for each of the $n$ nodes, and then the loop test fails at the sentinel, for $c_2 n + (c_1 + c_3)$. The walk is what costs time; any linked-list operation that has to find a node by value or by index, starting from one end, has the same worst case.

Nested loops and constant time

In some code, the counts do not depend on which input of size $n$ is given, and the best and worst cases are the same. Here $n$ is a number passed to the function:

def do_something(n):
    for i in range(n):
        for j in range(n):
            print(i, j)

The inner loop’s body, the print, takes some constant time. How many times does it run? For each of the $n$ iterations of the outer loop, the inner loop iterates $n$ times, so $n \cdot n = n^2$ times in all. The outer loop’s own step runs only $n$ times. The running time is $c_2 n^2 + c_1 n$, for every call with argument $n$.

At the other extreme, some code takes the same time regardless of its input:

def do_something_else(n):
    print("My name is Inigo Montoya")

Its running time is a constant $c$, whatever $n$ is.

Run-time analysis of insertion sort

In linear search and binary search, the best case is a constant: the key is found on the first look. Insertion sort, from CS 1, is an example where the best and worst cases are different and both grow with $n$.

def insertion_sort(the_list):
    n = len(the_list)
    for i in range(1, n):
        key = the_list[i]           # the value to insert
        j = i - 1                   # look to the left of position i
        while j >= 0 and the_list[j] > key:
            the_list[j + 1] = the_list[j]
            j -= 1
        the_list[j + 1] = key

Only the while loop’s section, $c_3$, runs a number of times that depends on the values in the list, not only on $n$.

Each iteration of the outer loop inserts one value into the sorted part of the list, but it does not work like inserting into a growing array. An array insertion needs a free slot at the end, and possibly a doubling, before it can shift values right to open a gap at a known index. Insertion sort needs neither. When it copies key = the_list[i] out of the list, position i becomes a hole, right next to the sorted part. Each iteration of the while loop copies one value one slot right, into the hole, which moves the hole one slot left. When the loop stops, key is written into the hole:

The hole also combines two jobs that an array insertion does separately: finding the position and shifting. Insertion sort does not know in advance where key belongs. It finds out by walking left from the end of the sorted part, and every comparison that does not stop the loop pays for one shift. Each shift is a single assignment, not a three-assignment swap.

In the worst case, the list starts out sorted in reverse order, and each item has to move all the way to the left: the item at position 1 moves 1 place, the item at position 2 moves 2 places, and so on, up to $n-1$ places. The total number of moves is $1 + 2 + \cdots + (n-1)$. To add this up, write the sum forward and then backward, and add the two rows column by column:

$\begin{array}{rcccccc} & 1 & + & 2 & + \cdots + & (n-1) \\ + & (n-1) & + & (n-2) & + \cdots + & 1 \\ \hline & n & + & n & + \cdots + & n \end{array}$

Each of the $n - 1$ columns adds up to $n$, so twice the sum is $n(n-1)$, and the sum is $\dfrac{n(n-1)}{2}$. The worst-case running time is

$c_3 \frac{n(n-1)}{2} + c_2 (n - 1) + c_1.$

In the best case, the list starts out already sorted. Each time the while loop is reached, the_list[j] is already less than or equal to key, so the loop body runs 0 times, and the best-case running time is

$c_2 (n - 1) + c_1.$

The best case here is a list of any length $n$ that happens to be sorted, not a short list. And it is a realistic input: lists that are already sorted, or nearly sorted, come up often in practice.

Asymptotic analysis of functions

We want to compare running times: is binary search faster than linear search, and by how much? Every analysis so far ended with a function of $n$ with constants in it, one for each case. (The constants belong to each analysis separately: $c_2$ in one row is not the same constant as $c_2$ in another.)

Algorithm	Best-case running time	Worst-case running time
Linear search	$c_1 + c_2$	$c_2 n + (c_1 + c_3)$
Binary search	$c_4 + c_5$	$c_4 \log_2 n + (c_4 + c_5 + c_6)$
find_node	$c_1 + c_2$	$c_2 n + (c_1 + c_3)$
do_something	$c_2 n^2 + c_1 n$	$c_2 n^2 + c_1 n$
do_something_else	$c$	$c$
Insertion sort	$c_2 (n-1) + c_1$	$c_3 \frac{n(n-1)}{2} + c_2(n-1) + c_1$

These functions are cumbersome to compare, and the constants depend on the computer and the Python implementation. As the next example shows, once $n$ is large enough, the constants matter much less than how fast each function grows. Asymptotic analysis compares functions by their growth as $n$ grows large, ignoring constant factors. It applies to any function of $n$: a best case, a worst case, or a function that has nothing to do with running time.

Constants matter less than growth

Which is faster, linear search or binary search? It depends on the input (linear search finds "Afghanistan" faster than binary search does, in the list above), on the length of the list, and on the constants. For the worst case, the question is: when is

$c_4 \log_2 n + (c_4 + c_5 + c_6) \quad\text{smaller than}\quad c_2 n + (c_1 + c_3)\,?$

The claim is that, whatever the constants, binary search is faster for large enough $n$. Suppose linear search runs on a blazingly fast, ultra-modern HAL 9000 that executes each iteration of the while loop in 0.00001 seconds, so $c_2 = 0.00001$, and let $c_1 + c_3 = 0$. Binary search runs on a vintage Apple II, which takes 0.01 seconds per iteration of its loop, so $c_4 = 0.01$: a thousand times slower per step. Let $c_5 + c_6 = 0$ as well.

$n$	$\log_2 n$	HAL 9000, linear search: $0.00001\,n$	Apple II, binary search: $0.01(\log_2 n + 1)$
1,024	10	0.01 s	0.11 s
16,384	14	0.16 s	0.15 s
1,000,000	about 20	10 s	about 0.21 s
1,000,000,000	about 30	about 2.8 hours	about 0.31 s

At $n = 16{,}384 = 2^{14}$, the Apple II already wins. It might look as though the constants were rigged to make that happen, but lengthening the list makes the point: no difference between $c_2$ and $c_4$ can overcome the Apple II’s advantage of a $\log_2 n$ factor where the HAL 9000 has a factor of $n$.

The two functions grow in different ways. Linear search’s worst case grows linearly: if the list doubles in length, the running time about doubles. Binary search’s worst case grows logarithmically: if the list doubles in length from $n$ to $2n$, the loop runs one more time, and the running time grows by only $c_4$. With equations,

$c_4 \log_2 (2n) = c_4 \log_2 n + c_4 \log_2 2 = c_4 \log_2 n + c_4,$

since $2^1 = 2$ means $\log_2 2 = 1$. The function $\log_2 n$ grows much more slowly than $n$, and for large enough $n$ that difference outweighs any difference in the constants.

The three notations below, big-O, big-Ω, and big-Θ, make “grows like” precise. Each compares a function $f(n)$, such as a running time, with a simpler function $g(n)$, such as $n$ or $n^2$.

Big-O: an upper bound

$f(n)$ is $O(g(n))$ (pronounced “big-O of $g(n)$”) if, for large enough $n$, $f(n)$ is at most a constant times $g(n)$. Precisely: there are positive constants $c$ and $n_0$ such that

$f(n) \le c \cdot g(n) \quad \text{for every } n \ge n_0.$

The constant $c$ is what lets big-O ignore constant factors, and $n_0$ is what lets it ignore small $n$. In the figure, $g(n) = n^2$ and $c = 2$. The wiggly function $f(n)$ crosses $2n^2$ several times while $n$ is small, but after about $n = 8.2$ it never goes back above. With $n_0 = 9$, $f(n) \le 2n^2$ for every $n \ge n_0$, so $f(n)$ is $O(n^2)$:

For example, the worst-case running time of linear search, $c_2 n + (c_1 + c_3)$, is $O(n)$: for every $n \ge 1$, it is at most $c_2 n + (c_1 + c_3) n = (c_1 + c_2 + c_3) n$. The worst-case running time of binary search is $O(\log_2 n)$. The best-case running time of linear search, $c_1 + c_2$, is a constant, at most $(c_1 + c_2) \cdot 1$, so it is $O(1)$.

When a function is a sum, such as

$c_1 n^2 + c_2 n + c_3,$

only the fastest-growing term matters: it is $O(n^2)$. The definition shows why. For every $n \ge 1$, $n \le n^2$ and $1 \le n^2$, so each lower-order term is at most its coefficient times $n^2$:

$c_1 n^2 + c_2 n + c_3 \le c_1 n^2 + c_2 n^2 + c_3 n^2 = (c_1 + c_2 + c_3)\, n^2.$

Choosing $c = c_1 + c_2 + c_3$ and $n_0 = 1$ satisfies the definition. The constant $c$ absorbs every coefficient, which is why both the lower-order terms and the leading constant can be dropped. So the running time of do_something, $c_2 n^2 + c_1 n$, is $O(n^2)$, and the running time of do_something_else, a constant $c$, is $O(1)$.

The base of a logarithm doesn’t matter either, because for any constants $a$ and $b$,

$\log_a n = \frac{\log_b n}{\log_b a}.$

Logarithms to base $a$ and base $b$ differ only by the constant factor $\log_b a$, and constant factors don’t matter in big-O notation. So the worst case of binary search is written $O(\log n)$.

Big-O is only an upper bound. It says that a function grows no faster than $g(n)$, and nothing about how much slower it might grow. Any function that is $O(\log n)$ is also $O(n)$, and also $O(n^2)$. It is true that the worst case of binary search is $O(n)$, but it is not the most informative statement; $O(\log n)$ is.

Big-Ω: a lower bound

$f(n)$ is $\Omega(g(n))$ (Ω is the Greek letter omega; “big-omega of $g(n)$”) if, for large enough $n$, $f(n)$ is at least a constant times $g(n)$. Precisely: there are positive constants $c$ and $n_0$ such that

$f(n) \ge c \cdot g(n) \quad \text{for every } n \ge n_0.$

This is big-O with the inequality reversed. In the figure, $g(n) = n^2$ and $c = 1/2$. The wiggly function $f(n)$ dips below $n^2/2$ twice while $n$ is small, but from $n_0 = 10$ on it stays at or above, so $f(n)$ is $\Omega(n^2)$:

For example, the running time of do_something, $c_2 n^2 + c_1 n$, is at least $c_2 n^2$ for every $n$, so it is $\Omega(n^2)$. The worst-case running time of linear search is at least $c_2 n$, so it is $\Omega(n)$.

Big-Ω is only a lower bound, in the same way that big-O is only an upper bound. The worst case of linear search is also $\Omega(\log n)$ and $\Omega(1)$, true statements that say less than $\Omega(n)$.

Big-Θ: both bounds

$f(n)$ is $\Theta(g(n))$ (Θ is the Greek letter theta) if it is both $O(g(n))$ and $\Omega(g(n))$: for large enough $n$, it is at least one constant times $g(n)$ and at most another constant times $g(n)$. It grows like $g(n)$, no faster and no slower.

The examples above already show both halves. The running time of do_something is $O(n^2)$ and $\Omega(n^2)$, so it is $\Theta(n^2)$. The worst-case running time of linear search is $O(n)$ and $\Omega(n)$, so it is $\Theta(n)$.

When a function is known exactly, as all the functions in the table above are, finding its big-Θ is the same work as finding its big-O: keep the fastest-growing term and drop its constant. Use big-Θ when you know the growth rate exactly, and big-O or big-Ω when you can bound it from only one side.

Ranking growth rates

For large enough $n$, any function that is $\Theta(n^2)$ is smaller than any function that is $\Theta(n^3)$, whatever the constants. So functions can be grouped into families by growth rate, and the families can be ranked:

$1 < \log n < \sqrt n < n < n \log n < n^2 < n^3 < 2^n < n!$

A table of values shows how quickly these families separate:

$n$	$\log_2 n$	$n \log_2 n$	$n^2$	$2^n$
10	about 3	about 33	100	1,024
100	about 7	about 664	10,000	about $1.3 \times 10^{30}$
1,000	about 10	about 9,966	1,000,000	about $1.1 \times 10^{301}$

An algorithm whose running time grows like $n!$ runs for a long time, even for moderate $n$.

The take-home method: choose the case, and compute its running time section by section, using constants. Then keep only the fastest-growing term and drop its constant. That case’s running time is big-Θ of that term.

Cases and bounds are separate questions

Putting the two steps together gives one tight bound for each case:

Algorithm	Best case	Worst case
Linear search	$\Theta(1)$	$\Theta(n)$
Binary search	$\Theta(1)$	$\Theta(\log n)$
find_node	$\Theta(1)$	$\Theta(n)$
do_something	$\Theta(n^2)$	$\Theta(n^2)$
do_something_else	$\Theta(1)$	$\Theta(1)$
Insertion sort	$\Theta(n)$	$\Theta(n^2)$

Read each entry as a sentence that names a case and a bound: “the best-case running time of linear search is $\Theta(1)$.” The case says which function; the bound says how fast it grows. Any case can be paired with any kind of bound:

• “The best-case running time of insertion sort is $O(n)$” is true. Big-O is used here for a best case.
• “The worst-case running time of linear search is $O(n^2)$” is also true. It is an upper bound, but not a tight one; $\Theta(n)$ says more.
• “The worst-case running time of insertion sort is $\Omega(n^2)$” is true. Big-Ω is used here for a worst case.
• “The worst-case running time of binary search is $\Theta(\log n)$” is a tight bound on a worst case.

So big-O does not mean worst case, big-Ω does not mean best case, and big-Θ does not mean average case. Big-O, big-Ω, and big-Θ describe a function; best case and worst case pick the function.

A bound can also be stated for all inputs at once, with no case named. Since no input takes longer than the worst case, a big-O bound on the worst case holds for every input. “Insertion sort runs in $O(n^2)$ time” is therefore true: no input of size $n$ takes more than a constant times $n^2$. But “insertion sort runs in $\Theta(n^2)$ time” is false as a statement about every input, because a sorted list takes time proportional to only $n$. For insertion sort, the strongest statement about all inputs is $O(n^2)$. Linear search is similar: it runs in $O(n)$ time on every input, but not in $\Theta(n)$ time on every input. For do_something, whose best and worst cases are the same, “do_something runs in $\Theta(n^2)$ time” is true for every input.

So a statement such as “linear search is $\Theta(n)$” is ambiguous: it is true of the worst case but false of the best case. Name the case, unless one bound holds for every input.

Amortized analysis: why doubling makes append constant-time

A growing array stores a list’s values in a block of fixed capacity, and when the block fills up, it allocates a larger block and copies the values into it. Why should the new block have double the capacity, rather than some fixed number of additional slots?

First, the individual costs. An array stores its items in order in one contiguous block of memory. Inserting at the front of an $n$-item array moves every item one slot to the right, so the worst-case cost of inserting is $\Theta(n)$; deleting from the front is $\Theta(n)$ for the same reason. Deleting from the end moves nothing, so pop() is $\Theta(1)$. Appending at the end usually moves nothing either. But if the block is full, and something else is stored in memory right after it, the whole list must be copied to a new, larger block, which takes $\Theta(n)$ time. So in the worst case, a single append takes $\Theta(n)$ time, much worse than the $\Theta(1)$ you might have expected.

The trick is to make that worst case rare. When the block of $n$ items fills up, the list requests room not for $n+1$ items but for $2n$. Overallocating like this may seem wasteful, since up to half of the block may be unused, but it means that a long sequence of appends copies the list only occasionally. Starting from capacity 1, appending 128 items resizes the block at capacity 1, 2, 4, 8, 16, 32, and 64, ending at 128.

To add up the cost, follow one block from one doubling to the next. Say that writing a value into a free slot costs 1, and that copying a value into a new block costs some constant $c$ per value. Suppose the block has just doubled to capacity 8, so it holds 4 values and has 4 free slots. Each of the next 4 appends finds a free slot and costs 1:

Now the block is full. The next append, of 90, must first copy all 8 values into a new block of capacity 16, at a cost of $8c$, and then write 90 into slot 8:

The append of 90 is expensive only because the 4 appends before it filled the block, so charge the cost of the copy to those 4 appends. Together they cost $4 + 8c$: $4$ to write their own values and $8c$ to copy the block they filled. That is $1 + 2c$ per append.

The same calculation works at every size. Right after a block doubles from capacity $m$ to capacity $2m$, it holds $m$ values and has $m$ free slots. The next $m$ appends cost 1 each, and the copy that follows costs $2m \cdot c$. Charged to those $m$ appends, that is $m + 2mc$ in total, or $1 + 2c$ per append, whatever $m$ is. (The very first appends, into blocks of capacity 1 and 2, don’t start half full, but they add only a constant to the total.)

The next figure shows the cost of each of the first 16 appends, starting from capacity 1, with $c = 1$. Most appends cost 1. The appends that find the block full, the 2nd, 3rd, 5th, and 9th, also copy every value, and cost 2, 3, 5, and 9. A single append can be expensive, but spread over the appends that filled each block, the copying never raises the average above $1 + 2c = 3$:

Since the appends cost at most the constant $1 + 2c$ each, on average, appending takes $\Theta(1)$ amortized time.

Doubling is not the only growth factor that works. Any constant factor greater than 1 gives $\Theta(1)$ amortized appends; a smaller factor wastes less memory but leaves each append with a larger share of the copying. Python’s own list grows by only about 12.5% at a time. After the block grows from capacity $m$ to $1.125m$, only $0.125m$ cheap appends happen before the next copy of $1.125m$ values, so each of them is charged $1 + 9c$ rather than $1 + 2c$. That is still a constant.

This kind of analysis is called an amortized running-time analysis: the occasional expensive append is paid for by many cheap appends, so that averaged over any long sequence of operations, each append costs $\Theta(1)$. It is a statement about the whole sequence, not about any one call: a single append can still take $\Theta(n)$ time.

Growing by a fixed number of slots instead, say 10, breaks this argument. After the block grows from capacity $m$ to $m + 10$, only 10 cheap appends happen before the next copy, and that copy moves all $m + 10$ values. Charged to those 10 appends, that is $1 + (m + 10)c/10$ per append, which grows with $m$. Once $n$ is large, nearly every tenth append copies the whole list, and $n$ appends cost $\Theta(n^2)$ in total: $\Theta(n)$ each, on average.

Source material for these notes

The analyses of linear and binary search, the review of logarithms, and the introduction to big-O are adapted from the CS 1 notes by Balkcom et al., chapter 14, “Analyzing algorithms”.

The rest of these notes were drafted by Claude (Anthropic) at Balkcom’s direction and then edited by Balkcom. This includes the comparison of the growing array and the linked list, the discussion of best, worst, and average cases, the analyses of find_node, nested loops, and insertion sort, big-Ω and big-Θ, the ranking of growth rates, the amortized analysis of doubling, the figures, and the practice problems. Claude’s drafts reflect the standard treatment of these topics, which can also be found in Cormen, Leiserson, Rivest, and Stein, Introduction to Algorithms, chapters 2 and 3 and the chapter on amortized analysis, and in Sedgewick and Wayne, Algorithms, section 1.4.

Further reading

• TimeComplexity: the running times of operations on Python’s built-in data structures
• timeit: for measuring the running time of small pieces of code

Practice problems

Try these on your own before checking the solutions. For each function, first compute the running time in terms of constants ($c_1$, $c_2$, and so on) and $n$, section by section. Then give the tightest big-Θ bound you can. If the best and worst cases differ, analyze both. (Every running time in these problems is $O(n!)$, but you can say something much more informative about each.)

Problem 1: Nested loops

def f(n):
    i = 0
    while i < n:
        j = 0
        while j < n:
            print(i, j)
            j += 1
        i += 1

(Solution)

Problem 2: A fixed inner loop

def f(n):
    i = 0
    while i < n:
        j = 0
        while j < 1000:
            print(i, j)
            j += 1
        i += 1

(Solution)

Problem 3: A loop that calls a function

def g(x):
    for i in range(x):
        print(i)

def f(n):
    i = n
    while i > 0:
        g(n)
        i -= 1

(Solution)

Problem 4: A doubling loop variable

def f(n):
    i = 1
    while i < n:
        i *= 2
        print(i)

(Solution)

Problem 5: Factors up to the square root

from math import sqrt

def f(n):
    i = 1
    while i <= sqrt(n):
        if n % i == 0:
            print(i, n // i)
        i += 1
    print("And that's all the integer factors of", n)

(Solution)

Problem 6: Linked-list operations

For a circular, doubly linked list with a sentinel and $n$ items, give the worst-case running time of __init__, insert_after, prepend, append, deleting a given node, get(i), and a length() that walks the list to count its items. For each, say in a sentence why.

(Solution)

Problem 7: Growing by a constant

A growing array that adds 10 slots each time it fills up, instead of doubling, starts empty with capacity 10. How many item copies does it make, in total, over the course of $n$ appends, where $n$ is a multiple of 10? Give an exact formula and a big-Θ bound, and compare with doubling.

(Solution)

Problem 8: Counting digits

Let $n$ be a positive integer.

def count_digits(n):
    count = 0
    while n > 0:
        n = n // 10
        count += 1
    return count

(Solution)

Problem 9: An inner loop that starts at i

def f(n):
    for i in range(n):
        for j in range(i, n):
            print(i, j)

(Solution)

Problem 10: Looking for a repeated value

Let $n$ be the length of the_list.

def has_duplicate(the_list):
    n = len(the_list)
    for i in range(n):
        for j in range(i + 1, n):
            if the_list[i] == the_list[j]:
                return True
    return False

(Solution)

Problem 11: Building a list at the front

Python’s list is a growing array. result.insert(0, i) shifts every value already in result one slot to the right before writing i into slot 0.

def countdown(n):
    result = []
    for i in range(n):
        result.insert(0, i)
    return result

How would you build the same list in $\Theta(n)$ time?

(Solution)

Problem 12: A doubling inner loop

def f(n):
    for i in range(n):
        j = 1
        while j < n:
            j *= 2

(Solution)

Problem 13: Cases and bounds

Say whether each statement is true or false, and why. has_duplicate is the function from Problem 10.

1. The worst-case running time of has_duplicate is $\Omega(n)$.
2. The best-case running time of has_duplicate is $\Omega(n)$.
3. has_duplicate runs in $O(n^2)$ time on every input.
4. has_duplicate runs in $\Omega(n^2)$ time on every input.
5. If the worst-case running time of an algorithm is $O(n \log n)$, then so is its best-case running time.
6. If the best-case running time of an algorithm is $\Omega(n)$, then every input of size $n$ takes $\Omega(n)$ time.

(Solution)

Problem 14: Shrinking a growing array

A growing array doubles its capacity when an append finds the block full. To avoid wasting memory after many deletions, it also shrinks: whenever a pop from the end leaves the block exactly half full, it copies the remaining values into a new block of half the capacity.

1. The array is full, with $m$ values and capacity $m$, where $m$ is a power of 2. Find a sequence of operations in which every operation copies $m$ values. What does this say about the amortized cost of appends and pops?
2. Suppose instead that a pop shrinks the block to half its capacity only when it leaves the block a quarter full. Explain why the sequence from part 1 no longer causes a copy on every operation. How many operations must happen, at least, between one copy and the next?

(Solution)