A9: Parsing with Grammars

Prerequisites: Grammars (BNF and EBNF), parse trees, recursion on trees

Learning Objective: See the value of describing a language with a formal grammar. You’ll write grammars for two notations for arithmetic, have a library generate a parser from each, and find that the two notations produce the same tree, which a single recursive function can evaluate. Changing the language will mean changing the grammar, not rewriting a parser.

Background

An arithmetic expression is a tree. In $(3 + 4) \times 2$ , the multiplication is done last, so it is the root; its left child is the subtree for $3 + 4$ , and its right child is the number 2. In this assignment, represent an expression tree in Python as a nested list, with the operator first and then the operands:

["*", ["+", 3, 4], 2]

A number is a Python int, and any other expression is a list whose first element is an operator string and whose remaining elements are expressions.

People write this tree in several notations. In prefix notation, used by the programming language Lisp, every operation is in parentheses with the operator first: (* (+ 3 4) 2). In infix notation, each operator goes between its two operands: (3 + 4) * 2.

Prefix notation lets + and * take any number of operands, as Lisp does: (+ 1 2 3) is $1 + 2 + 3 = 6$ . This makes sense because the grouping of a sum or a product doesn’t change its value: $(1 + 2) + 3$ and $1 + (2 + 3)$ are both 6, so there’s no need to say which to do first. Subtraction and division are different: $(10 - 4) - 3$ is 3, but $10 - (4 - 3)$ is 9. So - and / take exactly two operands.

Ordinary infix leaves most parentheses out, and relies on rules of precedence (does 3 + 4 * 2 mean $(3 + 4) \times 2$ or $3 + (4 \times 2)$ ?) and associativity (does 10 - 4 - 3 mean $(10 - 4) - 3$ or $10 - (4 - 3)$ ?) to say which tree a string means. A grammar can build those rules in, with a level for each precedence, as the notes on grammars show with the expr, term, and factor grammar; but getting such a grammar right takes care. This assignment sidesteps the problem with fully parenthesized infix: every operation, including the outermost one, is written in parentheses, as in ((3 + 4) * 2). The parentheses say exactly how to group, so, as in prefix notation, no precedence or associativity rules are needed.

You’ll write a grammar for each notation, prefix and fully parenthesized infix, and use the Lark library to generate a parser from it. Each parser produces a parse tree, which you’ll convert to the nested-list form. One evaluator then works for both.

Setup

Install Lark by running this in a terminal:

python3 -m pip install lark

(On Windows, use python or py instead of python3: whichever command runs Python for you.) Check that it worked by running import lark in Python. This is the same way you installed PyQt6 for A8, and the notes there, on installing into an editor’s Python and on the “externally-managed-environment” error, apply to Lark too.

A short guide to Lark

A Lark grammar is written in EBNF, as a string. Lark doesn’t write out a parser as code: calling Lark(...) with the grammar builds a parser object while your program runs, and parse uses it. A mistake in the grammar raises an error when Lark(...) runs, before any text is parsed. Here is a grammar for nested lists of numbers, like [1, [2, 3], []]:

from lark import Lark, Tree, Token

parser = Lark(r"""
    value: NUMBER | list
    list: "[" (value ("," value)*)? "]"

    %import common.NUMBER
    %import common.WS
    %ignore WS
""", start="value")

tree = parser.parse("[1, [2, 3], []]")
print(tree.pretty())

Each rule is a lowercase name, a colon, and alternatives separated by |. start="value" names the rule for a whole string of the language.
Terminals, the pieces of text, are either quoted strings like "[", or uppercase names. An uppercase name can be defined in the grammar, as in DIGIT: "0" | "1" | "2", or imported from Lark’s library of common definitions: %import common.INT gives INT, a nonnegative integer, and %import common.NUMBER gives NUMBER, which may also have a decimal point.
%import common.WS and %ignore WS let whitespace appear anywhere between tokens.
The EBNF operators work as usual: x* is zero or more xs, x+ is one or more, x? is an optional x, and parentheses group.

parser.parse(text) returns the parse tree. Each internal node is a Tree, with the name of its rule in tree.data and its children in the list tree.children. Each leaf is a Token, which is a string with its terminal’s name in token.type. Quoted terminals like "[" are left out of the tree. The code above prints:

value
  list
    value  1
    value
      list
        value  2
        value  3
    value
      list

Two features make trees easier to convert:

An alias, -> name, at the end of an alternative names the tree node for that alternative, in place of the rule’s name. With expr: INT -> number | "(" expr ")", a number gives a node with data == "number".
A rule whose name starts with ?, as in ?expr: ..., is inlined when an alternative has just one child: the child appears in the tree in place of the rule’s node. This keeps rules that only pass something along, like value above, out of the tree.

If the text isn’t in the language, parse raises an exception, a subclass of lark.exceptions.UnexpectedInput, whose message says where the text stopped matching the grammar and what was expected there.

A warning: Lark does not tell you when a grammar is ambiguous. If a string has more than one parse tree, Lark quietly picks one. The grammars in this assignment put every operation in parentheses, so they aren’t ambiguous, and this won’t come up. But it is one reason to test that a parser builds the right trees, not just that parsing succeeds.

Part 1: Evaluate

The recursion notes evaluate an expression tree made of BinaryNode objects, with + and *. Adapt that function to this assignment’s trees: write a recursive function evaluate(expr) that returns the value of an expression tree in nested-list form. For now, write the trees by hand, as in the tests below; Parts 2 and 3 build the same trees from text. The two cases are the same as in the notes:

Base case: expr is a number. It evaluates to itself.
Recursive case: expr is an operation, a list. Evaluate each of its operands, then combine their values with the operator.

Two things change. The operator and operands are in a list, expr[0] and expr[1:], rather than in the fields data, left, and right. And an operation can have more than two operands.

The operators follow Lisp:

"+" and "*" take one or more operands: ["+", 1, 2, 3] is 6, and ["*", 2, 3, 4] is 24. With a single operand, the value is that operand’s value: ["+", 5] and ["*", 5] are both 5.
"-" and "/" take exactly two operands: ["-", 10, 4] is 6, and ["/", 7, 2] is 3.5. Raise a ValueError if they get any other number of operands.

Here is a way to start. The loop makes the recursive calls, one per operand, and collects their values in a list:

def evaluate(expr):
    if isinstance(expr, int):        # base case: a number
        return expr
    operator = expr[0]
    values = []
    for operand in expr[1:]:         # evaluate each operand
        values.append(evaluate(operand))
    # Now combine the values, depending on the operator.

To finish it, combine values for each operator:

For "+", add up all the values: start a total at 0, and add each value to it. (Python’s sum(values) does the same thing.)
For "*", multiply them all: start a product at 1, and multiply it by each value.
For "-" and "/", first check that len(values) is 2, raising a ValueError if it isn’t. Then return values[0] - values[1], or values[0] / values[1].
For any other operator, raise a ValueError.

A single operand needs no special case: a total that starts at 0 and adds one value is that value.

print(evaluate(["*", ["+", 3, 4], 2]))              # 14
print(evaluate(["-", ["-", 10, 4], 3]))             # 3
print(evaluate(["+", 1, ["*", 2, 3, 4], 5]))        # 30

Part 2: Prefix notation

Write a grammar for arithmetic in prefix notation, with these rules:

A number is a nonnegative integer, such as 0, 7, or 42.
An operation is (, an operator, one or more expressions, and ). The operators are +, -, *, and /.
An expression is a number or an operation.
Whitespace may appear anywhere between tokens, and a space is needed only between two numbers.

First, write the grammar in BNF or EBNF, as you would on paper, in a comment in your file. Then compare it with this grammar for the same language, in Lark, which you should use for your parser:

prefix_parser = Lark(r"""
    ?expr: INT                      -> number
         | "(" OPERATOR expr+ ")"   -> operation
    OPERATOR: "+" | "-" | "*" | "/"

    %import common.INT
    %import common.WS
    %ignore WS
""", start="expr")

print(prefix_parser.parse("(* (+ 3 4) 2)").pretty())

This prints the parse tree:

operation
  *
  operation
    +
    number  3
    number  4
  number  2

Because of the aliases, each node of the parse tree is either a number node, whose one child is an INT token, or an operation node, whose first child is an OPERATOR token and whose remaining children are the operands. The expr+ lets an operation have any number of operands, one or more. The grammar doesn’t check that - and / get exactly two; evaluate does. If your BNF grammar differs from this one, explain in your comment whether it describes the same language.

Write a function parse_prefix(text) that parses the text with prefix_parser and returns the expression tree in nested-list form. Converting Lark’s parse tree to a nested list is a recursive function over the parse tree: a number node becomes an int, and an operation node becomes a list of its operator, as a string, followed by its converted operands.

print(parse_prefix("(* (+ 3 4) 2)"))           # ['*', ['+', 3, 4], 2]
print(parse_prefix("(+ 1 (* 2 3 4) 5)"))       # ['+', 1, ['*', 2, 3, 4], 5]
print(parse_prefix("42"))                      # 42
print(evaluate(parse_prefix("(/ (- 20 6) 4)")))  # 3.5

Strings that aren’t prefix expressions should make parse_prefix raise an exception: for example, (3 4), (+ 1 2, + 1 2, and (+).

Part 3: Fully parenthesized infix notation

Write a grammar for arithmetic in fully parenthesized infix notation, with these rules:

Numbers are nonnegative integers, as in Part 2.
An operation is (, an expression, an operator, an expression, and ). The operators are +, -, *, and /, and each operation has exactly two operands.
An expression is a number or an operation. Every operation is in parentheses, including the outermost one: ((3 + 4) * 2), not (3 + 4) * 2.
Whitespace may appear anywhere between tokens.

This time, write the grammar yourself: first in BNF or EBNF in a comment, then in Lark. It will look much like the prefix grammar, with the operator between the two operands instead of first. Write a function parse_infix(text) that returns the expression tree in the same nested-list form as parse_prefix.

Each infix string below should give the same tree as the prefix string beside it, and so the same value. Check that it does, by comparing the trees, not just the values:

Prefix	Infix	Tree	Value
`(+ 1 2)`	`(1 + 2)`	`['+', 1, 2]`	3
`(* (+ 3 4) 2)`	`((3 + 4) * 2)`	`['*', ['+', 3, 4], 2]`	14
`(+ 3 (* 4 2))`	`(3 + (4 * 2))`	`['+', 3, ['*', 4, 2]]`	11
`(- (- 10 4) 3)`	`((10 - 4) - 3)`	`['-', ['-', 10, 4], 3]`	3
`(- 10 (- 4 3))`	`(10 - (4 - 3))`	`['-', 10, ['-', 4, 3]]`	9
`(* (+ (+ 3 4) (* 4 2)) 5)`	`(((3 + 4) + (4 * 2)) * 5)`	`['', ['+', ['+', 3, 4], ['', 4, 2]], 5]`	75

Add at least four pairs of your own, including at least one that uses all four operators and one nested at least three levels deep. Strings that aren’t fully parenthesized infix expressions should make parse_infix raise an exception: for example, (3 +), (3 + 4, (3 4), 3 + 4 (no parentheses), and (1 + 2 + 3) (three operands).

A prefix operation with more than two operands, like (+ 1 2 3), has no single infix counterpart. Both ((1 + 2) + 3) and (1 + (2 + 3)) have its value, 6, but their trees, ['+', ['+', 1, 2], 3] and ['+', 1, ['+', 2, 3]], differ from each other and from ['+', 1, 2, 3].

Part 4: Change the language

Add an exponentiation operator, ^, to both notations, taking exactly two operands:

In prefix notation, (^ 2 10) is $2^{10} = 1024$ .
In infix notation, (2 ^ 10) is 1024, and the parentheses say how to group: ((2 ^ 3) ^ 2) is $8^2 = 64$ , and (2 ^ (3 ^ 2)) is $2^9 = 512$ .

Add ^ to both grammars, and extend evaluate to handle it, raising a ValueError unless it gets exactly two operands. (Python writes exponentiation as **.) Add prefix/infix test pairs for ^, including (^ 2 (^ 3 2)) with (2 ^ (3 ^ 2)), and (* 2 (^ 3 2)) with (2 * (3 ^ 2)).

Keep track of what you had to change, for the written questions.

What to do

Write evaluate, parse_prefix, and parse_infix as described above, in a file grammars.py, with both grammars also written out in BNF or EBNF in a comment.
Run the tests in each part, and add your own as described.
Answer these in a comment at the end of your file:
- Neither of your grammars needs rules of precedence or associativity. Why not? The ordinary infix string 10 - 4 - 3 * 2 leaves out the parentheses: what does a reader, or a grammar, need to know to decide which tree it means?
- Why does it make sense for + and * to take any number of operands, but not - and /? Give an example of a prefix expression using - with three operands, and two different values it could reasonably mean.
- What did you have to change to add ^ in Part 4: in each grammar, in your conversion functions, and in evaluate? What would you have had to change in a parser written by hand, without a grammar?

Feedback

At the top of your main Python file, add a comment that answers these questions:

About how long did the assignment take you?
What was hard?
What was easy?
Was anything in the assignment unclear?

We use your answers to improve the assignment.

Grading and requirements

evaluate is recursive, handles all five operators with the right numbers of operands, and raises ValueError otherwise.
Both grammars are written in BNF or EBNF in a comment, compared with the given prefix grammar, and the infix grammar is written in Lark.
parse_prefix and parse_infix return correct nested-list trees, and raise exceptions for strings not in their languages.
The infix grammar accepts exactly the fully parenthesized expressions, including ^, as shown by tests that compare trees and tests of strings it should reject.
All the given test pairs pass, and your own pairs cover all the operators and nested parentheses.
The written questions are answered correctly.

Challenge problems (no extra points, not required)

Ordinary infix. Write a grammar for infix without the extra parentheses, so that 3 + 4 * 2 and 10 - 4 - 3 parse correctly: * and / have higher precedence than + and -, all four are left-associative, and ^ has the highest precedence and is right-associative, so that 2 ^ 3 ^ 2 is 512. Parentheses are needed only to override these rules. The expr, term, and factor grammar in the notes on grammars is the place to start. Test it carefully, comparing trees: Lark won’t tell you if your grammar is ambiguous.
Printing in three notations. Write recursive functions to_prefix(expr), to_postfix(expr), and to_infix(expr) that turn a nested-list tree into a string in each notation. For to_infix, first put parentheses around every operation; then try to print only the parentheses that are needed, so that to_infix(parse_infix("(3 + 4) * 2 - 1")) is "(3 + 4) * 2 - 1".
Unary minus. Add negation to your ordinary infix grammar, so that -3 * -(2 + 1) means $(-3) \times (-(2 + 1))$ . How does it interact with ^? In most mathematical writing, $-2^2$ means $-(2^2) = -4$ .
Variables. Allow names like x and rate as operands in both notations, and give evaluate a second parameter, a dictionary from names to values.