.. _compiler:

The Compiler

The compilation process in MicroPython involves the following steps:

* The lexer converts the stream of text that makes up a MicroPython program into tokens.

* The parser then converts the tokens into an abstract syntax (parse tree).

* Then bytecode or native code is emitted based on the parse tree.

For purposes of this discussion we are going to add a simple language feature ``add1``

that can be use in Python as:

.. code-block:: bash

The ``add1`` statement takes an integer as argument and adds ``1`` to it.

Adding a grammar rule

MicroPython's grammar is based on the `CPython grammar <https://docs.python.org/3.5/reference/grammar.html>`_

and is defined in `py/grammar.h <https://github.com/micropython/micropython/blob/master/py/grammar.h>`_.

This grammar is what is used to parse MicroPython source files.

There are two macros you need to know to define a grammar rule: ``DEF_RULE`` and ``DEF_RULE_NC``.

``DEF_RULE`` allows you to define a rule with an associated compile function,

while ``DEF_RULE_NC`` has no compile (NC) function for it.

A simple grammar definition with a compile function for our new ``add1`` statement

looks like the following:

.. code-block:: c

The second argument ``c(add1_stmt)`` is the corresponding compile function that should be implemented

in ``py/compile.c`` to turn this rule into executable code.

The third required argument can be ``or`` or ``and``. This specifies the number of nodes associated

with a statement. For example, in this case, our ``add1`` statement is similar to ADD1 in assembly

language. It takes one numeric argument. Therefore, the ``add1_stmt`` has two nodes associated with it.

One node is for the statement itself, i.e the literal ``add1`` corresponding to ``KW_ADD1``,

and the other for its argument, a ``testlist`` rule which is the top-level expression rule.

.. note::

The ``add1`` rule here is just an example and not part of the standard

MicroPython grammar.

The fourth argument in this example is the token associated with the rule, ``KW_ADD1``. This token should be

defined in the lexer by editing ``py/lexer.h``.

Defining the same rule without a compile function is achieved by using the ``DEF_RULE_NC`` macro

and omitting the compile function argument:

.. code-block:: c

The remaining arguments take on the same meaning. A rule without a compile function must

be handled explicitly by all rules that may have this rule as a node. Such NC-rules are usually

used to express sub-parts of a complicated grammar structure that cannot be expressed in a

single rule.

.. note::

The macros ``DEF_RULE`` and ``DEF_RULE_NC`` take other arguments. For an in-depth understanding of

supported parameters, see `py/grammar.h <https://github.com/micropython/micropython/blob/master/py/grammar.h>`_.

Adding a lexical token

Every rule defined in the grammar should have a token associated with it that is defined in ``py/lexer.h``.

Add this token by editing the ``_mp_token_kind_t`` enum:

.. code-block:: c

Then also edit ``py/lexer.c`` to add the new keyword literal text:

.. code-block:: c

Notice the keyword is named depending on what you want it to be. For consistency, maintain the

naming standard accordingly.

.. note::

The order of these keywords in ``py/lexer.c`` must match the order of tokens in the enum

defined in ``py/lexer.h``.

Parsing

In the parsing stage the parser takes the tokens produced by the lexer and converts them to an abstract syntax tree (AST) or

*parse tree*. The implementation for the parser is defined in `py/parse.c <https://github.com/micropython/micropython/blob/master/py/parse.c>`_.

The parser also maintains a table of constants for use in different aspects of parsing, similar to what a

`symbol table <https://steemit.com/programming/@drifter1/writing-a-simple-compiler-on-my-own-symbol-table-basic-structure>`_

does.

Several optimizations like `constant folding <http://compileroptimizations.com/category/constant_folding.htm>`_

on integers for most operations e.g. logical, binary, unary, etc, and optimizing enhancements on parenthesis

around expressions are performed during this phase, along with some optimizations on strings.

It's worth noting that *docstrings* are discarded and not accessible to the compiler.

Even optimizations like `string interning <https://en.wikipedia.org/wiki/String_interning>`_ are

not applied to *docstrings*.

Compiler passes

Like many compilers, MicroPython compiles all code to MicroPython bytecode or native code. The functionality

that achieves this is implemented in `py/compile.c <https://github.com/micropython/micropython/blob/master/py/compile.c>`_.

The most relevant method you should know about is this:

.. code-block:: c

The compiler compiles the code in four passes: scope, stack size, code size and emit.

Each pass runs the same C code over the same AST data structure, with different things

being computed each time based on the results of the previous pass.

First pass

In the first pass, the compiler learns about the known identifiers (variables) and

their scope, being global, local, closed over, etc. In the same pass the emitter

(bytecode or native code) also computes the number of labels needed for the emitted

code.

.. code-block:: c

Second and third passes

The second and third passes involve computing the Python stack size and code size

for the bytecode or native code. After the third pass the code size cannot change,

otherwise jump labels will be incorrect.

.. code-block:: c

Just before pass two there is a selection for the type of code to be emitted, which can

either be native or bytecode.

.. code-block:: c

The bytecode option is the default but something unique to note for the native

code option is that there is another option via ``VIPER``. See the

:ref:`Emitting native code <emitting_native_code>` section for more details on

viper annotations.

There is also support for *inline assembly code*, where assembly instructions are

written as Python function calls but are emitted directly as the corresponding

machine code. This assembler has only three passes (scope, code size, emit)

and uses a different implementation, not the ``compile_scope`` function.

See the `inline assembler tutorial <https://docs.micropython.org/en/latest/pyboard/tutorial/assembler.html#pyboard-tutorial-assembler>`_

for more details.

Fourth pass

The fourth pass emits the final code that can be executed, either bytecode in

the virtual machine, or native code directly by the CPU.

.. code-block:: c

Emitting bytecode

Statements in Python code usually correspond to emitted bytecode, for example ``a + b``

generates "push a" then "push b" then "binary op add". Some statements do not emit

anything but instead affect other things like the scope of variables, for example

``global a``.

The implementation of a function that emits bytecode looks similar to this:

.. code-block:: c

We use the unary operator expressions for an example here but the implementation

details are similar for other statements/expressions. The method ``emit_write_bytecode_byte()``

is a wrapper around the main function ``emit_get_cur_to_write_bytecode()`` that all

functions must call to emit bytecode.

.. _emitting_native_code:

Emitting native code

Similar to how bytecode is generated, there should be a corresponding function in ``py/emitnative.c`` for each

code statement:

.. code-block:: c

The difference here is that we have to handle *viper typing*. Viper annotations allow

us to handle more than one type of variable. By default all variables are Python objects,

but with viper a variable can also be declared as a machine-typed variable like a native

integer or pointer. Viper can be thought of as a superset of Python, where normal Python

objects are handled as usual, while native machine variables are handled in an optimised

way by using direct machine instructions for the operations. Viper typing may break

Python equivalence because, for example, integers become native integers and can overflow

(unlike Python integers which extend automatically to arbitrary precision).