cmlzaGk - Python

Structural Pattern Matching in Python

2021-12-15T01:42:00-08:00

Python 3.10 was released in October 2021.

A major highlight of the release is a new language feature called Structural Pattern Matching.

Structural Pattern Matching was made famous in Scala but is found explicitly or implicitly in many languages.

It is a key paradigm of modern programming.

In this blog, I will share my thoughts on this feature.

I will cover why it is important, and how to approach it, but this is not a tutorial of the feature.

I encourage reading PEP 636, which is a complete tutorial of the feature IMO.

What is Structural Pattern Matching?

Structural Pattern Matching is a programming technique that takes a Subject and a set of Patterns and performs a pattern test on structure and state.

A successful pattern match results in execution of a block of code corresponding to the Pattern.

In typical usage, a side effect of pattern matching is name-binding, which live in enclosing scope.

Python 3.10 implements this by introduction of a new statement called the “match” statement. Typical usage is as follows

match subject_expr: 
    case pat1: 
        do_something()
    case pat2: 
        do_something_else()

In effect, match statement is a replacement for something like

if matches(expr1, pat1) : 
    do_something()
elif matches(expr2, pat2):
    do_something_else()

I will use “match statement" interchangeably with “Structural Pattern Matching” for the remaining sections of the blog.

Motivation and Applications

The match statement is usable in places where different blocks of code need to be executed based on the structure or state of the data. This means that it can be used in a lot of different places.

The motivation to use a match statement is to simplify the code and make it more readable.

The Subject is a regular python expression which can resolve to most sequences, dictionary or an object. Patterns can test for a match of length, type, specific index, key, attribute values, inheritance etc.

As a use case, a typical json handler checks data for states or properties and performs corresponding actions.

With match pattern, a handler for json data becomes very simple.

def process_media_blob(jsondata):
    match jsondata:
        case {'type':'audio', 'format' :'mp3', 'data': data}:
            render_mp3(data)

        case {'type':'video', 'format' : ('webm' | 'mkv') as videoformat, 'data': data}:
            render_video(data, videoformat)

        case _:
            raise MediaInputException('Usage: .... ')

The above match statement performed the necessary tests for presence of keys and their expected values elegantly.

It also introduced ‘data’ and ‘format’ values to the scope namespace as ‘data’ and ‘videoformat’ respectively.

The above is an example when the subject is a dictionary. PEP 636 exhausts use cases for data-structures like list, class, tuples etc.

Pattern is not an expression

An important thing to remember is that a Pattern is not a Python expression, even though under certain situations it looks exactly like a Python expression.

A Pattern has completely different grammar rules from an expression.

As an example below, Point(x=0, y=1) as an expression instantiates an object. The same as a Pattern statement is a rule check.

class Point:
    def __init__(self, x, y):
        self.x, self.y = x, y

p = Point(x=0, y=1)

match p:
    case Point(x=0, y=1):
        print('p is an object of class Point with attributes x as 0 and y as 1')
    case _:
        print(' p is something else')

Performance

The match clause is functionally equivalent to “if .. elif .. “ tests.

Even for potentially problematic patterns like sequences, constraints are in place to keep processing fast.

E.g., The greedy operator ‘*’ in sequence patterns can only be used once in a pattern.

The Pattern is a language specific rule and not a regular expression.

Hence performance is equivalent to the earlier alternate patterns of "if .. elif ..".

Conclusion

Structural Pattern Matching is a very exciting addition to the Python language.

Scala has already shown why this is a very popular and modern style of programming.

Portions of python language already use the match clause to simplify existing code.

Match statement is a language feature. Understanding it is not just useful for writing elegant code but also to read future Python code.

Scripting Matrix Rain On Good Old Terminals

2021-12-04T05:47:00-08:00

Matrix 4 release date is set for Dec 22. Matrix is one of the most influential movies of all time.

A feature of the movie was the Matrix Rain.

I went retro this week and scripted the Matrix Rain console application.

In this blog I will go over how to create matrix rain on a terminal.

What is Matrix Rain?

Matrix Rain is a visualization of characters falling downwards on a console in the form of rain.

Approach

The approach is simple

Prepare a buffer
Repeat forever
- Clear the screen
- Render the buffer
- Sleep for some time (refresh rate)
- Update the buffer so that all horizontal lines move downwards

This works for most part, except for some challenges that I will take about, as well as how to overcome them.

Character Set

The choice of Unicode characters and font-family for a terminal-based matrix rain is very important. Three conditions need to be met.

The font-family should be Mono-spaced, all characters should be printable on the terminal, and the terminal should be able to render the characters fast.

Mono-spaced means that the letters occupy same amount of horizontal space.

If this property is not true, then the lines that we print will be scattered, and there will not be an appearance of a vertical flow.

The default font "Consolas" on Windows terminal is Mono-spaced and meets the requirement.

I did a manual test to check which Unicode ranges can be displayed on the screen.

I built this range manually because I noticed that I could not rely isprintable() method of string in python.

e.g. My console was unable to print 0x16ef, but isprintable() returned True.

### https://stackoverflow.com/questions/1477294/generate-random-utf-8-string-in-python

include_ranges = [
    ( 0x0023, 0x0026 ),
    ( 0x0028, 0x007E ),
    ( 0x00A1, 0x00AC ),
    ( 0x00AE, 0x00FF ),
    ( 0x0100, 0x017F ),
    ( 0x0180, 0x024F ),
    ( 0x2C60, 0x2C7F ),
    ( 0x16A0, 0x16F0 ),
    ( 0x0370, 0x0377 ),
    ( 0x037A, 0x037E ),
    ( 0x0384, 0x038A ),
    ( 0x038C, 0x038C ),
]


# Used in manual testing
for current_range in include_ranges :
    for code_point in range(current_range[0], current_range[1] + 1):
        print('0x{:4x} {} {}'.format(code_point, chr(code_point), chr(code_point).isprintable()))


# Once include_ranges is determined
ALPHABETS = [
    chr(code_point) for current_range in include_ranges
        for code_point in range(current_range[0], current_range[1] + 1)

The above will also test whether a character takes a long time to render on the screen.

Unicode contains a lot of rich characters and if your range picks up such a character that takes time to render, you will not get the rain effect.

"Consolas" and the range I published should do the trick for Windows Command prompt.

Flickering Rain

The first goal was to script using dependencies from just the standard python library. With that restriction in place, the interesting part was how to clear the terminal.

The first approach is to invoke a shell call, and call 'cls' on windows, and that is not a bad start.

As can be seen, we have achieved a rain effect, but this level of flickering is unacceptable.

This flicker occurs because of the time it takes to do an inter-process call, clear and then display our content from another process.

We have to do better.

Enable VT100 mode on Windows 10+

The flicker occurs because of the delay within the refresh because clear screen happens in another process.

There is no standard API to clear the screen. However, starting Windows 10, console got support for VT100 escape sequences that can be sent to the console.

It can be enabled using system libraries, however I enabled it by setting a DWORD 'VirtualTerminalLevel' on registry key 'HKEY_CURRENT_USER\Console' to 0x1.

The following code can test if the console understands VT100 escape sequencing.

import sys
# send the escape sequence to clear the screen for VT100 compatible console
sys.stdout.write('This should not be visible\033[H\033[JThis should be first line of the screen')
sys.stdout.flush()

The flicker stops after replacing the 'cls' call with VT100 escape sequences.

VT100 controls also allows us to enter color codes, which completes the rendering, because we need that green matrix effect.

The Curses Library

The limitations of scripting using only the standard library can be summarized as below:

There is no standard method of determining the width and height of the terminal.
Terminal specific codes (e.g. VT100) need to be embedded to avoid flickering.
The buffer that gets printed on the screen needs to be organized manually.

It would be good if there was a library that provided a consistent interface and hid the porting implementation from the developer.

The Curses library does exactly that. Curses is not shipped with the standard Python distribution on Windows.

However, there are third party Windows Curses libraries that can be installed using pip.

The overall approach does not change with Curses, except rendering.

With Curses, I did something different - I decided to do async rendering and have vertical channels get drained asynchronously too.

The output of the Curses implementation is on Youtube

Conclusion

Console programming still has applications particularly in non-graphical systems that still exist.

Naturally, there are differences in considerations, restrictions and limitations from what developing even in Simple GUI systems.

Matrix Rain on terminal is a fun problem to learn the techniques. It is simple enough to solve and can have satisfying results.

See you at the theatres.

How To Parse Like A Compiler - Part I

2021-03-02T05:47:00-08:00

This week I helped a friend parse a file consisting of Poker hand histories. Instead of creating a hand parser, I wrote a small grammar and used a Python module called Lark to parse the file into a syntax tree. An integration like this used to be tedious in the past, but modern languages allow you to develop such a solution rather quickly.

The aim of this blog is to highlight how to use context free grammars as an alternative to hand parsers. In part I, I will focus on parsing and the next part will be on transformation.

This is a rather deep topic and I am by no means a language expert. I do not intend to cover theories of compilers, languages, parsers or syntax trees, but I will leave some practical references for those interested. I think that those with a basic understanding of context free grammars and Python can pick this solution, or you can read this again after doing a quick refresher on these topics.

Summary

Problem Statement
Lark
Context Free Grammar
Earley vs LALR
Building our grammar
Moving to LALR
Conclusion

For reference the entire project can be found here

Problem Statement

We will attempt to parse a poker hand history file.

Each hand begins with a handnumber and is seperated by other hands by an empty newline.

A hand consists of the complete state information of the game. The players, their positions, starting stacks, actions they took on each poker street, followed by the results.

I hand-created a sample file for this blog consisting of two hands.

The actual file that I parsed consisted a few hundred hands from one game.

Online Poker players can download these files from the sites they play in. Each site could have its own format and there are many paid Software online for parsing these files into databases.

Lark

Lark is a python module that needs be installed using PIP.

As always, I recommend creating a virtual python environment before installing any module to keep your global module space on your machine clean.

C:\~>mkdir samplelark

C:\~>cd samplelark

C:\~\samplelark>python -m venv venv

C:\~\samplelark>.\venv\Scripts\activate

(venv) C:\Users\rishim\samplelark>pip install lark
Collecting lark
  Using cached https://files.pythonhosted.org/packages/69/8b/9418c0d24df0e8261c24e4b7218e80369f175c68eb3dfa119bc89d53f7fc/lark-0.11.1-py2.py3-none-any.whl
Installing collected packages: lark
Successfully installed lark-0.11.1
You are using pip version 19.0.3, however version 21.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

(venv) C:\~\samplelark>

To understand how to use Lark, lets attempt to parse a nested list of words.

We will apply a simple grammar - a list consists of a list of single word or list, or it consists words and lists seperated by comma.

In different words, a list consists of a single word or list, followed by zero or more sequences of comma and a single list or word.

More formally

list -> [listelem ("," listelem)*]
listelem: list|WORD

A complete Lark based parser looks like this:

from lark import Lark
cfg = r'''
    list: "[" listelem ("," listelem)* "]"
    listelem: list|WORD

    %import common.WS
    %ignore WS
    %import common.WORD
'''

list_parser = Lark(cfg, start='list')
print(list_parser.parse('[test, me, [I, am, nested, [no, kidding]]]').pretty())

The parse() function above returned a syntax-tree.

The syntax tree is the parsed output, which we will explore in part II. In this part, we will just print a pretty output of the tree.

The output indicates, we were able to succesfully parse the word and capture all the terminals and non terminals that we defined.

(venv) C:\~\Samplelark>python nestedlist.py
list
  listelem      test
  listelem      me
  listelem
    list
      listelem  I
      listelem  am
      listelem  nested
      listelem
        list
          listelem      no
          listelem      kidding

This is the core principle of cfg parsing and allows us to cleanly implement a parser.

Context Free Grammar

A Context Free Grammar (CFG) consists of four things:

Terminals : The terminals make up the actual content of a sentence. "[", "," , WORD are all terminals in our example. Note that WORD is in implemented as a regex and we could have defined it ourselves instead of using LARK predefinitions. We will use our own regex Terminals in the actual solution.
NonTerminals : Phrase or a clause in grammar. eg. list, listelem
Production Rules : A mapping between NonTerminals and a set of Terminals and NonTerminals. eg. listelem: list|WORD is a production rule.
Start Elem: A special NonTerminal. This is essentially the rule we are trying to parse an input into.

The language is Context Free because its parsing does not depend on the actual value of a terminal.

It is not possible to create an XML parser using a CFG if the parser attempts to match the correctness of start element tag and end element tag. We can create a rule in CFG where a start and end element exist and are well nested, but we cannot guarentee X == Y.

Such would be possible with a context sensitive grammar. In reality, CFG is still used for XML, but matching of X and Y is left to the next stage, the transformer (or the compiler).

Parsers

The first stage of parsing is lexical analyzer where the input is converted into tokens, however the actual parsing is performed by the parser.

There are two main types of parsers supported by Lark - Earley and LALR.

Earley can parse any grammar that we can write in CFG because it performs backtracking. Essentially, if a rule does not match, it can backtrack and use the next available rule.

Such operation is clearly very expensive and is infact cubic in complexity. Unless performance is a major consideration, use Earley to begin with. It is infact default parser for Lark.

If you are able to construct grammar in a manner where a parser can deterministicly make a decision on which rule to use next, you could use the LALR parser.

LALR parses in linear time and fails fast.

In a nutshell, LALR is a shift-reduce parser and that can parse a language does not have any conflicts between shifting and reducing.

I will give an example of such a conflict later.

By definition, Earley can parse more languages than LALR, but IMO most parsing problems in day to day life can be parsed using LALR.

Lark will help you understand which particular construct of your grammar creates a conflict, and if you are able to modify your grammar, you should move to LALR.

Building a Grammar

For reference the sample input file can be found here

I started building the parser intitutively to begin with by reading the file.

I parsed one hand first, section by section, expanding my grammar.

The roadblock here were the statements begining with player names because the player names could be one or more words.

foobar checks
Mr Foo bets 12

This meant the lexical scanner will not be able to identify the player. If I used a rule such as a "oneword|twowords" to identify the player, the lexer will not be able to disambiguate between 'foobar checks' or 'Mr. foo'.

Eventually, I decided that since I was parsing games with static list of player names, I will just include player name as grammar terminals.

This allows me to be context free.

I can also create a pre-parser that will generate the player names as terminals before I start the main parser, so I was not very concerned about broader application of this approach.

As I started parsing more and more hands in the file, I encountered new conditions which I incorporated into my grammar and my final main rule took a form that was able to parse the entire file.

    handhistory: handdesc when table seats posts preflop [flop] [turn] [river] [boards] summary winner [shows]

For reference the earley parser implementation can be found here

Moving to LALR

The above implementation was Earley and was visibly slow. I decided to move my parser to LALR.

It is pretty straight-forward in Lark to move to LALR. I just needed to pass the parameter parser='lalr' to Lark constructor.

However, my parser broke immediately, and I noticed that asking the lexer to ignore whitespace meant that new statements were not easily recognized.

I conclude that Earley was doing a lot of backtracking between rules because of ignoring whitespace. I made the parser strict about whitespace and I was able to parse most of the structures.

I ran into a conflict resolution with my Non Terminal called 'shows'.

    handhistory: handdesc when table seats posts preflop [flop] [turn] [river] [boards] summary winner [shows]
    shows: showdown*

This directive happens when a player shows a hand. Square brackets in terminals indicate that the NonTerminal appears zero or one times.

The asterix symbol in the NonTerminal is a regular regex symbol indicating zero or more times.

Hence when there were no occurrences of shows, the parser didn't know which production rule to use, because both the rules matched.

An LALR parser needs to know with certainty whether to shift the square bracket rule or to reduce with the second production.

I resolved this by restructuring shows to have one or more occurences.

Hence 'shows' is only reduced when there is actually a showdown directive. If there is no showdown, a shift is performed.

The complete LALR parser implementation can be found here

Conclusion

At the end the complete parser was very quick to create and I was able to develop it organically and intituively.

I will admit that it is difficult to understand error conditions during building, and it requires a fair bit of step by step parsing.

The overall process is very quick because at the end of the day, the code is concise and quite frankly easier to understand

I have used this in parsing poker ranges and poker hand histories.

In the next part, I will explain the transformation technique. Afterall, the canonical use of parsing is in converting data from one form to another.

Python Project Layout For Flask And Everything

2019-09-03T10:20:00-07:00

I will walk through a sample web application built using python, flask and docker. The application invokes a specialized math function in a library co-developed during application development.

The aim of this blog is to highlight some project structure best practices, that I learnt and used in my own application.

For those not familiar with flask, flask is a small web framework that implements the python WSGI interface.

WSGI is an application level contract to develop web applications in python. The other side of the contract is implemented by WSGI webservers like UWSGI.

By no means, these practices are limited to a flask application, and could be applied everywhere.

Summary

I will highlight the following areas using a sample project which can be found here

Project layout.
setup.py
requirements.txt
__init__.py
Local Deployment
Intra-package and inter-package references.
Class factories
Test cases and code coverage
DockerFiles

Project Layout

The project layout looks like this.

.
├── docker-compose.yml
├── Dockerfile
├── Dockerfile.unittests
├── requirements.txt
├── run_coverage.sh
├── uwsgi.ini
└──-bettermath-lib
    ├── setup.py
    └── bettermathlib
        ├── better_random.py
        ├── __init__.py
    └── bettermathlib_tests
        ├── tests_bettermathlib.py
        ├── __init__.py
└── randomweb-app
    ├── setup.py
    └── randomwebapp_tests
        ├── tests_basic.py
        ├── tests_client.py
        ├── __init__.py
    └── randomweb_app
        ├── config.py
        ├── create_app.py
        ├── flask_app.py
        ├── random_creator.py
        ├── __init__.py
        └── main
            ├── views.py
            ├── __init__.py
            └── static
                ├── about.txt
            └── templates
                ├── base.html
                └── betterrandom.html

A distribution is a folder that can be installed and moved around anywhere. bettermath-lib and randomweb-app are the self-contained distributions.

A package is a directory contains other packages or modules and is also called "Import Package" because its importable.

The file __init__.py is a special file that is executed when a package is imported in the runtime.

Here bettermathlib, bettermathlib_tests are packages inside the distribution bettermath-lib. randomweb_app and randomwebapp_tests are packages inside the distribution randomweb-app, and finally main is a package inside the package randomweb_app.

setup.py

setup.py is part of setuptools and defines properties of a distribution.

from setuptools import setup, find_packages

setup(
    name='randomweb-app',
    version='1.0',
    description='Random Webapp',
    author='Fishy Baker',
    author_email='fishybaker@hotmail.com',
    packages=find_packages(),
    install_requires = [
        'Flask >= 1.1.1',
        'Flask-Bootstrap >= 3.3.7.1',
        'Flask-WTF >= 0.14.2',
        'jsonschema >= 3.0.1',
        'bettermath-lib >= 1.0'
    ],
    include_package_data=True,
    package_data={'randomweb_app': ['templates/*', 'static/*']}
)

There are three main parts of setup.py

    packages=find_packages(),

find_packages is a neat helper to automatically include all packages inside randomweb-app as part of the distributable. The other option would be to create a static python list with a list of packages.

    include_package_data=True,
    package_data={'randomweb_app': ['templates/*', 'static/*']}

The third importand section is install_requires. It is typically a list of minimum versions of other distributions that your package has a dependency on.

The canonical way to obtain this list is via pip freeze command.

The catch is that we will not use this list in our application environment.

The consumers of your distribution should ideally be free to choose the latest and greatest stable versions, and ensure that their environment is tested.

requirements.txt

requirements.txt is the list of dependencies that the application is tested with. In the production environment, you will want to install specific versions of those dependencies.

requirements.txt is not part of the distribution. That role is fullfilled by setup.py's install_requires.

Hence requirements.txt lives at a folder level higher than the distributions.

pip freeze > requirements.txt

The typical production deployment should execute a variant of pip install with -r requirements.txt.

pip install -r requirements.txt

init.py

__init__.py plays a crucial role in namespacing the package's contents appropriately.

The class BetterRandom is in module better_random.py. So a consumer of my distribution would have to import bettermathlib.better_random to access BetterRandom.

By directly importing BetterRandom in __init__.py we bring BetterRandom to bettermathlib namespace, and then "import bettermathlib" is a much better import style than "import bettermathlib.better_random"

from .better_random import BetterRandom

Local Deployment

Pip has an option -e, called the editable option, using which pip will reference the local source paths in the deployment index.

pip install  --no-deps -e bettermath-lib/
pip install  --no-deps -e randomweb-app/

After this, both the packages are self-contained and installed in the dev environment. We can continue to make code changes without re-installing the packages.

We also want to exclude these distributions in the applications requirements.txt in case we need to regenerate requirements.txt.

pip freeze --exclude-editable > requirements.txt

For co-developed distributions we will call pip seperately for these distributions.

Intra-package and inter package references.

References is now where we will see the advantage of keeping packages distributable come in.

For referencing modules within a package, we will use the intra-package references using the . notation.

We should be ok taking dependecies on where the relative paths of modules within a package are.

from ..random_creator import random_int

For inter-package references, we can now use absolute references without worrying about where the actual code for the package resides.

from bettermathlib import BetterRandom

Class Factories

This section is flask-app sepcific, but the principle applies everywhere.

This application is a flask micro-service, and does not need a "main" anywhere. Services like uwsgi and flask commandline expect to find a callable object in a module.

I created a module called flask_app.py inside randomwebapp which creates a global app object using a class factory which lives inside another module create_app.py.

Hence the module we want to define in uwsgi.ini becomes randomwebapp.flask_app.

module = randomweb_app.flask_app
callable = app

The class factory function create_app.create_app does flask initializaiton based on the passed configuration. This is now the only user object we want to expose in my package's namespace, as it will be useful in testing.

Test cases and code coverage

With distributed packages, there are a few options for testing. We cannot rely on test-discovery, as we want my tests to be deployable.

We can exploit the __init__.py again to bring all test classes directly into the namespace of the test package.

from .tests_bettermathlib import BetterMathTestCases

Now unittests can be run via the unittest module. We can choose to run both the test packages together or seperately.

python -m unittest randomwebapp_tests bettermathlib_tests

For Flask we can use a flask command-line-interface property to explicitly define all the test packages that verify this application.

app = create_app(os.getenv('FLASK_CONFIG') or 'default')

@app.cli.command()
def test():
    """Run the unit tests."""
    import unittest
    testmodules = [
        'bettermathlib_tests',
        'randomwebapp_tests',
    ]
    suite = unittest.TestSuite()
    for t in testmodules:
        suite.addTest(unittest.defaultTestLoader.loadTestsFromName(t))
    unittest.TextTestRunner(verbosity=2).run(suite)

Now flask tests starts working for me -

(venv) ...>flask test
test_better_random (bettermathlib_tests.tests_bettermathlib.BetterMathTestCases) ... ok
test_app_exists (randomwebapp_tests.tests_basic.BasicsTestCase) ... ok
test_app_is_testing (randomwebapp_tests.tests_basic.BasicsTestCase) ... ok
test_home_page (randomwebapp_tests.tests_client.FlaskClientTestCase) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.036s

OK

and code coverage

coverage run -m flask test
coverage report

DockerFiles

There are two DockerFiles one for production and one for unit-testing. The production docker files installs the packages non-editable.

RUN python3 -m pip install -r requirements.txt
RUN python3 -m pip install  --no-deps bettermath-lib/
RUN python3 -m pip install  --no-deps randomweb-app/

The unittest takes the production docker image and uninstalls the two packages and re-installs them editable. This allows us to retrieve code-coverage results from a container execution.

RUN python3 -m pip uninstall -y bettermath-lib
RUN python3 -m pip uninstall -y randomweb-app
RUN python3 -m pip install  coverage
RUN python3 -m pip install  --no-deps -e bettermath-lib/
RUN python3 -m pip install  --no-deps -e randomweb-app/

Thank you. The complete sample project is here.

cmlzaGk - Python

Structural Pattern Matching in Python

Links

Contents

What is Structural Pattern Matching?

Motivation and Applications

Pattern is not an expression

Performance

Conclusion

Scripting Matrix Rain On Good Old Terminals

Links

Contents

What is Matrix Rain?

Approach

Character Set

Flickering Rain

Enable VT100 mode on Windows 10+

The Curses Library

Conclusion

How To Parse Like A Compiler - Part I

Summary

Problem Statement

Lark

Context Free Grammar

Parsers

Building a Grammar

Moving to LALR

Conclusion

Python Project Layout For Flask And Everything

Summary

Project Layout

setup.py

requirements.txt

init.py

Local Deployment

Intra-package and inter package references.

Class Factories

Test cases and code coverage

DockerFiles