Motivation

Benchmarking has been used occasionally as hoc to decide between different solutions for implementing a feature. I would like to provide a way do this in our GitHub environment. This will also allow us to monitor performance degradation or improvement as the project matures.

Design proposal

I would like to have a benchmarking workflow that can run on GitHub actions and runs for all four of our supported lua versions (not fengari like the tests!). This workflow would have several benchmark scripts that it transpiled to Lua, and then measures the execution time of the Lua over a number of repetitions. I am interested in the mean execution time and standard deviation.

Accounting for GitHub actions machine differences

Assumption: GitHub action host performance stays roughly constant for the duration of one matrix build run.

We have no control over the way GitHub executes actions and on what machines, so we can not directly compare results between runs. That is a problem. To solve this I propose a 'baseline benchmark', which we can use as benchmark for all other benchmarks in the same build. So results are '1.3x baseline' instead of 1.3s.

Benchmarking code

I propose adding some benchmark scripts in the /benchmark directory in the repository. Their code would be similar to our test builders:

util.benchmark`
    const add = (x: number, y: number) => x + y;
    return add(1, 2);
`
    . repetitions (100);

This code would transpile the TS into lua, and measure the execution time of only the Lua execution.

Further ideas to investigate

We could maybe at some point consider setting some performance gate to display in pull requests. When performance compared to master degrades by a factor X, the run fails.