This pull request adds full support for RISC-V (#972) and ARM64 (#2024) assembly, including both backend support and full frontend support including assembled byte visualization.

It achieves this by using GNU as to assemble, GNU ld to link, objcopy to canonicalize the executable, and QEMU to run the final executable, as well as a custom assembly preprocessor that adds the DefAsm features that are missing in GNU as.

On the frontend, rather than using DefAsm, it uses a DefAsm-compatible implementation that uses those same tools compiled to WebAssembly, asks the assembler to produce debug information and uses objdump to read both the byte content and the mapping to source code lines.

While obviously this is less performant than the DefAsm approach (since it's non-incremental and uses heavier tools), in practice performance seems very good even on a relatively old machine. A drawback is that compiled bytes cannot be seen at all if there is any assembly error.

It is also possible to relatively easily add any other assembly language supported by both QEMU and the GNU toolchain.

General design

The code is designed to enable all possible ISA extensions supported by QEMU to make code golfing more interesting.

For ARM64, this might need some work as I'm not sure I truly got all of them, although I don't think it matters for code golfing.

For RISC-V, it is possible to add a comment in the first line of the source code to select RV32 or RV64 and to choose between sets of incompatible extensions (currently only Zcmp+Zcmt vs Zcd). I think adding support for RV32 as well makes code golfing a bit more interesting, and I think having a single language with a choice is much better than adding two extremely similar languages.

Backend design

The backend uses a custom as-ld-qemu.c driver that performs the following steps:

0. Preprocess the assembly using internal code (in lib/asm/asm.[ch])
1. Assemble the transformed assembly code with all QEMU-supported ISA extensions enabled, respecting the user choice for RISC-V
2. Link the ELF object file to produce an ELF executable with a custom linker script
3. Read the ELF executable with internal code to find out the entry point
4. Use objcopy to copy the ELF executable to a flat binary. This ensures that there are no side channels such as section names that can be used to smuggle information that is not counted in the solution length.
5. Count bytes in the binary, excluding zero bytes that are at the end of pages (to simulate ignoring padding after sections) and output the result on fd 3
6. Use objcopy to convert the flat binary to a second ELF object file
7. Link the second ELF object file to produce an ELF executable with another custom linker script, setting the entry point to the value we found out earlier
8. Run the final executable with QEMU in user emulation mode setting options to enable all possible ISA extensions. QEMU is patched to dump registers on crash like the x86-64 assembly language support does, and to work properly as pid 1 in a container

Frontend design

The frontend is implemented like this:

1. The @defasm/codemirror extension is replaced with an internal fork with slight changes to provide a non-x86-specific interface, to allow to pass any DefAsm-compatible backend in the constructor and to separate the syntax highlighting support that doesn't need an assembler from the other plugins that need it
2. A new extasm package is added that offers an interface similar to DefAsm, but implements it in terms of any assembler implementing assembler-interface. Unlike DefAsm, it is not incremental, but rather compiles everything every time and reconstructs StatementNode and ASMError objects from the assembler-interface results.
3. A new assembler-interface package that defines an interface for assembling code, tailored for command-line assemblers compiled to WebAssembly
4. A new assemble package that implements the aforementioned interface using aspp (a standalone Wasm-compiled version of the assembly preprocessing code in the backend), as, ld and objdump. It assembles using the same flags as the backend plus flags to enable debug information, links the object file so that relocated addresses are properly visible in the byte dump, and then uses objdump to decode the DWARF line number information and to dump sections contents, from which is reconstructs fragments consisting of bytes, an address, and a source location. It also parses as and ld error messages.
5. A new wasm-program package that provides an efficient implementation of the Emscripten support code, allowing to run emscripten-compiled Wasm programs without the Emscripten-generated JavaScript. Compared to the Emscripten-generated JavaScript, it supports running a program multiple times without recompiling or even reinstatiating it, and is much better written than the Emscripten code.
6. binutils/wasm and aspp docker packages that compile Binutils and the assembly preprocessor to WebAssembly using Emscripten. The binutils/wasm code includes very simple patches to GAS to make it generate debug information for non-.text sections, for multiple instructions on the same lines, and for directives.

Design rationale

The first choice is what assembler to use. x86 uses DefAsm, which has the advantage of being written in JavaScript and supports incremental and error-tolerant assembly, but requires to manually implement any other architecture, and also doesn't fully support the features of the usual assemblers like macros (although it's in fact very good, and has some extra features like UTF-8 character literals, which are implemented by the assembly preprocessor in this pull request).

Since I didn't feel like implementing RISC-V and other architectures in DefAsm and maintaining such code, this code instead chooses to use a normally used assembler, i.e. GNU as, and compile it to WebAssembly for the frontend. The main mostly unfixable loss is incremental assembly and error-tolerant assembly, but the gain is making it much easier to add any assembly language.

There is also a choice between GNU as and LLVM-MC and GNU as is the more standard program, and also seems better than the LLVM assembler at computing relative offsets between labels (although it is itself quite poor at it, being unfortunately a mostly single-pass assembler, but LLVM-MC seems to somehow be even worse).

Another choice is how to run the executables, and QEMU seems the obvious choice, since it provides Linux user emulation allowing to provide a similar environment to the current x86-64 assembly, and supports many architectures.

Finally there is the problem of how to run the assembler and other tools in the frontend. Here, there are two main choices, namely vanilla Emscripten and WASI SDK. Unfortunately they are both bad choices: the WASI SDK libc lacks basic functionality like an mktemp function in the libc and the WASI interface is somewhat limited; on the other hand, Emscripten has a good libc and interface, but the glue JavaScript code it generates is absolutely awful and doesn't allow to run programs multiple times without recompiling everything from scratch. Hence, I finally opted for a custom solution that leverages the Emscripten compiler to produce a WebAssembly module, but throws away the generated JavaScript code and instead uses a well-written and modular custom implementation of the required parts of the WASI and Emscripten ABIs (in the wasm-program package).

On the non-implementation side, there is a choice of which assembly languages to include.

RISC-V has been included since it's the most requested and it's the standard open-source ISA. It is much less interesting from a code golfing perspective than x86-64 assembly, but compressed instructions as well as the possibility of overlapping instructions still provide a bit of opportunity, and has the advantage of being much easier to learn and code golf than x86-64 assembly.

ARM64 has been included mostly because it's popular in actual CPUs and has been requested. Unlike RISC-V, instructions are always 4 bytes, which makes it kind of sad for code golfing, although it has some more complex instructions like post/pre-increment in memory access and comparisons with immediates. Overall I think it's not very interesting for code golfing, but I added anyway since it's easy to do it with this framework and it's popular in commercial systems (mainly smartphones, Apple computers and some cloud servers) so there is presumably a relatively significant number of people who know it.

ARM32 could be another option, seemingly more interesting due to the compressed instruction modes like THUMB2.

6502 (#628) is also requested, and is a bit more difficult since current QEMU doesn't support it, so an emulator needs to be chosen or implemented, probably by implementing a very simple one using one of the 6502 libraries available, providing I/O with a read/write port where reading past the end terminates the program. Regarding the assembler, LLVM-MOS seems a natural choice since it produces ELF files allowing to use the same objdump/objcopy toolchain.

It would also be possible to implement x86-64 assembly with this framework, and the assembly preprocessor should allow to do it without breaking current solutions, although QEMU might need some simple patches to make it behave like Zen 4 in particular for the undocumented flag setting behavior of division and multiplication. However, DefAsm works fine and has some advantages (incremental and error-tolerant compilation), so there doesn't seem to be a pressing reason to do this.