Skip to content

Conversation

@agatti
Copy link
Contributor

@agatti agatti commented Dec 20, 2025

Summary

This PR makes it possible to load native module compiled off-line without having to enable the code generation infrastructure.

Loading native modules was tied to the presence of a native emitter since all supported platforms had one. This assumption stopped being valid in 1.27.0 with the introduction of QEMU/RV64.

Platforms like AArch64 and RV64 are probably fast enough to run Python code through the VM and using external C/C++/Rust code via native modules, without having to resort to the native, viper, or inlineasm decorators (since those platforms are probably running a proper OS, direct machine access isn't really a thing anymore). The chances for both of those architectures to receive a native code generator are slim.

To work around this issue, the native loading framework had been modified a bit, and a new compiler definition called MICROPY_PERSISTENT_CODE_LOAD_NATIVE was introduced. By default it will track the state of the native emitter, but the real benefit for this can be had by disabling all code emitters and turning that define on. This way not only the firmware image won't have the native emitter code in there (no need if the binary code already lives elsewhere), but having no compiler in the firmware may be a bullet point in a security compliance checklist.

Testing

For each platform, all MICROPY_EMIT* configuration entries have been switched off and then MICROPY_PERSISTENT_CODE_LOAD_NATIVE and MICROPY_PERSISTENT_CODE_LOAD were turned on. Before each test run, a check for @micropython.native yielding an invalid decorator error was performed, to make sure the emitter framework was fully switched off.

This was tested on the following ports/boards:

  • esp32: all xtensawin natmod tests pass
  • esp32s3: all xtensawin natmod tests pass
  • esp32c3: all rv32imc natmod tests pass
  • esp8266: with a prelude file to make enough room for loading native code, xtensa natmod tests that can fit pass except for the following tests that crash the board [1]:
    • extmod/deflate_stream_error
    • extmod/re1
  • rp2/RPI_PICO: all armv6m natmod tests pass
  • rp2/RPI_PICO2: armv7emsp natmod tests pass except for the following tests:
    • extmod/random_extra_float [2]
  • rp2/RPI_PICO2 RISC-V: all rv32imc natmod tests pass
  • qemu/MPS2_AN385: all armv7m natmod tests pass
  • qemu/VIRT_RV32: all rv32imc natmod tests pass
  • unix/coverage x64: all x64 natmod tests pass.

[1] Haven't yet looked at that, probably it ran out of memory in ways it didn't expect to be possible
[2] Haven't looked at this in detail, chances are there's something related to floating point rounding or something like that.

What was not tested:

  • The nRF port received a minor change, but I have no suitable board to test that with
  • Same applies to Linux/ArmV7 - the Linux/Arm devices I've got are all V8.

Trade-offs and Alternatives

The micropython/import_mpy_native and micropython/import_mpy_native_gc tests use the fact that the native architecture index in sys.implementation._mpy is not 0. This means that unless those tests are modified to skip relevant platforms where loading is not an option (ie. windows and maybe macos), the mpy target index has to appear only if there's any native code loading capability in the current interpreter.

@codecov
Copy link

codecov bot commented Dec 20, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.38%. Comparing base (6341258) to head (19f6b02).

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #18596   +/-   ##
=======================================
  Coverage   98.38%   98.38%           
=======================================
  Files         171      171           
  Lines       22298    22298           
=======================================
  Hits        21937    21937           
  Misses        361      361           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link

github-actions bot commented Dec 20, 2025

Code size report:

Reference:  esp32/mpconfigport: Enable Zcmp opcodes for ESP32P4. [6341258]
Comparison: tests/run-natmodtests.py: Explicitly open prelude file. [merge of 19f6b02]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

@agatti agatti force-pushed the natmod-load-without-emitter branch from 7028855 to 271e27c Compare December 20, 2025 16:12
This commit lets the interpreter load MPY files containing native code
even if the target platform does not have a native emitter, or if native
code generation is disabled.

Native code loading has been tied to native code generation being
enabled as a discriminant to allow said operation.  This blocks native
code loading on platforms that could benefit from such a thing but they
don't (and probably won't) have a native code generation target written
for them (ie. AArch64 and RISC-V 64).  This also forces a firmware image
to have a full native code compiler present even if it doesn't need to
generate anything, as native modules already have all the code they will
ever need to load.

There is a new configuration setting,
MICROPY_PERSISTENT_CODE_LOAD_NATIVE, that if enabled it will allow
loading native code modules even if code generation
(MICROPY_EMIT_<platform> and MICROPY_EMIT_INLINE_<platform>) is
explicitly turned off.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit provides an alternate platform identification mechanism so
"sys.implementation._mpy" will report the running target type in all
cases.

Before these changes the target type was inferred from which kind of
native emitter was enabled, since there can be only just one available
at all times and it has to be the correct one otherwise generated code
would crash the target.  However, with the introduction of RV64 there is
now a platform without an emitter, and thus RV64 targets would not be
able to report their platform via "sys.implementation._mpy".  The target
is reported only if the interpreter is configured to load external MPY
code.

Whilst this is probably not directly affecting user-facing code, some
infrastructure components like the testing framework and its associated
feature scripts rely on this feature to properly execute test runs.
This is also a concern for situations in which loading external binary
code is needed but all code generation functions have to be disabled.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit forces a data/instruction cache flush after loading native
code blocks even if there's no emitter available.

Caches flush upon native code load was still tied to the presence of a
native emitter, but this assumption is no longer valid due to previous
commits decoupling native code loading from code generation
availability.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the Unix port use the new
MICROPY_PERSISTENT_CODE_LOAD_NATIVE configuration entry.

The Unix port needs to allocate memory with specific flags that let the
operating system run executable code from there.  This functionality was
gated behind the presence of a native emitter and thus its inclusion
condition had to be updated.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit modifies the QEMU port makefile to let the user provide
their own list of natmods to test as part of "test_natmod".

The makefile now will replace the list of natmods to test with the
contents of the "TEST_NATMODS" command line variable, so if there's a
specific subset of natmods causing problems test runs can be limited to
that very subset.  "TEST_NATMODS" accepts a whitespace-separated list of
natmod names for which there are available matching tests in
"tests/extmod" (eg. make test_natmod TEST_NATMODS="btree heapq").

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the ESP8266 port use the new
MICROPY_PERSISTENT_CODE_LOAD_NATIVE configuration entry.

The ESP8266 port needs a special procedure to allocate memory used to
hold executable native code.  This functionality was gated behind the
presence of a native emitter and thus its inclusion condition had to be
updated.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the nRF port use the new
MICROPY_PERSISTENT_CODE_LOAD_NATIVE configuration entry.

The nRF port needs a special procedure to allocate memory used to hold
executable native code.  This functionality was gated behind the
presence of a native emitter and thus its inclusion condition had to be
updated.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This change lets the natmod test runner report status information on
session end if a prelude script file is chosen.

The script serialises its input data as part of the end of session
reporting data, but since the prelude file is not a serialisable object
serialisation would fail (it's a file handle as far as the argument
container structure is aware).

Now the file is explicitly open by the script rather than relying on
argparse's file handle argument class wrapper.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
@agatti agatti force-pushed the natmod-load-without-emitter branch from 271e27c to 19f6b02 Compare December 20, 2025 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant