Skip to content

natmod: Add RV64 native modules support. #18428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
agatti wants to merge 5 commits into
base: master
Choose a base branch
from
Draft

natmod: Add RV64 native modules support. #18428

agatti wants to merge 5 commits into from
+94 −46

Conversation

@agatti
Copy link
Contributor

@agatti agatti commented Nov 17, 2025 •
edited
Loading

Summary

This PR adds natmod support for RV64, including the necessary changes to MicroPython's core, enabling this feature in the QEMU port, and test the feature as part of the CI jobs set.

Now, the interesting part. Natmod file loading has historically been tied to the presence of a native emitter, since there was no architecture that did not have such a thing. This, however, stopped being true with the introduction of RV64 support as no native emitter is available for said architecture.

Loading native modules shouldn't really depend on emitting code, since all the code to load is already there. For certain scenarios where native modules loading is needed, this could be something worthwhile, as the interpreter may be made smaller with all native code support generation not shipping with the main firmware image. The changes in this PR can selectively enable the code blocks needed to load and run native modules without the rest of the emitter framework being there, as needed for the RV64 architecture.

The native modules loader changes can also make possible supporting AArch64 without having to write a brand-new emitter, just like for RV64.

Along with core changes, tools/mpy_ld.py now has support for all needed relocation types (and got a bug fixed too!), the rv64imc value is recognised by py/dynruntime.mk's ARCH variable checks, the natmod test runner was updated to recognise RV64 as part of the potential targets, and the documentation plus the example natmods have been updated to mention the new architecture.

The Unix port hasn't been updated to load natmods when running the RV64 binary as I haven't done any on-device testing, I'll get around to that as soon as I remember where I put my MilkV-Duo board.

Testing

Besides manual testing on QEMU, the QEMU/RV64 CI job was updated to build and run natmods on RV64, just like RV32. RV64 tasks don't fail on my repo's branch, so hopefully they should work on this one too.

Trade-offs and Alternatives

I'm not really happy about the changes in py/ related to MICROPY_LOAD_NATIVE_MODULES, but I didn't really find any other obvious way to get that done with fewer code changes. I'm sure this can be done better, and I'm open to suggestions for this.

The same applies to the change made in py/persistentcode.h to let sys.implementation._mpy report that it's running on RV64. The target architecture in that tuple is based on what kind of native emitter is enabled, which may have been correct before, but it is not the case anymore.

I thought about factoring out the platform detection used in py/nlr.h and apply that to define MPY_FEATURE_ARCH, but that's a bit too much code to touch and it may break something else. I'd like some feedback about that - changing the relevant commit to do that instead isn't an issue at all.

@agatti
py/nativeglue: Enable natmod loading without an available emitter.
36eb1e1 
This commit lets the persistent code loader to load natmods even if
there is no emitter being present for the platform in use.

Native modules loading does not need an emitter being available, and
until the addition of RV64 to the architectures list there was no
platform without an associated native code emitter.  This also means
that native module loading for RV64 could not happen until a matching
emitter is written for it.

Still, that can be worked around by adding a new configuration define
("MICROPY_LOAD_NATIVE_MODULES") to short-circuit the define checks that
bring in native code loading to also be enabled if natmod support is
explicitly enabled.  The changes are a bit all over the place but this
seems to make it work for RV64.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
@agatti
tools/mpy_ld.py: Fix R_RISCV_GOT32_PCREL handling.
2b93456 
This commit fixes the implementation of the R_RISCV_GOT32_PCREL RISC-V
relocation type when linking native modules for RV32IMC.

The previous implementation of R_RISCV_GOT32_PCREL ended up not being
fully updated when the initial RV32 support was checked in.  Said
relocation was not emitted in the sample natmods that ship with the
MicroPython source tree, and since they're the testbed for CI jobs that
should check RV32 natmod support, this was not caught.

On the other hand, nobody raised an issue about this problem yet, so
hopefully it didn't cause much trouble.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
@agatti agatti mentioned this pull request Nov 17, 2025
Open
@github-actions
Copy link

github-actions bot commented Nov 17, 2025 •
edited
Loading

Code size report:

Reference:  esp32/mpconfigport: Enable Zcmp opcodes for ESP32P4. [6341258]
Comparison: qemu: Enable loading natmods on RV64. [merge of e689f41]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

@codecov
Copy link

codecov bot commented Nov 17, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.41%. Comparing base (336d463) to head (e689f41).
⚠️ Report is 131 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master   #18428      +/-   ##
==========================================
+ Coverage   98.38%   98.41%   +0.03%     
==========================================
  Files         171      171              
  Lines       22294    22965     +671     
==========================================
+ Hits        21933    22602     +669     
- Misses        361      363       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dpgeorge dpgeorge added the py-core Relates to py/ directory in source label Nov 19, 2025
@agatti
tools/mpy_ld.py: Add RV64 natmod support.
c61aa83 
This commit adds the ability to compile native modules for the RV64
platform, using "rv64imc" as its architecture name (eg.
"make ARCH=rv64imc" should build a RV64 natmod).

The rest of 64-bits relocations needed to build a native module are now
implemented, and all sample native modules build without errors and
warnings.  The same Picolibc caveats on RV32 also apply on RV64, thus
the documentation was updated accordingly.

RV64 are also built as part of the CI process, but not yet executed as
the QEMU port is not yet able to load and run native modules.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
@agatti
tests/run-natmodtests.py: Add RV64 natmod support.
993781e 
This commit extends the natmod test runner to also recognise the RV64
architecture when reported by the feature check script.

However, said script bases its findings on the output of
"sys.implementation._mpy", and the platform identifier reported in that
tuple depends on whether a native emitter is present or not.

Since natmod loading has been made independent of the presence of a
native emitter, RV64 needs a platform-specific definition check to
provide the correct value to "MPY_FEATURE_ARCH", and thus let the test
runner accept that architecture.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
@agatti
qemu: Enable loading natmods on RV64.
e689f41 
This commit lets the QEMU port's VIRT_RV64 board load and run native
modules, via the appropriate configuration changes in mpconfigport.h.

Now the CI test job for the QEMU/RV64 port can also run natmods and see
whether they actually work, so the CI tasks script has been updated to
bring RV64 to parity with RV32 as far as CI checks go.

Documentation was also updated, since now all supported boards in the
QEMU port should be able to run natmod tests.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
@dpgeorge
Copy link
Member

dpgeorge commented Dec 19, 2025

Thanks for this, it's a good addition. I did myself want to tackle this issue of emitting vs loading native code.

The way you approached it with MICROPY_LOAD_NATIVE_MODULES could probably be improved. One idea (which is still a bit hacky) would be to just add a new MICROPY_EMIT_RV64 and enable it, and then adjust py/compile.c enough so that it doesn't actually support @micropython.native/@micropython.viper for the case when MICROPY_EMIT_RV64 is enabled. Similarly, there would be no py/emitnrv64.c at all.

If you want to go for a more long term solution, I would suggest opening a separate PR that does a refactoring of the configuration options as follows:

  • adds a new MICROPY_PERSISTENT_CODE_LOAD_NATIVE option, that enables loading of native/machine code in .mpy files (just like MICROPY_PERSISTENT_CODE_LOAD enables loading of bytecode from .mpy files)
  • enabling this option would automatically select the correct value for MPY_FEATURE_ARCH (that may be tricky to do...)
  • adjust py/persistentcode.c as appropriate to use this new option instead of MICROPY_EMIT_MACHINE_CODE
  • enable this option automatically if MICROPY_EMIT_MACHINE_CODE is enabled

Then once that PR is merged, come back to this one.

@agatti
Copy link
Contributor Author

agatti commented Dec 20, 2025 •
edited
Loading

Thank you for the suggestions. I went for the long-term solution using your hints as a base, which does work on Unix but only partially on embedded targets. which so far works on Unix, QEMU, and on Xtensa with some caveats, I'll have to test this on ESP32, ESP32S3, ESP32C3, RP2040 and RP2350 before submitting a PR.

On QEMU tests/extmod/deflate_compress.py and tests/extmod/extmod/framebuf_ellipse.py crash for both Arm and RV32 (probably hiding more issues in the respective tests set), and on Xtensa nothing loads.

The difficulty here is that on QEMU using QEMU_DEBUG=1 for natmods doesn't properly attach the debugger, and cranking up the debugging information output stops mpremote from mounting the filesystem. The test runner not capturing the crash dump output on failure isn't helping either :)

I've resorted to merge my semihosting FS implementation PR and load natmods that way and running tests from the REPL by copying and pasting bits of code at a time, and it doesn't crash that way. (spoiler: It was native code pointer tracking that wasn't enabled...)

Anyway, there's plenty of work to do on this it seems... The changes I've made are here if you want to take a look when you've got some time: https://github.com/agatti/micropython/tree/natmod-load-without-emitter

I'll mark this PR as draft as it's not yet ready.

Edit: parent PR is ready (#18596)

@agatti agatti closed this Dec 20, 2025
@agatti agatti reopened this Dec 20, 2025
@agatti agatti marked this pull request as draft December 20, 2025 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

py-core Relates to py/ directory in source

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@agatti @dpgeorge