-
-
Notifications
You must be signed in to change notification settings - Fork 8.6k
py/persistentcode: Decouple native code loading from emitters' presence. #18596
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
agatti
wants to merge
8
commits into
micropython:master
Choose a base branch
from
agatti:natmod-load-without-emitter
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+103
−51
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #18596 +/- ##
=======================================
Coverage 98.38% 98.38%
=======================================
Files 171 171
Lines 22298 22298
=======================================
Hits 21937 21937
Misses 361 361 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Code size report: |
7028855 to
271e27c
Compare
This commit lets the interpreter load MPY files containing native code even if the target platform does not have a native emitter, or if native code generation is disabled. Native code loading has been tied to native code generation being enabled as a discriminant to allow said operation. This blocks native code loading on platforms that could benefit from such a thing but they don't (and probably won't) have a native code generation target written for them (ie. AArch64 and RISC-V 64). This also forces a firmware image to have a full native code compiler present even if it doesn't need to generate anything, as native modules already have all the code they will ever need to load. There is a new configuration setting, MICROPY_PERSISTENT_CODE_LOAD_NATIVE, that if enabled it will allow loading native code modules even if code generation (MICROPY_EMIT_<platform> and MICROPY_EMIT_INLINE_<platform>) is explicitly turned off. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit provides an alternate platform identification mechanism so "sys.implementation._mpy" will report the running target type in all cases. Before these changes the target type was inferred from which kind of native emitter was enabled, since there can be only just one available at all times and it has to be the correct one otherwise generated code would crash the target. However, with the introduction of RV64 there is now a platform without an emitter, and thus RV64 targets would not be able to report their platform via "sys.implementation._mpy". The target is reported only if the interpreter is configured to load external MPY code. Whilst this is probably not directly affecting user-facing code, some infrastructure components like the testing framework and its associated feature scripts rely on this feature to properly execute test runs. This is also a concern for situations in which loading external binary code is needed but all code generation functions have to be disabled. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit forces a data/instruction cache flush after loading native code blocks even if there's no emitter available. Caches flush upon native code load was still tied to the presence of a native emitter, but this assumption is no longer valid due to previous commits decoupling native code loading from code generation availability. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the Unix port use the new MICROPY_PERSISTENT_CODE_LOAD_NATIVE configuration entry. The Unix port needs to allocate memory with specific flags that let the operating system run executable code from there. This functionality was gated behind the presence of a native emitter and thus its inclusion condition had to be updated. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit modifies the QEMU port makefile to let the user provide their own list of natmods to test as part of "test_natmod". The makefile now will replace the list of natmods to test with the contents of the "TEST_NATMODS" command line variable, so if there's a specific subset of natmods causing problems test runs can be limited to that very subset. "TEST_NATMODS" accepts a whitespace-separated list of natmod names for which there are available matching tests in "tests/extmod" (eg. make test_natmod TEST_NATMODS="btree heapq"). Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the ESP8266 port use the new MICROPY_PERSISTENT_CODE_LOAD_NATIVE configuration entry. The ESP8266 port needs a special procedure to allocate memory used to hold executable native code. This functionality was gated behind the presence of a native emitter and thus its inclusion condition had to be updated. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the nRF port use the new MICROPY_PERSISTENT_CODE_LOAD_NATIVE configuration entry. The nRF port needs a special procedure to allocate memory used to hold executable native code. This functionality was gated behind the presence of a native emitter and thus its inclusion condition had to be updated. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This change lets the natmod test runner report status information on session end if a prelude script file is chosen. The script serialises its input data as part of the end of session reporting data, but since the prelude file is not a serialisable object serialisation would fail (it's a file handle as far as the argument container structure is aware). Now the file is explicitly open by the script rather than relying on argparse's file handle argument class wrapper. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
271e27c to
19f6b02
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes it possible to load native module compiled off-line without having to enable the code generation infrastructure.
Loading native modules was tied to the presence of a native emitter since all supported platforms had one. This assumption stopped being valid in 1.27.0 with the introduction of QEMU/RV64.
Platforms like AArch64 and RV64 are probably fast enough to run Python code through the VM and using external C/C++/Rust code via native modules, without having to resort to the native, viper, or inlineasm decorators (since those platforms are probably running a proper OS, direct machine access isn't really a thing anymore). The chances for both of those architectures to receive a native code generator are slim.
To work around this issue, the native loading framework had been modified a bit, and a new compiler definition called
MICROPY_PERSISTENT_CODE_LOAD_NATIVEwas introduced. By default it will track the state of the native emitter, but the real benefit for this can be had by disabling all code emitters and turning that define on. This way not only the firmware image won't have the native emitter code in there (no need if the binary code already lives elsewhere), but having no compiler in the firmware may be a bullet point in a security compliance checklist.Testing
For each platform, all
MICROPY_EMIT*configuration entries have been switched off and thenMICROPY_PERSISTENT_CODE_LOAD_NATIVEandMICROPY_PERSISTENT_CODE_LOADwere turned on. Before each test run, a check for@micropython.nativeyielding an invalid decorator error was performed, to make sure the emitter framework was fully switched off.This was tested on the following ports/boards:
extmod/deflate_stream_errorextmod/re1extmod/random_extra_float[2][1] Haven't yet looked at that, probably it ran out of memory in ways it didn't expect to be possible
[2] Haven't looked at this in detail, chances are there's something related to floating point rounding or something like that.
What was not tested:
Trade-offs and Alternatives
The
micropython/import_mpy_nativeandmicropython/import_mpy_native_gctests use the fact that the native architecture index insys.implementation._mpyis not 0. This means that unless those tests are modified to skip relevant platforms where loading is not an option (ie. windows and maybe macos), the mpy target index has to appear only if there's any native code loading capability in the current interpreter.