tests: add full, automated, top-level test runner for hardware#16112
Draft
dpgeorge wants to merge 61 commits into
Draft
tests: add full, automated, top-level test runner for hardware#16112dpgeorge wants to merge 61 commits into
dpgeorge wants to merge 61 commits into
Conversation
Member
Author
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #16112 +/- ##
=======================================
Coverage 98.47% 98.47%
=======================================
Files 176 176
Lines 22845 22845
=======================================
Hits 22497 22497
Misses 348 348 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
Octoprobe is ready so far to run these tests fully automated, including build of all firmware variants and to be triggered from github workflows. We have to agree on triggers (github action on PR, github manual action). I will start the discussion when I am back from vacation. |
b8ce03c to
2047b05
Compare
2047b05 to
39adb22
Compare
39adb22 to
31a289e
Compare
31a289e to
6ef6bd8
Compare
|
Code size report: |
This was referenced Aug 12, 2025
7875959 to
d668cbe
Compare
d668cbe to
943efd1
Compare
943efd1 to
63531a4
Compare
Signed-off-by: Damien George <damien@micropython.org>
TODO: make it a CLI option Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
There's already a --results-dir argument, the directory is already made, and a _results.json is already put there. So adding .exp/.out makes sense. Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
To reduce output clutter. Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
If it's too big PYBLITEV10 will run out of memory when running `extmod/deflate_decompress.py`. Actually, 32-7 is the default... so this ioctl could be removed. Really need to benchmark it to see if it makes any difference having it large. Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Fixed on tests-improvements-for-octoprobe. Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
TODO: can we remove this esp32-specific code? Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
pin.irq doesn't yet work, seems it files rising and falling regardless of config. Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
63531a4 to
2ddf4db
Compare
Contributor
|
On macOS it seems to hiccup on Should probably catch these exceptions and continue, like this but less grotty: diff --git a/lib/nrfx b/lib/nrfx
index 11f57e578..7a4c9d946 160000
--- a/lib/nrfx
+++ b/lib/nrfx
@@ -1 +1 @@
-Subproject commit 11f57e578c7feea13f21c79ea0efab2630ac68c7
+Subproject commit 7a4c9d946cf1801771fc180acdbf7b878f270093
diff --git a/tests/hwtest.py b/tests/hwtest.py
index c53b65a01..52d714ae6 100755
--- a/tests/hwtest.py
+++ b/tests/hwtest.py
@@ -12,7 +12,7 @@ import time
import serial_test
sys.path.append("../tools/mpremote")
-from mpremote.transport_serial import SerialTransport
+from mpremote.transport_serial import SerialTransport, TransportError
import mpremote.commands
import mpremote.mip
@@ -91,6 +91,7 @@ def map_device_name(d):
def detect_octoprobe(device):
+ print(f"detect_octoprobe({device})")
t = SerialTransport(device)
t.enter_raw_repl()
t.exec("import os")
@@ -106,9 +107,13 @@ def list_targets(args, exclude_devices):
for device in args:
for p in valid_serial_ports:
if device == p.device:
- if detect_octoprobe(device):
- print(f"{device} Octoprobe infrastructure device")
- targets.append(Target(device, p.serial_number))
+ try:
+ if detect_octoprobe(device):
+ print(f"{device} Octoprobe infrastructure device")
+ targets.append(Target(device, p.serial_number))
+ except TransportError:
+ print(f"{device} could not connect!")
+ continue
break
else:
print(f"{device} not found")
@@ -120,9 +125,13 @@ def list_targets(args, exclude_devices):
continue
if p.device in exclude_devices:
continue
- if detect_octoprobe(p.device):
- # If no devices specified, skip Octoprobe infrastructure.
- print(f"{p.device} Octoprobe infrastructure device, skipping")
+ try:
+ if detect_octoprobe(p.device):
+ # If no devices specified, skip Octoprobe infrastructure.
+ print(f"{p.device} Octoprobe infrastructure device, skipping")
+ continue
+ except TransportError:
+ print(f"{p.device} could not connect!")
continue
targets.append(Target(p.device, p.serial_number))
return targets |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With the aim to speed up testing for a release, I put together a script to automatically run tests on devices running MicroPython. We have a lot of different tests these days, with different ways to run them. The idea with this top-level test runner is:
Basically all you need to do is connect boards to your PC via USB and run:
Then it does the rest. It takes between 10 and 15 minutes per board, so if you have many connected it can take a few hours. You can also restrict to given devices by specifying them on the command line, eg:
(Probably the name of the script
hwtest.pycan change to something better.)Testing
This was used to test 16 distinct boards for the recent v1.24.0 release.
Trade-offs and Alternatives
This is a bit of a hacky script at the moment but can be improved over time.
Currently there are no boards that pass all tests (!!) and so over time we need to fix things and improve the tests so more of them can pass.