Skip to content

tests: add full, automated, top-level test runner for hardware#16112

Draft
dpgeorge wants to merge 61 commits into
micropython:masterfrom
dpgeorge:tests-full-test-runner
Draft

tests: add full, automated, top-level test runner for hardware#16112
dpgeorge wants to merge 61 commits into
micropython:masterfrom
dpgeorge:tests-full-test-runner

Conversation

@dpgeorge

Copy link
Copy Markdown
Member

Summary

With the aim to speed up testing for a release, I put together a script to automatically run tests on devices running MicroPython. We have a lot of different tests these days, with different ways to run them. The idea with this top-level test runner is:

  • detect all serial devices connected
  • detect what platform is running on each device and what its capabilities are (eg WLAN, BLE, ability to import .mpy files)
  • run as many tests as possible given the connected devices (eg if there are two WLAN boards connected, then run the network multitests against them, among other things)

Basically all you need to do is connect boards to your PC via USB and run:

$ cd tests
$ ./hwtest.py

Then it does the rest. It takes between 10 and 15 minutes per board, so if you have many connected it can take a few hours. You can also restrict to given devices by specifying them on the command line, eg:

$ ./hwtest.py a0 a1 u0

(Probably the name of the script hwtest.py can change to something better.)

Testing

This was used to test 16 distinct boards for the recent v1.24.0 release.

Trade-offs and Alternatives

This is a bit of a hacky script at the moment but can be improved over time.

Currently there are no boards that pass all tests (!!) and so over time we need to fix things and improve the tests so more of them can pass.

@dpgeorge dpgeorge added the tests Relates to tests/ directory in source label Oct 30, 2024
@dpgeorge dpgeorge marked this pull request as draft October 30, 2024 03:22
@dpgeorge dpgeorge changed the title tests: add full, automated test running for hardware tests: add full, automated, top-level test runner for hardware Oct 30, 2024
@dpgeorge

Copy link
Copy Markdown
Member Author

PR #16111 will help simplify this test runner a bit.

PR #15909 is included in this PR.

@codecov

codecov Bot commented Oct 30, 2024

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.47%. Comparing base (75555f4) to head (2ddf4db).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #16112   +/-   ##
=======================================
  Coverage   98.47%   98.47%           
=======================================
  Files         176      176           
  Lines       22845    22845           
=======================================
  Hits        22497    22497           
  Misses        348      348           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hmaerki

hmaerki commented Oct 30, 2024

Copy link
Copy Markdown
Contributor

Octoprobe is ready so far to run these tests fully automated, including build of all firmware variants and to be triggered from github workflows.

We have to agree on triggers (github action on PR, github manual action). I will start the discussion when I am back from vacation.

@dpgeorge dpgeorge force-pushed the tests-full-test-runner branch 2 times, most recently from b8ce03c to 2047b05 Compare November 4, 2024 05:56
@dpgeorge dpgeorge force-pushed the tests-full-test-runner branch from 2047b05 to 39adb22 Compare November 24, 2024 02:26
@dpgeorge dpgeorge force-pushed the tests-full-test-runner branch from 39adb22 to 31a289e Compare January 15, 2025 00:58
@dpgeorge dpgeorge force-pushed the tests-full-test-runner branch from 31a289e to 6ef6bd8 Compare April 15, 2025 03:56
@github-actions

Copy link
Copy Markdown

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

dpgeorge added 10 commits June 4, 2026 23:22
Signed-off-by: Damien George <damien@micropython.org>
TODO: make it a CLI option

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
There's already a --results-dir argument, the directory is already made,
and a _results.json is already put there.  So adding .exp/.out makes sense.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
To reduce output clutter.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
If it's too big PYBLITEV10 will run out of memory when running
`extmod/deflate_decompress.py`.

Actually, 32-7 is the default... so this ioctl could be removed.  Really
need to benchmark it to see if it makes any difference having it large.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
dpgeorge added 28 commits June 4, 2026 23:22
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Fixed on tests-improvements-for-octoprobe.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
TODO: can we remove this esp32-specific code?

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
pin.irq doesn't yet work, seems it files rising and falling regardless of
config.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge dpgeorge force-pushed the tests-full-test-runner branch from 63531a4 to 2ddf4db Compare June 4, 2026 13:23
@Gadgetoid

Copy link
Copy Markdown
Contributor

On macOS it seems to hiccup on /dev/cu.Bluetooth-Incoming-Port:

 ./hwtest.py   
WLAN_SSID and/or WLAN_PASS not provided
Selected tests: serial, bytecode, hardware, via-mpy, native, wlan, ble

================================================================
TARGETS TO BE TESTED
detect_octoprobe(/dev/cu.Bluetooth-Incoming-Port)
b''
Traceback (most recent call last):
  File "micropython/micropython/tests/./hwtest.py", line 447, in <module>
    main()
    ~~~~^^
  File "micropython/micropython/tests/./hwtest.py", line 334, in main
    targets = list_targets(selected_devices, [ref_target.device] if ref_target else [])
  File "micropython/micropython/tests/./hwtest.py", line 124, in list_targets
    if detect_octoprobe(p.device):
       ~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "micropython/micropython/tests/./hwtest.py", line 96, in detect_octoprobe
    t.enter_raw_repl()
    ~~~~~~~~~~~~~~~~^^
  File "micropython/micropython/tests/../tools/mpremote/mpremote/transport_serial.py", line 179, in enter_raw_repl
    raise TransportError("could not enter raw repl")
mpremote.transport.TransportError: could not enter raw repl

Should probably catch these exceptions and continue, like this but less grotty:

diff --git a/lib/nrfx b/lib/nrfx
index 11f57e578..7a4c9d946 160000
--- a/lib/nrfx
+++ b/lib/nrfx
@@ -1 +1 @@
-Subproject commit 11f57e578c7feea13f21c79ea0efab2630ac68c7
+Subproject commit 7a4c9d946cf1801771fc180acdbf7b878f270093
diff --git a/tests/hwtest.py b/tests/hwtest.py
index c53b65a01..52d714ae6 100755
--- a/tests/hwtest.py
+++ b/tests/hwtest.py
@@ -12,7 +12,7 @@ import time
 import serial_test
 
 sys.path.append("../tools/mpremote")
-from mpremote.transport_serial import SerialTransport
+from mpremote.transport_serial import SerialTransport, TransportError
 import mpremote.commands
 import mpremote.mip
 
@@ -91,6 +91,7 @@ def map_device_name(d):
 
 
 def detect_octoprobe(device):
+    print(f"detect_octoprobe({device})")
     t = SerialTransport(device)
     t.enter_raw_repl()
     t.exec("import os")
@@ -106,9 +107,13 @@ def list_targets(args, exclude_devices):
         for device in args:
             for p in valid_serial_ports:
                 if device == p.device:
-                    if detect_octoprobe(device):
-                        print(f"{device} Octoprobe infrastructure device")
-                    targets.append(Target(device, p.serial_number))
+                    try:
+                        if detect_octoprobe(device):
+                            print(f"{device} Octoprobe infrastructure device")
+                        targets.append(Target(device, p.serial_number))
+                    except TransportError:
+                        print(f"{device} could not connect!")
+                        continue
                     break
             else:
                 print(f"{device} not found")
@@ -120,9 +125,13 @@ def list_targets(args, exclude_devices):
                 continue
             if p.device in exclude_devices:
                 continue
-            if detect_octoprobe(p.device):
-                # If no devices specified, skip Octoprobe infrastructure.
-                print(f"{p.device} Octoprobe infrastructure device, skipping")
+            try:
+                if detect_octoprobe(p.device):
+                    # If no devices specified, skip Octoprobe infrastructure.
+                    print(f"{p.device} Octoprobe infrastructure device, skipping")
+                    continue
+            except TransportError:
+                print(f"{p.device} could not connect!")
                 continue
             targets.append(Target(p.device, p.serial_number))
     return targets

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests Relates to tests/ directory in source

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants