tests: add full, automated, top-level test runner for hardware by dpgeorge · Pull Request #16112 · micropython/micropython

dpgeorge · 2024-10-30T03:22:26Z

Summary

With the aim to speed up testing for a release, I put together a script to automatically run tests on devices running MicroPython. We have a lot of different tests these days, with different ways to run them. The idea with this top-level test runner is:

detect all serial devices connected
detect what platform is running on each device and what its capabilities are (eg WLAN, BLE, ability to import .mpy files)
run as many tests as possible given the connected devices (eg if there are two WLAN boards connected, then run the network multitests against them, among other things)

Basically all you need to do is connect boards to your PC via USB and run:

$ cd tests
$ ./hwtest.py

Then it does the rest. It takes between 10 and 15 minutes per board, so if you have many connected it can take a few hours. You can also restrict to given devices by specifying them on the command line, eg:

$ ./hwtest.py a0 a1 u0

(Probably the name of the script hwtest.py can change to something better.)

Testing

This was used to test 16 distinct boards for the recent v1.24.0 release.

Trade-offs and Alternatives

This is a bit of a hacky script at the moment but can be improved over time.

Currently there are no boards that pass all tests (!!) and so over time we need to fix things and improve the tests so more of them can pass.

dpgeorge · 2024-10-30T03:24:24Z

PR #16111 will help simplify this test runner a bit.

PR #15909 is included in this PR.

codecov · 2024-10-30T03:24:50Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.46%. Comparing base (2dc2e30) to head (63531a4).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #16112   +/-   ##
=======================================
  Coverage   98.46%   98.46%           
=======================================
  Files         176      176           
  Lines       22784    22784           
=======================================
  Hits        22435    22435           
  Misses        349      349

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

hmaerki · 2024-10-30T10:46:16Z

Octoprobe is ready so far to run these tests fully automated, including build of all firmware variants and to be triggered from github workflows.

We have to agree on triggers (github action on PR, github manual action). I will start the discussion when I am back from vacation.

github-actions · 2025-04-15T04:09:09Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Signed-off-by: Damien George <[email protected]>

TODO: make it a CLI option Signed-off-by: Damien George <[email protected]>

Signed-off-by: Damien George <[email protected]>

To reduce output clutter. Signed-off-by: Damien George <[email protected]>

Signed-off-by: Damien George <[email protected]>

If it's too big PYBLITEV10 will run out of memory when running `extmod/deflate_decompress.py`. Actually, 32-7 is the default... so this ioctl could be removed. Really need to benchmark it to see if it makes any difference having it large. Signed-off-by: Damien George <[email protected]>

Signed-off-by: Damien George <[email protected]>

To decrease the variation in time taken to send. Eg ARDUINO_NANO_33_BLE_SENSE fails without this change (but only when `tests/extmod/machine_uart_irq_txidle.py` is run before it...). Signed-off-by: Damien George <[email protected]>

On bare-metal lwIP targets, there's only enough buffer space for 4 groups. Signed-off-by: Damien George <[email protected]>

Signed-off-by: Damien George <[email protected]>

Fixed on tests-improvements-for-octoprobe. Signed-off-by: Damien George <[email protected]>

Signed-off-by: Damien George <[email protected]>

dpgeorge added the tests Relates to tests/ directory in source label Oct 30, 2024

dpgeorge marked this pull request as draft October 30, 2024 03:22

dpgeorge changed the title ~~tests: add full, automated test running for hardware~~ tests: add full, automated, top-level test runner for hardware Oct 30, 2024

dpgeorge mentioned this pull request Oct 30, 2024

tests/run-tests.py: Change --target/--device options to --test-instance. #16111

Merged

dpgeorge force-pushed the tests-full-test-runner branch 2 times, most recently from b8ce03c to 2047b05 Compare November 4, 2024 05:56

dpgeorge force-pushed the tests-full-test-runner branch from 2047b05 to 39adb22 Compare November 24, 2024 02:26

dpgeorge force-pushed the tests-full-test-runner branch from 39adb22 to 31a289e Compare January 15, 2025 00:58

dpgeorge force-pushed the tests-full-test-runner branch from 31a289e to 6ef6bd8 Compare April 15, 2025 03:56

This was referenced Aug 12, 2025

mimxrt: Add Quadrature Encoder and Pulse Counter classes. #12347

Merged

tests: Require SSL certificate file to be available for test to run. #17897

Merged

dpgeorge force-pushed the tests-full-test-runner branch 2 times, most recently from 7875959 to d668cbe Compare August 15, 2025 02:42

dpgeorge force-pushed the tests-full-test-runner branch from d668cbe to 943efd1 Compare September 8, 2025 04:24

dpgeorge added 12 commits March 26, 2026 00:07

tests/run-tests.py: Don't print info for tests that pass.

bbf8e15

Signed-off-by: Damien George <[email protected]>

tests/run-multitests.py: Suppress diff.

5f5b00e

TODO: make it a CLI option Signed-off-by: Damien George <[email protected]>

tests/run-tests.py: Capture KeyboardInterrupt for nicer exit.

b0bb19a

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Add a full, automatic test runner.

5716a66

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Make sure unittest is installed.

4d56d22

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Add command to build all natmods.

7932eff

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Improve CLI options.

06f1170

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Just print returncode on subprocess fail.

056695d

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Change args to selection then devices.

1747440

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Run hardware tests in native mode if selected.

0e69998

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Support targets without a filesystem.

c2faadb

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Run hardware tests after native tests.

da88829

Signed-off-by: Damien George <[email protected]>

dpgeorge added 29 commits March 26, 2026 00:07

tests/hwtest.py: Remove armv6 and armv7em.

f678771

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Fix target check.

f8860e2

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Skip Octoprobe infrastructure.

7880bd2

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Allow selecting natmods to build.

c5714bf

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Add argparse and support a reference board.

b7c0cd7

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Support any number of return values from target_info.

2fa522e

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Update to use --test-instance.

c210496

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Use new serial_test.py.

9e6f1a9

Signed-off-by: Damien George <[email protected]>

tests/hwtest.py: Update for new serial_test.py.

67b7861

Signed-off-by: Damien George <[email protected]>

tests/run-tests.py: Add option to not print list of skipped tests.

9d1a1de

To reduce output clutter. Signed-off-by: Damien George <[email protected]>

tests: Increase buffer to 249 bytes.

236a9c5

Signed-off-by: Damien George <[email protected]>

tests/run-tests.py: Exit with code 2 on error.

cd5fea2

Signed-off-by: Damien George <[email protected]>

tests/micropython/extreme_exc.py: Skip on ESP32 with PSRAM.

4784861

Signed-off-by: Damien George <[email protected]>

tests/multi_extmod: Update pins for I2CTarget test using octoprobe.

0c7e2a0

Signed-off-by: Damien George <[email protected]>

tests: Improve test support for renesas-ra boards.

3b6e59e

Signed-off-by: Damien George <[email protected]>

tests/extmod_hardware/machine_i2c_target.py: Improve pyboard support.

d21a123

Signed-off-by: Damien George <[email protected]>

tests/extmod/machine_uart_tx.py: Add tunings for zephyr boards.

84ad6d1

Signed-off-by: Damien George <[email protected]>

tests/target_wiring/UM_FEATHERS2.py: Add target_wiring.

b0500f1

Signed-off-by: Damien George <[email protected]>

tests/target_wiring/ZEPHYR_FRDM_K64F.py: Add target_wiring.

5e11a92

Signed-off-by: Damien George <[email protected]>

tests/run-multitests.py: Ignore DHCPS messages.

bb9b660

Signed-off-by: Damien George <[email protected]>

tests/multi_net/udp_data_multi.py: Reduce number of UDP groups.

9be364a

On bare-metal lwIP targets, there's only enough buffer space for 4 groups. Signed-off-by: Damien George <[email protected]>

tests/ports/stm32_hardware/dma_alignment.py: Convert to target_wiring.

39f7b68

Signed-off-by: Damien George <[email protected]>

tests/extmod_hardware: Add Pin test.

d77264b

Signed-off-by: Damien George <[email protected]>

tests//micropython/schedule_sleep.py: Fail on mimxrt, nrf, samd.

99067df

Fixed on tests-improvements-for-octoprobe. Signed-off-by: Damien George <[email protected]>

wip

5085ea0

wip

9059b7f

Signed-off-by: Damien George <[email protected]>

tests/target_wiring: Add more boards.

63531a4

Signed-off-by: Damien George <[email protected]>

dpgeorge force-pushed the tests-full-test-runner branch from 943efd1 to 63531a4 Compare April 2, 2026 05:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

tests: add full, automated, top-level test runner for hardware#16112

tests: add full, automated, top-level test runner for hardware#16112
dpgeorge wants to merge 44 commits into
micropython:masterfrom
dpgeorge:tests-full-test-runner

dpgeorge commented Oct 30, 2024

Uh oh!

dpgeorge commented Oct 30, 2024

Uh oh!

codecov Bot commented Oct 30, 2024 •

edited

Loading

Uh oh!

hmaerki commented Oct 30, 2024

Uh oh!

github-actions Bot commented Apr 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

dpgeorge commented Oct 30, 2024

Summary

Testing

Trade-offs and Alternatives

Uh oh!

dpgeorge commented Oct 30, 2024

Uh oh!

codecov Bot commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hmaerki commented Oct 30, 2024

Uh oh!

github-actions Bot commented Apr 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Oct 30, 2024 •

edited

Loading