zephyr: Add CI to run tests against zephyr with qemu.#15678
Merged
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #15678 +/- ##
=======================================
Coverage 98.43% 98.43%
=======================================
Files 163 163
Lines 21295 21295
=======================================
Hits 20961 20961
Misses 334 334 ☔ View full report in Codecov by Sentry. |
|
Code size report: |
dpgeorge
force-pushed
the
zephyr-add-ci-testing
branch
5 times, most recently
from
4504e6d to
22bd182
Compare
projectgus
approved these changes
Aug 27, 2024
Contributor
projectgus
left a comment
projectgus
left a comment
There was a problem hiding this comment.
Less code, more test coverage! 🚀
dpgeorge
force-pushed
the
zephyr-add-ci-testing
branch
from
22bd182 to
36a5d67
Compare
The pts line printed by qemu-system-arm goes to stdout, not stderr. Redirect stderr to stdout in case other tools do print to stderr. Signed-off-by: Damien George <[email protected]>
Both of these tests require slice to be enabled. Signed-off-by: Damien George <[email protected]>
Commit f573e73 rewored the zephyr port to build MicroPython as a cmake target, and since that commit the `make-bin-testsuite` helper script no longer works (it requires a Makefile) and hence the tinytest test framework can no longer be run. Instead of fixing this, remove the option to use tinytest. Boards running zephyr can use the standard `tests/run-tests.py` script to run tests in the usual way. Signed-off-by: Damien George <[email protected]>
It needs to be at least this big for `tools/pyboard.py` to work, which is used (among other things) by `tests/run-tests.py`. Signed-off-by: Damien George <[email protected]>
With this change, the zephyr port is tested against the standard test suite via the following scheme: - the zephyr port is built with the `qemu_cortex_m3` board and the `prj_minimal.conf` configuration - `qemu-system-arm` runs `zephyr.elf` - the zephyr console is redirected to a pts/pty - `tests/run-tests.py` is run in bare-metal mode against the pts/pty device This allows testing the zephyr port as though it were a physical board attached over a serial port. Signed-off-by: Damien George <[email protected]>
dpgeorge
force-pushed
the
zephyr-add-ci-testing
branch
from
36a5d67 to
c8838b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prior to this PR, the zephyr port was only compiled under CI, but no tests were run. There was some provision to run tests on zephyr using
make-bin-testsuite, but this hasn't been working since the zephyr port moved fully to cmake (and it was never run under CI, only manually).With this change, the zephyr port is tested against the standard test suite via the following scheme:
qemu_cortex_m3board and theprj_minimal.confconfigurationqemu-system-armrunszephyr.elftests/run-tests.pyis run in bare-metal mode against the pts/pty deviceThis allows testing the zephyr port as though it were a physical board attached over a serial port.
Testing
CI has been added and should run and pass.
Trade-offs and Alternatives
The existing tinytest framework for the zephyr port has been removed in favour of this new approach of running
qemu-system-armand redirecting console to a pts/pty device, then runningrun-tests.pyagainst that serial device. This makes running the tests just like any other bare-metal or unix port.I don't think it's worth continuing to maintain the tinytest framwork on the zephyr port because it's currently broken and needs effort to fix, it's different to everything else, puts an extra burden on the maintainers, and one day won't fit the test suite because it will overflow flash.