Improve DX for unit & integration testing of no_std and std targets

[mhatzl]

This issue was already briefly discussed in matrix.
https://matrix.to/#/!BHcierreUuwCMxVqOf:matrix.org/$7wPk8mbCr-l8k3nTCq4f6vCnEgj6LkYVZxkB7fiBxkI?via=matrix.org&via=catircservices.org&via=beeper.com

---

Infineon is trying to improve the tooling to test Rust code on embedded devices (target) and on desktop (host). Running tests on a host platform like Windows or Linux is quite easy with cargo test.
Testing code on embedded devices already becomes a bit harder, but defmt-test at least works well for integration tests. Unit tests are currently very limited, because it is unstable to include non-inline modules inside proc-macros, which kind of forces one to write all tests inside one inline test module.

At Infineon, the goal is to write tests once and run them on targets and on the host, because testing on the host is much faster, but target tests are needed for quality assurance and certifications.
This combination is not as easily done as it could be, because the harness setting in Cargo.toml is target independent.
Therefore, the default harness must be enabled/disabled if one wants to switch between testing on host or target.

I created a small demo project here that showcases the workarounds and pain points we faced.

These limitations seem quite hard to resolve without changes to rustc and Cargo.
Allowing non-inline modules inside proc-macros may help, but this seems like another workaround.
Ideally, defmt-test could be integrated for no_std targets similar to how libtest is for std targets.
Note that this is just a rough idea.

---

Feedback from matrix so far:

• Try embedded-test and probe-rs

  embedded-test has the same proc-macro problems as defmt-test.
  probe-rs is not always possible to use, because debug interface is not available on some chips.

• Custom proc-macro similar to defmt-test that collects tests from all files

  Taken from: https://matrix.to/#/!BHcierreUuwCMxVqOf:matrix.org/$8zL5AgFwcQ9L4D49ie0aEZ5fxXXE8yLmHHF3Sravs0M?via=matrix.org&via=catircservices.org&via=beeper.com

  The (revised) overall idea would be the following:

  • you annotate your mod tests in each file with a custom proc macro,

  • you set harness=false for lib in Cargo.toml

  • when running the unit tests for the host architecture, the proc macro around mod test can create the
    needed main function and use something like libtest_mimic in the background. You'll also have to
    collect the tests across all files, but this is possible.

  • when running the unit tests for the target architecture, the proc macro basically compiles the entire
    (unit test) file twice:

  • once for the target architecture, annotating the #[test] functions with something like defmt_test::test.

  • once for x86, resulting in code that flashes the target and checks the uart output and in turn output test failure/sucess for each test case (via libtest-mimic)

  Open: How to collect tests from all files under src? Resolving modules is hard and probably not doable correctly without building a stripped down Rust parser?

[t-moe]

• Open: How to collect tests from all files under src? Resolving modules is hard and probably not doable correctly without building a stripped down Rust parser?

You can probably use something like https://crates.io/crates/linkme to collect all the tests and then have a central main function which invokes them all or passes them to libtest-mimic.

[mhatzl]

You can probably use something like https://crates.io/crates/linkme to collect all the tests and then have a central main function which invokes them all or passes them to libtest-mimic.

Thanks. linkme looks really promising.

[mhatzl]

Ideally, code coverage should also be more integrated into testing.

At the moment, code coverage using RUSTFLAGS="-Cinstrument-coverage" returns the line coverage for a binary. Because tests in Rust code are currently combined to one binary per integration file, or one binary for all lib tests, it is not possible to get line coverage per test function.
It is also not possible to reverse-engineer this coverage, because there is no way to know which test resulted in which line coverage.

I am working on a tool for requirements traceability,
where I want to combine requirement traces with line coverage to detect what requirement is covered by
which test. But other tools might also benefit from this info e.g. to highlight which test covers more lines, showing the lines to kind of debug the path a test took, ...

A possible approach could be to run each test function as its own binary, but this would probably result in exploding compile times.
To reduce this compile time, it might be possible to compile each test function in isolation and then dynamically link the "application" code.

Not sure if this should be its own issue.

[BartMassey]

@mhatzl Sounds like a great idea. Definitely related to what's here, although it's kind of solving a different problem: probably worth its own linked issue. Seems like there's a more general need for figuring out what new infra in rustc and cargo (if any) would make building these kinds of testing solutions easier.

[mhatzl]

@BartMassey thanks. I will create a new issue and link to this one here.
Is the embedded WG the right group for this? might also be related to Rust tooling in general.