test: Support tests which can only be run manually

At present we normally write tests either in Python or in C. But most
Python tests end up doing a lot of checks which would be better done in C.
Checks done in C are orders of magnitude faster and it is possible to get
full access to U-Boot's internal workings, rather than just relying on
the command line.

The model is to have a Python test set up some things and then use C code
(in a unit test) to check that they were done correctly. But we don't want
those checks to happen as part of normal test running, since each C unit
tests is dependent on the associate Python tests, so cannot run without
it.

To acheive this, add a new UT_TESTF_MANUAL flag to use with the C 'check'
tests, so that they can be skipped by default when the 'ut' command is
used. Require that tests have a name ending with '_norun', so that pytest
knows to skip them.

Signed-off-by: Simon Glass <sjg@chromium.org>

1 files changed, 27 insertions, 0 deletions

diff --git a/doc/develop/tests_writing.rst b/doc/develop/tests_writing.rst
index 1ddf7a353a7..bb1145da268 100644
--- a/doc/develop/tests_writing.rst
+++ b/doc/develop/tests_writing.rst
@@ -74,6 +74,33 @@ NOT rely on running with sandbox, but instead should function correctly on any
 board supported by U-Boot.
 
 
+Mixing Python and C
+-------------------
+
+The best of both worlds is sometimes to have a Python test set things up and
+perform some operations, with a 'checker' C unit test doing the checks
+afterwards. This can be achieved with these steps:
+
+- Add the `UT_TESTF_MANUAL` flag to the checker test so that the `ut` command
+  does not run it by default
+- Add a `_norun` suffix to the name so that pytest knows to skip it too
+
+In your Python test use the `-f` flag to the `ut` command to force the checker
+test to run it, e.g.::
+
+   # Do the Python part
+   host load ...
+   bootm ...
+
+   # Run the checker to make sure that everything worked
+   ut -f bootstd vbe_test_fixup_norun
+
+Note that apart from the `UT_TESTF_MANUAL` flag, the code in a 'manual' C test
+is just like any other C test. It still uses ut_assert...() and other such
+constructs, in this case to check that the expected things happened in the
+Python test.
+
+
 How slow are Python tests?
 --------------------------