test: Support tests which can only be run manually

At present we normally write tests either in Python or in C. But most
Python tests end up doing a lot of checks which would be better done in C.
Checks done in C are orders of magnitude faster and it is possible to get
full access to U-Boot's internal workings, rather than just relying on
the command line.

The model is to have a Python test set up some things and then use C code
(in a unit test) to check that they were done correctly. But we don't want
those checks to happen as part of normal test running, since each C unit
tests is dependent on the associate Python tests, so cannot run without
it.

To acheive this, add a new UT_TESTF_MANUAL flag to use with the C 'check'
tests, so that they can be skipped by default when the 'ut' command is
used. Require that tests have a name ending with '_norun', so that pytest
knows to skip them.

Signed-off-by: Simon Glass <sjg@chromium.org>

2 files changed, 11 insertions, 1 deletions

diff --git a/include/test/test.h b/include/test/test.h
index c1853ce471b..4ad74614afc 100644
--- a/include/test/test.h
+++ b/include/test/test.h
@@ -28,6 +28,7 @@
  * @other_fdt_size: Size of the other FDT (UT_TESTF_OTHER_FDT)
  * @of_other: Live tree for the other FDT
  * @runs_per_test: Number of times to run each test (typically 1)
+ * @force_run: true to run tests marked with the UT_TESTF_MANUAL flag
  * @expect_str: Temporary string used to hold expected string value
  * @actual_str: Temporary string used to hold actual string value
  */
@@ -48,6 +49,7 @@ struct unit_test_state {
 	int other_fdt_size;
 	struct device_node *of_other;
 	int runs_per_test;
+	bool force_run;
 	char expect_str[512];
 	char actual_str[512];
 };
@@ -63,6 +65,12 @@ enum {
 	/* do extra driver model init and uninit */
 	UT_TESTF_DM		= BIT(6),
 	UT_TESTF_OTHER_FDT	= BIT(7),	/* read in other device tree */
+	/*
+	 * Only run if explicitly requested with 'ut -f <suite> <test>'. The
+	 * test name must end in "_norun" so that pytest detects this also,
+	 * since it cannot access the flags.
+	 */
+	UT_TESTF_MANUAL		= BIT(8),
 };
 
 /**
diff --git a/include/test/ut.h b/include/test/ut.h
index f7217aa8ac5..e0e618b58c2 100644
--- a/include/test/ut.h
+++ b/include/test/ut.h
@@ -409,9 +409,11 @@ void test_set_state(struct unit_test_state *uts);
  * @select_name: Name of a single test to run (from the list provided). If NULL
  *	then all tests are run
  * @runs_per_test: Number of times to run each test (typically 1)
+ * @force_run: Run tests that are marked as manual-only (UT_TESTF_MANUAL)
  * Return: 0 if all tests passed, -1 if any failed
  */
 int ut_run_list(const char *name, const char *prefix, struct unit_test *tests,
-		int count, const char *select_name, int runs_per_test);
+		int count, const char *select_name, int runs_per_test,
+		bool force_run);
 
 #endif