If the variable TESTS
is defined, its value is taken to be a
list of programs or scripts to run in order to do the testing.
Programs needing data files should look for them in srcdir
(which is both an environment variable and a make variable) so they
work when building in a separate directory (see Build Directories in The Autoconf Manual), and in
particular for the distcheck
rule (see Checking the Distribution).
For each of the TESTS
, the result of execution is printed along
with the test name, where PASS
denotes a successful test,
FAIL
denotes a failed test, XFAIL
an expected failure,
XPASS
an unexpected pass for a test that is supposed to fail,
and SKIP
denotes a skipped test.
The number of failures will be printed at the end of the run. If a given test program exits with a status of 77, then its result is ignored in the final count. This feature allows non-portable tests to be ignored in environments where they don’t make sense.
If the Automake option color-tests
is used (see Changing Automake’s Behavior)
and standard output is connected to a capable terminal, then the test
results and the summary are colored appropriately. The user can disable
colored output by setting the make
variable
‘AM_COLOR_TESTS=no’, or force colored output even without a connecting
terminal with ‘AM_COLOR_TESTS=always’.
The variable TESTS_ENVIRONMENT
can be used to set environment
variables for the test run; the environment variable srcdir
is
set in the rule. If all your test programs are scripts, you can also
set TESTS_ENVIRONMENT
to an invocation of the shell (e.g.
‘$(SHELL) -x’ can be useful for debugging the tests), or any other
interpreter. For instance the following setup is used by the Automake
package to run four tests in Perl.
TESTS_ENVIRONMENT = $(PERL) -Mstrict -I $(top_srcdir)/lib -w TESTS = Condition.pl DisjConditions.pl Version.pl Wrap.pl
You may define the variable XFAIL_TESTS
to a list of tests
(usually a subset of TESTS
) that are expected to fail. This will
reverse the result of those tests.
Automake ensures that each file listed in TESTS
is built before
any tests are run; you can list both source and derived programs (or
scripts) in TESTS
; the generated rule will look both in
srcdir
and .. For instance, you might want to run a C
program as a test. To do this you would list its name in TESTS
and also in check_PROGRAMS
, and then specify it as you would
any other program.
Programs listed in check_PROGRAMS
(and check_LIBRARIES
,
check_LTLIBRARIES
...) are only built during make check
,
not during make all
. You should list there any program needed
by your tests that does not need to be built by make all
. Note
that check_PROGRAMS
are not automatically added to
TESTS
because check_PROGRAMS
usually lists programs used
by the tests, not the tests themselves. Of course you can set
TESTS = $(check_PROGRAMS)
if all your programs are test cases.