Next: testsuite Invocation, Previous: Using an Autotest Test Suite, Up: Using Autotest
The testsuite.at is a Bourne shell script making use of special
Autotest M4 macros. It often contains a call to AT_INIT near
its beginning followed by one call to m4_include per source file
for tests. Each such included file, or the remainder of
testsuite.at if include files are not used, contain a sequence of
test groups. Each test group begins with a call to AT_SETUP,
then an arbitrary number of shell commands or calls to AT_CHECK,
and then completes with a call to AT_CLEANUP. Multiple test
groups can be categorized by a call to AT_BANNER.
Initialize Autotest. Giving a name to the test suite is encouraged if your package includes several test suites. In any case, the test suite always displays the package name and version. It also inherits the package bug report address.
State that, in addition to the Free Software Foundation's copyright on the Autotest macros, parts of your test suite are covered by copyright-notice.
The copyright-notice shows up in both the head of testsuite and in `testsuite --version'.
Log the file name and answer to --version of each program in space-separated list executables. Several invocations register new executables, in other words, don't fear registering one program several times.
Autotest test suites rely on PATH to find the tested program. This avoids the need to generate absolute names of the various tools, and makes it possible to test installed programs. Therefore, knowing which programs are being exercised is crucial to understanding problems in the test suite itself, or its occasional misuses. It is a good idea to also subscribe foreign programs you depend upon, to avoid incompatible diagnostics.
This macro identifies the start of a category of related test groups. When the resulting testsuite is invoked with more than one test group to run, its output will include a banner containing test-category-name prior to any tests run from that category. The banner should be no more than about 40 or 50 characters. A blank banner will not print, effectively ending a category and letting subsequent test groups behave as though they are uncategorized when run in isolation.
This macro starts a group of related tests, all to be executed in the same subshell. It accepts a single argument, which holds a few words (no more than about 30 or 40 characters) quickly describing the purpose of the test group being started. test-group-name must not expand to unbalanced quotes, although quadrigraphs can be used.
Associate the space-separated list of keywords to the enclosing test group. This makes it possible to run “slices” of the test suite. For instance, if some of your test groups exercise some `foo' feature, then using `AT_KEYWORDS(foo)' lets you run `./testsuite -k foo' to run exclusively these test groups. The title of the test group is automatically recorded to
AT_KEYWORDS.Several invocations within a test group accumulate new keywords. In other words, don't fear registering the same keyword several times in a test group.
If the current test group fails, log the contents of file. Several identical calls within one test group have no additional effect.
Determine whether the test is expected to fail because it is a known bug (for unsupported features, you should skip the test). shell-condition is a shell expression such as a
testcommand; you can instantiate this macro many times from within the same test group, and one of the conditions is enough to turn the test into an expected failure.
Initialize an input data file with given contents. Of course, the contents have to be properly quoted between square brackets to protect against included commas or spurious M4 expansion. The contents must end with an end of line. file must be a single shell word that expands into a single file name.
Execute a test by performing given shell commands. These commands should normally exit with status, while producing expected stdout and stderr contents. If commands exit with status 77, then the whole test group is skipped. Otherwise, if this test fails, run shell commands run-if-fail or, if this test passes, run shell commands run-if-pass.
This macro must be invoked in between
AT_SETUPandAT_CLEANUP.If status, or stdout, or stderr is `ignore', then the corresponding value is not checked.
The special value `expout' for stdout means the expected output of the commands is the content of the file expout. If stdout is `stdout', then the standard output of the commands is available for further tests in the file stdout. Similarly for stderr with `experr' and `stderr'.