2.2 Running Tests in Batch Mode

ERT supports automated invocations from the command line or from scripts or makefiles. There are two functions for this purpose, ert-run-tests-batch and ert-run-tests-batch-and-exit. They can be used like this:

emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit

This command will start up Emacs in batch mode, load ERT, load my-tests.el, and run all tests defined in it. It will exit with a zero exit status if all tests passed, or nonzero if any tests failed or if anything else went wrong. It will also print progress messages and error diagnostics to standard output.

You can also redirect the above output to a log file, say output.log, and use the ert-summarize-tests-batch-and-exit function to produce a neat summary as shown below:

emacs -batch -l ert -f ert-summarize-tests-batch-and-exit output.log

ERT attempts to limit the output size for failed tests by choosing conservative values for print-level and print-length when printing Lisp values. This can in some cases make it difficult to see which portions of those values are incorrect. Use ert-batch-print-level and ert-batch-print-length to customize that:

emacs -batch -l ert -l my-tests.el \
      --eval "(let ((ert-batch-print-level 10) \
                    (ert-batch-print-length 120)) \
                (ert-run-tests-batch-and-exit))"

Even modest settings for print-level and print-length can produce extremely long lines in backtraces, however, with attendant pauses in execution progress. Set ert-batch-backtrace-line-length to t to use the value of backtrace-line-length, nil to stop any limitations on backtrace line lengths (that is, to get full backtraces), or a positive integer to limit backtrace line length to that number.

By default, ERT in batch mode is quite verbose, printing a line with result after each test. This gives you progress information: how many tests have been executed and how many there are. However, in some cases this much output may be undesirable. In this case, set ert-quiet variable to a non-nil value:

emacs -batch -l ert -l my-tests.el \
      --eval "(let ((ert-quiet t)) (ert-run-tests-batch-and-exit))"

In quiet mode ERT prints only unexpected results and summary.

You can specify selectors to only run a subset of your tests (see Test Selectors). For example, the following would run all tests where the name of the test matches the regular expression “to-match”.

emacs -batch -l ert -l my-tests.el \
      -eval '(ert-run-tests-batch-and-exit "to-match")'

By default, ERT test failure summaries are quite brief in batch mode—only the names of the failed tests are listed. If the EMACS_TEST_VERBOSE environment variable is set and is non-empty, the failure summaries will also include the data from the failing test.

ERT can produce JUnit test reports in batch mode. If the environment variable EMACS_TEST_JUNIT_REPORT is set, ERT will produce for every test package my-tests.el a corresponding JUnit test report my-tests.xml. The function ert-summarize-tests-batch-and-exit collects all these package test reports into a new JUnit test report, with the respective name of that environment variable.