Testing and Debugging

This document is the reference manual for the LLVM testinginfrastructure. It documents the structure of the LLVM testinginfrastructure, the tools needed to use it, and how to add and runtests.

In order to use the LLVM testing infrastructure, you will need all of thesoftware required to build LLVM, as well as Python 2.7 orlater.

The LLVM testing infrastructure contains three major categories of tests:unit tests, regression tests and whole programs. The unit tests and regressiontests are contained inside the LLVM repository itself under llvm/unittestsand llvm/test respectively and are expected to always pass – they should berun before every commit.

The whole programs tests are referred to as the “LLVM test suite” (or“test-suite”) and are in the test-suite module in subversion. Forhistorical reasons, these tests are also referred to as the “nightlytests” in places, which is less ambiguous than “test-suite” and remainsin use although we run them much more often than nightly.

Unit tests are written using Google Testand Google Mockand are located in the llvm/unittests directory.

The regression tests are small pieces of code that test a specificfeature of LLVM or trigger a specific bug in LLVM. The language they arewritten in depends on the part of LLVM being tested. These tests are driven bythe Lit testing tool (which is part of LLVM), andare located in the llvm/test directory.

Typically when a bug is found in LLVM, a regression test containing justenough code to reproduce the problem should be written and placedsomewhere underneath this directory. For example, it can be a smallpiece of LLVM IR distilled from an actual application or benchmark.

The test suite contains whole programs, which are pieces of code whichcan be compiled and linked into a stand-alone program that can beexecuted. These programs are generally written in high level languagessuch as C or C++.

These programs are compiled using a user specified compiler and set offlags, and then executed to capture the program output and timinginformation. The output of these programs is compared to a referenceoutput to ensure that the program is being compiled correctly.

In addition to compiling and executing programs, whole program testsserve as a way of benchmarking LLVM performance, both in terms of theefficiency of the programs generated as well as the speed with whichLLVM compiles, optimizes, and generates code.

The test-suite is located in the test-suite Subversion module.

See the test-suite Guide for details.

The test suite contains tests to check quality of debugging information.The test are written in C based languages or in LLVM assembly language.

These tests are compiled and run under a debugger. The debugger outputis checked to validate of debugging information. See README.txt in thetest suite for more information. This test suite is located in thedebuginfo-tests Subversion module.

The tests are located in two separate Subversion modules. The unit andregression tests are in the main “llvm” module under the directoriesllvm/unittests and llvm/test (so you get these tests for free with themain LLVM tree). Use make check-all to run the unit and regression testsafter building LLVM.

The test-suite module contains more comprehensive tests including whole Cand C++ programs. See the test-suite Guide for details.

To run all of the LLVM unit tests use the check-llvm-unit target:

To run all of the LLVM regression tests use the check-llvm target:

In order to get reasonable testing performance, build LLVM and subprojectsin release mode, i.e.

If you have Clang checked out and built, youcan run the LLVM and Clang tests simultaneously using:

To run the tests with Valgrind (Memcheck by default), use the LIT_ARGS makevariable to pass the required options to lit. For example, you can use:

to enable testing with valgrind and with leak checking enabled.

To run individual tests or subsets of tests, you can use the llvm-litscript which is built as part of LLVM. For example, to run theInteger/BitPacked.ll test by itself you can run:

or to run all of the ARM CodeGen tests:

For more information on using the lit tool, see llvm-lit —helpor the lit man page.

To run debugging information tests simply add the debuginfo-testsproject to your LLVM_ENABLE_PROJECTS define on the cmakecommand-line.

The LLVM regression tests are driven by lit and are located in thellvm/test directory.

This directory contains a large array of small tests that exercisevarious features of LLVM and to ensure that regressions do not occur.The directory is broken into several sub-directories, each focused on aparticular area of LLVM.

The regression test structure is very simple, but does require someinformation to be set. This information is gathered via configureand is written to a file, test/lit.site.cfg in the build directory.The llvm/test Makefile does this work for you.

In order for the regression tests to work, each directory of tests musthave a lit.local.cfg file. lit looks for this file to determinehow to run the tests. This file is just Python code and thus is veryflexible, but we’ve standardized it for the LLVM regression tests. Ifyou’re adding a directory of tests, just copy lit.local.cfg fromanother directory to get running. The standard lit.local.cfg simplyspecifies which files to look in for tests. Any directory that containsonly directories does not need the lit.local.cfg file. Read the Litdocumentation for more information.

Each test file must contain lines starting with “RUN:” that tell lithow to run it. If there are no RUN lines, lit will issue an errorwhile running a test.

RUN lines are specified in the comments of the test program using thekeyword RUN followed by a colon, and lastly the command (pipeline)to execute. Together, these lines form the “script” that litexecutes to run the test case. The syntax of the RUN lines is similar to ashell’s syntax for pipelines including I/O redirection and variablesubstitution. However, even though these lines may look like a shellscript, they are not. RUN lines are interpreted by lit.Consequently, the syntax differs from shell in a few ways. You can specifyas many RUN lines as needed.

lit performs substitution on each RUN line to replace LLVM tool nameswith the full paths to the executable built for each tool (in$(LLVM_OBJ_ROOT)/$(BuildMode)/bin). This ensures that lit doesnot invoke any stray LLVM tools in the user’s path during testing.

Each RUN line is executed on its own, distinct from other lines unlessits last character is \. This continuation character causes the RUNline to be concatenated with the next one. In this way you can build uplong pipelines of commands without making huge line lengths. The linesending in \ are concatenated until a RUN line that doesn’t end in\ is found. This concatenated set of RUN lines then constitutes oneexecution. lit will substitute variables and arrange for the pipelineto be executed. If any process in the pipeline fails, the entire line (andtest case) fails too.

Below is an example of legal RUN lines in a .ll file:

As with a Unix shell, the RUN lines permit pipelines and I/Oredirection to be used.

There are some quoting rules that you must pay attention to when writingyour RUN lines. In general nothing needs to be quoted. lit won’tstrip off any quote characters so they will get passed to the invoked program.To avoid this use curly braces to tell lit that it should treateverything enclosed as one value.

In general, you should strive to keep your RUN lines as simple as possible,using them only to run tools that generate textual output you can then examine.The recommended way to examine output to figure out if the test passes is usingthe FileCheck tool. [The usage of grep in RUNlines is deprecated - please do not send or commit patches that use it.]

Put related tests into a single file rather than having a separate file pertest. Check if there are files already covering your feature and consideradding your code there instead of creating a new file.

If your test requires extra files besides the file containing the RUN:lines, the idiomatic place to put them is in a subdirectory Inputs.You can then refer to the extra files as %S/Inputs/foo.bar.

For example, consider test/Linker/ident.ll. The directory structure isas follows:

For convenience, these are the contents:

For symmetry reasons, ident.ll is just a dummy file that doesn’tactually participate in the test besides holding the RUN: lines.

Note

Some existing tests use RUN: true in extra files instead of justputting the extra files in an Inputs/ directory. This pattern isdeprecated.

It is easy to write a fragile test that would fail spuriously if the tool beingtested outputs a full path to the input file. For example, opt bydefault outputs a ModuleID:

ModuleID can unexpectedly match against CHECK lines. For example:

This test will fail if placed into a download directory.

To make your tests robust, always use opt … < %s in the RUN line.opt does not output a ModuleID when input comes from stdin.

Whenever adding tests that require the knowledge of a specific platform,either related to code generated, specific output or back-end features,you must make sure to isolate the features, so that buildbots thatrun on different architectures (and don’t even compile all back-ends),don’t fail.

The first problem is to check for target-specific output, for example sizesof structures, paths and architecture names, for example:

Also, if the test rely on any behaviour that is coded in any back-end, it mustgo in its own directory. So, for instance, code generator tests for ARM gointo test/CodeGen/ARM and so on. Those directories contain a speciallit configuration file that ensure all tests in that directory willonly run if a specific back-end is compiled and available.

For instance, on test/CodeGen/ARM, the lit.local.cfg is:

Other platform-specific tests are those that depend on a specific featureof a specific sub-architecture, for example only to Intel chips that support AVX2.

For instance, test/CodeGen/X86/psubus.ll tests three sub-architecturevariants: