Best Practices for Test Suites

When writing a test suite, there are a few things you should keep in mind:

• The test suite will be run together with many other test suites. It should, therefore, finish quickly (say, in under two minutes), and not use too much memory (so that it can run on a ``normal'' workstation).
• The test suite will be run automatically, often in situations where no one checks the screen output. All important output should be via grid variables that are written to files. Alternatively, if the test suite tests some low-level infrastructure, it may just abort the simulation if it fails; that will also be detected.
• Downloading many files is slow on many systems. A test suite should normally not have more than, say, hundred output files, and normally the output files should be small, so that there are not more than a few Megabytes of output files per test suite.
• The test suite output files should always be the same. That means that they should not contain time stamps, etc. It is, therefore, best to use the option IO::out_fileinfo="none".
• Norms are unfortunately quite insensitive to changes to a few grid points only, even if the changes are significant. It is necessary to output grid point values directly, not only norms.
• Try to use as few thorns as possible in a test case. For example, do not active 3D output thorns (unless you use it). The fewer thorns you use, the easier it is to run the test suite.
• It is not necessary that a test suite result is ``physically correct'', or that it uses parameters that ensure a stable time evolution. A test suite will usually take only a few time steps, so that a grid size of, e.g. 40#40 grid points without dissipation can be sufficient. Test suites cannot test whether the result of a simulation is physically feasible; they only test whether anything changed at all. Ensuring that the physics is still correct has to be handled by different mechanisms.