SimFactory Tutorial

This tutorial shows you how to run the WaveToy Cactus example using SimFactory. A familiarity with Cactus is assumed. For further information on SimFactory, see the SimFactory Documentation. The first part shows how to set up the Cactus directory structure and perform a basic configuration of SimFactory suitable for running the wave equation locally or remotely. If you already have a Cactus tree that you want to use with SimFactory, you can just omit the steps relating to downloading Cactus and the thorns.

1. Basic usage

Here we will create a Cactus tree and configure SimFactory to run a simple wave equation using the PUGH driver. The example is designed to run with only a C compiler (gcc), so it should be possible for anyone to run this.

Create the Cactus tree

First we will create the Cactus directory and download SimFactory into it from the SVN repository:

Now download the Cactus Utilities repository to obtain the useful GetCactus script:

Run the GetCactus script with one of the SimFactory standard thornlists. The thornlist contains information about which thorns to use in the configuration and where to download them from.

(Note: some users have had problems with the checkout hanging. If that happens, modify the GetCactus script to remove the -z9 from the CVS command.)

It is convenient to be able to run simfactory/sim using the much shorter "sim". A convenient method is to set up a shell alias:

in your .bashrc file, or

in your .cshrc file. Check that when you run

from your Cactus directory you get the SimFactory help output. This step is optional, but the remainder of the tutorial assumes you have done it. If not, just replace simfactory/sim with sim in the instructions that follow.

Configure SimFactory

SimFactory needs to be configured for each user and machine. This configuration is stored in the user database file, an example of which can be found as simfactory/udb.example.pm. You should copy this file as simfactory/udb.pm and customize it.

Open udb.pm in an editor and find the section entitled Customize your local machine here. This section creates a new machine profile based on a generic template. You should customize each entry in this section appropriate to your user and machine.

Once the udb.pm file has been customized, you can check that there are no syntax errors by running

You should see the SimFactory help message.

Build a configuration

You are now ready to compile a configuration. We will create a configuration called wavetoy and use the wavetoyc.th thornlist which comes with SimFactory, as it is very simple and has no dependencies on external software such as HDF5, MPI, a Fortran compiler etc.

This should take a few minutes.

Create and run a simulation

You can now create a simulation using this configuration and a parameter file. This example will use one of the WaveToy parameter files, wavetoyc_rad, which evolves a wave equation with simple radiative outer boundary conditions. We name the simulation after the parameter file; wavetoyc_rad.

Once the simulation has been created (you should see a directory wavetoyc_rad in your simulations directory), you can submit it. Submitting a simulation on a supercomputer will add it to the queue. If you are running on a laptop or workstation, the simulation will run immediately.

The standard output of the simulation will be in wavetoyc_rad/output-0000/wavetoyc_rad.out under the basedir specified in udb.pm. The output data will be in wavetoyc_rad/output-0000/wavetoyc_rad.

Running on a remote machine

Building a machine profile and optionlist to include all of the libraries commonly used by HPC codes is quite a lot of work (especially if it is to work remotely), so for many clusters, SimFactory contains machine profiles pre-configured. You can see a list of these in simfactory/mdb.pm. These machine profiles also come with optionlists in simfactory/optionlists and queue submission scripts in simfactory/scriptfiles.

The procedure for running on a remote machine supported by SimFactory is almost the same as for running locally. Certain details relating to the machine will need to be customized, for example, the username used to log in to the machine. See the section Essential customizations of other machines you will use in udb.pm and adapt it to your needs.

Once the machine is customized in udb.pm, the next step is to copy the Cactus tree to the remote machine:

All SimFactory commands can now be used as before, but with "remote machinename" as the first two arguments:

2. Using your own code with SimFactory

If you already have a thornlist, you can copy it into simfactory/thornlists and SimFactory will find it when you use the --thornlist option. However, unless your code is very simple, the "generic" machine profile used in the above example will not be capable of running your code locally. For example, this machine profile does not include HDF5, GSL, MPI, Fortran, LAPACK or BLAS. The lack of MPI means that Carpet, for example, will not run. This is due to the fact that there are no standards for the locations of these libraries on different machines, so it would be impossible to provide a profile which worked everywhere. However, the supercomputer profiles are much more complete, so it is possible to run on these without going through the stage of compiling the running on your local machine. If you do want to run on your local machine, and there is no suitable optionlist, you will need to adapt one of the existing ones to your setup.

3. Workstation profiles

[Needs work] The workstations at AEI and CCT have SimFactory profiles written for them. You can use the machines "duo", "quad" and "numrel".

4. Tips

• Once a simulation has been created, its parameter file cannot be modified. If you edit the parameter file, you must create a new simulation.
• If you are running remotely, remember to do a "sim sync" if you change any files locally, for example parameter files.
• You do not need to use the "remote" command. You can use "sim login machinename" and use the "sim" commands via ssh. However, if you modify files on the remote machine, the next sync will overwrite them.

[To do]

• Why does the -z9 CVS option not work all the time?
• Can we remove the "erik" REPOSITORY_USER from the thornlists?
• We should have a private follow-on document with details specific to the code (CCATIE, Whisky, McLachlan, Llama etc).
• We should try to provide installation instructions for the various packages for several common operating systems; e.g. Ubuntu, Mac OS X, as well as optionlists for these.
• Gives examples of simulations with multiple restarts, including presubmission.

Page last modified on 15-Oct-2009.