Thorn writers

To port an existing thorn using the method of lines, or to write a new thorn using it, should hopefully be relatively simple. As an example, within the MoL arrangement is WaveMoL which duplicates the WaveToy thorn given by CactusWave in a form suitable for use by MoL. In this section, ``the thorn'' will mean the user thorn doing the physics.

We start with some terminology. Grid functions are split into four cateogories.

1. The first are those that are evolved using a MoL form. That is, a right hand side is calculated and the variable updated using it. These we call evolved variables.
2. The second category are those variables that are set by a thorn at every intermediate step of the evolution, usually to respect the constraints. Examples of these include the primitive variables in a hydrodynamics code. Another example would be the gauge variables if these were set by constraints at every intermediate step (which is slightly artificial; the usual example would be the use of maximal slicing, which is only applied once every $N$ complete steps). These are known as constrained variables.
3. The third category are those variables that a thorn depends on but does not set or evolve. An example would include the metric terms considered from a thorn evolving matter. Due to the way that MoL deals with these, they are known as Save and Restore variables.
4. The final category are those variables that do not interact with MoL. These would include temporary variables for analysis or setting up the initial data. These can safely be ignored.

As a generic rule of thumb, variables for which you have a time evolution equation are evolved (obviously), variables which your thorn sets but does not evolve are constrained, and any other variables which your thorn reads during evolution is a Save and Restore variable.

MoL needs to know every GF that falls in one of the first three groups. If a GF is evolved by one thorn but is a constrained variable in another (for example, the metric in full GR Hydro) then each thorn should register the function as they see it. For example, the hydro thorn will register the metric as a Save and Restore variable and the spacetime thorn will register the metric as an evolved variable. The different variable categories are given the priority evolved, constrained, Save and Restore. So if a variable is registered as belonging in two different categories, it is always considered by MoL to belong to the category with the highest priority.

MoL needs to know the total number of GFs in each category at parameter time. To do this, your thorn needs to use some accumulator parameters from MoL. As an example, here are the paramaters from WaveMoL:

shares: MethodOfLines

USES CCTK_INT MoL_Num_Evolved_Vars
USES CCTK_INT MoL_Num_Constrained_Vars
USES CCTK_INT MoL_Num_SaveAndRestore_Vars

restricted:

CCTK_INT WaveMoL_MaxNumEvolvedVars \
    "The maximum number of evolved variables used by WaveMoL" \
    ACCUMULATOR-BASE=MethodofLines::MoL_Num_Evolved_Vars
{
  5:5           :: "Just 5: phi and the four derivatives"
} 5

CCTK_INT WaveMoL_MaxNumConstrainedVars \
    "The maximum number of constrained variables used by WaveMoL" \
    ACCUMULATOR-BASE=MethodofLines::MoL_Num_Constrained_Vars
{
  0:1           :: "A small range, depending on testing or not"
} 1

CCTK_INT WaveMoL_MaxNumSandRVars \
    "The maximum number of save and restore variables used by WaveMoL" \
    ACCUMULATOR-BASE=MethodofLines::MoL_Num_SaveAndRestore_Vars
{
  0:1           :: "A small range, depending on testing or not"
} 1

This should give the maximum number of variables that your thorn will register.

Every thorn should register every grid function that it uses even if you expect it to be registered again by a different thorn. For example, a hydro thorn would register the metric variables as Save and Restore, whilst the spacetime evolution thorn would register them as evolved (in ADM) or constrained (in ADM_BSSN), both of which have precedence. To register your GFs with MoL schedule a routine in the bin MoL_Register which just contains the relevant function calls. For an evolved variable the GF corresponding to the update term ( ${\bf L}({\bf q})$ in equation (A10.2)) should be registered at the same time. The appropriate functions are given in section A10.6.

These functions are provided using function aliasing. For details on using function aliasing, see sections B10.5 and F2.2.3 of the UsersGuide. For the case of real GFs, you simply add the following lines to your interface.ccl:

##########################################
### PROTOTYPES - DELETE AS APPLICABLE! ###
##########################################

CCTK_INT FUNCTION MoLRegisterEvolved(CCTK_INT EvolvedIndex, CCTK_INT RHSIndex)
CCTK_INT FUNCTION MoLRegisterConstrained(CCTK_INT ConstrainedIndex)
CCTK_INT FUNCTION MoLRegisterSaveAndRestore(CCTK_INT SandRIndex)
CCTK_INT FUNCTION MoLRegisterEvolvedGroup(CCTK_INT EvolvedIndex, \
                                          CCTK_INT RHSIndex)
CCTK_INT FUNCTION MoLRegisterConstrainedGroup(CCTK_INT ConstrainedIndex)
CCTK_INT FUNCTION MoLRegisterSaveAndRestoreGroup(CCTK_INT SandRIndex)
CCTK_INT FUNCTION MoLChangeToEvolved(CCTK_INT EvolvedIndex, CCTK_INT RHSIndex)
CCTK_INT FUNCTION MoLChangeToConstrained(CCTK_INT ConstrainedIndex)
CCTK_INT FUNCTION MoLChangeToSaveAndRestore(CCTK_INT SandRIndex)
CCTK_INT FUNCTION MoLChangeToNone(CCTK_INT RemoveIndex)

#############################################
### USE STATEMENT - DELETE AS APPLICABLE! ###
#############################################

USES FUNCTION MoLRegisterEvolved
USES FUNCTION MoLRegisterConstrained
USES FUNCTION MoLRegisterSaveAndRestore
USES FUNCTION MoLRegisterEvolvedGroup
USES FUNCTION MoLRegisterConstrainedGroup
USES FUNCTION MoLRegisterSaveAndRestoreGroup
USES FUNCTION MoLChangeToEvolved
USES FUNCTION MoLChangeToConstrained
USES FUNCTION MoLChangeToSaveAndRestore
USES FUNCTION MoLChangeToNone

Note that the list of paramters not complete; see the section on parameters for the use of arrays or complex variables. However, the list of functions is, and is expanded on in section A10.6. MoL will check whether a group or variable is a GF or an array and whether it is real or complex. Note that currently complex variable support is disabled.

Having done that, one routine (or group of routines) which we'll here call Thorn_CalcRHS must be defined. This does all the finite differencing that you'd usually do, applied to ${\bf q}$, and finds the right hand sides which are stored in ${\bf L}$. This routine should be scheduled in MoL_CalcRHS. The precise order that these are scheduled should not matter, because no updating of any of the user thorns ${\bf q}$ will be done until after all the RHSs are calculated. Important note: all the finite differencing must be applied to the most recent time level ${\bf q}$ and not to the previous time level ${\bf q}_p$ as you would normally do. Don't worry about setting up the data before the calculation, as MoL will do that automatically.

Finally, if you have some things that have to be done after each update to an intermediate level, these should be scheduled in MoL_PostStep. Examples of things that need to go here include the recalculation of primitive variables for hydro codes, the application of boundary conditions^part101, the solution of elliptic equations (although this would be a very expensive place to solve them, some sets of equations might require the updating of some variables by constraints in this fashion). When applying boundary conditions the cleanest thing to do is to write a routine applying the symmetries to the appropriate GFs and, when calling it from the scheduler, adding the SYNC statement to the appropriate groups. An example is given by the routine WaveToyMoL_Boundaries in thorn WaveMoL.

Points to note. The thorn routine Thorn_CalcRHS does not need to know and in fact should definitely not know where precisely in the MoL step it is. It just needs to know that it is receiving some intermediate data stored in the GFs ${\bf q}$ and that it should return the RHS ${\bf L}({\bf q})$. All the book-keeping to ensure that it is passed the correct intermediate state at that the GFs contain the correct data at the end of the MoL step will be dealt with by the MoL thorn and the flesh.