Ghost Zones

Cactus is based upon a distributed computing paradigm. That is, the problem domain is split into blocks, each of which is assigned to a processor. For hyperbolic and parabolic problems the blocks only need to communicate at the edges.

To illustrate this, take the example of the wave equation again. Equation B4.2 describes a possible time-evolution scheme. On examination, you can see that to generate the data at the point (14#14, 17#17) we need data from the four points (15#15, 17#17), (16#16, 17#17), (15#15, 18#18), and (15#15, 19#19) only. Ignoring the points at 17#17, which are obviously always available on a given processor, you can see that the algorithm requires a point on either side of the point 17#17, i.e. this algorithm has stencil size 1. Similarly algorithms requiring two points on either side have stencil size 2, etc.

Now, if you evolve the above scheme, it becomes apparent that at each iteration the number of grid points you can evolve decreases by one at each edge (see Figure B4.1).

Figure B4.1: Distributed wave equation with no ghostzones

[angle=0,width=8cm]1dnoghost

At the outer boundary of the physical domain, the data for the boundary point can be generated by the boundary conditions, however, at internal boundaries, the data has to be copied from the adjacent processor. It would be inefficient to copy each point individually, so instead, a number of ghostzones are created at the internal boundaries. A ghostzone consists of a copy of the whole plane (in 3D, line in 2D, point in 1D) of the data from the adjacent processor. That is, the array on each processor is augmented with copies of points from the adjacent processors, thus allowing the algorithm to proceed on the points owned by this processor without having to worry about copying data. Once the data has been evolved one step, the data in the ghostzones can be exchanged (or synchronised) between processors in one fell swoop before the next evolution step. (See Figure B4.2.) Note that you should have at least as many ghostzones as your stencil size requires.

Figure B4.2: Distributed wave equation with ghostzones

[angle=0,width=8cm]withghost