Processor Decomposition

By default PUGH will distribute the computational grid evenly across all processors (as in Figure I1.2a). This may not be efficient if there is a different computational load on different processors, or for example for a simulation distributed across processors with different per-processor performance.

Figure: Partitioning of the computational grid across processors, Figure a) is the default type of partition used by PUGH, Figure b) can be set manually, and Figure c) is not possible with PUGH

[angle=0,width=12cm]/home/cactus_web/CheckOut/CactusStable/arrangements/CactusPUGH/PUGH/doc/Partitioning

The computational grid can be manually partitioned in each direction in a regularly way as in Figure I1.2b.

The computational grid can be manually distributed using PUGH's string parameters partition_[1d_x|2d_x|2d_y|3d_x|3d_y|3d_z]. To manually specify the load distribution, set PUGH::partition = "manual" and then, depending on the grid dimension, set the remaining parameters to distribute the load in each direction. Note that for this you need to know apriori the processor decomposition.

The decomposition is easiest to explain with a simple example: to distribute a 30-cubed grid across 4 processors (decomposed as $2 \times 1 \times 2$, with processors 0 and 2 performing twice as fast as processors 1 and 3) as:

proc 2: $20 \times 30 \times 15$	proc 3: $10 \times 30 \times 15$
proc 0: $20 \times 30 \times 15$	proc 1: $10 \times 30 \times 15$

you would use the following topology and partition parameter settings:

  # the overall grid size
  PUGH::global_nsize = 30

  # processor topology
  PUGH::processor_topology      = manual
  PUGH::processor_topology_3d_x = 2
  PUGH::processor_topology_3d_y = 1
  PUGH::processor_topology_3d_z = 2     # redundant

  # grid partitioning
  PUGH::partition      = "manual"
  PUGH::partition_3d_x = "20 10"

Each partition parameter lists the number of grid points for every processor in that direction, with the numbers delimited by any non-digit characters. Note that an empty string for a direction (which is the default value for the partition parameters) will apply the automatic distribution. That's why it is not necessary to set PUGH::partition_3d_y = "30" or PUGH::partition_3d_z = "15 15" in the parameter file.

Because the previous automatic distribution gave problems in some cases (e.g. very long box in one, but short in other directions), there is now an improved algorithm that tries to do a better job in decomposing the grid evenly to the processors. However, it can fail in certain situations, in which it is gracefully falling back to the previous ("automatic_old") giving a warning. Note that, if one or more of the parameters PUGH::processor_topology_3d_* or PUGH::partition_3d_* are set, this mode automatically falls back to "automatic_old" without warning.