Next: Completing a Thorn Up: Cactus Application Interfaces Previous: Interpolation Operators Contents
Reduction Operators
A reduction operation can be defined as an operation on variables distributed across multiple processor resulting in a single number. Typical reduction operations are: sum, minimum/maximum value, and boolean operations. A typical application is, for example, finding the maximum reduction from processor local error estimates, therefore, making the previous processor local error known to all processors.
The exchange of information across processors needs the functionality of a communication layer, e.g. CactusPUGH/PUGH. For this reason, the reduction operation itself is not part of the flesh, instead, Cactus (again) provides a registration mechanism for thorns to register routines they provide as reduction operators. The different operators are identified by their name and/or a unique number, called a handle.
The registration mechanism gives the advantage of a common interface while hiding the individual communication calls in the layer.
In Cactus, reduction operators can be applied to grid functions, arrays and scalars, as well as to local arrays. Note that different implementations of reduction operators may be limited in the objects they can be applied to. There is a fundamental difference between the reduction operation on grid functions and quantities as arrays.
Currently the flesh supports the new and old reduction specification. The old APIs will be deprecated in the next beta cycle in favour of the new specification.
New Reduction Specification API documentation In the new reduction specification, there are two different flesh APIs for reduction, depending on whether the data arrays are Cactus grid arrays or processor-local, programming language built-in arrays, and on what assumptions are made about the topology and spacing of the grid (these descriptions are for 3D, but the generalisations to other numbers of dimensions should be obvious):
- CCTK_ReduceGridArrays()
- Reduces Cactus grid arrays, with the topology of the
grid implicitly specified by a Cactus coordinate system.
This API doesn't provide a reduction functionality itself, it only takes care of the interprocessor communication necessary when reducing distributed grid arrays, and invokes the CCTK_ReduceLocalArrays() API on the each processor's local patch of the data.
- CCTK_ReduceLocalArrays()
- Reduces processor-local arrays with various options including offsets, strides and masks.
The flesh provides an API to register local reduction operators:
- CCTK_RegisterLocalArrayReductionOperator()
- Register a
CCTK_ReduceLocalArrays()interpolation operator
Each local reduction operator is registered under a character string name;
at registration, the name is mapped to a unique integer handle, which
may be used to refer to the operator. CCTK_LocalArrayReductionHandle()
is used to get the handle corresponding to a given character string
name.
Old Reduction Specification API Documentation
Obtaining the reduction handle
Before calling the routine which performs the reduction operation, the handle, which identifies the operation, must be derived from its registered name.
int CCTK_ReductionHandle(const char *reduction_name); integer reduction_handle character*(*) reduction_name call CCTK_ReductionHandle(reduction_handle, reduction_name) int CCTK_ReductionArrayHandle(const char *reduction_name); integer reduction_handle character*(*) reduction_name call CCTK_ReductionArrayHandle(reduction_handle, reduction_name)
- reduction_handle
- in Fortran, the name of the variable will contain the handle value after the call. In C, this value is the function value.
- reduction_name
- is the name under which the operator has been registered by the providing thorn. The only thorn in the standard Computational Toolkit release, which provides reduction operators, is CactusPUGH/PUGHReduce.
- error checking
- negative handle value indicates failure to identify the correct operator.
Get a integer handle corresponding to a given reduction operator. The operator is identified by the name it was registered with. (Note that although it would appear to be far more convenient to pass the name of the reduction operator directly to the following function call to CCTK_Reduce this causes problems with the translation of strings from Fortran to C with variable argument lists).
The general reduction interface. The main interfaces for reduction operations are quite powerful (and hence rather complicated). To ease the use of these main interfaces, wrappers designed for specific and more restricted use are described below. If uncertain, you should use these.
int CCTK_Reduce( const cGH *GH,
int proc,
int operation_handle,
int num_out_vals,
int type_out_vals,
void *out_vals,
int num_in_fields,
...);
call CCTK_Reduce( int returnvalue,
cctkGH,
int processor,
int operation_handle,
int num_out_vals,
int type_out_vals,
out_vals,
int num_in_fields,
... )
int CCTK_ReduceArray( const cGH *GH,
int proc,
int operation_handle,
int num_out_vals,
int type_out_vals,
void *out_vals,
int num_dims,
int num_in_arrays,
int type_in_arrays,
... )
call CCTK_ReduceArray(int returnvalue,
cctkGH,
int processor,
int operation_handle,
int num_out_vals,
int type_out_arrays,
void out_vals,
int num_dims,
int num_in_arrays,
int type_in_arrays,
... )
- int returnvalue
- the return value of the operation. Negative value indicates failure to perform reduction. Zero indicates a successful operation.
- cctkGH
- in Fortran, the pointer to the grid hierarchy structure. Can not be used within Fortran, but can be used from within C. Since this name is fixed, write it out as shown.
- cGH *GH
- in C, it is the pointer to the grid hierarchy.
- int processor
- the processor which collects the information, a negative value (35#35) will distribute the data to all processors.
- int operation_handle
- the number of the reduction operation handle, needs to be found by calling CCTK_ReductionHandle or CCTK_ReductionArrayHandle.
- int num_out_vals
- integer defining the number of output values.
- int type_out_arrays, type_in_arrays
- specifies the type of the gridfunction you are communicating. Use the values as specified in Section B10.8. Note: Do not mix data types, e.g. in Fortran, do not declare a variable as integer and then specify the type CCTK_VARIABLE_INT in the reduction command. These types need not be the same on some architectures and will conflict.
- out_vals
- an array that will contain the output values.
- int num_in_fields
- specifies the number of input fields.
- ...
- indicates a variable argument list: specify the arrays which will be reduced, the number of specified arrays must be the same as the value of the num_in_fields variable.
- error checking
- a return value, other than zero, indicates failure to perform the operation.
Special reduction interfaces. The routines are designed for the purpose of reducing scalars, arrays and grid functions. They hide many of the options of the generic interface described above.
Reduction of local scalars across multiple processors. The result of the reduction operation will be on the specified processor or on all processors.
int CCTK_ReduceLocScalar (const cGH *GH,
int processor,
int operation_handle,
void *in_scalar,
void *out_scalar,
int data_type)
call CCTK_ReduceLocScalar(int returnvalue,
cctkGH,
int processor,
int operation_handle,
in_scalar,
out_scalar,
int data_type)
- in_scalar
- the processor local variable with local value to be reduced
- out_scalar
- the reduction result: a processor local variable with the global value (same on all processors), if processor has been set to 35#35. Otherwise, processor will hold the reduction result.
- data_type
- specifies the type of the gridfunction you are communicating. Use the values as specified in Section B10.8.
Reduction of local 1d arrays to a local arrays. This reduction is carried out element by element. The arrays need to have the same size on all processors.
int CCTK_ReduceLocArrayToArray1D( const cGH *GH,
int processor,
int operation_handle,
void *in_array1d,
void *out_array1d,
int xsize,
int data_type)
call CCTK_ReduceLocArrayToArray1D(int returnvalue
cctkGH,
int processor,
int operation_handle,
in_array1d,
out_array1d,
int xsize,
int data_type)
- in_array1d
- one dimensional local arrays to be reduced across a processors, element by element.
- out_array1d
- array holding the reduction result. out_array1d[1] = Reduction(in_array[1]).
- xsize
- the size of the one dimensional array.
Reduction of local 2d arrays to a local 2d array. This reduction is carried out element by element. The arrays need to have the same size on all processors.
int CCTK_ReduceLocArrayToArray2D( const cGH *GH,
int processor,
int opertaion_handle,
in_array_2d,
out_array2d,
int xsize,
int ysize,
int data_type)
call CCTK_ReduceLocArrayToArray2D( int returnvalue
cctkGH,
int processor,
int operation_handle,
in_array2d,
out_array2d,
int xsize,
int ysize,
int data_type)
- in_array1d
- two dimensional local arrays, to be reduced across a processors, element by element.
- out_array1d
- two dimensional array holding the reduction result. out_array2d[i,j]= Reduction(in_array2d[i,j]).
- xsize
- the size of the one dimensional array in x direction.
- ysize
- the size of the one dimensional array in y direction.
Reduction of local 3D arrays to a local 3D array. This reduction is carried out element by element. The arrays need to have the same size on all processors.
int CCTK_ReduceLocArrayToArray3D(const cGH *GH,
int processor,
int opertaion_handle,
in_array_3d,
out_array3d,
int xsize,
int ysize,
int zsize,
int data_type)
call CCTK_ReduceLocArrayToArray3D(int returnvalue
cctkGH,
int processor,
int operation_handle,
in_array3d,
out_array3d,
int xsize,
int ysize,
int zsize,
int data_type)
- in_array3d
- two dimensional local arrays, to be reduced across a processors, element by element.
- out_array3d
- two dimensional array holding the reduction result. out_array3d[i,j,k]= Reduction(in_array3d[i,j,k]).
- xsize
- the size of the one dimensional array in x direction.
- ysize
- the size of the one dimensional array in y direction.
- ysize
- the size of the one dimensional array in z direction.
Some brief examples:
Reduction of a local scalars: a local error is reduced across all processors with the maximum operation. The variable tmp will hold the maximum of the error and is the same on all processors. This quantity can then be reassigned to normerr.
CCTK_REAL normerr, tmp
integer ierr, reduction_handle
call CCTK_ReductionArrayHandle(reduction_handle,"maximum")
if (reduction_handle.lt.0) then
call CCTK_WARN(CCTK_WARN_ALERT, "Cannot get reduction handle for maximum operation.")
endif
call CCTK_ReduceLocScalar(ierr, cctkGH, -1,
. reduction_handle,
. normerr, tmp, CCTK_VARIABLE_REAL)
if (ierr.ne.0) then
call CCTK_WARN(CCTK_WARN_ALERT, "Reduction of norm failed!");
endif
normerr = tmp
Reduction of a local 2D array: a two dimensional array 36#36 is reduced, reduction results (array of same size: bla_tmp) are seen on all processors (35#35 entry as the third argument); also demonstrates some simple error checking with the CCTKi_EXPECTOK macro.
CCTK_REAL bla(2,3),bla_tmp(2,3);
integer ierr, sum_handle
call CCTK_ReductionArrayHandle(sum_handle,"sum")
bla = 1.0d0
write (*,*) "BLA ",bla
call CCTK_ReduceLocArrayToArray2D(ierr, cctkGH, -1, sum_handle,
. bla, bla_tmp, 2, 3, CCTK_VARIABLE_REAL)
call CCTKi_EXPECTOK(ierr, 0, 1, "2D Reduction failed")
bla = bla_tmp
write (*,*) "BLA ",bla
Note that the memory for the returned values must be allocated before the reduction call is made.
Next: Completing a Thorn Up: Cactus Application Interfaces Previous: Interpolation Operators Contents