PETSc in src code

Use PETSc as normal, Use the PUGH communicator if a routine needs a communictor. On first pass, you need to make a call to PETScSetCommWorld() and PetscInitialize() to set the PETSc communicator and initialize PETSc.

This could be a seperate routine scheduled early in schedule.ccl at BASEGRID eg. PetscInitialize() requires the commandline parameters as input. It allows you to pass through the flags, etc. (I have not ried this feature.) Initialize the PETSc communicator with the Cactus communicator. You end up having code like this:

  
  /* The pugh Extension handle */	
  pGH *pughGH;
  
  /* Get the link to pugh Extension */
  pughGH = (pGH*)GH->extensions[CCTK_GHExtensionHandle("PUGH")];

  if (first_trip==0) 
  {
    int argc;
    char **argv;

    /* Get the commandline arguments */
    argc = CCTK_CommandLine(&argv);

    /* Set the PETSc communicator to set of 
       PUGH and initialize PETSc */
    ierr = PetscSetCommWorld(pughGH->PUGH_COMM_WORLD); CHKERRA(ierr);
    PetscInitialize(&argc,&argv,NULL,NULL);
    
    CCTK_INFO("PETSc initialized");
  }