PETSc: Building Your Own

Description

We recommend that whenever possible, you use the PETSc release provided by the vendor (Cray). Please see the PETSc page for more information on using the Cray PETSc build that is optimized for Cray systems.

If you need to build the latest release or a specific version then please follow the instructions below.

 

How to build PETSc

Downloading PETSc from the PETSc webpage,

Download a specific version of PETSc tarball, and extract the sources as follows: 
% wget http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.7.4.tar.gz
% tar –zxvf  petsc-3.7.4.tar.gz
To move to the PETSc directory use:
% cd petsc-3.7.4

Downloading PETSc using Git (an alternative way)

To download the release use:  
% git clone -b maint https://bitbucket.org/petsc/petsc petsc
To move to the PETSc directory use: 
% cd petsc
To obtain new patches that have been added since your "git clone" or last "git pull" use:
% git pull

Installing PETSc with PrgEnv-gnu

To start an interactive mode use:
% qsub -I -l nodes=1:ppn=32:xe -l walltime=2:00:00
To update the programming environment on Blue Waters use:
% module unload PrgEnv-cray
% module load PrgEnv-gnu
% module load cmake
% export CRAYPE_LINK_TYPE=dynamic
% export CRAY_ADD_RPATH=yes
To set up PETSC_DIR and PETSC_ARCH use:
% export PETSC_DIR=[the PETSc directory]
% export PETSC_ARCH=BW-gnu
You can specify PETSC_ARCH with any character string. For each build, you have to specify different PETSC_ARCH. See an example for another build with PrgEnv-intel at the bottom.  
 
To configure PETSc with some external packages (e.g., superlu_dist, mumps, hypre, spooles, ml, metis, and parmetis libraries) use:
% cd ${PETSC_DIR}
% ./configure \
  --with-cc=cc --with-cxx=CC --with-fc=ftn \
  --download-fblaslapack=1 --download-blacs=1 --download-metis=1 --download-parmetis=1 \
  --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --download-hypre=1 \
  --download-spooles=1 --download-ml=1 --with-pic=1 --with-debugging=0 \
  --with-shared-libraries=1 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 \
  --with-mpiexec=aprun
This step may take around 40 to 60 minutes on Blue Waters due to downloading and compiling external libraries. 
 
To build and test PETSc use:
% make all test

Installing another PETSc with PrgEnv-intel (optional)

To update the programming environment on Blue Waters use:
% module unload PrgEnv-gnu
% module load PrgEnv-intel
To update PETSC_ARCH use:
% export PETSC_ARCH=BW-intel
To configure, build and test PETSc, repeat the corresponding steps presented for PreEnv-gnu at the above. 

 

How to use PETSc

To set up the PETSc environment use:

% module unload PrgEnv-cray
% module load PrgEnv-gnu            (or, % module load PrgEnv-intel)
% export CRAYPE_LINK_TYPE=dynamic
% export CRAY_ADD_RPATH=yes
% export PETSC_DIR=[the PETSc directory]
% export PETSC_ARCH=BW-gnu          (or, % export PETSC_ARCH=BW-intel)

 

Examples

Linear Poisson equation on a 2D grid

To compile and build the example use: 
% cd ${PETSC_DIR}/src/ksp/ksp/examples/tutorials
% make ex50
To run the example use:
% qsub –I –l nodes=2:ppn=32 –l walltime=00:30:00
% cd $PBS_O_WORKDIR
% aprun -n 32 -N 16 -S 4 -j 1 ./ex50  -da_grid_x 100 -da_grid_y 100 \
  -pc_type lu -pc_factor_mat_solver_package superlu_dist -ksp_monitor -ksp_view

 

Additional Information / References