VASP

A. Description

VASP is a package for performing ab initio quantum mechanical molecular dynamics. It offers a wide range of Density Functional Theory (DFT) methods along with a number of electron correlation methods, and specializes in material simulation. VASP code is developed and comercially distributed by VASP Group at the University of Vienna. See VASP home page http://www.vasp.at for additional information.

B. How to obtain VASP

Due to commercial nature of VASP package, Blue Waters does not provide precompiled binaries. However, help on compiling VASP package on Blue Waters will be provided to those users who already have the VASP license. The current VASP licensee may obtain VASP 5.4.1 that includes GPU support.

C. How to build VASP

On Blue Waters Cray XE6 platform, VASP can be built under Cray or Intel programming environments. The Cray compiler currently provides the best performing code on Blue Waters. 

C.1 Building under Cray environment

Upon opening a new terminal on Blue Waters login node, Cray programming environment is loaded by default. At the time of writing this document, these are PrgEnv-cray/4.2.34, cce/8.3.3, cray-libsci/13.0.1, and cray-mpich/7.0.3. Additional module to be loaded is fftw. Tests showed, that Cray programming environment is fully compatible with VASP 5.x, and that compatibility will presumably remain. At the present moment, using the latest (default) version of modules should be fine, and one can simply resort to using the default modules without worrying about their particular version number. Just for the record, loading default version of module fftw gives fftw/3.3.4.0 at this particular moment.
 
Following are step-by-step instructions for the freshly opened terminal on Blue Waters login node:
 
module add fftw
 
tar zxvf vasp.5.lib.tar.gz
tar zxvf vasp.5.3.3.tar.gz
cd vasp.5.lib
 
cp /u/staff/anisimov/vasp/builds/Makefile.cray.vasp5.3.3.lib Makefile
make
 
cd ../vasp.5.3
cp /u/staff/anisimov/vasp/builds/Makefile.cray.vasp5.3.3.src Makefile
make
 
Successful compilation will create a vasp binary file in the source code directory.

C.2 Building under Intel environment

In a freshly opened terminal, we will swap the default Cray programming environment to Intel. Since we intend to use Intel MKL, it is necessary to unload cray-libsci module.
 
module swap PrgEnv-cray PrgEnv-intel
module unload cray-libsci
 
At the time of writing this document, the programming environment get loaded with PrgEnv-intel/4.2.34, intel/15.0.090, and cray-mpich/7.0.3. The MKL library is the one that comes with this particular version of Intel compiler. We will need to compile FFTW interface comping with MKL library.
 
cp -r $MKLROOT/interfaces/fftw3xf/ .
cd fftw3xf
make libintel64
 
The compilation will create libfftw3xf_intel.a in directory fftw3x/
 
cd ..
tar zxvf vasp.5.lib.tar.gz
tar zxvf vasp.5.3.3.tar.gz
cd vasp.5.lib
 
cp /u/staff/anisimov/vasp/builds/Makefile.intel.vasp5.3.3.lib Makefile
make
 
cd ../vasp.5.3
cp /u/staff/anisimov/vasp/builds/Makefile.intel.vasp5.3.3.src Makefile
make
 
Successful compilation will create a vasp binary file in the source code directory.

D. Sample test

VASP tests can be downloaded from the public repository.

git clone https://github.com/egplar/vasptest

The tests come without the POTCAR library. The README.md file inside vasptest/ directory explains how to get that library.
 
Following is sample script to run vasp jobs on Blue Waters. Create run.pbs file having the following content:
#!/bin/bash
#PBS -l nodes=4:ppn=32:xe
#PBS -l walltime=00:30:00
#PBS -q debug
cd $PBS_O_WORKDIR
aprun -n 64 -N 16 -d 2 ../vasp.5.3/vasp > job.out
 
Submit the job

qsub run.pbs

Note the use of 16 cores per node, as instructed by aprun commands -N16 -d2. Using 32 cores per node leads to suboptimal performance.