GROMACS

A. Description

GROMACS is a popular classical molecular dynamics code, which is designed for computer simulation of proteins, lipids, and nucleic acids. GROMACS source code is released under the terms of GNU Lesser General Public License. See GROMACS home page http://www.gromacs.org/ for additional information.

B. How to obtain GROMACS

The following instructions assume the use of bash shell. To download a stable version of the code following steps should be performed:

export GROMACS=$HOME/gromacs
export VERSION=gromacs-5.0.4
 
mkdir $GROMACS
cd $GROMACS
wget ftp://ftp.gromacs.org/pub/gromacs/${VERSION}.tar.gz
tar zxvf ${VERSION}.tar.gz
cd $VERSION
 
This will download and unpack GROMACS source code. 

C. How to build GROMACS

On Blue Waters, GROMACS can be built under GNU programming environment. See GROMACS manual for additional insallation instructions.

C.1 Building the CPU-code under GNU environment

- Configure programming environment and set up environment variables
 
module swap PrgEnv-cray PrgEnv-gnu
module add fftw
module add cmake
module add boost
 
export CRAYPE_LINK_TYPE=dynamic
export CRAY_ADD_RPATH=yes
export CXX=CC
export CC=cc
export CMAKE_PREFIX_PATH=$FFTW_DIR/../
export FLAGS="-dynamic -O3 -march=bdver1 -ftree-vectorize -ffast-math -funroll-loops"
 
- Run cmake. Make sure $GROMACS and $INSTALL are defined
 
export INSTALL=$GROMACS/$VERSION/build-cpu
mkdir $INSTALL
cd $INSTALL
 
cmake ../ -DGMX_MPI=ON -DGMX_OPENMP=ON -DGMX_GPU=OFF -DBUILD_SHARED_LIBS=OFF -DGMX_PREFER_STATIC_LIBS=ON -DGMX_X11=OFF -DGMX_DOUBLE=OFF -DCMAKE_SKIP_RPATH=YES -DCMAKE_INSTALL_PREFIX=$INSTALL -DCMAKE_C_FLAGS="$FLAGS" -DCMAKE_CXX_FLAGS="$FLAGS" -DGMX_CPU_ACCELERATION="AVX_128_FMA"
 
- Compile and install
 
make -j4
make install
 
Successful compilation will create bin, include, lib64, and share directories under directory $INSTALL
 
The latest successful compilation was conducted under the following environment:
PrgEnv-gnu/5.2.40
gcc/4.8.2
cray-mpich/7.0.3
fftw/3.3.4.0
cmake/3.1.3
boost/1.53.0

C.2 Building the GPU-code under GNU environment

- Configure programming environment and set up environment variables
 
module swap PrgEnv-cray PrgEnv-gnu
module add fftw
module add cmake
module add boost
module add cudatoolkit
 
export CRAYPE_LINK_TYPE=dynamic
export CRAY_ADD_RPATH=yes
export CXX=CC
export CC=cc
export CMAKE_PREFIX_PATH=$FFTW_DIR/../
export INSTALL=$GROMACS/$VERSION/build-gpu
export FLAGS="-dynamic -O3 -march=bdver1 -ftree-vectorize -ffast-math -funroll-loops -Wl,--rpath,$INSTALL/lib64"
 
- Run cmake. Make sure $GROMACS and $INSTALL are defined as above
mkdir $INSTALL
cd $INSTALL
 

cmake ../ -DGMX_MPI=ON -DGMX_OPENMP=ON -DGMX_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=$CUDATOOLKIT_HOME -DBUILD_SHARED_LIBS=OFF -DGMX_PREFER_STATIC_LIBS=ON -DGMX_X11=OFF -DGMX_DOUBLE=OFF -DCMAKE_SKIP_RPATH=YES -DCMAKE_INSTALL_PREFIX=$INSTALL -DCMAKE_C_FLAGS="$FLAGS" -DCMAKE_CXX_FLAGS="$FLAGS" -DGMX_SIMD="AVX_128_FMA" -DCUDA_HOST_COMPILER=$CRAYPE_DIR/bin/cc

 
- Compile and install
 
make -j4
make install
 
Successful compilation will create bin, include, lib64, and share directories under directory $INSTALL
 
The latest successful compilation was conducted under the following environment:
PrgEnv-gnu/5.2.40
gcc/4.8.2
cray-mpich/7.2.0
fftw/3.3.4.1
cmake/3.1.3
boost/1.53.0
cudatoolkit/6.5.14-1.0502.9613.6.1

D. Tests

Sample run.pbs file for XE-nodes:

#!/bin/bash
#PBS -l nodes=2:ppn=32:xe
#PBS -l walltime=00:05:00
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=1
aprun -n 32 -N 16 -d 2 $HOME/gromacs/gromacs-5.0.4/build-cpu/bin/mdrun_mpi 
 
Sample run.pbs file for XK-nodes:
 
#!/bin/bash
#PBS -l nodes=2:ppn=16:xk
#PBS -l walltime=00:05:00
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=16
aprun -n 2 -N 1 -d $OMP_NUM_THREADS $HOME/gromacs/gromacs-5.0.4/build-gpu/bin/mdrun_mpi 
 
Submit the job:

qsub run.pbs