FLASH
A. Description
The FLASH code is a publicly available high performance application code which has evolved into a modular, extensible software system from a collection of unconnected legacy codes. FLASH consists of inter-operable modules that can be combined to generate different applications. The FLASH architecture allows arbitrarily many alternative implementations of its components to co-exist and interchange with each other. A simple and elegant mechanism exists for customization of code functionality without the need to modify the core implementation of the source. A built-in unit test framework combined with regression tests that run nightly on multiple platforms verify the code.
B. How to download FLASH
FLASH can be downloaded from the University of Chicago's FLASH center's homepage. Documentation is available there as well.
C. How to build FLASH
Blue Waters does not provide precompiled FLASH binaries. The users are expected to obtain their own copy of the code and install it for the personal use.
Compilation starts from the default Cray environment. Following steps lead to the compilation of FLASH on the Cray platform. Since there is no site configured for Blue Waters in the FLASH code, the site configuration for NERSC's Edison machine, which is a similar Cray system to Blue Waters is used instead. If you would prefer to use the GNU compiler instead of the Intel compiler used by Edison's Makefile, please use this Makefile.h .
module swap PrgEnv-cray PrgEnv-intel module load cray-hdf5-parallel module load cray-parallel-netcdf # obtain FLASH4.5.tar.gz from http://www.flash.uchicago.edu/site/flashcode/ wget http://flash.uchicago.edu/site/flashcode/user_support/FLASH4.5-a.diff tar -xzf FLASH4.5.tar.gz cd FLASH4.5/ patch -p0 <../FLASH4.5-a.diff ./setup Sedov -auto -site=edison.nersc.gov cd object make -j6
Last successful compilation was performed under the following environment:
1) modules/3.2.10.4 19) intel/16.0.3.210 2) eswrap/1.3.3-1.020200.1280.0 20) craype-interlagos 3) cray-mpich/7.5.0 21) craype-network-gemini 4) torque/6.0.4 22) craype/2.5.8 5) moab/9.1.2-sles11 23) cray-libsci/16.11.1 6) java/jdk1.8.0_51 24) udreg/2.3.2-1.0502.10518.2.17.gem 7) globus/6.0 25) ugni/6.0-1.0502.10863.8.28.gem 8) gsissh/7.5p1b 26) pmi/5.0.10-1.0000.11050.179.3.gem 9) xalt/0.7.6.local 27) dmapp/7.0.1-1.0502.11080.8.74.gem 10) scripts 28) gni-headers/4.0-1.0502.10859.7.8.gem 11) OpenSSL/1.0.2m 29) xpmem/0.1-2.0502.64982.5.3.gem 12) cURL/7.59.0 30) dvs/2.5_0.9.0-1.0502.2188.1.113.gem 13) git/2.17.0 31) alps/5.2.4-2.0502.9774.31.12.gem 14) wget/1.19.4 32) rca/1.0.0-2.0502.60530.1.63.gem 15) user-paths 33) atp/2.0.4 16) gnuplot/5.0.5 34) PrgEnv-intel/5.2.82 17) darshan/3.1.3 35) cray-parallel-netcdf/1.7.0 18) gcc/4.9.3 36) cray-hdf5-parallel/1.10.0
D. Tests
Sample job script:
#!/bin/bash #PBS -l nodes=1:xe:ppn=32 cd $PBS_O_WORKDIR OMP_NUM_THREADS=8 aprun -n 4 -d $OMP_NUM_THREADS ./flash4
Sample output (sedov.log):
FLASH log file: 01-24-2019 16:18:45.132 Run number: 1
==============================================================================
Number of MPI tasks: 4
MPI version: 3
MPI subversion: 1
Dimensionality: 2
Max Number of Blocks/Proc: 1000
Number x zones: 8
Number y zones: 8
Number z zones: 1
[...]
guardcell Barrier 0.000 0.000 0.000 147
guardcell internal 1.525 1.503 1.514 147
amr_guardcell 1.325 1.312 1.316 147
eos gc 0.108 0.104 0.105 147
amr_refine_derefine 6.527 6.506 6.516 147
updateData 0.005 0.005 0.005 147
updateParticleRefinement 0.000 0.000 0.000 147
Driver_computeDt 0.084 0.065 0.070 295
==============================================================================
[ 01-24-2019 16:24:10.103 ] LOGFILE_END: FLASH run complete.
