2019 Blue Waters Symposium: Abstracts and Presentations

 

A   B   C   D   E   F   G   H   I   J   K   L   M   N   O   P   Q   R   S   T   U   V   W   X   Y   Z

 

 

 

Resolving the Structure of Viral Genomes with Atomic Precision

Project PI: Aleksei Aksimentiev, University of Illinois at Urbana-Champaign

Abstract: Atomistic structures of protein capsids have been resolved for many viral species, but structural organization of their genomes still remains largely unclear. One such viral species is HK97 bacteriophage, for which experiments have characterized the packaging mechanism and resolved its protein capsid with atomistic resolution. Here, we report a computational reconstruction of the HK97 genome organization at atomistic resolution, obtained from a series of simulations gradually increasing in resolution. An all-atom explicit solvent molecular dynamics simulation was performed using an implicit representation for the capsid, allowing the internal pressure and ions to reach equilibrium. The simulation was continued with the explicit atomistic protein capsid, revealing the pattern of DNA-protein capsid interactions. Measurements of internal pressure throughout the coarse-grained simulation were consistent with experimental data, and the SAXS profile derived from the resulting structures matched experiment. The complete atomistic structure of a packaged virus particle uncovers the detailed genome-capsid interactions, offering exciting new avenues for the development of antiviral drugs.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

Multiscale Simulations of Electronic and Fluidic Nanoscale Systems

Project PI: Narayana Aluru, University of Illinois at Urbana-Champaign

Presented by: Amir Taqieddin, University of Illinois at Urbana-Champaign

Abstract: Controlling fluid transport at nanoscale is crucial to enhance the performance of several applications such as water treatment and bio-electronic devices. Understanding this transport requires multiscale modeling to allow better integration of the associated multi-physics. In this talk, we focus on two different systems. First, we present a new power-law relation between the ionic conductance and concentration in CNT. This system is modeled using both continuum and molecular dynamics approaches in OpenFOAM and LAMMPS, respectively. The obtained results are validated against experimental measurements. Second, we present ab initio molecular dynamic modeling, using CP2K package, of water transport in graphene nanochannel with a slit gap equivalent to the carbon inter-atom distance (~3.4 Å). We show how the chemistry of graphene surface plays a role in determine the water properties. In each part, we show the scaling of the computational tools and the unique calculations that were performed using Blue Waters.

Field of Science: Material Science, Fluid Systems, Engineering

View the presentation slides (PDF)

 

Looking Out for the Little Guy: A Comprehensive Study of Star Formation in Dwarf Galaxies

2018 Graduate Fellow: Elaad Applebaum, Rutgers University

Abstract: Cosmological hydrodynamic simulations pose unique computational challenges owing to processes spanning many orders of magnitude in both space and time. In addition to solving the equations of gravity and hydrodynamics, these codes use analytic prescriptions to model processes taking place below the resolution of the simulation. Though implementation details differ, in large galaxies the results tend to converge. However, these prescriptions have not been tested in the low-mass limit. In new simulations capable of resolving the faintest known galaxies, we test how different star formation recipes affect the resulting galaxy distributions. Using Blue Waters we run different models, then compare galaxies across simulations. We find that the robustness of the models depends on the galaxy environment, with results being model-dependent in isolated regions but converged in dense environments. These results highlight the need for simulators to use caution when interpreting the results of their simulations in faint galaxies.

Field of Science: Astronomy and Astrophysics

View the presentation slides (PDF)

 

Simulating Two-Fluid MHD Turbulance and Dynamos and a Novel Paradigm for Geodesic Mesh MHD

Project PI: Dinshaw Balsara, University of Notre Dame

Abstract: Our understanding of the star formation process has reached the point where advanced observational capabilities and high-resolution simulations that include the appropriate two-fluid physics of giant molecular clouds (GMCs) are required. We have shown that while magnetic fields grow exponentially in a fully ionized plasma, they grow more slowly in a partially ionized. Another part of our research consists of realizing that most astrophysical systems are spherical and should, therefore, be simulated on meshes that are optimal for such a mesh. We have developed high order MHD for such meshes. vSeveral large-scale simulations have been run or are running on Blue Waters, and key results have been published. The present work is a substantial improvement over previous work in terms of resolution and in the details of input physics and accuracy. Developing geomesh MHD and publishing it was also a prime accomplishment. GPU ready codes have also been developed via an attendance at an NCSA Hackathon.

Field of Science: Astronomy & Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Characteristics of Tropical and Midlatitude Out-of-Cloud Convectively-Induced Turbulence from High Resolution Convective Simulations

2018 Graduate Fellow: Katelyn A. Barber, University of North Dakota

Abstract: Convectively-induced turbulence is an aviation hazard that is a forecasting challenge because operational weather models are too coarse to resolve turbulence. Turbulence indices are commonly used to aid pilots in avoiding turbulence through nowcasting and limited forecasting systems. Over tropical oceans, turbulence prediction systems rely heavily upon nowcasting where satellite observations identify hazardous regions near convection and diagnostics calibrated for midlatitude turbulence. In this study, seven high resolution simulations of tropical oceanic and midlatitude continental convection are performed to characterize the turbulent environment near various convective types. This study addresses the current shortcomings in turbulence diagnostics and determines the limitations of midlatitude continental diagnostics currently used for turbulence prediction over tropical oceanic regions. The environmental characteristics including stability, vertical wind shear, velocity in the vertical and horizontal, and temperature around convection during the developing and mature stages are analyzed to determine if stage specific thunderstorm guidelines are needed.

Field of Science: Atmospheric and Climate Science

View the presentation slides (PDF)

 

The Use of Petaflop Simulations in Optimization of Amber Force Field

Project PI: Jerry Bernholc, North Carolina State University

Presented by: Victor Anisimov, University of Illinois at Urbana-Champaign

Abstract: The Amber empirical force field is the major workhorse of modern biomolecular simulation packages. However, tuning its parameters to the properties of biological materials represents a major methodological and computational challenge, due to the size of the required petascale-level simulations. This work employs a combination of experimental data, quantum chemistry calculations of finite systems, and large-scale RMG DFT-level simulations of a crystal of Guanine to generate an extended set of data for refinement of the empirical force field. Simultaneous charge fitting to the electrostatic potential of multiple clusters incorporates the condensed phase effect into the empirical parameters. The created extensive dataset of dimer, trimer and tetramer clusters includes configurations that probe the curvature of the potential energy profile of the crystal in the vicinity of its thermodynamic minimum that are not directly accessible from experiment. These runs, along with the Lennard-Jones parameter sweep, require petascale compute capability.

Field of Science: Chemistry

Watch the presentation | View the presentation slides (PDF)

 

Direct Numerical Simulation of Turbulence Suppression in Rotating Pipe Flows

Project PI: Christoph Brehm, University of Kentucky

Presented by: Jefferson M. Davis, University of Kentucky

Abstract: In past research studies, rotation of turbulent flows has been shown to provide a turbulence suppression. The axially rotating pipe is an exemplary prototypical model problem that exhibits these complex turbulent flow physics. For this flow, the rotation causes a region of turbulence suppression which is particularly sensitive to the rotation rate and Reynolds number. The physical mechanisms causing turbulence suppression are not well-understood, and a deeper understanding of these mechanisms is of great value for many practical examples involving swirling or rotating flows, e.g. swirl generators, wing-tips, axial compressors, hurricanes, etc. DNS at moderate Reynolds numbers and several rotation rates were conducted. Turbulence statistics are used to quantify the effects of rotation on turbulence. Finally, an overview of the second phase of this research project is provided where the entire reverse transition process is simulated on considering ~30 billion points as part of the recently awarded NSF PRAC 2019 allocation.

Field of Science: Fluid Systems

View the presentation slides (PDF)

 

Efficient Strategy for Tracking Wavepackets in Hypersonic Boundary Layers

Project PI: Oliver M. F. Browne, University of Kentucky

Abstract: A robust and computationally efficient transition prediction methodology for hypersonic boundary layers is being developed, verified and validated at the University of Kentucky. This framework includes the Adaptive Mesh Refinement Wave-Packet Tracking (AMR-WPT) technique which is used for tracking linear and nonlinear instabilities in hypersonic boundary layers and, subsequently, for predicting the transition to turbulence. As opposed to classical fixed mesh approaches, an AMR method is employed for greater computational efficiency. The presentation will show the research that has been conducted using the Blue Waters HPC allocation. It has allowed for the AMR-WPT approach to be validated for tracking strongly nonlinear wavepackets in hypersonic boundary layers. Results from far more computationally costly Direct Numerical Simulations were used for validating the AMR-WPT method. Furthermore, by utilizing the Blue Waters HPC allocation, we have been able to optimize the tracking procedure and the algorithms which govern the AMR-WPT method.

Field of Science: Fluid Systems

 

Three-Dimensional Supernova Explosion Simulations

Project PI: Adam Burrows, Princeton University

Abstract: Using our state-of-the-art code Fornax on Blue Waters we have simulated the collapse and explosion of the cores of many massive-star models in three spatial dimensions. This is the most comprehensive set of realistic 3D core-collapse supernova simulations yet performed and has provided very important insights into the mechanism and character of this 50-year-old astrophysical puzzle. I will present detailed results from this suite of runs and the novel conclusions derived from our new capacity to simulate many 3D, as opposed to 2D and 1D, full physics models every year. This new capability, enabled by this new algorithm and modern HPC assets such as Blue Waters, is destined to transform our understanding of this central phenomenon.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Climate Policy in a Dynamic Stochastic Economy

Project PI: Yongyang Cai, Ohio State University

Abstract: There are significant uncertainties in the climate and economic system. Integrated Assessment Models (IAMs) of climate and economy aim to analyze the impact and efficacies of policy responses to climate change. We develop and solve new computational IAMs with more than ten-dimensional continuous state space, which incorporate spatial temperature system, climate tipping points, economic risks, carbon capture and storage, and regional economic activities. We then analyze the optimal policy under uncertainty and risks and how such a policy impacts the economic activities. We find that tipping points and sea level rise significantly increase social cost of carbon (SCC), but efficient adaptation and carbon capture and storage can significantly decrease SCC, whileignoring spatial heat transfer leads to nonnegligible bias. Moreover, we solve dynamic stochastic cooperative and non-cooperative equilibrium and find that non-cooperation leads to much lower carbon tax and then much higher temperature in future.

Field of Science: Social Sciences

Watch the presentation | View the presentation slides (PDF)

 

Accretion Dynamics of Black Hole Binaries

Project PI: Manuela Campanelli, Rochester Institute of Technology

Presented by: Scott Noble, University of Tulsa

Abstract: Binary systems of supermassive black holes hold the key to resolving open questions regarding galactic evolution and massive black hole growth. Here, we present the first credible calculations of the light emitted by nearby gas as two supermassive black holes approach merger. By so doing, we will give observers strong clues about how to find such systems, potentially identifying examples well before their gravitational wave emission can be detected. The galactic environment of supermassive black hole binaries should, in many cases, supply ample gas to be accreted, liberating large amounts of energy in the process. This should produce photons spanning the spectrum from visible light to hard X-rays; the power output can be great enough that they can be seen from across the Universe. The magnetohydrodynamical equations of gas flows near binary supermassive black holes are extremely complicated. Fortunately, we have developed novel codes in the past few years on Blue Waters enabling us to realistically simulate these systems in 3-D using general relativity for the first time. Our simulations on Blue Waters provide the first astrophysically-accurate predictions of the time- and energy-dependence of the electromagnetic precursor to supermassive black hole binary merger.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Reproducibility, Convergence and Accuracy in Biomolecular Simulation of Nucleic Acids Conformational Ensembles

Project PI: Thomas Cheatham, University of Utah

Abstract: From 2013-2018, molecular dynamics and enhancing sampling simulations of nucleic acids have allowed us to assess, validate, and improve the force fields for nucleic acids. Using various model systems, we have demonstrated through various ensemble simulations and replica exchange molecular dynamics (REMD), including multidimensional replica exchange (with temperature and Hamiltonian exchanges), the ability to reproducibly converge, from different initial conditions, the conformational ensembles of various DNA duplexes and mini-dumbells and of various RNA dinucleotides, tetranucleotides and tetraloops. We have also better refined NMR models to their published experimental data, explored salt dependence of DNA duplexes and their interactions with various ligands, and demonstrated that the use of minimal experimental restraints can improve fidelity with experiment. All of this has been enabled by advancements in the AMBER biomolecular simulation codes including excellent performance on the Blue Waters GPUs.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

Phase Transitions in Dense Hydrogen

Project PI: David Ceperley, University of Illinois at Urbana-Champaign

Abstract: The phase diagram of high pressure hydrogen is of great interest both for fundamental research, such as in astrophysics and for energy applications. The existence and precise location of a phase transition in the liquid phase between a molecular insulating fluid and a monoatomic metallic fluid is relevant for planetary models. Recent experiments reported contrasting results about its location and results based on Density Functional Theory are very scattered. We performed Coupled Electron-Ion Quantum Monte Carlo calculations of this transition, finding results that lay between the two experimental predictions. The transition is signaled by a discontinuity in the specific volume, a sudden dissociation of the molecules, a jump in electrical conductivity and in electron localization. During the past year our prediction of a transition was verified by a new experiment and our recent results of its optical properties reinterpret the other experiments.

Field of Science: Material Science

View the presentation slides (PDF)

 

A Massively Parallel Evolutionary Metropolis-Try Markov Chain Monte Carlo Algorithm for Spatial State Space Traversal

Project PI: Wendy Cho, University of Illinois at Urbana-Champaign

Abstract: We develop an Evolutionary Markov Chain Monte Carlo (EMCMC) algorithm for sampling from large, idiosyncratic, and multi-modal state spaces. Our algorithm combines the advantages of evolutionary algorithms (EAs) as optimization heuristics for state space traversal and the theoretical convergence properties of Markov Chain Monte Carlo algorithms for sampling from unknown distributions. We encompass these two algorithms within the framework of a Metropolis-Try Markov Chain with a generalized Metropolis-Hastingratio. We harness the computational power of massively parallel architecture by integrating a parallel EA framework that guides Markov chains running in parallel. Our algorithm has applications in many different fields of science.

Field of Science: Social Sciences

 

Source Processes of Intermediate-Depth Earthquakes

2018 Graduate Fellow: Shanna Chu, Stanford University

Abstract: The physical mechanisms that generate intermediate-depth earthquakes (70-300 km) within subducting slabs are not well understood, because the processes that enable shallow earthquakes are prohibited at extremely high temperatures and pressures. Because there can be large intraslab earthquakes in regions without mapped faults, it is important to better elucidate their mechanisms to better understand ground motions and hazards. Seismological studies of intermediate-depth earthquakes primarily focus on analyzing data assuming a very simple source model, but the physics of these models are based on shallow earthquake physics and intermediate-depth earthquakes have shown signatures of very complex ruptures. We attempt to create better physical models of intermediate-depth earthquakes using the SORD dynamic rupture code on Blue Waters. We show the results from source inversions of large intermediate depth earthquakes in Japan, as well as the effects of modeling these earthquakes with alternate source models, such as pulse-like ruptures.

Field of Science: Earth Sciences

Watch the presentation | View the presentation slides (PDF)

 

Computational Fluid Dynamics Modeling Reveals a Unique Net-Unidirectional Pulmonary Airflow Patterns in Monitor Lizards (Varanidae)

2018 Graduate Fellow: Robert L. Cieri, The University of Utah

Abstract: Unidirectional pulmonary airflow, a condition where lung gases travel in the same direction through most of the airways throughout the respiratory cycle, has long been of interest to comparative physiologists. Recent work has discovered this phenomenon beyond birds and raised questions about the underlying fluid dynamical phenomena occurring in unidirectional lungs. Computational fluid dynamics, which simulates patterns of flow from prescribed boundary conditions and the laws of fluid motion, provide a powerful tool to study airflow through these complex structures. Here, computed tomography scans were segmented into a detailed computational mesh, representing the major and minor airways of monitor lizards, Varanidae. The surface of the computational meshes expanded and contracted to simulate lung motion during ventilation and provided the boundary conditions for flow. Our models show unidirectional flow in many regions of the lung and reveals airflow patterns in chambers that are too small or are inaccessible with traditional techniques.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

Investigating the Molecular Mechanisms of Drug Induced Cardiac Arrhythmias

Project PI: Colleen Clancy, University of California, Davis

Presented by: Igor Vorobyov, University of California, Davis

Abstract: Induction of potentially deadly abnormal cardiac rhythms is one of the most common and dangerous risks for drugs in development and clinical use. It has been tightly associated with the conduction block of the cardiac potassium channel hERG, which leads to delayed action potential repolarization, manifesting in the prolongation of the QT interval on the ECG. However, not all hERG blocking and QT prolonging drugs cause cardiac arrhythmias. This leads to withdrawal from the development of safe and effective pharmaceuticals. We have developed a multi-scale computational pipeline for safety pharmacology, which allows to determine drug proclivity for arrhythmogenesis based on its chemical structure and fundamental mode of interaction with hERG channel. All-atom enhanced sampling molecular dynamics simulations of hERG drug interactions, simultaneously running on multiple Blue Waters nodes, allowed us to determine state-dependent drug binding affinities and rates, which we used to predict emergent cardiac arrhythmias on cardiac tissue scale.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

Nutrient Loads from Estuaries to the Coastal Ocean: The Role of Resolution and Vegetation on Numerical Estimates

2017 Graduate Fellow: Salme Cook, University of New Hampshire

Abstract: The coastal ocean includes diverse ecosystems encompassing both terrestrial and marine habitats that support approximately a third of the world's population. We are only beginning to understand the economic and environmental value of these resources and how to protect these ecosystems in the face of climate change, sea level rise, extreme storm events, and increased human impact and pollution associated with population growth. These highly nonlinear systems and are difficult to observe, however the advent of numerical models and increased computational resources has made predicting the dynamics of these systems more accessible. An open question this research address is the required resolution to capture the shallow water dynamics, and how to couple these higher resolution models to coarser regional and global models. We employ a validated high-resolution numerical model of a New Hampshire estuary to estimate the nutrient loading from sediments in the estuary to the coastal ocean.

Field of Science: Earth Sciences

Watch the presentation | View the presentation slides (PDF)

 

Continental-Scale Remote Monitoring of Invasive Species Dynamics through Petascale High Performance Computing System

Project PI: Chunyuan Diao, University of Illinois at Urbana-Champaign

Abstract: The rapid expansion of exotic saltcedar along riparian corridors has drastically altered landscape structures and ecosystem functions throughout the United States. Conducting the continental-scale monitoring of spatio-temporal dynamics of this invasive species over the past 40 years is essentially critical to understand its invasion mechanism. Previous studies indicated that the leaf senescence stage is the optimal time window to remotely monitor saltcedar distributions. However, the computational complexity in predicting the leaf senescence timing, along with the massive volume of satellite data in both spatial and temporal dimensions, makes large-scale invasive species monitoring prohibitive. This project aims to develop a parallel computational algorithm on Blue Waters that can accommodate the spatio-temporal variation in plant phenology to facilitate the continental-scale monitoring of invasive species dynamics. Results indicates that the Blue Waters provides unprecedented opportunities to achieve continental-scale monitoring of saltcedar distribution to revolutionize our understanding of saltcedar invasion processes.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

BlueTides Simulation: Predictions for the First Galaxies and Quasars at the Cosmic Dawn

Project PI: Tiziana Di Matteo, Carnegie Mellon University

Presented by: Yueying Ni, Carnegie Mellon University

Abstract: Detecting and understanding the first galaxies and supermassive black holes are one of the major challenges in galaxy formation. Our team has led the development of cosmological codes optimized to petascale/Blue Waters and used these resources to understand how supermassive black holes and galaxies formed, from the smallest to the rarest and most luminous. With close to one trillion particles we have carried out the BlueTides simulation on Blue Waters. BlueTides has been run successfully on the entire set of compute nodes on Blue Waters using the latest version of our MP-Gadget code. With its large volume and high resolution, the BlueTides simulation is the only one that probes directly the first and rarest observed quasars in the early universe. With the increasing rate of discovery of galaxies and quasars at this early epoch, the BlueTides dataset continues to increases its tremendous value as we evolve into the later Universe. We discuss the BlueTides predictions for the first quasars counterparts in the regime at current and upcoming frontier of observational facilities. BlueTides can probe the host galaxies of the first quasars, as well as the gas inflows/outflow that regulate the growth and regulation processes around the first supermassive black holes.

Field of Science: Astronomy & Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Advancing First-Principle Symmetry-Guided Nuclear Modeling for Studies of Nucleosynthesis and Fundamental Symmetries in Nature

Project PI: Jerry Draayer, Louisiana State University

Presented by: Tomas Dytrych, Louisiana State University<

Abstract: Many ongoing and future experiments utilize light and medium mass atomic nuclei as laboratories for exploring the fundamental symmetries of nature and searching for signals of new physics beyond the Standard Model. Determining structure of these nuclei from first principles (or ab initio), that is, using realistic internucleon interactions derived from Quantum Chromodynamics considerations, represents a great computational challenge. We utilized the computational power of the Blue Waters to carry out large-scale first-principle modeling of a range of nuclei previously inaccessible to ab initio theory and experimental studies. We applied a highly scalable hybrid MPI/OpenMP implementation of an innovative many-body technique, that exploits dominant collective symmetries of many-nucleon systems, to provide nuclear structure information for isotopes from oxygen through titanium key to probing physics beyond the Standard model, including exotic unstable isotopes—focus of next-generation large-scale nuclear experimental facilities for studying details of nucleosynthesis processes.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

 

Direct Numerical Simulation of Pressure Fluctuations Induced by Supersonic Turbulent Boundary Layers

Project PI: Lian Duan, Missouri University of Science and Technology

Presented by: Junji Huang, Missouri University of Science and Technology

Abstract: Understanding the physics of the pressure fluctuations induced by turbulent boundary layers is of major theoretical and practical importance. From a practical point of view, an in-depth knowledge of the nature of boundary-layer-induced pressure fluctuations is essential to the structural design of launch vehicles and to the generation of background noise in wind-tunnel facilities. From a theoretical point of view, pressure fluctuations are an important ingredient in turbulence as they appear in statistical correlations such as the pressure-strain correlation terms that redistribute turbulence among different components of fluctuating velocity. In the presentation, I will talk about our use of the Blue Waters supercomputer to conduct direct numerical simulations that provide the basis for an in-depth understanding of the global pressure field induced by turbulent boundary layers at supersonic speeds. I will also discuss about our effort to maximize node-level parallelism on the Blue Waters.

Field of Science: Fluid Systems

View the presentation slides (PDF)

 

Enabling Exascale Astrophysics: Galaxy-Scale Simulations with Enzo-E

2018 Graduate Fellow: Andrew Emerick, Columbia University

Abstract: Developing codes that can take advantage of future generations of computing environments is essential to advancing our understanding of the Universe. The size and scope of simulations possible with current generations of Astrophysical codes are often hampered by substantial scaling limitations beyond 1,000 or 10,000 cores. The Enzo-E / Cello project seeks to push beyond these restrictions to develop an astrophysical hydrodynamics code capable of taking advantage of exascale systems. As part of a Blue Waters Graduate Fellowship I helped to develop necessary physics modules in Enzo-E (including star formation and stellar feedback) with the long-term goal of running large-scale cosmological simulations of galactic evolution. I will discuss this development, present initial scaling results for simulations of an isolated Milky Way mass galaxy, and discuss future plans and development work.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Harnessing Viscous Streaming in Complex Active Systems: Mini-Bots in Fluids

Project PI: Mattia Gazzola, University of Illinois at Urbana-Champaign

Abstract: Recent proof-of-concept demonstrations of artificial and bio-hybrid (partly synthetic, partly biological) miniaturized swimming robots have underscored their potential as diagnostic and therapeutic vehicles, as well as the lack and need of rigorous engineering methods for design and flow control. A brief survey reveals that most prototypes operate in flow regimes where viscous streaming can be leveraged. Streaming generates steady flows in response to body-fluid oscillations and offers powerful control options for transport, mixing, drug delivery, assembly. Here we present a novel approach to co-design bio-hybrid compliant mini robots and associated flow fields so as to control and manipulate the surrounding environment.

Field of Science: Engineering

View the presentation slides (PDF)

 

Simulations of the Structure of Magnetic Fields in Galaxy Clusters

2019 Graduate Fellow: Forrest W. Glines, Michigan State University

Abstract: Magnetic fields have been observed on galactic scales. However, their coupling to the gas dynamics and turbulence in and around galaxies is still debated, especially concerning small-scale dynamo action and turbulent cascades. Resolving the turbulent cascade in magnetohydrodynamic (MHD) simulations of galaxies requires computational resources much larger than current supercomputers. Simplified models of turbulence are needed to reproduce the effects of magnetized turbulence below the grid resolution. High resolution simulations of MHD turbulence can be used to develop these subgrid models. To efficiently simulate astrophysical plasmas on current and next generation supercomputers, we have developed K-Athena, a performance portable version of Athena++ that also runs on GPUs. Using K-Athena, I have simulated the Taylor-Green vortex in the ideal compressible MHD regime to study transfer of magnetic and kinetic energies between scales in an idealized setup. This work will be used to inform high resolution galaxy simulations and subgrid turbulence models for Enzo-E, a cosmology code built for exascale.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Modeling Quasar Radiative Feedback during Cosmic Reionization

Project PI: Nick Gnedin, University of Chicago

Presented by: Huanqing Chen, University of Chicago

Abstract: Recent observational discoveries of lack of Lyman-alpha emitters near z~6 quasars raised questions about how quasars impact galaxy formation during the cosmic reionization. To accurately model this problem requires both a large cosmological simulation volume and high spatial resolution to resolve galaxies, as well as fully self-consistent radiative transfer (RT). Here I present the first attempt to model quasar proximity zones in full RT hydrodynamic cosmological simulations. We find that a quasar suppresses star formation in low mass halos (M<10⁹Msun) through dissolving molecular hydrogen. However, due to the decreased stellar feedback, star formation catches up after ~50 Myr. The effect on the galaxy luminosity function is observable at faint magnitudes, but also depends on the quasar lifetime. This effect can be observed by the JWST in the near future, and can be used as a novel probe of cosmic reionization.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

GPU Accelerated Time-Dependent Chemistry in the Context of Galaxy Formation with WIND

2018 Graduate Fellow: Alex Gurvich, Northwestern University

Abstract: Massive outflows of gas from galaxies known as galactic winds are crucial in regulating galactic star formation rates. These outflows are observed to consist of multiple phases, including both hot (~million K) and cold, molecular, gas. Previous simulations either lacked the resolution or the chemistry treatment necessary to capture these phases. This limits the ability of previous work to produce powerful galactic winds and also accurately predict observations of their chemical diagnostics. In response, we present WIND-CHIMES, a new GPU-accelerated version of the time-dependent chemistry code CHIMES that enables us to run high resolution simulations that, for the first time, will capture the full range of phases of galactic winds. We will show initial results of applying WIND-CHIMES to the context of galaxy formation, including convergence tests and speed comparisons along with a description of the new aspects of the code and its key features.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

The Hepatitis B Capsid through the Computational Microscope

Project PI: Jodi Hadden, University of Delaware

Abstract: Hepatitis B virus is a major cause of liver disease. Researchers are endeavoring to develop new treatments against the virus that target its capsid, a protein shell that encases the viral genome and drives its delivery to the host cell-nucleus. Numerous drugs that disrupt the capsid have been identified, but their mechanisms of action are incompletely understood. The PI has been utilizing Blue Waters as a computational microscope since 2015 to investigate the intact hepatitis B capsid and the effect of drugs on its structure, dynamics, and assembly. Here, the PI will describe her continued efforts to characterize the capsid as a drug target, including important new discoveries regarding drug binding modes, cooperativity in drug-uptake, and mechanisms of drug resistance. Importantly, the results produced by this study were inaccessible to experimental methods and were made possible only through molecular dynamics simulations of multimillion-atom capsid systems enabled by Blue Waters.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

Tilted Disks around Black Holes: Investigating the Alignment Mechanism

Project PI: John Hawley, University of Virginia

Abstract: When matter orbits around a black hole obliquely with respect to the holes spin axis, a relativistic torque causes the orbits to precess. Forty years ago, astrophysicists proposed that the orbital angular momentum in accreting gas should align with the mass' spin near the hole. The location of the alignment front is determined by a balance between the torque, the resulting differential precession, and warp-induced inward mixing of misaligned angular momentum from the outer to the inner disk. Analytic approaches are restricted to linear one-dimensional analysis, using viscosity for dissipation, despite the fact that accretion disks are not viscous systems, but MHD turbulent. Numerical simulations are required, but three-dimensional high-resolution simulations required have only recently become possible. We are investigating alignment through MHD and hydrodynamic simulations of mis-aligned disks. Our approach has been to use a semi-Newtonian method, using idealized disks that can probe the physical process in detail.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Hybrid MD/Spectroscopic Refinement of Heterogeneous Conformational Ensembles on Blue Waters

2017 Graduate Fellow: Jennifer Hays, University of Virginia

Abstract: Multi-structured biomolecular systems play crucial roles in many cellular processes but have resisted traditional methods of structure determination, which often resolve only a few low-energy states. High-resolution structure determination of conformational ensembles using experimental methods that report on heterogeneity, such as double electron-electron resonance (DEER), remains extremely challenging. We have therefore developed a computational method to integrate these sparse, non-parametric experimental data to obtain estimates of conformational ensembles using molecular dynamics (MD) simulation. Existing methods of incorporation capture side-chain fluctuation rather than backbone conformational change; our method is particularly designed to address this outstanding challenge. Using Blue Waters, we have tested our method by incorporating DEER data obtained on the SNARE protein syntaxin-1a into MD simulation. We have found that our method of integration substantially outperforms existing state-of-the-art methods in capturing syntaxins open/closed conformational equilibrium. Our improved refinement method will greatly accelerate the structural understanding of such systems.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

Manipulating Small Droplets in Microchannels with Complex Fluids

Graduate Fellow: Michael Howard, University of Texas at Austin

Abstract: Controlled particle migration in a microchannel has important applications in separation technologies like filtration, cell sorting, and fractionation. It also has implications for physical processes like the margination of cells in the blood stream and for multiphase flows in geological formations (enhanced oil recovery). We investigated the migration of a small fluid droplet in a parallel-plate microchannel using large-scale dissipative particle dynamics computer simulations. In a Newtonian solvent, the droplet moved toward the channel walls, in agreement with theoretical predictions and recent simulations. However, the droplet focused onto the channel centerline in a viscoelastic complex fluid (dilute polymer solution). Focusing was typically enhanced for longer polymers and higher polymer concentrations, with a nontrivial flow-rate dependence due to droplet and polymer deformability, but the droplet position fluctuated because of Brownian motion. Such viscoelastic focusing is shown to be a viable method to manipulate small fluid droplets in microchannels.

Field of Science: Engineering

View the presentation slides (PDF)

 

Deep Learning at Scale: A Paradigm Shift for Multi-Messenger Astrophysics

Project PI: Eliu Huerta Escudero, University of Illinois at Urbana-Champaign

Abstract: I present new developments at the interface of deep learning, high performance computing, numerical relativity and electromagnetic surveys to push the frontiers of Multi-Messenger Astrophysics. I discuss the application of deep learning at scale to characterize binary black hole mergers through gravitational wave observations, developing the first generation of neural network models that are trained with over a trillion waveforms. It is shown that once fully trained, these state-of-the-art neural network models enable gravitational wave parameter estimation within a few milliseconds using a single V100 GPU. The combination of these algorithms with the construction of galaxy catalogs using deep learning algorithms is discussed in the contest of gravitational wave cosmology.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

The Power of Many: The Next Frontier

Project PI: Shantenu Jha, Rutgers University

Abstract: Current computing trends make the ensemble computational model highly relevant; it has the ability to overcome limitations of single task applications, and to achieve significant performance gains on large-scale parallel machines. Not surprisingly, the concept of running ensembles on large-scale HPC systems is thus gaining in importance. We discuss some of the challenges in executing ensembles at scale. We discuss the abstractions (pilot-systems), software systems (RADICAL-Cybertools) and execution models that we have developed to address many of these challenges. We will discuss how RADICAL-Cybertools, along with advances in statistical and adaptive algorithms have enabled ensemble-based applications to overcome limitations oftraditional single task applications. In spite of order(s) of magnitude efficiency gains, much greater improvements are still needed. We will close with a brief mention of novel application architectures that hold promise to provide the needed improvements.

Field of Science: Computer Science

View the presentation slides (PDF)

 

High-fidelity Numerical Simulations of Collapsing Cavitation Bubbles near Solid and Elastically Deformable Objects

Project PIs: Eric Johnsen, University of Michigan; and Zhen Xu, University of Michigan

Presented by: Mauro Rodriguez, University of Michigan

Abstract: The violent collapse of cavitation bubbles can damage its surroundings and causing cavitation erosion on material surfaces. To understand this phenomenon, we have developed a novel numerical model to conduct efficient, high-fidelity simulations to investigate the collapse of single and multiple bubbles near (i) semi-infinite rigid and (ii) finite-size elastic objects. The numerical model solves the three-dimensional compressible Navier-Stokes equations for a multi-component system and includes visco(elasticity) to investigate the interactions between an object and the bubble collapse fluid flow. The Blue Waters system's computational power and uniqueness was essential to conduct the high-resolution simulations to quantify the non-spherical bubble morphology, maximum pressure/temperature fields and elastic components of stress therein produced. As a result, these fundamental simulations have furthered our ability to quantify the damage mechanisms and lead to strategies to better control cavitation-induced erosion in a wide-range of naval and medical engineering and energy science applications.

Field of Science: Fluid Systems

View the presentation slides (PDF)

 

Development of Large Scale Electronic Structure Computations and Applications

Project PI: Harley T. Johnson, University of Illinois at Urbana-Champaign

Presented by: Purnima Ghale, University of Illinois at Urbana-Champaign

Abstract: Extending the size of systems for which electronic structure can be computed has been an ongoing effort at different levels of electronic structure theory. Within tight-binding, system sizes are limited by the linear algebraic problem of computing the density matrix whose rank increases with the number of electrons. In recent work, an implicit algorithm that scales linearly with system size, and relies entirely on sparse matrix-vector multiplications (SpMVs) was presented. Now, using the PETSC matrix library optimized for Blue Waters, it is possible to simulate 10⁶ atoms in a few minutes (<100 nodes). Simulations approaching 10⁷ atoms have been demonstrated, and further increase in system sizes by one or two orders of magnitude is also possible. Electronic structure simulations of this size allow us to simulate microscale systems with atomic resolution in particular, we are interested in microscale plasma generators, which have several technological applications.

Field of Science: Materials Science

Watch the presentation | View the presentation slides (PDF)

 

Efficient Use of HPC Resources for Turbulent Mixing Simulations

Project PI: Tulin Kaman University of Arkansas

Abstract: Numerical simulations of turbulent mixing play an important role in understanding the chaotic mixing dynamics driven by the fluid interface instabilities and in predicting the growth rate that describes the outer edge of the mixing zone. The essential features of the algorithm used in our numerical studies are front tracking, to achieve resolution of a steep and sharp discontinuity in density gradients and Large Eddy Simulations with subgrid scale to model the diffusive transport corrections to the mesh averaged Navier-Stokes equations. The focus of this project is to improve the scalability of the front tracking code for turbulent mixing simulations on HPC systems. The performance measurements and analysis using TAU and CPMAT allow us to observe the performance on Blue Waters, identify the most computationally expensive parts and enhance the performance of collective operations and high order accurate weighted essentially non-oscillatory numerical scheme.

Field of Science: Fluid Systems

View the presentation slides (PDF)

 

Improving Virtually Guided Certification for Product Design with Implicit FEA Software LS-DYNA

Project PI: Seid Koric, University of Illinois at Urbana-Champaign

Presented by: Erman Guleryuz, University of Illinois at Urbana-Champaign

Abstract: This project addresses the applied research goal of virtual certification pursued by NCSA Industry program and their academic and industrial collaborators, a key challenge in the design and manufacturing of gas turbine engines, in particular for aerospace applications. We are advancing the state of the art in computationally intensive implicit finite element analysis by studying real-world jet engine models with LS-DYNA software, having as many as 200 million degrees of freedom, and identifying and removing performance scaling barriers. This is a continuation of our previous work on Blue Waters that demonstrated the feasibility of efficiently solving extreme-size real-world multiphysics problems at peta- and potentially exa-scale levels thus adding novel value to engineering research by enabling the creation of large-scale, high-fidelity models that yield accurate and detailed insight into the performance and safety of a proposed engineering design.

Field of Science: Engineering

View the presentation slides (PDF)

 

Electron Density-Based Machine Learning for Accelerating Quantum Calculations

2019 Graduate Fellow: Joshua L. Lansford, University of Delaware

Abstract: The electronic density distribution completely specifies a chemical system's state and can be calculated using density functional theory. We are developing an atom-centered machine learning (ML) algorithm, trained on electron density, which can generate catalytic reaction mechanisms and kinetic models at reduced computational cost. The ML-based model will accelerate the three most computationally intensive components of the reaction energy profile: transition states, global minima, and entropy. This methodology will be the first to combine structural and density data to enhance ML convergence. Due to the atom-centered nature, the representation of the molecular systems will be invariant to rotations, translations, and reordering of atoms. Because we will use distances, partial charges, and atomic dipoles, the model will be generalizable to any size system.

Field of Science: Engineering

View the presentation slides (PDF)

 

Connecting Microscale Processes to Mesoscale Phenomena: Improving Cold Pool Parameterizations

Project PI: Sonia Lasher-Trapp, University of Illinois at Urbana-Champaign

Presented by: Holly Mallinson, University of Illinois at Urbana-Champaign

Abstract: Global circulation models operate at scales requiring parameterizations to represent the effects of convective storms that occur over distances too small to be directly resolved. These parameterizations lack sufficient representation of the primary convective components. Of particular interest here is the cold pool, produced by the melting or evaporation/sublimation of ice/raindrops falling from the storm. Improving parameterizations requires a better understanding of these microscale processes contributing to the larger-scale cold pool, requiring a set of high-resolution simulations over large domains with various modifications of precipitation processes, a feat that is best accomplished with petascale computing on Blue Waters. Results from the 12 simulations show that (i) the sublimation of graupel within downdrafts contributes the most to the cold pool; (ii) evaporation of rain better explains its depth and propagation speed; and (iii) the importance of the warm rain process in creating the earliest precipitation that modulates its first appearance.

Field of Science: Atmospheric and Climate Science

Watch the presentation | View the presentation slides (PDF)

 

Electronic Detection Of DNA-Nicks using 2D Solid-State Nanopore Transistors

Project PI: Jean-Pierre Leburton, University of Illinois at Urbana-Champaign

Presented by: Nagendra Athreya, University of Illinois at Urbana-Champaign

Abstract: DNA double-strand breaks make up a vast majority of lesions a normal human cell undergoes every day. Here, we propose to employ graphene nanopore transistor to detect nicks in the DNA backbone efficiently using electronic sheet currents obtained across the membrane. For this purpose, we use molecular dynamics simulations accompanied with electronic transport calculations, data denoising and signal detection. While the ionic currents do not show any distinct signature from the nicked-site in the signal, a clear dip is seen in the transverse sheet current signal corresponding to the location of the breakage, thereby enabling us to detect and map these damages, electronically. For this research work, we rely greatly on supercomputers such as Blue Waters nodes to run our MD simulations and electronic transport calculations. We strongly believe such a detection mechanism enables the development of versatile sensor for early cancer detection caused by structural modification of the genome.

Field of Science: Engineering

View the presentation slides (PDF)

 

Variations in Massive Star Explosions

Project PI: Eric Lentz, University of Tennessee

Abstract: I will present the results of our PRAC work on Blue Waters evolving the collapse and explosion of massive stars as core-collapse supernovae using large, long-running, three-dimensional, radiation hydrodynamics simulations. I will focus on the diversity of outcomes and how they are tied to the variety of input conditions that represent stars of different masses and compositions that experience core collapse throughout the galaxy and universe, and also show how these simulations fit into a long-term simulation program and can be used for years to come.

Field of Science: Astronomy & Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Extending Plate Tectonics into the Earth's Interior using Supercomputers

Project PI: Lijun Liu, University of Illinois at Urbana-Champaign

Presented by: Zebin Cao, University of Illinois at Urbana-Champaign

Abstract: The theory of plate tectonics has become the foundation of modern Earth sciences. However, many questions remain on the fate of subducted plates and their effects on the Earth's surface geology. Answers to this question begs for a quantitative understanding of the internal dynamics of the solid Earth, a process only becoming tangible in the past decade due to the fast development of supercomputing. At the University of Illinois, my group has been using Blue Waters to construct such quantitative and deterministic geodynamic models. By now, we have developed the largest in-scale models of its kind, with up to 10,000 MPI cores for a single simulation. Most of our models use data assimilation that merges data from geology, tectonics and geophysics into the same geodynamic framework. Notable examples include our simulation of subduction and convection on a global scale, mantle processes below Yellowstone, and the long-term evolution of continental lithosphere.

Field of Science: Earth Sciences

Watch the presentation | View the presentation slides (PDF)

 

Magnetic Field Amplification in Binary Neutron Star Merger Remnants

Project PI: Philipp Moesta, University of California, Berkeley

Abstract: Binary neutron-star mergers belong to the most energetic transients in the universe. They are key targets for time-domain astronomy surveys and advanced LIGO in the era of multimessenger observations. Magnetic fields play a crucial role in the merger and postmerger and are the key ingredient to determine the lifetime of the hypermassive neutron star remnant, the composition of ejected material, and the formation of a sGRB engine. I will discuss the unique challenges in both input physics and computational modeling for these systems involving all four fundamental forces and present results from recent breakthrough 3D simulations performed on Blue Waters.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Employing Microsecond-Level Simulations of Membrane Proteins to Capture Their Millisecond-Level Behaviors

Project PI: Mahmoud Moradi, University of Arkansas

Abstract: We employ all-atom molecular dynamics simulations along with novel enhanced sampling techniques to characterize the large-scale conformational changes of membrane transporters and their coupling to chemical events. We have specifically used Blue Waters to study the conformational landscape of a bacterial membrane transporter involved in peptide transport. Proton-coupled oligopeptide transporters (POTs) couple the inwardly directed proton flow to the transport of small peptides and peptide-like molecules. The human POT transporters PepT1 and PepT2 provide the main route through which the body absorbs and retains dietary proteins. Employing enhanced sampling techniques and path-finding algorithms within an ensemble-based molecular dynamics framework, we have been able to characterize the conformational free energy landscape of GkPOT, a homolog of human POTs, and its dynamic behavior at the millisecond timescale. A detailed description of GkPOT conformational landscape achieved by these simulations sheds light on the structure-function relationships in POTs and other membrane transporters.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

The World Is Upside Down: Why We Have Better Topography of the Poles than the Rest of the Planet

Project PI: Paul Morin, University of Minnesota

Abstract: For years, those of us that made maps of the Poles apologized. We apologized for the blank spaces on the maps, we apologized for mountains being in the wrong place and that the information was out of date. Over the past 10 years the situation improved. The Landsat image mosaic was produced, BEDMAP provided an improving view under the Antarctic ice sheet and a constellation of satellites started to stream data at ever higher resolution, at an increasing tempo and even during the long Polar winters.

Now a diverse collaboration of US science and intelligence agencies, universities and a geospatial software company have produced REMA—the Reference Elevation Model of Antarctica and ArcticDEM—using open source software to process licensed imagery on Blue Waters to produce a Digital Elevation Model (DEM) that is at an 2m posting, with an accuracy of ~0.35cm, with that repeats 90% of the poles eight times over five years. This project was too large for any one agency, university or company. It required the equivalent of running the fastest desktop computer for 50,000 years, four satellites that collected sub-meter optical imagery for five years and petabytes of storage.

We never thought that we would be in this place and at a time where the science community has better topography for ice than most of the land on earth and better topography for Marie Byrd Land than we have for the Rocky Mountains. Even we as the creators of REMA are having a difficult time understanding what we have made. The use of it in logistics and facilities management was a surprise. We are as dumbfounded as anyone at the degree that the bed of the Antarctic and Greenland Ice Sheets are seen in the surface topography. Incredibly, the volume and repeat DEM of the data is causing all of us to reassess how we manage and analyze geospatial in general.

We now apologize to the polar science community for a different reason. They have to keep up and the current DEMs are only the beginning. Antarctic topography has been processed to a 2m posting and the data from ICESat-2, the NASA mission that will provide ground control and increase the accuracy with near coincident altimetry, will provide yet another leap forward providing 16 times the resolution and 16 times the file size with a potential of increased accuracy. In addition, we now face an avalanche of imagery in an increasingly complex landscape of small-sats launch by the dozen by commercial start-ups and governments with data licensing that ranges from open to buy it by the pixel. It is a complex, exciting time.

Field of Science: Earth Sciences

 

Deep Learning for Higgs Boson Identification and Searches for New Physics at the Large Hadron Collider

Project PI: Mark Neubauer, University of Illinois at Urbana-Champaign

Abstract: The quest to understand the fundamental building blocks of nature and their interactions is one of the oldest and most ambitious of human scientific endeavors. As the world's most powerful particle collider, the Large Hadron Collider (LHC) represents a huge step forward in this quest. The 2012 discovery of the Higgs boson demonstrates the great scientific value of the LHC and the Higgs particle itself represents a new tool for discovery. In this talk, we present our work using deep learning techniques on the Blue Waters supercomputing to develop and optimize a novel method of identifying the decays of highly-boosted Higgs bosons produced at the LHC a signature of new particles and/or phenomena at the energy frontier of particle physics. We also discuss our ongoing work using Blue Waters to develop scalable cyberinfrastructure for sustainable and reproducible data analysis workflows through the NSF-funded IRIS-HEP Institute and SCAILFIN project.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

 

Realistic Simulations of the Intergalactic Medium: A Scalable Gravity Solver for Enzo-E

Project PIs: Michael L. Norman and James Bordner, UC San Diego

Abstract: In our continuing quest for more comprehensive models of the intergalactic medium including resolved galaxies, we are developing Enzo-P/Cello (now renamed Enzo-E), the petascale/exascale fork of the popular Enzo cosmological AMR code. Enzo-E employs an array-of-octrees AMR mesh for improved scalability, and is parallelized using Charm++. We report on progress implementing a scalable Poisson solver for Enzo-E for cosmological applications. The solver consists of two steps: in a first step, the global gravitational potential on the uniform based grid is computed using geometric multigrid. In a second step, the local gravitational potential in each octree of the array is solved using a biconjugate gradient method subject to boundary conditions taken from the global mesh. We report on the weak and strong scaling properties of the DD solver, and compare its speed and accuracy to the original Enzo code.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Exploring the Extremes of Galaxy Formation using Blue Waters

Project PI: Brian O'Shea, Michigan State University

Presented by: Forrest W. Glines, Michigan State University

Abstract: Galaxy formation is quite numerically challenging, due to the wide range of physics and physical scales involved. In this talk I present two results from our collaboration's simulation campaign to explore galaxy collaboration over the entire age of the universe on Blue Waters, which has spanned three PRAC grants, one GLCPC allocation, and the entirety of the Blue Waters lifecycle. The first result is the discovery of a new mechanism for the formation of supermassive black holes in the early universe (Wise et al. 2019, Nature), which unlike much previous work provides a natural explanation for the abundance of incredible massive black holes observed in the universe roughly a billion years after the Big Bang. The second result explores the diffuse circumgalactic plasma around Milky Way galaxies using incredibly high spatial resolution, and shows that this increased (and expensive!) resolution is crucial to resolving hydrothermal instabilities in this gas, thus understanding observations of present-day galaxies. These results are exemplars of the type of theoretical studies that are only possible on capability-scale supercomputers, and I close the talk by highlighting two codes, Enzo-E and K-Athena, that build on lessons learned with Blue Waters in order to be both scalable and highly performant on future exascale supercomputers.

Field of Science: Astronomy & Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Utilizing Machine Topology in Numerical Algorithms

Project PI: Luke Olson, University of Illinois at Urbana-Champaign

Abstract: The focus of this talk is on improving simulation performance by exposing algorithms to machine layout, as found in modern machines such at Blue Waters. The solution to large sparse linear systems, for example, often demands a high amount of communication. By reorganizing the communication in a node-centric manner, the number of costly intranode messages can be reduced. The talk will highlight the benefits of using node aware communication in the context of a multigrid solver. The solve times can be reduced in comparison to conventional approaches. The results also extend to a wider range of sparse matrix operations and a general communication package will be presented.

Field of Science: Computer Science

Watch the presentation | View the presentation slides (PDF)

 

Simulation and Analysis of Severe Thunderstorms on Blue Waters

Project PI: Leigh Orf, University of Wisconsin

Presented by: Kelton Halbert, University of Wisconsin

Abstract: Thunderstorms are responsible for causing severe winds resulting from tornadoes and strong downdrafts called downbursts. Over the past five years our group has conducted simulations of supercell thunderstorms at resolutions in which fully resolved tornadoes form. A file system utilizing lossy floating point compression of 3D arrays was developed in order to accommodate the large amount of data produced by these simulations. Due to the large data volume, new analysis techniques have been developed that leverage Graphics Processing Units (GPUs) to conduct fast quantitative analysis. One such tool for analyzing saved model data in a Lagrangian particle framework has been developed, allowing parcels of air to be tracked from simulation data in which data has been saved every model time step of 1/6s. In this talk, recent analysis of tornado and downburst producing thunderstorms will be presented, along with details on data analysis on the GPU.

Field of Science: Atmospheric and Climate Science

View the presentation slides (PDF)

 

Forecast Geomagnetic Secular Variation using NASA GEMS Data Assimilation System

Project PI: Nikolaos K. Pavlis, National Geospatial-Information Agency

Presented by: Weijia Kuang, NASA Goddard Space Flight Center

Abstract: Earth's intrinsic magnetic field, called the geomagnetic field, is observed to vary on broad spatial and temporal scales, and is generated by vigorous convection in the Earth's liquid outer core (geodynamo). Understanding and prediction of such variations are very important for fundamental scientific research and for societal applications. This project, under the collaboration of NASA and NGA, aims at investigating the properties of covariances of large ensembles of geodynamo model solutions, and their implications to the forecast accuracies of geomagnetic secular variations on time scales of 5 to 20 years. The model used in this project is the GEMS (Geomagnetic Ensemble Modeling System) developed in NASA GSFC. It includes three components: a 3D geodynamo model to provide forecasts, the EnKF model to provide analysis with model forecast covariances, and the driver to manage interactions between the other two components. This project requires large amount of computing resources: for a single test with the ensemble size N_e and the spatial resolution N³, thus the total flops required are ~ 5 X 10⁶ x N_e x N³. We plan to carry out three ensemble tests initially with N_e≥ 200 and N ~ 160. More tests will be carried out if needed. The final tested system will be used to generate candidate model to International Geomagnetic Reference Model (IGRF) and World Magnetic Model (WMM).

Field of Science: Earth Sciences

Watch the presentation | View the presentation slides (PDF)

 

Seven Years of HIV Research on Blue Waters: Past, Present and Future

Project PI: Juan Perilla, University of Delaware

Abstract: The essential conundrum of modern biology, namely the question of how life emerges from myriad molecules whose behavior is governed by physical law alone, is embodied within a single cell—the quantum of life. The rise of scientific supercomputing has allowed for the study of the living cell in unparalleled detail, from the scale of the atom to a whole organism and at all levels in between. In particular, the past three decades have witnessed the evolution of molecular dynamics simulations as a "computational microscope," which has provided a unique framework for the study of the phenomena of cell biology in atomic (or near-atomic) detail.

Here I present an overview of our use of Blue Waters, starting from the early science days until the present date, to determine the molecular details during the life-cycle of multiple infectious diseases.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

The Dark Energy Survey: Using Blue Waters for Experimental Astrophysics

Project PI: Donald Petravick, University of Illinois at Urbana-Champaign

Abstract: For years, extreme scale high-performance computing has been used to investigate cosmology—the origin and development of the universe via simulations. Optical Astronomical Surveys, such as the Dark Energy Survey, collect data used to compare cosmological simulations with observation based measurements. The Dark Energy Survey recently completed six years of observations primarily targeted at the phenomena known as dark energy, the continuing acceleration of the universe. Blue Waters is a crucial infrastructure used for the most ongoing demanding tasks to derive physical measurements from the observations of the Dark Energy Survey. My goal for attendance—to continue to work/understand issues with HPC machines in the context of experimental, rather than simulation science.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Modeling Physical Processes in the Solar Wind and Local Interstellar Medium

Project PI: Nikolai Pogorelov, University of Alabama in Huntsville

Abstract: The challenge of our project is to perform multi-scale simulations of the solar wind (SW) from the solar surface to the boundary of the heliosphere, and its interaction with the local interstellar medium (LISM). Our simulations have both fundamental and practical applications: (1) they involve discontinuous flows of partially ionized, collisional and collisionless plasmas, and (2) help interpret spacecraft observations and predict hazardous conditions affecting humans and electronics in space. Our accomplishments: (1) we removed the heliospheric contribution to the TeV cosmic ray (CR) anisotropy observed in the Tibet air shower experiment and identified a source of the original anisotropy; (2) developed a data-driven model for coronal mass ejections; and (3) studied the SW-LISM interaction effect on CRs. MS-FLUKSS solves self-consistently the system of MHD equations for the ion-electron mixture, kinetic Boltzmann equation for neutral atoms, and two separate systems for PUIs and turbulence generated by them.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Exploring Sea Spray Aerosols with a Computational Microscope

Project PI: Kimberly Prather, University of California, San Diego

Abstract: Sea spray aerosols (SSA) make up a significant portion of atmospheric aerosols. However, our understanding of how they impact clouds and climate represents one of the largest sources of uncertainty in current climate models. Aerosols directly impact the global energy balance by interacting with incoming solar radiation. Indirectly, they serve as nuclei for cloud formation and provide surfaces for heterogeneous chemistry influencing the composition of our atmosphere. Recent developments in both field and laboratory studies have led to a greater understanding of the chemical components and size distributions of SSA particles, as well as the physical mechanisms that lead to the transfer of organic and biological material from the ocean to the atmosphere. However, due to limitations in single particle analysis methods, there is still an incomplete understanding of individual SSA chemical composition and morphology and how these directly influence the climate-relevant properties of SSA. It has been shown experimentally that SSA particles are coated with organic surfactants, and that these surfactants significantly impact the transfer of water and reactive uptake of gases through the SSA surface. Additionally, recent evidence has shown that enzymes are present and remain highly active in SSA. Thus a major outstanding goal of CAICE is to develop a better understanding of how the chemical complexity of SSA impacts the reactions that take place at the air-water interface and on the surfaces of SSA. I will discuss how integration of computational and experimental tools in CAICE are being used to characterize SSA, plans for modeling these particles at the molecular level, and our goals in using Blue Waters to simulate the dynamics of these modeled SSA particles. Our efforts provide a major shift in approach to existing current studies at the molecular scale, and will provide the first glimpses into the detailed time-dependent dynamics of nascent SSA at unprecedented scales of size and complexity.

Field of Science: Chemistry

 

Unified Modeling of Galaxy Populations in Clusters

Project PI: Thomas Quinn, University of Washington

Abstract: Modeling galaxies and the Intra-Cluster Medium (ICM) in galaxy clusters is a formidable challenge. The morphology of the galaxies, and the energy and metal content of the gas is ultimately controlled by star formation processes that happen on molecular cloud scales of less than a million solar masses. On the other, total cluster masses exceed 10¹⁵ solar masses; hence a dynamic range in mass of over a billion is necessary for consistently modeling galaxies within this context.

With Blue Waters we have tackled this challenge and produced the highest resolution hydrodynamic simulations of galaxy clusters run to date. The simulations have given us new insight into the connected evolution of the cluster galaxies, the ICM and the active galactic nuclei within the cluster.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Mapping Proton Quark Structure using Petabytes of COMPASS Data

Project PI: Caroline Riedl, University of Illinois at Urbana-Champaign

Abstract: COMPASS probes proton substructure by scattering high-energy pion- and muon-beams on nuclear targets at CERN. The experiment explores the momentum and coordinate phase space of quarks inside the proton. Observing correlations between proton spin and the intrinsic transverse momentum of quarks will shed light on the quark dynamics inside the proton and will provide a critical test of fundamental predictions derived from Quantum Chromo Dynamics, the quantum field theory describing the strong nuclear force. The measurements produce 10 petabytes of experimental and simulated data. Blue Waters' balance of processing capabilities and data storage and handling is well suited for the analysis of the large COMPASS data samples as these require significant algorithmic processing per pion/muon-proton scattering event. In addition to raw data processing and physics-level analysis, COMPASS carries out extensive studies of systematic effects as Blue Waters allows for a detailed simulation of COMPASS detector and environmental properties.

Field of Science: Physics

View the presentation slides (PDF)

 

Improved Initial Lapse and Shift for Binary Black Hole Simulations

2019 Graduate Fellow: Nicole Rosato, Rochester Institute of Technology

Abstract: We present new initial lapse and shift by constructing analytic formulas that mimic their expected settled behavior in the moving puncture gauge. These initial values allow for improvements in the initial shape of both the lapse and the shift. Current simulations set initial β = 0 and initial lapse to be isotropic asymptotically; our new initial construction of α sets these initial values to be closer to the settled shape of α and β for boosted, spinning black holes. To evaluate the effects of this initial gauge choice on evolutions, the Hamiltonian and Momentum constraint violations are monitored. This method shows improvements in a simulation of an unequal-mass, q = 3, nonspinning system and also in an equal-mass system with spins a = 0.8. We expect that this method will reduce the amount of time required for the settling of the lapse and shift, and will make simulations of both high mass-ratio and near-extremally spinning systems more efficient.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Spectral/Discontinuous Galerkin Approach to Fully Kinetic Simulations of Plasma Turbulence with Reduced Velocity Space

Project PI: Vadim Roytershteyn, Los Alamos National Laboratory

Presented by: Oleksandr Koshkarov, Los Alamos National Laboratory

Abstract: Plasma turbulence is ubiquitous in the Universe. In most space and astrophysical systems, turbulence dissipates energy and momentum via microscopic collective processes operating at small (kinetic) scales. Correct description of these processes is only possible via 6D Vlasov equation, whose high dimensionality coupled with large scale separation prohibits the use of many standard numerical tools for simulations with parameters typical for space environment. We describe a new method for the solution of the kinetic equations for a magnetized plasma. It is based on spectral expansion of the velocity space with Asymmetrically Weighted Hermite Polynomials (AWHP), together with discontinuous Galerkin (DG) approximation for coordinate space. The spectral AWHP expansion allows significant reduction in number of degrees of freedom required to represent velocity space, while still retaining kinetic effects. Moreover, the spectral expansion is isomorphic to classical fluid moment expansion, hence providing a fine control over fluid/kinetic regime transition with number of used polynomials/moments. DG approximation guarantees high accuracy (arbitrary order) required for turbulence simulation. At the same time, DG high locality ensures parallel efficiency leading to optimal scalability, which is desirable for large scale kinetic simulations on modern HPC architectures. Here, we demonstrate properties and capabilities of Spectral Plasma Solver framework based on DG approximation (SPS-DG) on classical example of decaying magnetized plasma turbulence, the Orszag-Tang vortex test. We further compare SPS-DG with fully more standard tools, such as Particle-In-Cell codes.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

First-Principles Study of Voltage-Induced Switching, Optical Properties, and Heat Capacity of Antiferromagnetic Materials

Project PI: Andre Schleife, University of Illinois at Urbana-Champaign

Presented by: Kisung Kang, University of Illinois at Urbana-Champaign

Abstract: Antiferromagnetic materials (AFM) have fascinating properties like robust properties against external stimulations and absence of stray fields, which makes them interesting candidates for future memory devices. To understand the properties of AFMs, two different studies are implemented involving Blue Waters: First, voltage-induced switching of antiferromagnetically ordered topological orthorhombic CuMnAs is explored. Model Hamiltonian theoretical studies indicate that a chemical potential change can induce the reorientation of the Néel vector and transition from semi-metallic to semiconducting. First-principles density functional theory (DFT) is performed to verify model Hamiltonian analysis in orthorhombic CuMnAs. Second, optical properties and heat capacity of metallic antiferromagnetic Fe₂As are analyzed. In collaboration with experiment, the magneto-optic response of Fe₂As to ultrafast temperature excursions is investigated. Using electron and phonon dispersion from DFT, the electron and phonon heat capacity was obtained and subtracted from the total heat capacity from experiment, to obtain the magnon heat capacity.

Field of Science: Materials Science

Watch the presentation | View the presentation slides (PDF)

 

Hydrodynamics beyond Navier-Stokes: Mass and Energy Transport in Nanofluidic Flows through the Lens of the Numerical Model

Graduate Fellow: Sean Seyler, Arizona State University

Abstract: In microfluidic flows, fluctuations in stress and heat flux continuously drive the fluid out of local thermodynamic equilibrium (LTE). The stochastic Landau-Lifshitz Navier-Stokes (LLNS) equations have helped reveal how such fluctuations, amplified by nonlinearities, underlie unique phenomena such as giant fluctuations, enhanced diffusive transport, and nanojet droplet breakup. At yet finer spatial discretization, fluids appear granular and, with very few particles per grid cell, leveraging particle-based simulations seems simpler than justifying the usual constitutive laws (Newton's law of viscosity and Fourier's law), the continuum approximation, and LTE. Yet our work suggests that a fluctuating hydrodynamics model can not only be computationally viable and economical, but also physically preferable because its structure explicitly encodes the underlying transport physics. Our numerical model, HERMESHD, naturally extends LLNS and is seen to ultimately emerge from higher-moment approximations to the Boltzmann transport equation.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

 

Dark Matter Interactions with Nuclei

Project PI: Phiala Shanahan, Massachusetts Institute of Technology

Presented by: Michael Wagman, Massachusetts Institute of Technology

Abstract: There is strong evidence for the existence of dark matter, but most of its properties are unknown. Experiments around the world are therefore searching for signals of atomic nuclei recoiling from collisions with dark matter. To constrain theories of dark matter with existing null results and reliably interpret future experimental discoveries, the interactions between nuclei and possible types of dark matter must be understood and accurately modeled. Nuclear structure effects on these interactions can be calculated from the Standard Model of particle physics using lattice QCD, and our first calculations predict unexpectedly large nuclear structure effects in light nuclei. These exploratory calculations used unphysically heavy particle masses to reduce computational cost, and we are now using Blue Waters HPC resources and GPU optimization techniques including JIT compilation to embark on calculations with physical particle masses in order to definitively calculate the interaction strengths of light nuclei with dark matter.

Field of Science: Physics

View the presentation slides (PDF)

 

Escaping from an Ultracold Inferno: Computational Chemistry at a New Frontier in the Ultracold KRb Dimer Reaction

2019 Graduate Fellow: Micheline Soley, Harvard University

Abstract: The investigation of ultracold chemical reactions requires a combination of experiment, theory, and computation. Computational simulation of many ultracold chemical reactions is currently beyond reach with time-dependent quantum mechanics. The memory requirements scale exponentially with the size of the chemical system. This scaling has been termed the "curse of dimensionality." Blue Waters offers the opportunity to break this curse. Parallelization allows a unique view into how the reaction proceeds by enabling simulation of thousands of trajectories with classical mechanics and diagonalization of large random matrices in time-independent quantum mechanics. By using insight from classical mechanics and R-matrix theory, we are employing Blue Waters to predict and analyze the results of the ultracold KRb dimer reaction. This research offers opportunities for applications from development of quantum computers to investigation of fundamental constants of nature.

Field of Science: Chemistry

Watch the presentation | View the presentation slides (PDF)

 

Communication-Optimal QR Factorization: Performance and Scalability on Varying Architectures

Project PI: Edgar Solomonik, University of Illinois at Urbana-Champaign

Presented by: Edward Hutter, University of Illinois at Urbana-Champaign

Abstract: Parallel numerical linear algebra libraries are a key component of the software stack on HPC systems. We are pursuing development of novel communication avoiding algorithms and libraries that encapsulate them. This talk will focus on advances in parallel QR factorization, one of the most widely used matrix factorizations and a critical component of linear least squares and eigenvalue problems. Our new communication-avoiding CholeskyQR2 algorithm is the first practical algorithm that is communication optimal given any amount of memory and matrix dimensions. The algorithm requires slightly more computation but achieves a much higher arithmetic intensity, making it especially promising on emerging HPC architectures. We demonstrate its benefit via strong and weak scaling studies on Blue Waters and Stampede2, achieving speed-ups of up to 3.3x when using 65,000 processors.

Field of Science: Computer Science

Watch the presentation | View the presentation slides (PDF)

 

Full-Scale Biophysical Modeling of Hippocampal Networks during Spatial Navigation

Project PI: Ivan Soltesz, Stanford University

Presented by: Ivan Raikov, Stanford University

Abstract: The hippocampus provides the basis for episodic memory in the brain, remembering events experienced in the past and linking them with their spatio-temporal context. Our project combines experimental data with computational modeling to elucidate how neurons in the hippocampus generate episodic memory sequences. We have constructed 1:1 scale computational models of two hippocampal subregions with detailed neuronal morphology and biophysics for most neuron types. Input to these models is provided by an idealized representation of the environmental information coded by cortical neurons projecting into the hippocampus, which matches the statistical features of experimental observations. Model parameters have been constrained so that simulations of neural activity during locomotion show highly sparse coding of position information that is specific to the hippocampus. Our simulations provide a complete picture of the known neurons within the hippocampus and allow the tracking of subpopulations representing individual trajectories and sensory cues associated with them.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

Simulation of Viral Infection Propagation through Air Travel

Project PI: Ashok Srinivasan, University of West Florida

Abstract: Air travel has been identified as a leading factor in the spread of infections. We had earlier shown that changes to boarding and deplaning procedures could substantially reduce the risk of transmitting Ebola and SARS on flights. This was accomplished through a fine-scaled model that tracks the movement of passengers in airplanes. Our approach incurs a large computational cost to deal with inherent uncertainties in human behavior, which is addressed through massive parallelization. We will discuss new techniques, such as low dimensional parameter sweep, to reduce the above computational cost. In addition, optimizations using number-theoretic properties of low-discrepancy sequences reduce load imbalance on Blue Waters. These optimizations enable our simulations to produce timely results as needed in emergencies that threaten public health.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

An Examination of Midlatitude Storms and Storm Tracks in the CESM1.3: Resolution Dependence, Coupling Sensitivity, and Projected Future Change

Project PI: Ryan Sriver, University of Illinois at Urbana-Champaign

Presented by: Susan Bates, NCAR

Abstract: High-resolution simulations using the Community Earth System Model, version 1.3 (CESM1.3) were conducted on Blue Waters for the purpose of advancing the study of climate change and its potential impacts on our planet, and more specifically for the purposes of examining climate extremes. This presentation will serve to update the Blue Waters community on the important role this project plays in a much larger effort to provide the climate community with ensembles of climate simulations of varying ocean and atmosphere resolution and complexity. Current research on midlatitude storms and storm tracks in both hemispheres will also be discussed. This research utilizes a 0.25° atmosphere and 0.1° ocean, the highest resolution possible with the CESM. Discussion will include: dependence on model resolution of processes involved in future change of these storms, namely atmospheric stability and steering, and sensitivity to model coupling and modeled atmospheric physics on storm track strength and location.

Field of Science: Atmospheric and Climate Science

Watch the presentation | View the presentation slides (PDF)

 

The Response of Tropical Cyclone Activity to Increasing CO₂ in the Community Earth System Model (CESM)

Project PI: Ryan Sriver, University of Illinois at Urbana-Champaign

Abstract: The project explores the response of global tropical cyclone activity to increased atmospheric carbon dioxide in the coupled Community Earth System Model (CESM), which in previous Blue Waters experiments has been shown capable of simulating realistic tropical cyclone circulations and global cyclone activity metrics (number, intensity distribution, seasonality, etc.). We performed two thirty-year CESM simulations, using the 25 km atmosphere coupled to the nominal 1-degree ocean model. In one simulation, atmospheric carbon dioxide is held fixed at pre-industrial levels, while the other simulation prescribes carbon dioxide levels equal to four times pre-industrial values. Here we outline key differences TC activity between model simulations, and we highlight the effects of enhanced temperature on large scale dynamics and tropical cyclones for different regions. Results shed important light on how tropical cyclones may change in the future under global warming using a state-of-the-art fully-coupled climate model.

Field of Science: Atmospheric and Climate Science

View the presentation slides (PDF)

 

High Resolution Simulation of the Last Glacial Maximum

Project PI: Clay Tabor, University of Connecticut

Abstract: In the coming decades, models predict an increase in climate extremes in Southwest North America (SWNA), with both more severe droughts and more intense precipitation events. However, confidence in model projections of SWNA climate are limited by an incomplete understanding of the complex interactions driving regional climate variability and the unprecedented nature of future climate perturbation. By studying the past, we can improve our understanding of the mechanisms driving current and future hydroclimate change in SWNA. This region has been particularly well surveyed, with some of the densest and most diverse coverage of paleoclimate proxy records available. Nevertheless, comparisons between climate model outputs and proxy records remain difficult because 1) the resolution of the models is often too low to resolve the topography of the sample sites and 2) the models do not directly simulate the quantities measured in the proxy records. Here, we overcome these limitations by employing a 0.25° resolution, water isotope-enabled Earth system model for simulations of the Last Glacial Maximum and preindustrial. These simulations are computationally expensive, requiring the use of the Blue Waters supercomputer.

Field of Science: Atmospheric and Climate Science

Watch the presentation | View the presentation slides (PDF)

 

Towards Self-Learning Agents in Era of High-Throughput Omics

Project PI: Ilias Tagkopoulos, University of California Davis

Presented by: Ameen Eetemadi, University of California Davis

Abstract: The renaissance in artificial intelligence and robotics combined with the rise of high-throughput omics and high-performance computing have put us on the verge of a paradigm shift in biological research. In this new paradigm, omics-based self-learning agents not only act as powerful predictors but also identify next set of experiments to be done by the robot. In this talk, we first present the Genetic Neural Network, an artificial neural network for predicting genome-wide gene expression given gene knockouts and master regulator perturbations. Then we introduce optimal experimental design framework to identify set of experiments that maximize prediction performance given gene expression data. Our methods result in approximately 40% improvement both in terms of prediction accuracy and number of data points necessary to achieve same accuracy compared to common baselines. Validation of our methods was enabled by HPC parallel computing platform using in vitro data sets and in silico simulations.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

Incorporating Proteins into Geometrically Complex, Cell-Scale Membrane Models

Project PI: Emad Tajkhorshid, University of Illinois at Urbana-Champaign

Presented by: Noah Trebesch, University of Illinois at Urbana-Champaign

Abstract: The membranes that surround cells and organelles play a vital role in biological function, acting as semipermeable barriers that separate and protect cells and organelles from their surroundings. Molecular dynamics (MD) simulations provide the means to investigate the fundamental molecular mechanisms that give rise to membranes' biological functions, but it is exceedingly difficult to build MD models that capture cellular membranes' immense scales and complex geometries. To address this challenge, we previously introduced software called xMAS (Experimentally-Derived Membranes of Arbitrary Shape) Builder, and we used it to generate a ~4.5 billion atom (~1.9³) model of a piece of the endoplasmic reticulum with a helicoidal shape called a Terasaki ramp. We also ran a long-term equilibrium MD simulation on a smaller test system to demonstrate that xMAS Builder produces stable, high quality models. Used to develop and test the base capabilities of xMAS Builder, these previous models were somewhat limited in that they were composed entirely of lipids. We now introduce an updated xMAS Builder capable of incorporating proteins into its membrane models.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

Simulations of Tilted Black Hole Accretion

Project PI: Alexander Tchekhovskoy, Northwestern University

Abstract: Black holes are responsible for a wide variety of astrophysical phenomena. They devour stars, eject relativistic jets, affect star formation and galaxy evolution, and enrich the Universe with heavy elements. In many of these situations, the angular momentum axis of the infalling gas is tilted relative to that of the black hole to some degree. Particularly poorly understood is the physics of bright tilted accretion flows, such as those that power active galaxies, or quasars. Such disks are very thin, with height-to-radius ratio, h/r ~ 0.01, which makes simulating them a long-standing computational challenge. The unique combination of Blue Waters GPUs and our recently developed GPU-accelerated general relativistic magnetohydrodynamic code H-AMR allowed us to carry out the simulations of the thinnest disks to date, with h/r ~ 0.015 - 0.03. We show that contrary to the standard expectations, such disks break up into multiple individually precessing sub-disks, upending the standard picture of luminous black hole accretion. We discuss the observational and physical consequences of the disk breaking.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Unraveling Functional Hole Hopping Pathways in the [4Fe4S]-Containing DNA Primase

2018 Graduate Fellow: Darius Teo, Duke University

Abstract: [Fe4S4]-containing proteins participate in charge transfer and related redox signaling relevant to cell function. The Y345C mutation in the DNA-binding domain of primase (which has been identified in gastric tumors) can affect the binding of the RNA/DNA duplex to the C-terminal domain of primase and alter charge hopping pathways between the [Fe4S4] cluster and the duplex. This in turn impacts the signaling dynamics and endogenous repair function of primase. In this study, we first developed a python module that is able to map and rank hopping pathways in proteins according to the mean residence time of the transferring charge in such pathways. We then parameterized the high-potential [Fe4S4] cluster in its 2+ and 3+ redox states for molecular dynamics simulations. The effects of the mutation on the structure, dynamics, hole hopping pathways between the [Fe4S4] cluster and the duplex, and DNA-binding free energy will be described. We will also examine the electrostatic contributions to the binding free energies of the duplex to primase in both redox states. This analysis aims to examine the molecular origins of the experimentally observed strong dependence of the DNA binding affinity on the charge state of the [Fe4S4] cluster.

Field of Science: Chemistry

View the presentation slides (PDF)

 

Multiphase Modeling of Defect Formation Mechanisms in Steel Continuous Casting

Project PI: Brian G. Thomas, University of Illinois at Urbana-Champaign

Presented by: Seong-Mook Cho, Colorado School of Mines

Abstract: Continuous casting is a mature and sophisticated technological process, used to manufacture over 96% of the world's steel. Thus, even small improvements to understanding of defect formation in the process can have huge impact on the steel industry. Due to the harsh environment of the process and many process-variables, computational modeling is an ideal tool to quantify the complex defect-related phenomena. This project aims to develop sophisticated and accurate multiphase models of the process for more realistic simulations and scientific investigations of the formation of various defects. Transport and capture of over 5-million particles and micro-scale steel-solidification structures are simulated together with multiphase turbulent flow in a huge caster-domain (>2m³), using both the multi-GPU in-house code, CUFLOW on Blue Waters XK-node and the commercial CFD-code ANSYS-Fluent with numerous user-defined functions on XE-node. Speed-up breakthrough (>3,000X) on Blue Waters is greatly contributing to conducting very high-resolution simulations of numerous cases simultaneously.

Field of Science: Engineering

View the presentation slides (PDF)

 

The Transport and Dynamics of Wave-Driven Reef Jets under the Influence of Stratification and Rotation

2018 Graduate Fellow: Walter Torres, Duke University

Abstract: The physics of a plume or jet entering a quiescent fluid is a classical problem in fluid dynamics at both small and geophysical scales. In the coastal ocean, jets are particularly ecologically and physically important; they facilitate the exchange of heat, salt, nutrients, and larvae between the nearshore environment and open ocean. Here, we investigate the behavior of a momentum jet driven by the interaction of surface gravity waves with coral reef topography using a 3D coupled wave-circulation numerical model of an idealized island domain. Simulations conducted over a range of background density stratification and rotational (Coriolis force) conditions are used to characterize possible flow regimes and dynamical balances. The results highlight the role of stratification and rotation in significantly modifying the spatial structure of reef jets, as well as identifying the leading-order importance of wave-current interaction for jet dynamics in the near-surface.

Field of Science: Fluid Systems

View the presentation slides (PDF)

 

OpenMP Parallelization of the Complex Magnetohydrodynamic Model BATS-R-US

Project PI: Gabor Toth, University of Michigan

Abstract: The BATS-R-US magnetohydrodynamic (MHD) code can solve various forms of the MHD equations on block-adaptive grids with a variety of spatial and temporal discretizations. The code consists of over 200,000 lines of Fortran 90+ code. Scaling to very large problems has been limited to about 16,000 to 32,000 cores on Blue Waters due to memory constraints: the global block tree information is stored on every MPI process, and the tree becomes larger as the number of grid blocks is increased. Using OpenMP can mitigate this issue, and allow scaling to 500,000 cores. Our strategy of OpenMP parallelization exploits the grid blocks: the loops over the blocks are multi-threaded instead of individual grid cells. With relatively modest modifications of the source code, a lot of testing and debugging, all done by graduate student Hongyang Zhou, we achieved our goal. I will discuss issues, solutions, and show some extremely promising results.

Field of Science: Physics

View the presentation slides (PDF)

 

Implementation and Use of a Global Nonhydrostatic Model (MPAS) for Extended Prediction

Project PI: Robert J. Trapp, University of Illinois at Urbana-Champaign

Abstract: I will discuss our implementation and then recent use of the Model for the Prediction Across Scales (MPAS). MPAS is a new, non-hydrostatic weather and climate model that allows for local grid refinement. Because MPAS is also a global model, it is well suited for extended range predictions, as will be illustrated with the MPAS predictions made during the RELAMPAGO field campaign. A configuration detail of particular relevance during RELAMPAGO was the specification of 3-km gridpoint spacing over the entirety of South America, with 15-km gridpoint spacing elsewhere around the globe. The 3-km spacing is considered to be "convection allowing," and thus we were effectively able to resolve thunderstorms over large domains.

Field of Science: Atmospheric and Climate Science

View the presentation slides (PDF)

 

A CyberShake Probabilistic Seismic Hazard Model for Northern California

Project PI: John Vidale, University of Southern California

Presented by: Philip Maechling, University of Southern California

Abstract: The Southern California Earthquake Center (SCEC) used Blue Waters to perform physics-based probabilistic seismic hazard analysis (PSHA) using the CyberShake method which executes a complex suite of seismological software and scientific workflow middleware. Physics-based PSHA results are used to improve hazard estimates, which are used in building design, urban planning, community earthquake awareness, and disaster preparation. During the last year, we completed CyberShake Study 18.8, the first 3D physics-based PSHA model for the San Francisco Bay region. This study used over 3.7 million Blue Waters node hours to calculate hazard results for 869 locations in California, and produced hundreds of millions of seismograms for hundreds of thousands of individual earthquakes. These data products enable us to investigate the effect of basin structures and rupture directivity on hazard, improve upon standard attenuation-based methods of calculating seismic hazard, and identify research targets to further improve PSHA estimate accuracy.

Field of Science: Earth Sciences

Watch the presentation | View the presentation slides (PDF)

 

Coarse-Grained Simulations of TRIM5 Restriction of HIV Capsids

Project PI: Gregory Voth, University of Chicago

Presented by: Alvin Yu, University of Chicago

Abstract: The human immunodeficiency virus (HIV) uses enclosing capsid (CA) proteins to package and release its genetic information into newly infected host cells. CA forms fullerene cones that compartmentalize viral RNA. TRIM5 is an important antiretroviral restriction factor found in rhesus macaques that blocks HIV infection. Recent cryo-electron tomography analysis demonstrate cytoplasmic TRIM5 self-assembles into hexagonally patterned nets that encage mature capsids, with a symmetry and spacing that matches the underlying CA lattice. In this work, we use coarse-grained and atomistic simulations to probe the structural and thermodynamic properties TRIM5 assembly and the molecular mechanisms of how TRIM5 recognizes and restricts viral capsid.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

Charge-Spin Coupling as a Probe of Correlated Materials

Project PI: Lucas K. Wagner, University of Illinois at Urbana-Champaign

Presented by: Joao N. B. Rodrigues, University of Illinois at Urbana-Champaign

Abstract: The study of the quantum properties of materials is all about understanding how collective behavior arises in systems with a large number of interacting particles (electrons and nuclei). In some materials these interactions give rise to unique correlated quantum states which are behind fascinating macroscopic phenomena such as magnetism or superconductivity. We will discuss a novel efficient way of probing correlated materials which relies on quantifying how electrons in a material respond to each other. In particular, we will see how known high-temperature unconventional superconductors such as the copper oxides, iron-pnictides and iron-chalcogenides have distinctive electronic correlations that set them apart from most other materials. Such observation allows for the construction of a data-based prediction model that might help in the search for new high-temperature unconventional superconductors.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

 

Scaling Methods for Phylogeny Estimation to Large-Datasets using Divide-and-Conquer

Project PI: Tandy Warnow, University of Illinois at Urbana-Champaign

Abstract: Over the last years, the availability of genomic sequence data from thousands of different species has led to hopes that a phylogenetic (i.e., evolutionary) tree of all life might be achievable. Yet, the most accurate methods for estimating phylogenies are heuristics for NP-hard optimization problems, many of which are too computationally intensive to use on large datasets. In research supported by Blue Waters, we have developed new divide-and-conquer approaches for scaling the leading species tree estimation methods to large datasets. In this talk, I will describe these strategies and their theoretical properties, present open problems, and discuss opportunities for impact in large-scale phylogenetic estimation using these and similar approaches.

Field of Science: Computer Science

Watch the presentation | View the presentation slides (PDF)

 

Simulation of Bluff-Body-Stabilized Flames using PeleC, a Combustion Code for Exascale Computing

2018 Graduate Fellow: Samuel H.R. Whitman, University of Colorado Boulder

Abstract: Flame stabilization is important for maintaining steady combustion in gas turbines. However, computational combustion models do not capture the dynamic complexity of stabilized flames with sufficient generality, limiting attempts to improve turbine performance. To understand this complexity and provide validation data, we adapt PeleC, an exascale compressible combustion code, to Blue Waters to develop direct numerical simulations of stabilized reacting and non-reacting flows surrounding a triangular bluff body, currently under experimental investigation. Adaptive mesh refinement (AMR) is incorporated via the AMReX block-structured framework to locally resolve the physics of interest at reduced cost compared to static mesh approaches. We show results of a convergence study for the non-reacting case and preliminary findings from the reacting case, in which heat release interacts with shear-layer instabilities and vortex formation in the wake. AMR is necessary to accurately resolve the interactions among shear-layer, vorticity dynamics, reaction-rate chemistry, and flame formation at a computationally manageable cost.

Field of Science: Engineering

View the presentation slides (PDF)

 

Attacking the Shell Merger Problem in Massive Stars with PPMstar

Project PI: Paul R. Woodward, University of Minnesota

Abstract: In collaboration with Falk Herwig's team at the University of Victoria, in Canada, we have used Blue Waters for the last six years to simulate in 3D brief events in the deep interiors of stars in which new or additional fuel is ingested into a convection zone above a nuclear burning shell, giving rise to violent and aspherical nuclear combustion and associated nucleosynthesis. We have proposed to exploit the increased power of NSF's follow-on to the Blue Waters system to expand our investigation to the study of interactions between multiple convection zones and their nuclear burning shells. A glimpse of our early test runs in gearing up for this expanded study will be shown and the key new numerical techniques that we will employ will be discussed. Most important among these new techniques is a hybrid advection scheme for concentrations of components of the gas in which composite fluids are advected with the full-blown PPB moment-conserving technique, while the various constituents of each such composite are advected with a less elaborate and less accurate PPM technique. One additional quantity is advected for each composite fluid as a new means of maintaining high accuracy while exclusively employing 32-bit precision arithmetic.

Field of Science: Astronomy & Astrophysics

 

The Evolving Air Quality Under the Changing Climate: Enhanced Understanding through Blue Waters

Project PI: Donald Wuebbles, University of Illinois at Urbana-Champaign

Abstract: Past studies have shown that projected climate change could affect air quality, but little is known about resulting effects on health—this study examines exceedances for upper limits of exposure set by environmental policy. We use a fully coupled model of the Earth's climate system with interactive atmospheric chemistry; Blue Waters is one of the few computers that can be used for such a complex study. Exceedances for surface ozone and particulate matter (PM) concentrations are examined for two different climate projections. We especially focus the United States, India and China. Along with an overall increase in ozone exceedances, the study also projects a significant shift in seasonality of the events. PM showed a significant increasing trend in all the regions in the future with an overall increase of number of exceedance events annually. We also show that a switch to clean energy by mid-century would greatly improve air quality.

Field of Science: Atmospheric and Climate Science

Watch the presentation | View the presentation slides (PDF)

 

High-fidelity Numerical Simulations of Collapsing Cavitation Bubbles near Solid and Elastically Deformable Objects

Project PIs: Zhen Xu, University of Michigan; and Eric Johnsen, University of Michigan

Presented by: Mauro Rodriguez, University of Michigan

Abstract: The violent collapse of cavitation bubbles can damage its surroundings and causing cavitation erosion on material surfaces. To understand this phenomenon, we have developed a novel numerical model to conduct efficient, high-fidelity simulations to investigate the collapse of single and multiple bubbles near (i) semi-infinite rigid and (ii) finite-size elastic objects. The numerical model solves the three-dimensional compressible Navier-Stokes equations for a multi-component system and includes visco(elasticity) to investigate the interactions between an object and the bubble collapse fluid flow. The Blue Waters system's computational power and uniqueness was essential to conduct the high-resolution simulations to quantify the non-spherical bubble morphology, maximum pressure/temperature fields and elastic components of stress therein produced. As a result, these fundamental simulations have furthered our ability to quantify the damage mechanisms and lead to strategies to better control cavitation-induced erosion in a wide-range of naval and medical engineering and energy science applications.

Field of Science: Fluid Systems

 

Computing Petascale Turbulence on Blue Waters: Advances Achieved and Lessons Learned

Project PI: P. K. Yeung, Georgia Institute of Technology

Abstract: Turbulence in fluid flow is extremely complex, of vital importance in many problems of nature and engineering, and—even in its simplest forms—a grand challenge problem for high performance computing. Precious resources on Blue Waters made available through two successive PRAC grants have enabled simulations at state-of-the-art grid resolution in the order of 1 trillion grid points. The computational approach is primarily based on Fourier pseudo-spectral methods, which are of high accuracy but communication intensive, especially at large problem sizes where massive parallelism is essential. Outstanding support from the Blue Waters staff have also allowed us to perform simulations efficiently up to 8,192 Cray XE nodes on Blue Waters. These simulations have led to significant advances in fundamental physical understanding, of the fluid flow (velocity field), of turbulent mixing especially of substances of low molecular diffusivity, and the dispersion of collections of infinitesimal fluid elements both forwards and backwards in time. In this talk I plan to review the lessons we learned through code development and optimization on Blue Waters, and to discuss some of the new insights that we have obtained through the analysis of large datasets produced from this work. Specific topics will include the dynamics of four-particle clusters known as tetrads, and the statistics of locally averaged energy dissipation rates as well as circulation around closed contours of linear size in the so-called inertial range of high Reynolds number turbulence.

Field of Science: Fluid Systems

View the presentation slides (PDF)

 

Sensitivity of Arctic Sea Ice Simulations to Treatment of Sea Ice Dynamics

Project PIs: Xiangdong Zhang, University of Alaska Fairbanks

Abstract: Arctic sea ice is an important component of the global climate system, which is highly sensitive to any internal and external forcings associated with and feeds back to amplify global climate changes. During recent decades, significant shrinking of Arctic sea ice extent and thinning of sea ice thickness have occurred. However, climate models have shown large discrepancies in simulating Arctic sea ice properties, which therefore hinders understanding and prediction of sea ice changes. In this study, we employed the sea ice-ocean component model of the Community Earth System Model (CESM 1.2) with the Elastic-Viscous-Plastic (EVP) rheology, forced by a repeating climatological forcing constructed using the ERA-Interim reanalysis dataset. We conducted a series of model sensitivity experiments and found that sea ice thickness distribution and associated sea ice motion and extent are highly sensitive to nonlinear interplays between intensity of wind stress and treatments of sea ice internal dynamics

Field of Science: Atmospheric and Climate Science

View the presentation slides (PDF)

Blue Waters is supported by the National Science Foundation (awards OCI-0725070 and ACI-1238993) the State of Illinois, and the National Geospatial-Intelligence Agency.

Contact Blue Waters Team with questions regarding this page.
Copyright 2024 Board of Trustees of the University of Illinois. All rights reserved. Web privacy notice.