Consisting of 48 racks, 786,432 processors, and 768 terabytes of memory, Mira is 20 times faster than Intrepid, its IBM Blue Gene/P predecessor at the ALCF. Mira’s 49,152 compute nodes have a PowerPC A2 1600 MHz processor containing 16 cores, each with 4 hardware threads, running at 1.6 GHz, and 16 gigabytes of DDR3 memory. A 17^th core is available for the communication library.

IBM’s 5D torus interconnect configuration—with 2GB/s chip-to-chip links—connects the nodes, enabling highly efficient computation by reducing the average number of hops and latency between compute nodes. The Blue Gene/Q system also features a quad floating point unit (FPU) that can be used to execute scalar floating point instructions, four-wide SIMD instructions, or two-wide complex arithmetic SIMD instructions. This quad FPU provides higher single thread performance for some applications.

In addition to being one of the fastest computers in the world, Mira is also among the most energy-efficient. The supercomputer saves considerable energy through innovative chip designs and a unique water-cooling system.

Enabled by Mira’s massively parallel architecture, advanced algorithms and accurate intermolecular potentials, the team’s calculations expedited hundreds of thousands of virtual experiments for highly complex systems.

Using Mira, we are able to use our computer simulations to compress decades of research in the lab into a total of about a day’s worth of computing.

—Ilja Siepmann, a University of Minnesota chemistry professor and director of the DOE-funded Nanoporous Materials Genome Center

The researchers gained access to Mira through the ALCF’s Director’s Discretionary program. ALCF assistant computational scientist Chris Knight worked closely with the team to ensure optimal performance on the ALCF supercomputer. A co-author on the paper on the work published in Nature Communications article, Knight assisted in porting their MCCCS-MN (Monte Carlo for Complex Chemical Systems‒Minnesota) code to Mira, guided the developers in adding OpenMP support to permit hybrid MPI/OpenMP parallelism, and helped design an MPI-based framework to allow high-throughput calculations capable of using all of Mira’s 786,432 cores.

• MCCCS‒MN allows the simulation of multicomponent chain systems. It uses the configurational-bias Monte Carlo method efficiently to sample phase space for linear, branched and cyclic chain molecules; the adiabatic nuclear and electronic sampling Monte Carlo method to treat many-body polarization effects; and the aggregation-volume-bias Monte Carlo algorithm to efficiently sample the spatial distribution of associating molecules. MCCCS‒MN is under constant development.

• OpenMP is an Application Program Interface (API) jointly defined by a group of major computer hardware and software vendors. OpenMP provides a portable, scalable model for developers of shared memory parallel applications.

• The Message Passing Interface (MPI) is a message-passing library standard based on the consensus of the MPI Forum, which has more than 40 participating organizations, including vendors, researchers, software library developers, and users. The goal of the Message Passing Interface is to establish a portable, efficient, and flexible standard for message passing that will be widely used for writing message passing programs. As such, MPI is the first standardized, vendor-independent, message-passing library.

The code performance enhancements allowed Siepmann and his team to carry out simulations for the two zeolite applications of industrial relevance. For the purification of ethanol from fermentation broths, the simulations pointed them to a zeolite with a pore/channel system that effectively accommodates ethanol molecules while discouraging hydrogen bonding with water molecules.

Snapshots of representative hydrocarbon configurations inside zeolite frameworks and performance scores predicted for adsorption occurring from a liquid phase containing an equimolar hydrocarbon mixture. Views facing the main channel axis (a–c) and along the main channel axis (d–f) are shown for ATO (a,d), MTW (b,e) and PCOD-8113534 (c,f). Zeolite frameworks are depicted as grey lines, and C18, C24, C30, (2C17 and 4C17) and 22C16 molecules as cyan, purple, blue, red and green spheres, respectively. (g) Performance scores for two hypothetical PCOD zeolites and ten zeolite frameworks from the database of the International Zeolite Association (materials discovered in this project and those previously known for hydroisomerization processes are shown in blue and yellow, respectively). Performance scores are a heuristic measure of the zeolites’ effectiveness based on chemical properties such as adsorption. Image credit: P. Bai and J.I. Siepmann, University of Minnesota. Click to enlarge.

With the ability to purify ethanol in a single separation step, this material displays the potential to replace an energy-intensive, multi-step distillation process currently used by industry. To validate the simulation results, University of Minnesota researchers synthesized and tested the promising zeolite, providing experimental data that was in very good agreement with the predictions.

For the second study, the team investigated potential zeolite catalysts for a de-waxing process called hydroisomerization, in which linear long-chain alkanes are transformed into slightly branched alkanes to reduce the pour point and increase the viscosity of lubricant oils and other fuel products. Their simulations identified zeolites with up to 100 times better adsorption capability than current technology used for this process.

In both cases, the simulations led us to the identification of zeolites that could possibly improve the efficiency of operations for industry. Our findings demonstrate how advanced computational screening methods can be used to increase the speed of materials discovery.

—Ilja Siepmann

The vast amount of data generated for this effort will be made publicly available through the Nanoporous Materials Explorer, an application that is part of the DOE-funded Materials Project.

This research received financial support through a Predictive Theory and Modeling award from DOE’s Office of Science, which also provided support for the computing time at ALCF.