Nek5000 hackathon at UIUC
The inaugural Nek5000 Hackathon will be held at NCSA Building, University of Illinois, Urbana-Champaign (UIUC), IL on Nov 12-14, 2017.
The event will bring together users and developers with the intent to help new and current users bring their applications into production mode. It will also give users a chance to provide the development team with feedback on what features or enhancements they would like to see in future releases.
At the event, each of the participants or team will be asked to develop a small example demonstrating the phenomena of interest that could be shared with the rest of the user community through the NekExamples Suite. There will be several rounds of short presentations describing the problems of interest, the features that make the problem interesting and challenging, and the anticipated ways in which the Nek team can help.
For more details, see the Nek5000 hackathon website.
CEED organizing minisymposium at ICOSAHOM 2018
CEED is organizing a minisymposium, Efficient High-Order Finite Element Discretizations at Large Scale, at the International Conference on Spectral and High-Order Methods (ICOSAHOM 2018) in London UK, Jul 9-13, 2018.
The goal of the minisymposium is to discuss the next-generation high-order discretization algorithm and software, based on finite/spectral element approaches that will enable a wide range of important scientific applications to run efficiently on future architecture.
Best Paper Award at NURETH-17
CEED researchers (P. Fischer, E. Merzari, A. Obabko) won a Best Paper Award at the 17th International Topical Meeting on Nuclear Reactor Thermal Hydraulics (NURETH-17), held in China in September 2017, with a paper entitled High-Fidelity Simulation of Flow Induced Vibrations in Helical Steam Generators for Small Modular Reactors.
CEED attending Cray, AMD and Intel Deep-Dives
- Cray deep-dive in Bloomington, MN on Oct 18-19
- AMD deep-dive in Austin, TX on Oct 24-25
- Intel deep-dive in Hudson, MA on Oct 21-Nov 2
Topics of discussion include advanced technology and memory design, strong scaling considerations and the porting and evaluation of CEED's bake-off problems and miniapps (Nekbone, Laghos, NekCEM CEEDling).
New Nekbench repository
The user provides ranges for important parameters ranges (e.g., processor counts and local problem size ranges) and a test type (e.g., scaling or ping-pong test). Nekbench will run the given test in the given parameter space using a Nek5000 case file which is also given by the user (in the ping-pong tests, the case file is optional).
Nekbench is written using bash scripting language and runs any Unix-like operating system that supports bash. It has been successfully tested on Linux laptops/desktops, ALCF Theta, NERSC Cori (KNL and Haswell), and NERSC Edison machines for scaling tests.
Planned extensions for Nekbench include adding more machine types like ANL's Cetus, additional support for the ping-pong test type, and automated plot generation (e.g., scaling study graphs) for each test run.
GPU ports of Nek and Laghos
GPU acceleration is a main focus of the performance optimization efforts in CEED. Recent progress in this direction include GPU ports of CEED's Nek5000 application and the Nekbone and Laghos miniapps.
For Nek5000, an initial GPU-enabled version has been developed based on OpenACC. For Nekbone a pure OpenACC implementation as well as a hybrid OpenACC/CUDA implementation with a CUDA kernel for matrix-vector multiplication has been developed.
For Laghos, a GPU-enabled version of has been released using the OCCA interface. With this approach, the user is able to run Laghos distributively using varying device types per MPI process, whether serial C++, OpenMP, or CUDA.
First CEED annual meeting held at LLNL
CEED held its first annual meeting in August, 2017 at the HPC Innovation Center of Lawrence Livermore National Laboratory.
The goal of the meeting was to report on the progress in the center, deepen existing and establish new connections with ECP hardware vendors, ECP software technologies projects and other collaborators, plan project activities and brainstorm/work as a group to make technical progress.
In addition to gathering together many of the CEED researchers, the meeting included representatives of the ECP management, hardware vendors, software technology and other interested projects.
CEED researchers at ATPESC17
Six CEED researchers presented at the 2017 edition of the Argonne Training Program on Extreme-Scale Computing, now part of the Exascale Computing Project.
The CEED presentations covered a wide variety of topics, from overview of Theta, to GPU programming, dense and sparse linear algebra, and high-order discretizations on unstructured meshes.
Videos of all 2017 talks are available on YouTube. CEED researchers have also participated in past editions of the meeting.
CEED BPs and benchmarks repository released
CEED released an initial set of bake-off (BP) problems, which are simple kernels designed to test and compare the performance of high-order codes, both internally in CEED, as well as in the broader high-order community.
In addition to the benchmark descriptions on the CEED BPs page, a benchmarks repository is publicly available with several implementations of the CEED bake-off problems. Currently, MFEM, Nek5000 and deal.ii are included, see directories tests/mfem_bps, tests/nek5000_bps and tests/dealii_bps respectively.
New Laghos and NekCEM CEEDling miniapps released
Two new miniapps developed in CEED were released in June 2017: Laghos and NekCEM CEEDling.
Laghos (LAGrangian High-Order Solver) is a new miniapp developed in CEED that solves the time-dependent Euler equations of compressible gas dynamics in a moving Lagrangian frame using unstructured high-order finite element spatial discretization and explicit high-order time-stepping. In CEED, Laghos serves as a proxy for a sub-component of the MARBL/LLNLApp application.
NekCEM CEEDling is a new NekCEM miniapp, solving the time-domain Maxwell equation for electromagnetic systems.
Paper with MPICH at SC17
Joint paper with the MPICH group, Why is MPI so Slow? Analyzing the fundamental limits in implementing MPI-3.1 accepted in Supercomputing 2017. The paper provides an in-depth analysis of the software overheads in the MPI performance-critical path and exposes mandatory performance overheads that are unavoidable based on the MPI-3.1 specification.
STRUMPACK support in MFEM
Support for the sparse direct solver and preconditioner STRUMPACK has been integrated in MFEM.
STRUMPACK is being developed at LBNL and is part of the ECP project Factorization Based Sparse Solvers and Preconditioners (Xiaoye Sherry Li and Pieter Ghysels). The STRUMPACK solver is based on multifrontal sparse Gaussian elimination and uses hierarchically semi-separable matrices to compress fill-in. It can be used as an exact direct solver or as an algebraic, robust and parallel preconditioner for a range of discretized PDE problems.
2017 PETSc User Meeting
Over 75 participants from all over the world attended the PETSc User Meeting, held June 14-16 in Boulder, CO. Hosted by the University of Colorado Boulder, the event consisted of a one-day tutorial on the solver library PETSc and showcased the latest research enabled by the functionality available in PETSc. The meeting agenda covered a total of 15 talks, four posters, and two panels.
Thanks to generous support from Intel and Tech-X, 22 students received travel grants and got to learn about the latest techniques on the large-scale numerical solution of partial differential equations.
PETSc is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations. It has become one of the most widely used numerical software packages of its kind and has users in application areas ranging from acoustics and arterial flow to seismology and semiconductors.
GPU Hackathon at BNL
Nek/CEED team participated the GPU Hackathon 2017 that was held in Brookhaven National Laboratory on June 5-9, 2017. Our team focused on performing and tuning GPU-enabled Nek5000/Nekbone/NekCEM version on large-scale GPU systems for small modular reactor, thermal fluids, and meta-materials modeling.
Workshop on Batched, Reproducible, and Reduced Precision BLAS
The second Workshop on Batched, Reproducible, and Reduced Precision BLAS was held in Atlanta, GA on February 23-25, 2017 including many members of the CEED MAGMA team.
The goal of this workshop was to touch on extending the Basic Linear Algebra Software Library (BLAS). The existing BLAS have proven to be very effective in assisting portable, efficient software for sequential and some of the current class of high-performance computers. New computational needs in many applications have motivated the need to investigate the possibility of extending the currently accepted standards to provide greater parallelism for small size operations, reproducibility, and reduced precision support.
Of particular interest to CEED is the use of batched BLAS for finite element tensor contractions, and thus our team is interested in the establishment of a batched BLAS standard, highly-optimized implementations, and support from vendors on various architectures.
This is the second workshop of an open forum to discuss and formalize details related to batched, reproducible, and reduced precision BLAS. The agenda and the talks from the first workshop can be found here.
Software release: MFEM v3.3
Version 3.3 of MFEM, a lightweight, general, scalable C++ library for finite element methods and a main partner in CEED, was released on January 28, 2017 at http://mfem.org
The goal of MFEM is to enable high-performance scalable finite element discretization research and application development on a wide variety of platforms, ranging from laptops to exascale supercomputers.
It has many features, including:
- 2D and 3D, arbitrary order H1, H(curl), H(div), L2, NURBS elements.
- Parallel version scalable to hundreds of thousands of MPI cores.
- Conforming/nonconforming adaptive mesh refinement (AMR), including anisotropic refinement, derefinement and parallel load balancing.
- Galerkin, mixed, isogeometric, discontinuous Galerkin, hybridized, and DPG discretizations.
- Support for triangular, quadrilateral, tetrahedral and hexahedral elements, including arbitrary order curvilinear meshes.
- Scalable algebraic multigrid, time integrators, and eigensolvers.
- Lightweight interactive OpenGL visualization with the MFEM-based GLVis tool.
Some of the new additions in version 3.3 are:
- Comprehensive support for the linear and nonlinear solvers, preconditioners, time integrators and other features from the PETSc and SUNDIALS suites.
- Linear system interface for action-only linear operators including support for matrix-free preconditioning and low-order-refined spaces.
- General quadrature and nodal finite element basis types.
- Scalable parallel mesh format.
- Thirty six new integrators for common families of operators.
- Sixteen new serial and parallel example codes.
- Support for CMake, on-the-fly compression of file streams, and HDF5-based output following the Conduit mesh blueprint specification.
CEED co-design center announced
The Exascale Computing Project (ECP) announced on November 11, 2016 its selection of four co-design centers, including CEED: the Center for Efficient Exascale Discretizations, which is a research partnership between Lawrence Livermore National Laboratory; Argonne National Laboratory; the University of Illinois Urbana-Champaign; Virginia Tech; University of Tennessee, Knoxville; Colorado University, Boulder; and the Rensselaer Polytechnic Institute (RPI).
R&D 100 Award for NekCEM / Nek5000
NekCEM/Nek5000: Scalable High-Order Simulation Codes received a 2016 R&D 100 Award, given by R&D Magazine to 100 top new technologies for the year.
The R&D 100 citation reads: "NekCEM/Nek5000: Release 4.0: Scalable High-Order Simulation Codes is an open-source simulation-software package that delivers highly accurate solutions for a wide range of scientific applications including electromagnetics, quantum optics, fluid flow, thermal convection, combustion and magnetohydrodynamics. It features state-of-the-art, scalable, high-order algorithms that are fast and efficient on platforms ranging from laptops to the world’s fastest computers. The size of the physical phenomena that can be simulated with this package ranges from quantum dots for nanoscale devices to accretion disks surrounding black holes. NekCEM provides simulation capabilities for the analysis of electromagnetic and quantum optical devices, such as particle accelerators and solar cells. Nek5000 provides turbulent flow simulation capabilities for a variety of thermal-fluid problems including nuclear reactors, internal combustion engines, vascular flows, and ocean currents."
See the ANL press release for more information.