Nek5000 hackathon at UIUC

The inaugural Nek5000 Hackathon will be held at NCSA Building, University of Illinois, Urbana-Champaign (UIUC), IL on Nov 12-14, 2017.

The event will bring together users and developers with the intent to help new and current users bring their applications into production mode. It will also give users a chance to provide the development team with feedback on what features or enhancements they would like to see in future releases.

At the event, each of the participants or team will be asked to develop a small example demonstrating the phenomena of interest that could be shared with the rest of the user community through the NekExamples Suite. There will be several rounds of short presentations describing the problems of interest, the features that make the problem interesting and challenging, and the anticipated ways in which the Nek team can help.

For more details, see the Nek5000 hackathon website.

CEED organizing minisymposium at ICOSAHOM 2018

CEED is organizing a minisymposium, Efficient High-Order Finite Element Discretizations at Large Scale, at the International Conference on Spectral and High-Order Methods (ICOSAHOM 2018) in London UK, Jul 9-13, 2018.

The goal of the minisymposium is to discuss the next-generation high-order discretization algorithm and software, based on finite/spectral element approaches that will enable a wide range of important scientific applications to run efficiently on future architecture.

Best Paper Award at NURETH-17

CEED researchers (P. Fischer, E. Merzari, A. Obabko) won a Best Paper Award at the 17th International Topical Meeting on Nuclear Reactor Thermal Hydraulics (NURETH-17), held in China in September 2017, with a paper entitled High-Fidelity Simulation of Flow Induced Vibrations in Helical Steam Generators for Small Modular Reactors.

CEED attending Cray, AMD and Intel Deep-Dives

CEED researchers and representatives of the Nek and MFEM teams will attend the October 2017, ECP vendor deep-dive meetings:

Topics of discussion include advanced technology and memory design, strong scaling considerations and the porting and evaluation of CEED's bake-off problems and miniapps (Nekbone, Laghos, NekCEM CEEDling).

New Nekbench repository

New Nekbench repository has been released to provide scripts that simplify the benchmarking of Nek5000.

The user provides ranges for important parameters ranges (e.g., processor counts and local problem size ranges) and a test type (e.g., scaling or ping-pong test). Nekbench will run the given test in the given parameter space using a Nek5000 case file which is also given by the user (in the ping-pong tests, the case file is optional).

Nekbench is written using bash scripting language and runs any Unix-like operating system that supports bash. It has been successfully tested on Linux laptops/desktops, ALCF Theta, NERSC Cori (KNL and Haswell), and NERSC Edison machines for scaling tests.

Planned extensions for Nekbench include adding more machine types like ANL's Cetus, additional support for the ping-pong test type, and automated plot generation (e.g., scaling study graphs) for each test run.

GPU ports of Nek and Laghos

GPU acceleration is a main focus of the performance optimization efforts in CEED. Recent progress in this direction include GPU ports of CEED's Nek5000 application and the Nekbone and Laghos miniapps.

For Nek5000, an initial GPU-enabled version has been developed based on OpenACC. For Nekbone a pure OpenACC implementation as well as a hybrid OpenACC/CUDA implementation with a CUDA kernel for matrix-vector multiplication has been developed.

For Laghos, a GPU-enabled version of has been released using the OCCA interface. With this approach, the user is able to run Laghos distributively using varying device types per MPI process, whether serial C++, OpenMP, or CUDA.

First CEED annual meeting held at LLNL

CEED1AM picture

CEED held its first annual meeting in August, 2017 at the HPC Innovation Center of Lawrence Livermore National Laboratory.

The goal of the meeting was to report on the progress in the center, deepen existing and establish new connections with ECP hardware vendors, ECP software technologies projects and other collaborators, plan project activities and brainstorm/work as a group to make technical progress.

In addition to gathering together many of the CEED researchers, the meeting included representatives of the ECP management, hardware vendors, software technology and other interested projects.

CEED researchers at ATPESC17

Six CEED researchers presented at the 2017 edition of the Argonne Training Program on Extreme-Scale Computing, now part of the Exascale Computing Project.

The CEED presentations covered a wide variety of topics, from overview of Theta, to GPU programming, dense and sparse linear algebra, and high-order discretizations on unstructured meshes.

YouTube YouTube YouTube YouTube YouTube YouTube

Videos of all 2017 talks are available on YouTube. CEED researchers have also participated in past editions of the meeting.

CEED BPs and benchmarks repository released

CEED released an initial set of bake-off (BP) problems, which are simple kernels designed to test and compare the performance of high-order codes, both internally in CEED, as well as in the broader high-order community.

In addition to the benchmark descriptions on the CEED BPs page, a benchmarks repository is publicly available with several implementations of the CEED bake-off problems. Currently, MFEM, Nek5000 and deal.ii are included, see directories tests/mfem_bps, tests/nek5000_bps and tests/dealii_bps respectively.

New Laghos and NekCEM CEEDling miniapps released

Two new miniapps developed in CEED were released in June 2017: Laghos and NekCEM CEEDling.

Laghos (LAGrangian High-Order Solver) is a new miniapp developed in CEED that solves the time-dependent Euler equations of compressible gas dynamics in a moving Lagrangian frame using unstructured high-order finite element spatial discretization and explicit high-order time-stepping. In CEED, Laghos serves as a proxy for a sub-component of the MARBL/LLNLApp application.

NekCEM CEEDling is a new NekCEM miniapp, solving the time-domain Maxwell equation for electromagnetic systems.

For more details, see the CEED miniapps page and the Laghos and NekCEM CEEDling repositories on GitHub.

Paper with MPICH at SC17

Joint paper with the MPICH group, Why is MPI so Slow? Analyzing the fundamental limits in implementing MPI-3.1 accepted in Supercomputing 2017. The paper provides an in-depth analysis of the software overheads in the MPI performance-critical path and exposes mandatory performance overheads that are unavoidable based on the MPI-3.1 specification.


Support for the sparse direct solver and preconditioner STRUMPACK has been integrated in MFEM.

STRUMPACK is being developed at LBNL and is part of the ECP project Factorization Based Sparse Solvers and Preconditioners (Xiaoye Sherry Li and Pieter Ghysels). The STRUMPACK solver is based on multifrontal sparse Gaussian elimination and uses hierarchically semi-separable matrices to compress fill-in. It can be used as an exact direct solver or as an algebraic, robust and parallel preconditioner for a range of discretized PDE problems.

2017 PETSc User Meeting

Over 75 participants from all over the world attended the PETSc User Meeting, held June 14-16 in Boulder, CO. Hosted by the University of Colorado Boulder, the event consisted of a one-day tutorial on the solver library PETSc and showcased the latest research enabled by the functionality available in PETSc. The meeting agenda covered a total of 15 talks, four posters, and two panels.

Thanks to generous support from Intel and Tech-X, 22 students received travel grants and got to learn about the latest techniques on the large-scale numerical solution of partial differential equations.

PETSc is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations. It has become one of the most widely used numerical software packages of its kind and has users in application areas ranging from acoustics and arterial flow to seismology and semiconductors.

GPU Hackathon at BNL

Nek/CEED team participated the GPU Hackathon 2017 that was held in Brookhaven National Laboratory on June 5-9, 2017. Our team focused on performing and tuning GPU-enabled Nek5000/Nekbone/NekCEM version on large-scale GPU systems for small modular reactor, thermal fluids, and meta-materials modeling.

Workshop on Batched, Reproducible, and Reduced Precision BLAS

The second Workshop on Batched, Reproducible, and Reduced Precision BLAS was held in Atlanta, GA on February 23-25, 2017 including many members of the CEED MAGMA team.

The goal of this workshop was to touch on extending the Basic Linear Algebra Software Library (BLAS). The existing BLAS have proven to be very effective in assisting portable, efficient software for sequential and some of the current class of high-performance computers. New computational needs in many applications have motivated the need to investigate the possibility of extending the currently accepted standards to provide greater parallelism for small size operations, reproducibility, and reduced precision support.

Of particular interest to CEED is the use of batched BLAS for finite element tensor contractions, and thus our team is interested in the establishment of a batched BLAS standard, highly-optimized implementations, and support from vendors on various architectures.

This is the second workshop of an open forum to discuss and formalize details related to batched, reproducible, and reduced precision BLAS. The agenda and the talks from the first workshop can be found here.

Software release: MFEM v3.3

Version 3.3 of MFEM, a lightweight, general, scalable C++ library for finite element methods and a main partner in CEED, was released on January 28, 2017 at

The goal of MFEM is to enable high-performance scalable finite element discretization research and application development on a wide variety of platforms, ranging from laptops to exascale supercomputers.

It has many features, including:

Some of the new additions in version 3.3 are:

MFEM is being developed in CASC, LLNL and is freely available under LGPL 2.1. For more details, see the interactive documentation and the full CHANGELOG.

CEED co-design center announced

The Exascale Computing Project (ECP) announced on November 11, 2016 its selection of four co-design centers, including CEED: the Center for Efficient Exascale Discretizations, which is a research partnership between Lawrence Livermore National Laboratory; Argonne National Laboratory; the University of Illinois Urbana-Champaign; Virginia Tech; University of Tennessee, Knoxville; Colorado University, Boulder; and the Rensselaer Polytechnic Institute (RPI).

Additional news coverage can be found in LLNL Newsline and the ANL press release.

R&D 100 Award for NekCEM / Nek5000

NekCEM/Nek5000: Scalable High-Order Simulation Codes received a 2016 R&D 100 Award, given by R&D Magazine to 100 top new technologies for the year.

The R&D 100 citation reads: "NekCEM/Nek5000: Release 4.0: Scalable High-Order Simulation Codes is an open-source simulation-software package that delivers highly accurate solutions for a wide range of scientific applications including electromagnetics, quantum optics, fluid flow, thermal convection, combustion and magnetohydrodynamics. It features state-of-the-art, scalable, high-order algorithms that are fast and efficient on platforms ranging from laptops to the world’s fastest computers. The size of the physical phenomena that can be simulated with this package ranges from quantum dots for nanoscale devices to accretion disks surrounding black holes. NekCEM provides simulation capabilities for the analysis of electromagnetic and quantum optical devices, such as particle accelerators and solar cells. Nek5000 provides turbulent flow simulation capabilities for a variety of thermal-fluid problems including nuclear reactors, internal combustion engines, vascular flows, and ocean currents."

See the ANL press release for more information.