Publications and Outreach
CEED Documents

CEED's highorder Benchmarks and Miniapps.

Activities in the Applications, Hardware, Software and Finite Element thrusts.

CEEDproposed highorder Operator and Visualization formats.
 CEEDMS1 report: Engage first wave ECP/CEED apps.
 CEEDMS6 report: Identify initial kernels, benchmarks and miniapps.
 CEEDMS8 report: Initial integration of CEED software in ECP apps.
 CEEDMS10 report: Initial CEED API.
 CEEDMS13 report: Public release of CEED 1.0.
 CEEDMS18 report: Propose highorder mesh/data format.
 CEEDMS20 report: Performance tuning of CEED software and first wave apps.
 CEEDMS23 report: Engage second wave ECP/CEED applications.
 CEEDMS25 report: Public release of CEED 2.0.
 CEEDMS29 report: Improved Support for Parallel Adaptive Simulation in CEED.
 CEEDMS32 report: Performance tuning of CEED software and 1st and 2nd wave apps.
 CEEDMS34 report: Improve performance and capabilities of CEEDenabled ECP applications on Summit/Sierra.
Publications
2020
 V. Barra, J. Brown, J. Thompson, Y. Dudouit, Highperformance operator evaluations with ease of use: libCEED's Python interface, Proc. 19th Python in Science Conf., 8590, 2020
2019
 P. Fischer, M. Min, T. Rathanayake, S. Dutta, Tz. Kolev, V. Dobrev, J.S. Camier, M. Kronbichler, T. Warburton, K. Swirydowicz, and J. Brown, Scalability of HighPerformance PDE Solvers, Int. J. High Perf. Comput. Appl., 2020.
 H. Hajduk, D. Kuzmin, Tz. Kolev, and R. Abgrall, Matrixfree subcell residual distribution for Bernstein finite elements: Loworder schemes and FCT, Comp. Meth. Appl. Mech. Eng., in press, 2019.
 P. BelloMaldonado and P. Fischer, Scalable loworder finite element preconditioners for highorder spectral element Poisson solvers, SIAM J. Sci. Comp., to appear, 2019.
 V. A. Dobrev, T. V. Kolev, C. S. Lee, V. Z. Tomov, and P. S. Vassilevski, Algebraic Hybridization and Static Condensation with Application to Scalable H(div) Preconditioning, SIAM Journal on Scientific Computing, 41(3):B425–B447, 2019.
 J. Cerveny, V. Dobrev, and Tz. Kolev, NonConforming Mesh Refinement For HighOrder Finite Elements, SIAM Journal on Scientific Computing, 41(4):C367–C392, 2019.
 V. Dobrev, P. Knupp, Tz. Kolev, and V. Tomov, Towards SimulationDriven Optimization of HighOrder Meshes by the TargetMatrix Optimization Paradigm, Roca X., Loseille A. (eds) 27th International Meshing Roundtable. IMR 2018. Lecture Notes in Computational Science and Engineering, vol 127, pp. 285302, Springer, 2019.
 I. Masliah, A. Abdelfattah, A. Haidar, S. Tomov, M. Baboulin, J. Falcou, and J Dongarra, Algorithms and optimization techniques for highperformance matrixmatrix multiplications of very small matrices, Parallel Computing, vol. 81, pp. 121, 2019.
 V. Dobrev, P. Knupp, Tz. Kolev, K. Mittal, V. Tomov, The TargetMatrix Optimization Paradigm for HighOrder Meshes, SIAM J. Sci. Comp., 41(1), pp. B50–B68, 2019.
 J. Martinez, Y. Lan, E. Merzari, M. Min, On the use of LESbased turbulent thermalstress models for rod bundle simulations, Int. J. Heat and Mass Transfer, 142:118399, 2019.
 J. Gong, M. Min, P. Fischer, P. Schlatter, and E. Laure, OpenACC acceleration for the PnPn2 algorithm in Nek5000, J. Parallel and Distributed Computing, vol. 132, pp. 6978, 2019.
 L. Lu, M. Nazarov, P. Fischer, Nonlinear artificial viscosity for spectral element methods , Comptes Rendus Mathematique, vol 357, pp. 646654, 2019.
 K. Mittal, S. Dutta, and P. Fischer, K. Mittal, S. Dutta, and P. Fischer, Nonconforming Schwarzspectral element methods for incompressible flow, Computers and Fluids, 191(15):104237, 2019.
 S. Patel, P. Fischer, M. Min and A. Tomboulides, A CharacteristicBased Spectral Element Method for MovingDomain Problems, J. Sci. Comput., vol 79, pp. 564592, 2019.
 K. Mittal and P. Fischer, Mesh smoothing for the spectral element method, J. Sci. Comput., 78(2), pp. 11521173, 2019.
 G.K. Giannakopoulos, C.E. Frouzakis, P. Fischer, A. Tomboulides and K. Boulouchos, LES of the gasexchange process inside an internal combustion using a highorder method, Flow, Turbulence and Combustion, pp. 120, 2019.
2018
 P. BelloMaldonado and P. Fischer, Scalable LowOrder Finite Element Preconditioners for HighOrder Spectral Element Poisson Solvers, submitted, 2018.
 A. Karakus, N. Chalmers, J.S. Hesthaven and T. Warburton, Discontinuous Galerkin Discretizations of the Boltzmann Equations in 2D: semianalytic time stepping and absorbing boundary layers, submitted, 2018.
 A. Haidar, S. Tomov, and J. Dongarra, Optimizing GPU Kernels for Irregular Batch Workloads: A Case Study for Cholesky Factorization, HPEC'18, submitted, 2018.
 A. Karakus, N. Chalmers, K. Swirydowicz, T. Warburton, GPU Acceleration of a HighOrder Discontinuous Galerkin Incompressible Flow Solver, submitted, 2018.
 A. Barker, V. Dobrev, J. Gopalakrishnan and T. Kolev, A scalable preconditioner for a primal discontinuous PetrovGalerkin method, SIAM J. Sci. Comp., 40(1), pp. B32B58, 2018.
 R. Anderson, V. Dobrev, Tz. Kolev, R. Rieben and V. Tomov, HighOrder MultiMaterial ALE Hydrodynamics, SIAM J. Sci. Comp., 40(1):B32B58, 2018.
 V. Dobrev, Tz. Kolev, D. Kuzmin, R. Rieben and V. Tomov, Sequential limiting in continuous and discontinuous Galerkin methods for the Euler equations, Journal of Computational Physics, 356:372390, 2018.
2017
 K. Swirydowicz, N. Chalmers, A. Karakus, T. Warburton, Acceleration of tensorproduct operations for highorder finite element methods, submitted, 2017
 J. Solberg, E. Merzari, H. Yuan, A. Obabko and P. Fischer, S. Lee, J. Lai, M. Delgado, S. J. Lee, and Y. Hassan, HighFidelity Simulation of Flow Induced Vibrations in Helical Steam Generators for Small Modular Reactors, Best Paper Award at The 17th International Topical Meeting on Nuclear Reactor Thermal Hydraulics (NURETH17), September, 2017.
 S. Patel, P. Fisher, M. Min, and A. Tomboulides, A Characteristicbased, spectral element method for movingdomain problems, J. Sci. Comp., submitted, 2017.
 I. Masliah, A. Abdelfattah, A. Haidar, S. Tomov, M. Baboulin, J. Falcou, and J. Dongarra, Algorithms and optimization techniques for highperformance matrixmatrix multiplications of very small matrices, Parallel Computing (PARCO), submitted, 2017.
 K. Raffenetti et. al, P. Fischer, M. Min, and P. Balaji, Why is MPI so slow? Analyzing the fundamental limits in implementing MPI3.1, accepted, SC'17, 2017.
 P. Fischer, M. Schmitt, and A. Tomboulides, Recent developments in spectral element simulations of movingdomain problems, vol. 79, Fields Institute Communications, 213–244, 2017.
 S. Lomperski, A. Obabko, P. Fischer, E. Merzari, and W.D. Pointer, Jet stability and wall impingement flow field in a thermal striping experiment, Int. J. Heat Mass Transfer, 115A:1125– 1136, 2017.
 V. Makarashvilia, E. Merzari, A. Obabko, A. Siegel, and P. Fischer, A performance analysis of ensemble averaging for high fidelity turbulence simulations at the strong scaling limit, Comp. Phys. Comm., in press, 2017.
 A. Abdelfattah, A. Haidar, S. Tomov, J. Dongarra, Factorization and Inversion of a Million Matrices using GPUs: Challenges and Countermeasures, Proceedings of the 2017 International Conference on Computational Science, ICCS'17, Zürich, Switzerland, June 1214, Procedia Computer Science, 2017.
 A. Abdelfattah, M. Baboulin, V. Dobrev, J. Dongarra, A. Haidar, I. Karlin, Tz. Kolev, I. Masliah, and S. Tomov, Small Tensor Operations on Advanced Architectures for Highorder Applications, Technical report UTEECS17749, EECS Department, University of Tennessee, 2017.
 R. Anderson, V. Dobrev, Tz. Kolev, D. Kuzmin, M. Quezada de Luna, R. Rieben and V. Tomov, Highorder local maximum principle preserving (MPP) discontinuous Galerkin finite element method for the transport equation, Journal of Computational Physics, 334:102–124, 2017.
 Bazilevs, Y., Kamran, K., Moutsanidis, G., Benson, D. J., & Oñate, E, A new formulation for airblast fluid–structure interaction using an immersed approach. Part I: basic methodology and FEMbased simulations, Computational Mechanics, 118, 2017.
 Abdelfattah, A., Haidar, A., Tomov, S., and Dongarra, J. Novel HPC Techniques to Batch Execution of Many Variable Size BLAS Computations on GPUs, International Conference on Supercomputing (ICS'17), ACM, Chicago, Illinois, pp. 110, June 1416, 2017.
 Ibanez, D., Shephard, M.S., Modifiable Array Data Structures for Mesh Topology, SIAM Journal on Scientific Computing, 39(2):C144C161, 2017.
 Granzow, B.N., Shephard M.S., Oberai, A.A., Outputbased error estimation and mesh adaptation for variational multiscale methods, Computer Methods in Applied Mechanics and Engineering, Vol. 322, pp. 441459, 2017.
2016
 E. Merzari, A. Obabko, P. Fischer, N. Halford, J. Walker, A. Siegel, and Y. Q. Yu, Largescale large eddy simulation of nuclear reactor flows: Issues and perspectives, Nuclear Engineering and Design, page 13, Oct., 2016.
 E. Merzari, P. Fischer, H. Yuan, K. Van Tichelen, S. Keijers, J. De Ridder, J. Degroote, J. Vierendeels, H. Doolaard, V. R. Gopala, and F. Roelofs, Benchmark exercise for fluid flow simulations in a liquid metal fast reactor fuel assembly, Nuclear Engineering and Design, 298(3):218–228, 2016.
 V. Dobrev, Tz. Kolev, R. Rieben and V. Tomov, Multimaterial closure model for highorder finite element Lagrangian hydrodynamics, Int. J. Numer. Meth. Fluids, 82(10), pp. 689–706, 2016.
 A. Abdelfattah, M. Baboulin, V. Dobrev, J. Dongarra, C. Earl, J. Falcou, A. Haidar, I. Karlin, Tz. Kolev, I. Masliah, S. Tomov, Highperformance Tensor Contractions for GPUs, Procedia Computer Science, Volume 80, Pages 108118, ISSN 18770509, 2016.
 M. B.E., Y. Peet, P. Fischer, and J. Lottes, A spectrally accurate method for overlapping grid solution of incompressible NavierStokes equations, J. Comp. Phys., 307:60–93, 2016.
 M. Otten, J. Gong, A. Mametjanov, A. Vose, J. Levesque, P. Fischer, and M. Min, An MPI/OpenACC implementation of a high order electromagnetics solver with GPUDirect communication, The International Journal of High Performance Computing Application, 30(3):320–334, 2016.
 J. Gong, S. Markidis, E. Laure, M. Otten, P. Fischer, M. Min, Nekbone Performance on GPUs with OpenACC and CUDA Fortran Implementations, Special issue on Sustainability on Ultrascale Computing Systems and Applications: Journal of Supercomputing, 2016.
 Ibanez, Daniel A., et al. PUMI: Parallel unstructured mesh infrastructure. ACM Transactions on Mathematical Software (TOMS) 42.3 2016.
 Smith, Cameron W., et al. Inmemory Integration of Existing Software Components for Parallel Adaptive Unstructured Mesh Workflows. Proceedings of the XSEDE16 Conference on Diversity, Big Data, and Science at Scale. ACM, 2016.
 Ibanez, D., Dunn, I., Shephard M.S., Hybrid MPIthread parallelization of adaptive mesh operations, Parallel Computing, 52:133143, 2016.
2015 and earlier
 P. Fischer, K. Heisey, and M. Min, Scaling limits for PDEbased simulation, In 22nd AIAA Computational Fluid Dynamics Conference, AIAA Aviation, AIAA 20153049, 2015.
 A. Kraus, S. Aithal, A. Obabko, E. Merzari, A. Tomboulides, and P. Fischer, Erosion of a largescale gaseous stratified layer by a turbulent jet  Simulations with URANS and LES approaches, volume 2, pp. 1448–1461. American Nuclear Society, 2015.
 E. Merzari, P. Fischer, and J. Walker, Largescale simulation of rod bundles: Coherent structure recognition and stability analysis, volume 1. American Society of Mechanical Engineers, 2015.
 M. Otten, R. A. Shah, N. F. Scherer, M. Min, M. Pelton, and S. K. Gray, Entanglement of two, three and four plasmonically coupled quantum dots, Physical Review B, 92:125432, 2015.
 D. A. May, J. Brown, and L. Le Pourhiet. pTatin3D: Highperformance methods for longterm lithospheric dynamics, In Proceedings of SC14: International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 2014.
 R. Anderson, V. Dobrev, Tz. Kolev and R. Rieben, Monotonicity in highorder curvilinear finite element ALE remap, Int. J. Numer. Meth. Fluids, 77(5), pp. 249–273, 2014.
 Kamran, Kazem, et al. A compressible Lagrangian framework for modeling the fluid–structure interaction in the underwater implosion of an aluminum cylinder. Mathematical Models and Methods in Applied Sciences 23.02 2013.
 Tz. Kolev and P. Vassilevski, Parallel auxiliary space AMG solver for H(div) problems, SIAM J. Sci. Comp., 34, pp. A3079–A3098, 2012.
 V. Dobrev, Tz. Kolev and R. Rieben, Highorder curvilinear finite element methods for Lagrangian hydrodynamics, SIAM J. Sci. Comp., 34, pp. B606–B641, 2012.
 J. Brown, Efficient nonlinear solvers for nodal highorder finite elements in 3D, Journal of Scientific Computing, 45:48–63, 2010. doi:10.1007/s1091501093968
 Tz. Kolev and P. Vassilevski, Parallel auxiliary space AMG for H(curl) problems, J. Comput. Math., 27, pp. 604623, 2009.
Presentations
2017
 Tz. Kolev and M. Shephard, Conforming & Nonconforming Adaptivity for Unstructured Meshes, Argonne Training Program on ExtremeScale Computing, Aug 7, 2017.
 Tz. Kolev and M. Shephard, Unstructured Mesh Technologies, Argonne Training Program on ExtremeScale Computing, Aug 7, 2017.
 B. Smith, Nonlinear and Krylov Solvers, Argonne Training Program on ExtremeScale Computing, Aug 7, 2017.
 J. Dongarra, Adaptive Linear Solvers and Eigensolvers, Argonne Training Program on ExtremeScale Computing, Aug 7, 2017.
 T. Warburton, An Intro to GPU Architecture and Programming Models, Argonne Training Program on ExtremeScale Computing, Aug 3, 2017.
 S. Parker, Architectures of the Argonne Cray XC40 KNL System "Theta", Argonne Training Program on ExtremeScale Computing, Jul 31, 2017.
 S. Tomov and A. Haidar, MAGMA Tensors and Batched Computing for Accelerating Applications on GPUs, GPU Technology Conference (GTC'17), Session S7728, May 811, 2017.
 A. Abdelfattah, M. Baboulin, V. Dobrev, J. Dongarra, C. Earl, J. Falcou, A. Haidar, I. Karlin, Tz. Kolev, I. Masliah, S. Tomov, Accelerating Tensor Contractions in HighOrder FEM with MAGMA Batched, SIAM Conference on Computer Science and Engineering (SIAM CSE'17), Atlanta, GA, Feb 26Mar 3, 2017.
 P. Fischer, Efficiency of HighOrder Methods on the 2nd Generation Intel Xeon Phi Processor, SIAM Conference on Computer Science and Engineering (SIAM CSE'17), Atlanta, GA, Feb 26Mar 3, 2017.
 M. Min, Spectral Element Simulation for Nanowire Solar Cells on HPC Platforms, SIAM Conference on Computer Science and Engineering (SIAM CSE'17), Atlanta, GA, Feb 26Mar 3, 2017.
 C. Smith, G. Diamond and M.S. Shephard, Fast Dynamic Load Balancing Tools for Extreme Scale System, SIAM Conference on Computer Science and Engineering (SIAM CSE'17), Atlanta, GA, Feb 26Mar 3, 2017.
 S. Tendulkar, O. Klaas, M.W. Beall, and M.S. Shephard, Parallel Geometry and Meshing Adaptation with Application to Problems with Evolving Domains, SIAM Conf. on Computational Science and Engineering, Atlanta, GA, Mar 3, 2017.
 P. Fischer, CFD, PDEs, and HPC: A 30Year Perspective, Argonne Training Program on ExtremeScale Computing, Aug 2, 2016.
Highlights
 ECP article: CEED’S Impact On Exascale Computing Project (ECP) Efforts Is WideRanging, Jul 2019.
 PC magazine article highlighting supercomputing at LLNL, Jul 2019.
 DEIXIS article about CEED: Scaling the Unknown, Jul 2018.
 ECP article: CoDesign is Key to ECP's Holistic Approach to Capable Exascale Computing, Apr 2018.
 MFEM highlighted in LLNL's Science & Technology Review magazine, including on the cover, Jan/Feb 2018.
 GCN article: Exascale a "main priority" for DOE, Jan 2018.
 ECP article: CoDesign Center Develops NextGeneration Simulation Tools, also in HPCwire, Nov 2017.
 Work with LLNL's Center for Design and Optimization mentioned in LLNL Newsline, Oct 2017.
 Highlight in CASC Newsletter #3, Oct 2017.
 Highlight in LLNL’s 65th Anniversary Book (2017 page), Oct 2017.
 GPU work highlight in LLNL's COMP News page and Livermore_Comp's Twitter feed, May 2017.
 Work with Cardioid mentioned in LLNL's Science & Technology Review magazine, Mar 2017.
 News coverage of CEED announcement in LLNL Newsline and the ANL press release, Nov 2016.
 Work with BLAST mentioned in LLNL's Science & Technology Review magazine, Sep 2016.
Other Resources
 CEEDtagged topics on ECP's website.
 ANL's exascale computing website.
 LLNL's exascale computing website.
 Blog of Virginia Tech's Parallel Numerical Algorithms research group.
 U.S. Department of Energy Exascale Initiative.