DSD-INT 2019 Parallelization project for the USGS - Verkaik
1 like•200 views
The document outlines a parallelization project for MODFLOW 6 undertaken by USGS and Deltares aimed at improving groundwater simulation through distributed memory parallel computing on high-performance supercomputers. Key components include the development of a parallel linear solver, the partitioning of large-scale models into subdomains, and the integration of new modeling capabilities for global groundwater applications. The project emphasizes reducing computing time and memory usage, with various results and ongoing plans for future development presented.
DSD-INT 2019 Parallelization project for the USGS - Verkaik
- 1. i M O D U s e r D a y 2 0 1 9 – D S D - I N T 2 0 1 9 Parallelization project for the USGS Jarno Verkaik (Deltares, groundwater management department) SURFsara Cartesius supercomputer (47,776 cores, 130TB RAM)
- 2. Why (distributed memory) parallel computing? MODFLOW grid Serial computing Parallel computing 256 GB RAM 64 GB RAM 64 GB RAM 64 GB RAM 64 GB RAM MPI iMODUserDay2019–DSD-INT2019 1 day computing @ 256 GB machine 6 hours computing @ 64 GB machines MPI = Message Passing Interface 2
- 3. Contents • Organization • Project results and plans • Global scale application iMODUserDay2019–DSD-INT2019 3
- 4. How it started… • 2010: Email correspondence on parallel MT3DMS • 2013: Visit to USGS, start of joined code development (in kind) • 2015: Start development of Parallel Krylov Solver for MODFLOW-2005 and MODFLOW-USG → poster @ AGU Fall Meeting 2015, San Francisco • 2016: First application of PKS at national and global scale → poster @ AGU Fall Meeting 2016, San Francisco • Jul.2017: PKS as main feature for iMOD 4.0 & applied as default solver in National Water Model • Oct.2017: Start parallelization of MODFLOW 6 → funded by USGS through USGS-Deltares co-op iMODUserDay2019–DSD-INT2019 4 AGU Fall Meeting 2015 AGU Fall Meeting 2016
- 5. Organization through (coastal morphology) USGS-Deltares co-op iMODUserDay2019–DSD-INT2019 5 Robert McCall Applied Morphodynamics, Delft Kees Nederhoff Deltares USA, Silver Spring Martijn Russcher Numerical Simulation Software, Delft Jarno Verkaik Groundwater management, Utrecht Joseph D. Hughes Integrated Modeling and Prediction, Reston Christian D. Langevin Integrated Modeling and Prediction, Mounds View Li Erikson Pacific Coastal and Marine Science Center, Santa Cruz
- 6. USGS project FY2018 (Oct.2017 – Sep.2018) • Start parallelization of MODFLOW 6 • Such that it can be part of a future release • Target application: CONUS model by Wesley Zell and Ward Sanford (USGS) • USGS requirements: - Proof of concept applicable to CONUS model - Low code footprint - Version controlled code at GitHub - Easy to use - Not depending on 3rd party libraries iMODUserDay2019–DSD-INT2019 6
- 7. USGS project FY2018 (Oct.2017 – Sep.2018) • Proof of concept was developed, applicable to CONUS • Parallelization of IMS linear solver using Schwarz domain decomposition (similar to Parallel Krylov Solver in iMOD) • Repos: https://github.com/verkaik/modflow6-parallel.git →MODFLOW 6 framework refactoring required for exchanges between models (subdomains): - That is generic for both serial and parallel computing - Such that numerical schemes can be evaluated more easily at model interfaces - Such that XT3D option can be used with multiple models (serial and parallel) iMODUserDay2019–DSD-INT2019 7 Halo v2. concept
- 8. USGS project FY2019 & FY2020 • FY2019 (Oct.2018 – Sep.2019) • Support XT3D option with multi-models (serial only) • Development of interface model concept (revised halo v2) • FY2020 (Oct.2019 – Sep.2020) (To be determined) • Continue working on parallel MODFLOW • Development of Basic Model Interface iMODUserDay2019–DSD-INT2019 8 M1 M2
- 9. USGS project FY2018 results: circle test 1250M cells iMODUserDay2019–DSD-INT2019 9
- 10. USGS project FY2018 results: circle test 12.5M cells iMODUserDay2019–DSD-INT2019 10
- 11. Related work to USGS project iMODUserDay2019–DSD-INT2019 • PhD project (start 2018) “Towards Exascale Computing for Large Scale Groundwater Simulation” Goal: development of distributed parallel methods applying to large real-life groundwater models of O(106)–O(109) cells. • Mainly funded by Deltares research • Directly relates to MODFLOW 6 kernel development for new iMOD 6 (see next presentation by Otto de Keizer) Prof. Marc Bierkens (Utrecht University) Prof. Hai Xiang Lin (Delft University of Technology) Gualbert Oude Essink, PhD (Deltares) 11
- 12. Contributions from PhD project iMODUserDay2019–DSD-INT2019 Short term coding: • Improve linear solver convergence when using many subdomains: → add coarse grid parallel preconditioner (implementation largely done) • Option to check parallel implementation → add serial block Jacobi preconditioner (first implementation done) • Code profiling & optimizing parallel performance (ongoing) Longer term coding: • Robustness option when using many subdomains: → add recovery mechanism for failing hardware • Add physics-based parallel preconditioner Short term modeling: • Run USGS CONUS model in parallel @ 250 m 12
- 13. • Development of PCR-GLOBWB global groundwater model having 1km x 1km resolution, O(108) cells • First experience with parallel MODFLOW 6 with this scale: • Physics based subdomain partitioning • Model generation (pre-processing) • Parallel computing • Visualization of model results → Big data! Typical raster: 43200 columns x 21600 rows, 3 GB binary Global groundwater model @ 1km and 428M cells Ref: Verkaik, J., Sutanudjaja, E.H., Oude Essink, G.H.P., Lin, H.X., and Bierkens, M.F.P., 2019. Parallel global hydrology and water resources PCR-GLOBWB-MODFLOW model at hyper-resolution scale (1 km): first results, in: EGU General Assembly Conference Abstracts. p. 13397. iMODUserDay2019–DSD-INT2019 13
- 14. MODFLOW 6 model characteristics: • Steady-state, 2 layers, subsurface down-scaled from 10km • Unstructured DISU grid met only “land cells”, total 428M • CHD for sea, RIV in layer 1 + DRN in layer 1 & 2 (HydroSHEDS) Global groundwater model @ 1km and 428M cells iMODUserDay2019–DSD-INT2019 Parallel pre-processing using 128 subdomains 14
- 15. Global groundwater model @ 1km and 428M cells Can we come up with predefined subdomain boundaries (e.g. hydrologically / administrative boundary) such that they are useful for both modeler and parallel computing? → How to partition the world into 1024 subdomains using 1.8M catchments? → How to solve a sub-optimal optimization problem (load + edge cuts)? 1. Determine independent regions for groundwater flow (continents, islands) → ~20k regions 2. Further divide large regions/catchments using a lumped graph method → define parallel models 3. Cluster small regions → define serial models iMODUserDay2019–DSD-INT2019 15
- 16. Global groundwater model @ 1km and 428M cells • Partitioning results in 52 separate MODFLOW 6 models: • 38 serial, small islands • 13 parallel, 3 largest on super computer 428M (2 layers) Small parallel+ serial models 2. America 120M cells 286 cores 1min 36sec 112GB memory 1. Africa+EurAsia 256M cells 612 cores 3min 31sec 390 GB memory 3. Australia 20M cells 48 cores 33 sec 13 GB memory 5% 28% 60% iMODUserDay2019–DSD-INT2019 16
- 17. Global groundwater model @ 1km and 428M cells iMODUserDay2019–DSD-INT2019 Simulated Groundwater Table subdomain boundary (total: 1024) 17
- 18. Global groundwater model @ 1km and 428M cells iMODUserDay2019–DSD-INT2019 Simulated Groundwater Table subdomain boundary (total: 1024) Take home message: USGS and Deltares are making progress on MPI parallelization of the MODFLOW 6 multi-model capability for reducing computing times & memory usage THANK YOU! 18