Progress in Fast, Accurate Multi-scale Climate Simulations
W. D. Collins
1, H. Johansen
1, K. J. Evans
2, C. S. Woodward
3, and P. M.
Caldwell
31 Lawrence Berkeley National Laboratory, Berkeley, CA, USA
[email protected], [email protected]
2 Oak Ridge National Laboratory, Oak Ridge, TN, USA
3 Lawrence Livermore National Laboratory, Livermore, CA, USA
[email protected], [email protected]
Abstract
We present a survey of physical and computational techniques that have the potential to con-tribute to the next generation of high-fidelity, multi-scale climate simulations. Examples of the climate science problems that can be investigated with more depth with these computational improvements include the capture of remote forcings of localized hydrological extreme events, an accurate representation of cloud features over a range of spatial and temporal scales, and parallel, large ensembles of simulations to more effectively explore model sensitivities and un-certainties. Numerical techniques, such as adaptive mesh refinement, implicit time integration, and separate treatment of fast physical time scales are enabling improved accuracy and fidelity in simulation of dynamics and allowing more complete representations of climate features at the global scale. At the same time, partnerships with computer science teams have focused on taking advantage of evolving computer architectures such as many-core processors and GPUs. As a result, approaches which were previously considered prohibitively costly have become both more efficient and scalable. In combination, progress in these three critical areas is poised to transform climate modeling in the coming decades. These topics have been presented within a workshop titled, ”Numerical and Computational Developments to Advance Multiscale Earth System Models (MSESM ’15),” as part of the International Conference on Computational Sci-ences, Reykjavik, Iceland, June 1-3, 2015.
Keywords: earth system models, multi-scale climate, time integration, many-core
1
Introduction
Some of the greatest challenges in projecting the future of the Earth’s climate result from the significant and complex interactions between small-scale features and large-scale structures throughout the Earth system. In addition, significant interactions occur between structures of the ocean, atmosphere, land surface, and cryosphere. In order to advance Earth system
Volume 51, 2015, Pages 2006–2015
ICCS 2015 International Conference On Computational Science
2006 Selection and peer-review under responsibility of the Scientific Programme Committee of ICCS 2015 c
science, a new generation of models that capture the structure and evolution of the climate system across a broad range of spatial and temporal scales is required. The climate community needs to produce better models for these critical processes, to better support science inves-tigations that span everything from cloud systems to grounding line retreat in ice sheets at global scales, through improved physical and computational implementations. In tandem with new model features is the requirement of sufficient spatial resolution to capture the smallest unparametrized scales and their feedbacks onto the global Earth system. Recent work has demonstrated improvements to the representation of multi-scale events including precipitation [10] and large scale dynamics [4] when finer spatial scales are captured in global atmospheric models. At the same time, these models and algorithms must be capable of running on the largest and fastest computer resources available. Producing these models will require forming integrated teams of climate and computational scientists to accelerate the development and integration of parameterizations and dynamics into Earth System Models (ESMs). This article surveys pathways for introducing accurate and computationally efficient treatments of multi-scale features into ESMs in order to address the increasing complex science questions these models are being tasked to address.
The increase in resolution via multi-scale implementations could facilitate solutions for sev-eral of the greatest challenges in modeling the significant two-way interactions between local and global scales in the climate system. For example, there are abundant observational stud-ies [8, 50, 51], theoretical derivations [5, 35, 36, 37], and emerging computational results [40] demonstrating that the tropical ocean-atmosphere system is characterized by multi-scale dy-namics.
Some of the largest feedbacks in the climate system may be due to clouds [7, 13, 48], yet clouds remain unresolved and are highly parameterized in current climate models. The effects of clouds on the climate system are one to two orders of magnitude larger than the effects from anthropogenic greenhouse gases that cause the observed warming [17]. As a result, small cloud changes in response to this warming could greatly amplify or ameliorate climate change over the 21st century. Clouds also play critical roles in the Earth’s hydrological cycle, for they are the origins of rain, snow, and other forms of precipitation. One of the key uncertainties in climate projections is how the hydrological cycle, and in particular the intensity and frequency of rainfall, will evolve in a warmer world [1]. Resolving this will also require major advances in the treatment of atmospheric convection, another critical process in the climate system that is still unresolved at the resolutions typical of current climate models. Fluid motions in convective clouds, which overturn air containing condensed water, span over five orders of magnitude in space ranging from a few meters to over one hundred kilometers. It is not possible to resolve all of these scales simultaneously in one model. Thus, representing clouds and convection remains an extreme scientific challenge for climate modeling.
Quasi-uniform Almost Cloud Resolving global models with resolutions of approximately 2km in the horizontal direction and equations similar to those in the cloud resolving and LES models reproduce many cloud features, but they do not accurately represent shallower clouds and they are not appropriate for climate studies of years or longer. Early evidence from proof-of-concept tests of cloud-system-permitting and cloud-resolving models suggest that the cloud feedback in these ultra-high-resolution frameworks differ systematically from the feedbacks exhibited by conventional models [38, 62]. Prototypes [38] are being evaluated but they run about a million times slower than current climate models [45], and it is an enormous computational challenge to produce a simulation of just a few months.
New viable computational approaches are needed to simulate multi-scale dynamics at reso-lutions approaching characteristic length scales ranging from order 1 to 10 km and time scales
ranging from minutes to hours. Promising research areas include: taking advantage of advanced computational hardware (§2), focusing computational resources on key phenomena (§3), and time integration (§4). For example, most climate models use explicit time integration for most terms, and treat selected tendencies semi-implicitly, thereby worsening stability constraints at high resolution. To address this, implicit integration, sub-cycling and implicit-explicit (IMEX) approaches are all being explored. In variable resolution models, these techniques could be blended to manage time integration in both coarse and fine resolution regions of the domain [25].
Development of multi-scale models poses challenges and opportunities for the numerical and physical formulations of climate processes, such as cloud physics. Their parameterizations in current climate models are highly specific and tuned for a given set of horizontal and ver-tical grid spacings and must be adjusted with a change in resolution. So-called scale-aware parameterizations are required in order to adapt the physics to a range of resolutions or for variable resolutions within a single simulation. Such simulations have the promise of resolving key features, such as orographic effects and extreme atmospheric events in a way that global uniform resolutions would not. In addition, the assumption that small scale processes are in quasi-equilibrium with the grid-scale boundary conditions becomes increasingly less tenable at higher resolutions, and so the effects of small-scale stochastic fluctuations must be included. Finally, as we move to spatial scales of a few km or less, some processes begin to be resolved and will not need to be parameterized. The combination of scale-aware, stochastic, and explicit process representations constitutes a major paradigm shift in the design of climate models, and will force a retooling across the significant software infrastructure that the climate community has developed. For fixed resolution models, the introduction of scale-aware parameterizations alone will require reproducible techniques for optimizing the physical fidelity and computational performance of the model physics across a wide range of scales than have been employed to date [31, 32, 59]. This transition could be advanced through the adoption of techniques for sen-sitivity analyses, sampling of high-dimensional parameter spaces, and model calibration from the field of uncertainty quantification (UQ). These efforts would also benefit from weak scaling on new architectures, which would enable additional resources to be applied to refinement of sub-grid structures in the ocean and atmosphere.
2
Scalability and computational trends
High performance computing (HPC) architectures continue to evolve along with the complex-ity of ESM’s. Often the science simulation capabilities of multi-scale models depend on the newest and largest computational resources. Unfortunately, this leads to recurring challenges to upgrade both programming models to achieve good performance with existing discretiza-tions and algorithms that best exploit the capabilities of new architectures. The current trend in high-end computation points to a many-core crisis [28] with codes needing to transform to leverage hybrid nodes with both heavy-weight processors and highly threaded co-processors, such as GPU’s. Analysis of the Top 500 computers shows alarming trends for ESM’s: FLOP growth is 2x faster than memory and memory bandwidth. On most high-end systems, this ratio is currently below 0.1 GB / GFLOP, rapidly on its way to 0.01, with memory capacity per core expected to drop by 30% every two years. ESM’s are already more communication-bound than compute-bound for some configurations [11], and will need to explicitly manage access to complex memory hierarchies. This shift will cause ESMs to benefit more from algorithms and higher-order discretizations with higher arithmetic intensity.
limita-tions based on the maximum wave speed, which is in the range of 300− 500 m/s. At around 20km horizontal grid spacing, where cloud physics and tropical cyclones are predominantly represented by parameterizations, the time step is O(100 s). The corresponding vertical grid spacing may beO(100 m) in the lower atmosphere, so these codes, using traditional fully explicit methods, must take punishingly small time steps,O(1 s). In addition, these methods have low arithmetic intensity, which inhibits strong scaling on memory- and bandwidth-limited architec-tures. Higher-order methods are underway to address these issues, including algorithms that are fully explicit but enable larger time steps, such as ADER schemes [44]. HEVI (horizontal-explicit, vertical-implicit) time integrators also have potential to overcome time step limitations while still resolving large-scale features. The increased computational cost when the time step size is restricted - which is 8x for every doubling of resolution in all 3 spatial dimensions - is also driving research into fully implicit approaches and asymptotic simplifications, such as low-Mach number approximations. Both raise questions of the frequency and computational expense of coupling to physics parameterizations at these scales. Similar analyses apply to the ocean and other ESM components, with trade offs on weak- and strong-scaling discretizations, and the implied disruptive software changes to ESM’s to optimize communication, computation, and memory.
Another consideration is that the volume of data produced by O(1 km) simulations would be staggering: approximately 100B degrees of freedom, updated with frequency of 1 s −15 m of simulation time. It is not reasonable to write this volume of data to disk (at the rate of 1T B/s), so it is clear that feasible scientific data plans must include reasonably selective output, such as high resolution only around key features, and less I/O-intensive climate statistics.
2.1
The Critical Role of Numerical Software Packages
Fast-changing computer hardware architectures and lagged use of software libraries have made it increasingly difficult for science application developers to build high-performance software. The climate community is no exception. High resolution simulations require state-of-the-art hardware, but the newest high end computing systems are undergoing rapid technology changes. As a result, most large multi-scale and multi-physics parallel applications must adapt to ever-evolving computing platforms in order to try to realize the potential of extreme-scale supercom-puters [19, 12]. Unfortunately, adaptations necessary to realize the promise of new architectures require substantial time, effort, and specialized expertise. Recent workshops sponsored by U.S. Dept. of Energy offices, one from the high performance computing community, and one from the terrestrial science community, identified the need and priorities for attention to software productivity [23].
One critical priority identified at these workshops was the need for significant scientific application simulators to make use of reusable numerical libraries. The main advantage in using these is the ability to rely on these packages to adapt to changing hardware while allowing the application code to remain largely intact, preserving much of the validation and verification progress. For example, the Trilinos library [18] includes methods for solution of linear and nonlinear systems, graph partitioning, and uncertainty quantification. The PETSc library specializes in methods for linear and nonlinear system solution but has been extended to include time integration [3]. The SUNDIALS library includes methods for solution of nonlinear systems, time integration of ordinary and differential-algebraic systems, and sensitivity analysis [52]. The CHOMBO library provides infrastructure and support for structured adaptive mesh refinement [9]. Other useful packages for linear system solvers include the hypre [20] and SuperLU libraries [33].
Based on their scalability, availability on large-scale HPC, and support for discretizations commonly used in ESM’s, new dynamical cores are being developed that extensively take ad-vantage of these libraries [24, 43, 49]. In addition, much solver work in existing cores is also leveraging parallel and optimized software libraries [63, 34, 2]. By taking advantage of these and other libraries, large-scale climate simulations are evolving into software ecosystems that bridge between the diverse ESM components and the rapidly-evolving extreme-scale platforms on which they must run to achieve their science goals. A key component of this evolution is the encapsulation of architecture-dependent optimizations. Through encapsulation, these ecosystems can minimize the error-prone work of adaption to any given machine since the most specialized code is well-separated from the physics models. Leveraging of numerical and other computational libraries also allows for easier inclusion of new algorithms and technologies. These libraries have the potential to provide significant benefits in the minimization of time both to bring new algorithms into production and to bring optimizations to what we expect to be rapidly changing computer architectures.
3
Adaptive Mesh Refinement
At very high resolutions of atmosphere models, non-hydrostatic effects become important, and flows become less dominated by large horizontal aspect ratios and are more equally three-dimensional in nature. A critical requirement is accurately representing wave phenomena that affect climate [55]. BelowO(10 km) the interaction of gravity waves and cloud-resolving models may require spatial resolution ofO(1 km). At uniform global resolution these calculations are infeasible, so “regional” static mesh refinement to increase resolution only where such features must be resolved is being developed [64]. Similar research is underway in ocean modeling to provide regional, high-resolution, “eddy-allowing,” configurations [46]. Opportunities exist also for dynamic or adaptive mesh refinement (AMR), which adapts both grid refinement and time stepping to maintain acceptable error estimates and/or to track features such as tropical cyclones or squall lines. Implementing high-order methods is useful in preserving accuracy in multi-resolution simulations within static or dynamic refined configurations so they are gaining attention. They also provide arithmetic intensity and better capture correct dispersive behavior of low-to mid-frequency waves. Many challenges listed in [58] are starting to be overcome, including criteria for where and when to refine, distortion of waves across refinement boundaries, and discretizations that are scale-appropriate and accurate across 2-3 orders of magnitude of refinement in space and time [54]. Dynamic load balancing issues are more complex than static local refinement [47], and although “feature-based” adaptivity is desired it is complex, and load balancing AMR calculations remains an open research topic [58, 56]. Combining numerical techniques with AMR, such as HEVI schemes, is an active area of research [22]. It remains to be seen if the resulting complexity of these codes creates challenges in software maintenance and performance portability, which would hinder their adoption in the broader ESM community.
4
Time integration
Properly treating the time dimension is a crucial aspect of accurate, multi-scale modeling. Explicit methods are easy to implement, but errors can accumulate over long simulations if the order of accuracy and level of filtering are not treated carefully. For example, century-long simulations with high-resolution coupled ESMs needed to analyze biogeochemical cycles or sea level variations requireO(10M) time steps to maintain stability when using a fully-explicit
single-stage method. Due to necessary filtering at the 2Δx scale of the grid, the commonly used explicit leapfrog method is effectively first order, and so floating point errors can accumulate unacceptably to the level of the climate variability over this many time steps. This effect has been observed for simple problems [21]. In addition, due to the CFL constraint on the time step size, lack of parallelism in time, and additional scale interactions with spatial refinement, explicit methods suffer from a superlinear deficiency in weak scaling [26].
A diversity of ESMs have addressed this issue by using semi-implicit [14, 53] and sub-cycled explicit methods [41], where the fastest resolved atmospheric features reside (e.g. gravity waves and diffusion in the case of the atmosphere). They enable the faster time scale portions to be solved separately so that other parts operating on longer time scales, e.g. cloud physics and tracer transport, can be solved with larger time step sizes. However, the coupling of the dynam-ics to other features, either through process or time-splitting, limits the effective temporal error to the order of the coupling scheme, rather than the order of the numerical method. Quanti-fying the errors of each method individually as well as their combination requires substantial analysis. To date, this analysis has only begun and the issue is not well documented in most simulation results.
As for the atmosphere, while the equations governing fluid flow are well-understood, rela-tively less attention has been spent on the numerical implementation of physics paramateriza-tions, and strict numerical convergence criteria are frequently relaxed due to the under-resolved, statistical nature of the system. Lack of rigorous formulation and testing strategies mean that model behavior could be dominated by numerical error. Additionally, the complexity of physical processes means that each scheme is typically developed by a separate team of domain experts and coupling between processes often falls through the cracks. For example, physical processes in version 5 of the Community Atmosphere Model (CAM5) are sequentially split, meaning they are updated in serial using a time step of 30 minutes. This splitting causes large errors, par-ticularly between the condensation scheme and microphysics. Substepping these processes is a crude but effective way to reduce splitting errors [16]. Development of more efficient methods to accurately couple physical processes is an active area of research. Parameterizations often depend on assumptions about temporal and spatial scale that are at odds with the concept of convergence with decreasing mesh and temporal discretizations. As a result, physics pa-rameterizations have seldom been subjected to temporal discretization convergence tests; Wan et al. [57] do this for CAM5 and find a global convergence rate of only 0.4 due primarily to problems within the microphysics scheme. As such, one can only expect modest improvement in numerical solution quality with refinement of the time step. Another active research area is in development of methods that will achieve high convergence rates. Although the example here is the atmosphere, all components within ESMs contain multiple types of time stepping methods, and all face the complexities of multiple scales of resolved, coupled behavior.
Alternatively, work is being pursued on advanced time-split and implicit methods for these problems. HEVI methods address the stringent vertical CFL restriction for non-hydrostatic models [39], while retaining explicit methods in the horizontal plane. Implicit methods lack the restrictive stability requirement of standard explicit methods discussed in§2. The tradeoff is the requirement of solving a large system of nonlinear, algebraic equations within each time step. Effective implicit strategies exploit Newton-Krylov methods for their fast convergence and scalability for large, parallel problems [27, 60]. These methods have been used within atmosphere [15, 63], ocean [42], and ice sheet dycores [29, 24], and show stability with large time steps and accuracy at the order of the discretization scheme. The Krylov method, however, often requires custom preconditioning for efficient performance. Recent work has addressed this issue for shallow water and other simpler climate systems, but preconditioners for the full three
dimensional climate dynamics remain an active area of research [63, 43, 34].
5
Looking Ahead
Further research in multi-scale models will enable new insights through improved simulation of interacting phenomena that impact climate change. For example, cloud radiative feedbacks are the largest sources of uncertainty in simulations of climate response to a known radiative forcing [7, 13]. Much of the uncertainty is due to the multi-scale nature of cloud feedbacks, which involves interactions between radiation, cloud microphysics, turbulence, shallow convection, deep convection, organized cloud systems, and global circulation modes. Examples of multi-scale models have been used to estimate cloud feedback in idealized experiments [62, 61], but further analysis [6] found that the estimate was sensitive to horizontal resolution. A scale-aware multi-scale ESM has the potential to overcome this limitation, but it must be demonstrated. The climate community should conduct coupled and uncoupled experiments with a multi-scale ESM to quantify the cloud feedback and its sensitivity to resolution, with and without the multi-scale treatment of clouds, and quantify the uncertainty of cloud feedbacks with advanced mathematical techniques.
Future investigations will also exploit multi-scale ESM capabilities with the goal of demon-strating scale-awareness, convergence, and parallelism across the hydrostatic to non-hydrostatic transition. A major focus of future work in this area should include processes related to moist convection and a quantification of scale separation assumptions as one moves from more tradi-tional “resolution” configurations of the resolved atmospheric motion field to more high-resolution LES-like configurations. This work also includes using appropriate numerical meth-ods to capture newly resolved processes, and the coupling of cloud physics and radiation to the larger scale flow field needs to be implemented and tested. The scale of these brute-force reference simulations will tax state-of-the-art heterogeneous high-performance computing archi-tectures, requiring close collaboration among applied mathematicians and computational and computer scientists. Large-scale coupling of ESM components is also a rich area of research, although only limited work to understand and address the associated coupling issues exists [30]. Ideas to take advantage of new computing platforms also center around large, parallel ensembles, which can enable a better understanding of the degree of variability around these multiscale behaviors. Strategies to treat the volumes of new I/O and data storage required to process output from these multiscale models, especially when running multiple ensembles, will also be required. Adoption of the resulting research codes in the broader climate commu-nity will require sustained investments to support scientific research, as well as the software ecosystem that will enable simulations on extreme-scale computing architectures.
References
[1] R. Allan and B. Soden. Atmospheric warming and the amplification of precipitation extremes.
Science, 321:1481–1484, 2008.
[2] R. Archibald, K. J. Evans, and A. Salinger. Accelerating time integration for climate modeling using GPUs. Procedia Computer Science, (same volume), 2015.
[3] S. Balay, et. al., PETSc Web page. http://www.mcs.anl.gov/petsc, 2014.
[4] J. Berckmans, T. Woollings, M.-E. Demory, P.-L. Vidale, and M. Roberts. Atmospheric blocking in a high resolution climate model: influences of mean state, orography and eddy forcing. Atmos.
[5] J.A. Biello and A.J. Majda. A new multiscale model for the Madden-Julian oscillation. Journal
of the Atmospheric Sciences, 62(6):1694–1721, 2005.
[6] P.N. Blossey, C.S. Bretherton, and M.C. Wyant. Subtropical low cloud response to a warmer climate in a superparameterized climate model. Part II: Column modeling with a cloud resolving model. J. Adv. Model. Earth Syst, 1, 2009.
[7] S. Bony, R. Colman, V. Kattsov, R. Allan, C. Bretherton, J.-L. Dufresne, A. Hall, S. Hallegatte, M. Holland, W. Ingram, D. Randall, B. Soden, G. Tselioudis, and M. Webb. How well do we understand and evaluate climate change feedback processes? J. Climate, 19:3445–3482, 2006. [8] S.S. Chen, R.A. Houze, and B.E. Mapes. Multiscale variability of deep convection in relation to
large-scale circulation in TOGA COARE. J. Atmos. Sci., 53(10):1380–1409, 1996.
[9] P. Colella, D. T. Graves, D. Modiano, D. B. Serafini, and B. van Straalen. Chombo software package for AMR applications. Technical report, Lawrence Berkeley National Laboratory, 2000. http://seesar.lbl.gov/anag/chombo/.
[10] T. Delworth, et. al., Simulated climate and climate change in the GFDL2.5 High-Resolution Coupled Climate Model. J. Climate, 25:2755–2781, 2012.
[11] J. Dennis, J. Edwards, K. J. Evans, O. Guba, P.H. Lauritzen, A. Mirin, A. St.-Cyr, M.A. Taylor, and P. H. Worley. A scalable spectral element dynamical core for the Community Atmosphere Model. Internat. J. High Perf. Comput. Appl., 26:74–89, 2012.
[12] J. Dongarra, et. al., The international exascale software project roadmap. Int. J. High Perform.
Comput. Appl., 25(1):3–60, February 2011.
[13] J.L. Dufresne and S. Bony. An assessment of the primary sources of spread of global warming estimates from coupled atmosphere-ocean models. J. Climate, 21:5135–5144, 2008.
[14] ECMWF. IFS Documentation - Cy38r1, Part III: Dynamics and Numerical procedures. Technical Report cy38ra, European Centre for Medium-Range Weather Forecasting, June 2012.
[15] K. J. Evans, M. A. Talyor, and J. B. Drake. Accuracy analysis of a spectral element atmospheric model using a fully implicit solution framework. Mon. Wea. Rev., 138:3333–3341, 2010.
[16] A. Gettelman, H. Morrison, S. Santos, P. Bogenschutz, and P. M. Caldwell. Advanced two-moment bulk microphysics for global models. part II: Global model solutions and aerosol-cloud interactions.
J. Climate, 2014.
[17] D. L. Hartmann, M. E. Ockert-Bell, and M. L. Michelsen. The effect of cloud type on earth’s energy balance: Global analysis. Journal of Climate, 5(11):1281–1304, 1992.
[18] M.A. Heroux, et. al., An overview of the Trilinos project. ACM Trans. Math. Soft., 31(3):397–423, 2005.
[19] X.S. Hu, R.C. Murphy, S. Dosanjh, K. Olukotun, and S. Poole. Hardware/software co-design for high performance computing: Challenges and opportunities. In Hardware/Software Codesign
and System Synthesis (CODES+ISSS), 2010 IEEE/ACM/IFIP International Conference, pages
63–64, Oct 2010.
[20] hypre: High performance preconditioners. http://www.llnl.gov/CASC/hypre/.
[21] J. Jia, J.C. Hill, K.J. Evans, G. I. Fann, and M. A. Taylor. A spectral deferred correction method applied to the shallow water equations on a sphere. Mon. Wea. Rev., page In press., 2013. [22] H. Johansen. A higher-order finite volume nonhydrostatic dynamical core with space-time
refine-ment. MSESM Workshop within the Int’l Conf. Comp. Sci., Reykjavic, Iceland, 2015.
[23] H. Johansen and L. Curfman McInnes (co chairs). ASCR workshop on software produc-tivity for extreme-scale science, January 2014. http://www.orau.gov/swproductivity2014/ SoftwareProductivityWorkshopReport2014.pdf.
[24] I. Kalashnikova, R. Tuminaro, M. Perego, A. Salinger, and S. Price. On the scalability of the Albany/FELIX first-order Stokes approximation ice sheet solver for large-scale simulations of the Greenland and Antarctic ice sheets. Procedia Computer Science, (same volume), 2015.
C., 27(1):4–83, 2013.
[26] D.E. Keyes, D.R. Reynolds, and C.S. Woodward. Implicit solvers for large-scale nonlinear prob-lems. J. Phys. Conf. Series, 46:433–442, 2006.
[27] D.A. Knoll and D.E. Keyes. Jacobian-free Newton-Krylov methods: A survey of approaches and applications. J. Comput. Phys., 193:357–397, 2004.
[28] Peter Kogge and John Shalf. Exascale computing trends: Adjusting to the “new normal” for computer architecture. Comput. Sci. Eng., 15(6):16–26, 2013.
[29] P.H Lauritzen, A. A. Mirin, J. Truesdale, K. Raeder, , J.L. Anderson, J. Bacmeister, and R. B. Neale. Implementation of new diffusion/filtering operators in the CAM-FV dynamical core.
Internat. J. High Perf. Comput. Appl., 2011. to appear.
[30] F. Lemarie, E. Blayo, and L. Debreu. Analysis of ocean-atmosphere coupling algorithms : consis-tency and stability. Procedia Computer Science, (same volume), 2015.
[31] F. Li, W.D. Collins, M.F. Wehner, D.L. Williamson, and J.G. Olson. Response of precipitation extremes to idealized global warming in an aqua-planet climate model: Towards a robust projection across different horizontal resolutions. Tellus Series A-Dynamic Meteorology and Oceanography, 63(5):876–883, 2011.
[32] F. Li, W.D. Collins, M.F. Wehner, D.L. Williamson, J.G. Olson, and C. Algieri. Impact of horizontal resolution on simulation of precipitation extremes in an aqua-planet version of Com-munity Atmospheric Model (CAM3). Tellus Series A-Dynamic Meteorology And Oceanography, 63(5):884–892, 2011.
[33] X. Li, J. Demmel, J. Gilbert, L. Grigori, P. Sao, M. Shao, and I. Yamazaki. SuperLU web page. http://crd.lbl.gov/ xiaoye/SuperLU.
[34] P.A. Lott, C.S. Woodward, and K.J. Evans. Algorithmically scalable block preconditioner for fully implicit shallow water equations in cam-se. Comp. Geosci., accepted, 2014.
[35] A.J. Majda. New multiscale models and self-similarity in tropical convection. Journal of the
Atmospheric Sciences, 64(4):1393–1404, 2007.
[36] A.J. Majda and J.A. Biello. A multiscale model for tropical intraseasonal oscillations. Proceedings
of the National Academy of Sciences of the United States of America, 101(14):4736–4741, 2004.
[37] A.J. Majda and R. Klein. Systematic multiscale models for the tropics. J. Atmos. Sci., 60(2):393– 408, 2003.
[38] H. Miura, H. Tomita, T. Nasuno, S. Iga, M. Satoh, and T. Matsuno. A climate sensitivity test using a global cloud resolving model under an aquaplanet condition. Geophys. Res. Lett., 32(19):4, 2005.
[39] R. Nair, L. Bao, and M. Toy. A time-split discontinuous galerkin transport scheme for global atmospheric model. Procedia Computer Science, (same volume), 2015.
[40] T. Nasuno, H. Tomita, S. Iga, H. Miura, and M. Satoh. Multiscale organization of convection simulated with explicit cloud processes on an aquaplanet. J. Atmos. Sci., 64(6):1902–1921, 2007. [41] R.B. et al. Neale. Description of the Community Atmosphere Model (CAM 5.0). NCAR Technical
Note, TN-486+STR, 2010.
[42] C. Newman and D. A. Knoll. Physics-based preconditioners for ocean simulation. SIAM J. Sci.
Comp., 35:S445–S464, 2013.
[43] C. Newman, G. Womeldorff, L. Chacon, and D. Knoll. High-order / low-order methods for ocean modeling. Procedia Computer Science, (same volume), 2015.
[44] M. Norman. Arbitrarily high-order-accurate, hermite WENO limited, boundary-averaged multi-moment constrained finite-volume (BA-MCV) schemes for 1-D transport. MSESM Workshop within the Int’l Conf. Comp. Sci., Reykjavic, Iceland, 2015.
[45] D. Randall, M. Khairoutdinov, A. Arakawa, and W. Grabowski. Breaking the cloud parameteri-zation deadlock. Bulletin of the American Meteorological Society, 84(11):1547–1564, 2003. [46] T. Ringler, M. Peterson, R.L. Higdon, D. Jacobsen, P.W. Jones, and M. Maltrud. A mutliresolution
approach to ocean modeling. Ocean Modelling, 211-232, 2013.
[47] A. Sarje, S. Song, D. Jacobsen, K. Huck, J. Hollingsworth, A. Malony, S. Williams, and L. Oliker. Parallel performance optimizations on unstructured mesh-based simulations. Procedia Computer
Science, (same volume), 2015.
[48] S. Solomon, D. Qin, M. Manning, Z. Chen, M. Marquis, K.B. Averyt, M.Tignor, and H.L. Miller (eds.). Climate Change 2007: The Physical Science Basis. Contribution of Working Group I to
the Fourth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge
University Press, Cambridge, United Kingdom and New York, NY, USA, 2007.
[49] W. Spotz, T. Smith, and I. Demeshko. Aeras: A next generation global atmosphere model.
Procedia Computer Science, (same volume), 2015.
[50] C.H. Sui and K.M. Lau. Multiscale phenomena in the tropical atmosphere over the western Pacific.
Monthly Weather Review, 120(3):407–430, 1992.
[51] C.H. Sui, X. Li, K.M. Lau, and D. Adamec. Multiscale air-sea interactions during TOGA COARE.
Monthly Weather Review, 125(4):448–462, 1997.
[52] SUNDIALS: Suite of nonlinear and differential / algebraic equation solvers. http://computation.llnl.gov/casc/sundials/main.html.
[53] B. Tsydenov, A. Kay, and A. Starchenko. Numerical modelling of pollutant propagation in Lake Baikal during the spring thermal bar. MSESM Workshop within the Int’l Conf. Comp. Sci., Reykjavic, Iceland, 2015.
[54] P. Ullrich and J. Guerra. Exploring the effects of a high-order vertical coordinate in a non-hydrostatic global model. Procedia Computer Science, (same volume), 2015.
[55] P.A. Ullrich. Understanding the treatment of waves in atmospheric models, Part I: The shortest resolved waves of the 1D linearized shallow water equations. Quart. J. Roy. Meteor. Soc., 140, 2014.
[56] B. Van Straalen, P. Colella, D. T. Graves, N. Keen. Petascale Block-Structured AMR Applica-tions Without Distributed Meta-data. Euro-Par Proceedings, Part II. Lecture Notes in Computer
Science 6853, Bordeaux, France, Springer. ISBN 978-3-642-23396-8, 2011.
[57] H. Wan, P. J. Rasch, M. A. Taylor, and C. Jablonowski. Short-term time step convergence in a climate model. J. Adv. Model. Earth Syst., 2015.
[58] H. Weller, T. Ringler, M. Piggott, and N. Wood. Challenges facing adaptive mesh modeling of the atmosphere and ocean. Bull. Amer. Meteorol. Soc., 91(1):105–+, January 2010.
[59] D.L. Williamson. Convergence of aqua-planet simulations with increasing resolution in the Com-munity Atmospheric Model, Version 3. Tellus A, 60(5):848–862, 2008.
[60] C. S. Woodward, D. Gardner, and K. J. Evans. On the use of finite difference matrix-vector products in Newton-Krylov solvers for implicit climate dynamics with spectral elements. Procedia
Computer Science, (same volume), 2015.
[61] M.C. Wyant, C.S. Bretherton, and P.N. Blossey. Subtropical low cloud response to a warmer climate in a superparameterized climate model. Part II: Column modeling with a cloud resolving model. J. Adv. Model. Earth Syst, 1, 2009.
[62] M.C. Wyant, M. Khairoutdinov, and C.S. Bretherton. Climate sensitivity and cloud response of a GCM with a superparameterization. Geophysical Research Letters, 33(6):4, 2006.
[63] C. Yang, J. Cao, and X.-C. Cai. A fully implicit domain decomposition algorithm for shallow water equations on the cubed sphere. SIAM J. Sci. Comput., 32:418–438, 2010.
[64] C. Zarzycki, C. Jablonowski, D. R. Thatcher, and M. Taylor. Assessing the model climatology of a multidecadal variable-resolution global resolution atmospheric general circulation model simula-tion. J. Climate, in press, doi=10.1175/JCLI-D-14-00599.1, 2015.
