In our interactions with climate scientists, we have focused our efforts in two main areas. First TSTT scientists at ORNL have been working on strategies to create high-quality meshes whose grid points are focused over regions of interest, in this case, over regions of high altitude in the world. Second, joint work between scientists at NCAR and at ANL has resulted in a new preconditioner for spectral element simulations based on low-order finite element discretizations.

Smooth Grid Refocusing

Comprehensive Design and Development of the Community Climate System Model for Tera-scale Computers

TSTT Personnel: Ahmed Khamayseh (ORNL)

Climate Personnel: John Drake (ORNL), Daniel Guo North Carolina at Wilmington, formerly of ORNL)

The accuracy and convergence of computational solutions using mesh-based numerical methods are strongly dependent on the quality of the mesh being used. Our efforts at ORNL and part of TSTT efforts include the development of several algebraic and PDE-based elliptic methods for optimizing meshes that are comprised of elements of arbitrary hybrid polygonal and polyhedral type. These methods provide the ability to generate and focus mesh resolution over areas of particular interest yet strive to equidistribute the node densities of the mesh while improving the aspect ratios and quality of mesh cells. The numerical methods that solve the partial differential equations perform node redistribution on the mesh to maximize the equidistribution of a weighted function of geometric and solution parameters.

Applications of this capability include the generation and adaptation of smooth grid transformations for General Circulation Models (GCMs) that attempt to simulate the Earth's climate system. Mesh adaptation can play a crucial role in atmospheric-ocean-land models that calculate physical quantities such as temperature, humidity, wind speeds which have direct effect on sea ice cover, soil moisture, and cloud formation. In climate modeling, mesh adaptation can also reduce the simulation error in prediction of (1) the dynamics of the climate system that describe the large-scale movement of air masses and transport of energy and momentum; (2) the physics of the climate system such as radiation transmission through the atmosphere, thermodynamics, and evaporation; and (3) other factors such as air-sea interaction, topography, and vegetation parameters.

Our efforts in this area have focused on developing two adaptive methods for climate modeling and simulation. The first method developed is an algebraic method in which the mesh is adapted to reduce the error in the solution while the mesh remains orthogonal. This method works on logically structured meshes where the mesh is first adapted along the boundaries of the domain and then algebraically interpolated into the interior. The second method is based on solving an adaptive elliptic mesh generation system coupled with the climate physical model. These methods are currently utilized at ORNL for the generation of refocused adaptive meshes applied to fine scale processes in climate modeling. In particular, we concentrated our efforts on producing high quality meshes that are adapted to the earth’s orography field, i.e., earth’s surface height. We have generated a variety of meshes (structured and unstructured) with the feature of mesh refocusing in and around areas high altitude without changing the resolution of the initial meshes (see Figure 1). In return, the pay off in computational cost is reduced while achieving the desired simulation results. The climate code would spend much less computational time using adapted optimized mesh to obtain a high quality solution verses using non-adapted mesh.

Figure 1

Figure 1 Meshes adapted to the earth's orography field

Low-Order Discretizations used as Preconditioners

TSTT Personnel: Paul Fischer, Henry Tufo (ANL)

Climate Personnel: Steve Thomas, J. Dennis (NCAR)

Climate and weather models have traditionally been based on spectral methods that exploit the underlying symmetries of spherical geometry by representing the solution as a tensor product of global basis functions. Such methods provide a high degree of accuracy per grid point and, through the use of transforms, make the implicit steps of the underlying PDEs easy to solve. Unfortunately, the transforms, needed for both operator evaluation and inversion, require all-to-all communication on multicomputers, which is a potential bottleneck on systems having limited bisectional bandwidth. Spectral element methods (SEMs), which employ local spectral expansions interior to quadrilateral or hexahedral elements, offer another approach to achieving the high degree of numerical accuracy required for long time integrations typical of climate simulations. The SEM discretization requires only C0 continuity and hence has low communication requirements. The spectral element method has been proven effective in geophysical fluid dynamics (GFD) by Iskandarani et al. (1995), Taylor et al. (1997), and by Thomas and Loft (2002).

The ANL TSTT group has been working with Thomas and Loft at NCAR to further develop semi-implicit spectral element methods for GFD. We have been investigating finite-element- (FE-) and block-Jacobi-based preconditioning strategies for a semi-implicit treatment of the shallow-water equations that is designed to overcome the timestep-size restrictions imposed by fast gravity waves. The FE-based preconditioner exploits the spectral equivalence between the high-order SEM discretization and a low-order FEM discretization based on the same set of interpolation points. We have shown (Thomas et al., 2002) that, at moderately high resolution, the FEM-based approach, coupled with an additive overlapping Schwarz strategy, outperforms block-Jacobi preconditioning, even in large-scale parallel environments. We are currently investigating the potential of a two-level additive Schwarz method, in which the preconditioner is augmented with a coarse-grid problem. The coarse-grid problem is defined by discretizing the original PDE using bilinear interpolants on each spectral element.