Distributed State

DART project logo

Jump to DART Documentation Main Index
version information for this file:
$Id: distributed_state.html 11427 2017-03-31 21:24:30Z nancy@ucar.edu $

The key part of RMA DART is having a state that is physically distributed across processors. The location in memory of any part of the state vector (which processor and where in memory on that processor) is completely under the control of filter, not model_mod.

Implications of this:

So, how does the model_mod access the state without having the vector and not knowing the state order? - state accessor routines.

State Accessor Routines

Getting the dart index
function get_dart_vector_index(i, j, k, dom_id, var_id)

get_dart_vector_index returns the dart index for a given i,j,k of a variable. Note if the variable is 1D j and k are ignored. If a variable is 2D k is ignored. Note only variables upto 3D are supported, but this could be extended to support upto 7 dimensional variables (or whatever fortran and netcdf will support).

Getting the state at a given dart index
function x = get_state(index, state_handle)

get_state returns the state x at the given index. state_handle is a derived type which conatins the state information. state_handle is passed to the model_mod from above. get_state returns an array of values (the whole ensemble at index) during model_mod and a single value (the mean) during get_close_obs or vert_convert.

If you have an array of indices, for example a forward operator which is located in different levels on different ensemble members you can use get_state_array. An example of this is in CAM when an observation is in pressure, the level an observation is in depends on the state and so can vary across the ensemble.

subroutine get_state_array(x(:), index(:), state_handle)

The code inside get_state_array will do the minimum amount of communication to get you the indices you need. For example if

  index = [ 3 4 3 3 4 3]

get_state_array will only do 2 mpi communications and return

  x = [state(3), state(4), state(3), state(3), state(4), state(3)]

A limited module diagram is shown below. A -> B means A uses B:

Filter_mod and assim_tools_mod take care of making data available for use with get_state. Note get_state will only return data during model_interpolate, get_close_obs, or vert_convert. If you use get_state outside these routines you will get and error.

Compliation Notes

The Remote Memory Access programming model we are using uses mpi_windows. There are 2 ways to compile window mods for mpi and non-mpi filter. This is taken care of automatically when you run quickbuild.csh or an mkmf_* with -mpi or -nompi. However, if you use mpi, there is a choice of mpi_window mods:

We have these two modules that you can swap in your path_names files because the MPI 2 standard states:
Implementors may restrict the use of RMA communication that is synchronized by lock calls to windows in memory allocated by MPI_ALLOC_MEM.
MPI_ALLOC_MEM uses cray pointers, thus we have supplied a window module that uses cray pointers. However, no_cray_win_mod.f90 is the default since some versions of gfortran (4.9.0) do not support cray pointers. These different modules will go away when we swap to MPI 3.


Terms of Use

DART software - Copyright UCAR. This open source software is provided by UCAR, "as is", without charge, subject to all terms of use at http://www.image.ucar.edu/DAReS/DART/DART_download

Contact: DART core group
Revision: $Revision: 11427 $
Source: $URL: https://svn-dares-dart.cgd.ucar.edu/DART/releases/Manhattan/documentation/html/distributed_state.html $
Change Date: $Date: 2017-03-31 15:24:30 -0600 (Fri, 31 Mar 2017) $
Change history:  try "svn log" or "svn diff"