Contact: | Jeff Anderson, Tim Hoar, Nancy Collins |
Revision: | $Revision: 2856 $ |
Source: | $URL: http://subversion.ucar.edu/DAReS/DART/trunk/doc/html/Jamaica_diffs_from_I.html $ |
Change Date: | $Date: 2007-04-12 09:53:09 -0600 (Thu, 12 Apr 2007) $ |
Change history: | try "svn log" or "svn diff" |
This document describes the changes from the "Iceland" or "Post-Iceland" DART releases
to the new jamaica release.
A word about MPI: It is very important to remember that all
the MPI routines are in one module -- mpi_utilities_mod.f90.
If you are NOT using MPI - there is a simple equivalent module named
null_mpi_utilities_mod.f90 which is the default.
1. adaptive_inflate_mod.f90: This module was almost completely
rewritten although fundamental operations remain unchanged. The only change to have
direct impact on users is that there is no longer a namelist for this module
(inflation is now controlled through the filter_nml) and inflation restarts
are always in a single file, no matter what the choice of multiple restart
files for state restarts is.
There have been a number of structural and algorithmic changes. First,
the module is now somewhat more object-oriented with an adaptive_inflate_type
introduced that contains all the information about a particular type of
inflation. The adaptive_inflate_type includes: the inflation_flavor
(observation, spatially fixed state space, or spatially-varying state space)
the unit open for inflation diagnostic output, logicals that control
whether an inflation restart is read or written, and a logical that
controls whether the inflation is done with a deterministic or stochastic
algorithm. The file names for the input and output restarts and the
diagnostic file are also included. The mean value and standard deviation
of the inflation for the spatially fixed state space are in the type. Also
a lower bound on the standard deviation and lower and upper bounds on the
mean inflation value are included. All calls to apply or adjust inflation
values now require an inflate_handle of type adaptive_inflate_type.
Several algorithmic improvements were made to attempt to increase the
numerical stability of the inflation algorithms. First, an improved
quadratic solver was added. The rest of the inflation code was restructured
to reduce the loss of precision in computing the mean of an updated
inflation. A minor code change removed the possibility of inflation values
of exactly 1.0 introducing inflation through round-off on some compilers.
Numerical stability for extreme outliers in ensembles was greatly improved.
There is evidence that inflation computations can continue to be sensitive
to numerical round-off when run with (r4) precision instead of (r8). There
may be a need to switch to enforced higher precision for challenging
applications that require the use of (r4) as the default precision (like
some WRF applications).
2. assim_model_mod.f90:
There were several changes to the public interfaces.
First, get_close_states and get_num_close_states no longer exist. This is
part of moving the default ability to find close points out of model_mod and
into the location mod. Second, an additional optional argument,
override_write_format, was added to function open_restart_write. This allows
a caller to force a write in either binary or ascii independent of the namelist
setting.
A number of non-algorithmic changes were made internal to the code. These
include using the utilities_mod routines to check arguments for netcdf calls
and addition of output control for the mpi implementation. More precision was
added in diagnostic output of days and seconds to avoid a problem with WRF.
3. assim_tools_mod.f90:
Large portions of this module were entirely rewritten. The parallel
assimilation algorithm is implemented in the subroutine filter_assim which is
now the ONLY public interface into assim_tools. The new parallel assimilation
algorithm and its associated inflation computations are documented in the
manuscript by Anderson and Collins that is to appear in the
Journal of Atmospheric and Oceanic Technology and in section 22 of
the tutorial. The manual definition of 'regions' no longer exists.
assim_region.csh no longer exists.
The namelist has also seen major changes. Namelist entries do_parallel,
num_domains and parallel_command no longer exist. Namelist entry num_close_threshold
has been renamed as adaptive_localization_threshold and has a default value of
-1 which means no adaptive localization is in use. Namelist entry
print_every_nth_obs is new and requests that a message be printed after every
nth observation is assimilated in order to monitor progress of large test
jobs.
Algorithmically, the obs_increment_???? routines are unchanged but the
ensemble mean and variance are passed in as arguments now to avoid a redundant
computation. The update_from_obs_increment routine also has the mean and
variance for the observed variable input to avoid extra computation while the
correl argument has been made optional and the correlation is only computed
if it is requested in the call.
An additional observation space increment option has been added along with
the supporting subroutine obs_increment_hybrid subroutine. This interface
is not documented at present and should not be used without consulting the
DART development team. Additional support for a class of particle filters is
provided by subroutine update_ens_from_weights. This routine is also
undocumented and should not be used without consulting the DART development
team.
4. types_mod.f90: Types i4, r4, c4 and c8 were added to the public list and additional comment documentation added.
5. cov_cutoff_mod.f90: The interface to comp_cov_factor was modified to add four new arguments, obs_loc, obs_kind, target_loc, and target_kind. These arguments are not used in the default implementation but are made available for applications in which users want to implement more complicated cutoff functions that can be a function of the location and type of both the observation and the state variable.
6. diagnostics/oned/obs_diag.f90: The default value of rat_cri in the namelist was changed from 3.0 to 4.0 to avoid throwing out too many observations as outside the expected range. Observation-space diagnostics have been extended to handle the fact that inflation can happen in the prior or posterior (by keying on the new dart_qc variable). The new dart_qc values were used to determine which observations were assimilated for posterior and prior estimates and a histogram of the qc values is output. The namelist variable qc_threshold was renamed input_qc_threshold to further distinguish the fact there are now two types of QC values: dart_qc and whatever comes with the observation to begin with.
7. diagnostics/threed_sphere/obs_diag: Observation-space diagnostics have been extended to handle which observations were assimilated for posterior and/or prior estimates (by keying on the new dart_qc variable). The namelist variable qc_threshold was renamed input_qc_threshold to further distinguish the fact there are now two types of QC values: dart_qc and whatever comes with the observation to begin with.
8. ensemble_manager_mod.f90: This module has been almost
entirely rewritten for the mpi implementation. Its continues to manage ensemble
storage but adds a general transpose capability that is central the the parallel
implementation.
The namelist retains the single_restart_file_in and single_restart_file_out
but drops the in_core entry and adds perturbation_amplitude which controls
the standard deviation of perturbations added to the state when starting
from a single state estimate and generating a default ensemble. See the html
documentation for details on the new interfaces.
9. filter.f90: There are major changes throughout
filter for the mpi implementation. Foremost is the fact that the async=3
option no longer exists, which removes the need for filter_server.csh;
nor do manually-defined 'regions' exist, which removes the need for
assim_region.csh.
Changes to namelist: for full details see the html documentation for filter
and the tutorial documentation for adaptive inflation. The namelist can be
divided into two parts, a new part that controls inflation (previously done in the
now defunct adaptive_inflate.nml) and an old part that controls other aspects of
the assimilation. In the old part, the namelist entries output_state_ens_mean,
output_state_ens_spread, output_obs_ens_mean and output_obs_ens_spread have
been removed. Mean and spread are now always output. New entries
first_obs_days, first_obs_seconds, last_obs_days and last_obs_seconds have
been added. These specify the time of the first and last observation in the
obs_sequence file that are to be used. The default values of -1 indicate that
these are to be ignored (see html documentation). The entry input_qc_threshold
has been added: observations with an associated qc field in the input obs_sequence
that is larger than the threshold are not assimilated. The entry
output_forward_op_errors (default false) has been added: it outputs a detailed
file containing a list of all failed forward observation operators in the
assimilation when true. The entry output_timestamps has been added: when true
it generates diagnostic ouput bounding each model advance call in filter.
The inflation portion of the namelist is divided into two columns, the first
controlling prior inflation and the second controlling posterior inflation.
Details of these controls can be found in the html documentation. They are
related to the defunct namelist values that were found in adaptive_inflate_mod.nml
in the post-I release which supported only prior inflation.
A number of other internal details were changed. The main program is now a
single statement that calls a subroutine, filter_main. This allows for better
memory management and avoids lots of shared 'module' storage from the main
program. In order to support a fully modular smoother capability, the
subroutine filter_state_space_diagnostics has been moved to smoother_mod.f90
and smoother_mod is used by filter. The arguments for all previously existing
internal subroutines have been modified and are described in the html
documentation.
10. integrate_model.f90: The namelist integrate_model.nml no longer exists. The old namelist had entries for the input and output files, but they are now hard-coded to 'temp_ic' and 'temp_ud'. The target time for the advance is now taken directly from the input file and the old namelist entries for target_time_days and target_time_seconds are no longer needed. It is essential that integrate_model be compiled WITHOUT the parallel mpi_utilities.
11. oned/location_mod.f90: location modules are now
responsible for providing interfaces to (efficiently) find a set of locations
that are close to a given location. The new and modified public interfaces
are: get_close_obs,
get_close_obs_destroy,
get_close_maxdist_init, and
get_close_obs_init. In addition, a new type, get_close_type, is defined
to store information that helps to do this type of search efficiently given
a fixed set of locations to check. The oned location_mod doesn't have a need
to do this efficiently, so these new interfaces are generally just stubs. The
old get_close_obs still works as before to find locations that are within
a certain distance of a given location.
The get_dist interface has been changed to include two new arguments, kind1
and kind2. These are the kinds associated with the two locations. These
arguments are not used in the default implementation of get_dist, but are made
available for users who want to define distances using not only the location
but also the kinds.
12. threed_sphere/location_mod.f90:
The location module is now primarily responsible for the efficient search for
close states / obs. A series of new interfaces have been added, along with
namelist modifications, to support finding a subset of locations that are close
to a single given location. This can be used both for get_close_obs and
get_close_state computations and replaces the get_close_state interfaces that
were in the model_mod.
A new type, the get_close_type, is defined in a partially object-oriented
fashion. For the threed_sphere, the algorithm works by partitioning the surface
of the sphere using a longitude/latitude equally-spaced grid. This grid
divides the sphere's surface into a set of nlon by nlat boxes. The first step
in the efficient search computes the minimum distance between points in boxes
that are separated by a given number of boxes in latitude and in longitude.
This is accomplished by the new interface get_close_maxdist_init. This routine
also accepts a cutoff radius and keeps a list of all box offsets from a box
at a given latitude that are possibly within the radius. The second step takes
a list of locations and places them into the appropriate boxes. This is
performed by new interface get_close_obs_init. Finally, interface get_close_obs
finds all observations that are less than the cutoff distance from a single
input location and returns their indices in the original list of locations,
along with the distance between them and the single base location if
requested. An interface, get_close_obs_destroy, is also provided to destroy
an instance of the get_close_type.
Three new namelist entries control the number of boxes used in the search,
nlon and nlat, and allow for detailed diagnostic output of the performance
of the close search, output_box_info. The public interface print_get_close_type
is also provided for debug and diagnostic use.
The get_close_obs algorithm only partitions a subset of the sphere's surface
into boxes if the input set of locations is confined to a small region of the
surface. The algorithm works most efficiently when the average number of
locations in a box is small compared to the total number of locations being
searched but large compared to 1. Additional guidance in tuning the nlon and
nlat control over the number of boxes is available from the DART development
team.
The get_dist interface has been changed to include two new arguments, kind1
and kind2. These are the kinds associated with the two locations. These
arguments are not used in the default implementation of get_dist, but are made
available for users who want to define distances using not only the location
but also the kinds.
13. model_mod.f90: Unfortunately, there are minor
changes to the model_mod
public interfaces required to work with the switch to using the location_mod
to find close locations. The public interface model_get_close_states is no
longer required. Three new interfaces:
get_close_maxdist_init,
get_close_obs_init, and
get_close_obs are required to use the new location
module. In a minimal implementation, these three interfaces can just be
satisfied by using the interfaces of the same name in the location module.
The models/template/model_mod.f90 demonstrates this minimal implementation.
Large models can implement their own modified algorithms for the get_close
interfaces if desired for efficiency or correctness purposes. Examples of
this can be seen in the model_mod.f90 for cam or wrf.
An additional new interface, ens_mean_for_model has also been added. This
routine is used to pass the ensemble mean state vector into model_mod by
the filter before each assimilation step. This allows the model_mod to save
this ensemble mean state if it is needed for computing forward operators
as in some large atmospheric models (see cam). For low-order models, this
interface can be a stub as shown in the template/model_mod.f90.
14. PBL_1d/create_real_network.f90 originated from create_fixed_network. It uses module_wrf to get obs from smos file, with file, date, and interval controlled via the wrf1d namelist. Note that an obs_def is still required to control which obs are actually written out. Normally, this would be created with create_obs_sequence. This would be run in place of both create_fixed_network and perfect_model_obs.
15. cam/model_mod.f90:
model_mod can now automatically handle the eulerian and finite volume CAMs
(and should handle the Semi-Lagrangian core too), both single threaded
and MPI-enabled. The latter enables efficient use of more processors than
there are ensemble members. This situation is becoming more common, since DART can now
assimilate using smaller ensembles, and massively parallel machines are becoming more
common. This new mode of running jobs
(async=4) replaces the async=3 and requires only 2 scripts instead of 4.
The multi-core capability required reorganizing the state vector, so new filter_ic files
will be necessary. These can be created, from the CAM initial
files for the relevant dynamical core, using program trans_sv_pv_time0.
The namelist has changed:
state_names_pert has been replaced by pert_names, pert_sd and pert_base_vals
as described in the cam/model_mod.html page.
highest_obs_pressure_mb, which prevents obs above this height to be assimilated,
has been joined by 2 other parameters. Observations on heights (or model
levels) will have their vertical location converted to pressure, and be
restricted by highest_obs_pressure_mb.
highest_state_pressure_mb damps the influence of all obs on state variables
above this height.
max_obs_lat_degree restricts obs to latitudes less than this parameter,
which is needed by some GPS observation sets.
If more fields from the CAM initial files are to be added to the state vector,
it may be necessary to add more 'TYPE's in model_mod, and more 'KIND's in
DART/obs_kind/DEFAULT_obs_kind_mod.F90, and possibly a new obs_def_ZZZ_mod.f90.
There is a new program, trans_pv_sv_pert0.f90, which can be useful in parameterization
studies. It takes a model parameter, which has been added
to the CAM initial files, and gives it a spread of values among the filter_ic
files that it creates.
16. wrf/model_mod.f90: several researchers had their own subtly-different versions of WRF model_mod.f90. These versions have been integrated (assimilated? ;) into one version. The new version performs vertical localization, support for soil parameters, and a host of other features. Hui Liu (DAReS), Altug Aksoy, Yongsheng Chen, and David Dowell of NCAR's MMM group are extensively using this model.
17. DEFAULT_obs_def_mod.F90: A new public interface, get_obs_def_key, was added to return the integer key given an obs_def_type. The interface to read_obs_def has an additional intent(inout) argument, obs_val. This is NOT used in the default implementation, but is required for the implementation of certain special observations like radar reflectivity.
17a. obs_def_radar_mod.f90: added nyquist velocity metadata field to radial velocity.
17b. obs_def_QuikSCAT_mod.f90: New module for the SeaWinds instrument (on the QuikSCAT satellite) data as available in the NCEP BUFR files.
17c. obs_def_reanalysis_bufr_mod.f90: Added land surface and ATOVS temperature and moisture values.
17d. obs_def_GWD_mod.f90: module to define 'observations' of gravity wave drag that are needed to perform parameter estimation studies.
18. DEFAULT_obs_kind_mod.F90: Added in several new raw variable types including KIND_CLOUD_LIQUID_WATER, KIND_CLOUD_ICE, KIND_CONDENSATION_HEATING, KIND_VAPOR_MIXING_RATIO, KIND_ICE_NUMBER_CONCENTRATION, KIND_GEOPOTENTIAL_HEIGHT, KIND_SOIL_MOISTURE, KIND_GRAV_WAVE_DRAG_EFFIC, and KIND_GRAV_WAVE_STRESS_FRACTION.
19. obs_model_mod.f90: There were major internal changes to this routine which implements the shell interface advance of models that is used in the new async = 2 and async = 4 advance options. The public interfaces also changed with the old get_close_states being removed and the advance_state interface being made public. The advance_state interface appears the same except that the intent(in) argument model_size no longer exists. The advance_state interface allows the model to be advanced to a particular target_time without going through the move_ahead interface that uses the observation sequence to drive the advance.
20. merge_obs_seq.f90: This routine is now MUCH faster for both insertions and simple 'appends' and can now handle multiple input files. Conversion between ASCII and binary formats (even for a single file) is now supported. Sorting by time and the removal of unused blocks of observations is also possible.
21. obs_sequence_mod.f90:
The obs_sequence_mod presents abstractions for
both the obs_sequence and the obs_type.
For the observation sequence, public interfaces delete_seq_head,
delete_seq_tail, get_next_obs_from_key and get_prev_obs_from_key were
added. Given a time and a sequence, the delete_seq_tail deletes all
observations later than the time from the sequence and delete_seq_head
deletes all observations earlier than the time. Given a sequence and an
integer key, get_next_obs_from_key returns the next observation after the
one with 'key' and get_prev_obs_from_key returns the previous observation.
A bug in get_obs_time_range that could occur when the entire time range
was after the end of the sequence was corrected. A bug that occurred when
the only observation in a sequence was deleted with delete_obs_from_seq was
corrected.
For the obs_type section, public interfaces replace_obs_values and replace_qc
were added. These replace the values of either the observations or the qc
fields given a sequence and a key to an observation in that sequence. The
interface read_obs had an optional argument, max_obs, added. It allows error
checking to make sure that the maximum storage space in the sequence is not
exceeded during a read. The value of the observation is now passed as an
argument to read_obs_def where it can be used to make observed value
dependent modifications to the definition. This is only used at present
by the doppler velocity obs_def_mod when it does unfolding of aliased
doppler velocities.
22. perfect_model_obs.f90: There were major internal changes to be consistent with the new ensemble_manager_mod and to use a one-line main program that calls a subroutine to avoid lots of shared storage. The namelist has 4 additional arguments, first_obs_days, first_obs_seconds, last_obs_days and last_obs_seconds. These specify times before which and after which observations in the input obs_sequence should be ignored. The default value is -1 which implies that all observations are to be used.
23. random_nr_mod.f90: Converted to use digits12 for real computations to avoid possible change in sequences when reduced precision is used for the r8 kind defined in types_mod.f90.
24. random_seq_mod.f90: Interface init_random_seq was modified to accept an additional optional argument, seed, which is the seed for the sequence if present.
25. utilities_mod.f90:
Several modules had duplicate netCDF error checking
routines; these have been consolidated into an nc_check() routine in the
utilities module. A new set_output() routine can control which tasks in a
multi-task MPI job write output messages (warnings and errors are written
from any task). The default is for task 0 to write and all others not to.
A routine do_output() returns .true. if this task should write messages.
This is true by default in a single process job, so user code can always
safely write: if (do_output()) write(,) 'informative message'
In an MPI job only task 0 will return true and only one copy of the message
will appear in the log file or on standard output.
In an MPI job messages written via the error_handler() will prefix the
message with the task number. The initialize_utilities() routine now
takes an alternative log filename which overrides the default in the
input.nml namelist; this allows utility programs to select their own
separate log files and avoid conflicts with other DART programs.
The MPI initialization and finalize routines call the utility init
and finalize routines internally, so programs which use the MPI utilities
no longer need to initialize the utilities separately.
26. mpi_utilities_mod.f90:
A new module which isolates all calls to
the MPI libraries to this one module. Includes interfaces for sending and
receiving arrays of data, broadcasts, barriers for synchronization,
data reduction (e.g. global sum), and routines for identifying the
local task number and total number of tasks. Also contains a block
and restart routine for use with the async=4 mode of interacting with
a parallel MPI model advance. Programs using this module must generally
be compiled with either an MPI wrapper script (usually called mpif90)
or with the proper command line flags. Some MPI installations use an
include file to define the MPI parameters, others use an F90 module.
If the mpi_utilities_mod does not compile as distributed search the
source code of this module for the string 'BUILD TIP' for more
detailed suggestions on getting it to compile.
When using MPI the call to initialize_mpi_utilities() must be made as
close to the start of the execution of the program as possible, and
the call to finalize_mpi_utilities() as close to the end of execution
as possible. Some implementations of the MPICH library (which is
common on Linux clusters) require that MPI be initialized before
any I/O is done, and other implementations (SGI in particular) will
not allow I/O after MPI is finalized. These routines call the
normal utilities init and finalize routines internally, so at the
user level only the mpi versions need to be called.
27. null_mpi_utilities_mod.f90: A module which has all the same entry points as the mpi_utilities_mod but does not require the MPI library. A program which compiles with this module instead of the real MPI utilities module can only be run with a single task since it cannot do real parallel communication, but does not require the MPI libraries to compile or link. This is the default module -- you cannot simultaneously use both the mpi_utilities_mod and the null_mpi_utilities_mod.
28. mkmf/mkmf: The mkmf program takes a new -w argument. If specified, the resulting makefile will call 'wrappers' for the fortran compiler and loader. The default compiler and loader are $(FC) and $(LD); with the -w flag they will become $(MPIFC) and $(MPILD). In the mkmf.template file you can then define both the MPI wrappers (generally 'mpif90') and the regular F90 compiler.
29. mkmf.template.*: The mkmf.template files have been consolidated where possible and heavily commented to reflect nuances of implementations on different comiler/OS combinations. The habit of appending individual platform names (which led to file creep) is hopefully broken.
30. input.nml: All the default input.nml namelists are now easily 'diff'ed against the corresponding input.nml.*_template files. This enhances the ability to determine what values are different than the default values.
31. DART/shell_scripts/DiffCVS_SVN:
is a new script that identifies differences in two parallel instantiations of
source files. I used it so many times during the migration from CVS to SVN
that I decided to add it to the DART project. This script compares all the
source files in two directories ... after removing the trivial differences
like the form of the copyright tags (its a new year, you know) and the
fact that SVN has different revision numbers than CVS.
If you set an environment variable XDIFF to a graphical comparator like
'xdiff' or 'xxdiff' it is used to display the differences between the files.
With no arguments, a usage note is printed.
The hope is that you can use it see on your 'old' sandboxes in the directories
where you have modified code and compare that to the new code to see if
there are any conflicts. The directories do not have to be under
CVS or SVN control, by the way.
32. models/PBL_1d/src/*: These source files are 'directly' from other developers and had file extensions that required the use of special compilation flags. There is now a script PBL_1d/shell_scripts/ChangeExtensions.csh that not only changes the file extensions (to something the compilers understand i.e. F90) it also modifies the path_names_* files appropriately. The original files had an extension of .F even though they used F90-like formatting. .F is generally understood to mean the contents of the file abide by the F77 fixed-format syntax ... columns 2-5 are for line numbers, column 7 is a line-continuation flag ... etc. Now if we can only get them to not rely on the 64bit real autopromotion ...
33. DART/matlab:
The matlab scripts have experienced no major overhauls, but do handle
a few more models than previously. The next release is scheduled to
have full matlab support for CAM, and WRF. The ReadASCIIObsSeq.m function
has a couple (backwards compatible) tweaks to accomodate some changes
in R2006a, R2006b.
The biggest change is the addition of
plot_observation_locations.m,
which facilitates exploring observation locations (by type) for
any 3D (i.e. real-world) observation sequence. This ability is derived by
running obs_diag with a namelist variable set such
that a matlab-readable dataset is written out.
34. models/ikeda: There is a whole new model - courtesy of Greg Lawson of CalTech. A nice 2-variable system that does not require any fancy time-stepping routines. Thanks Greg!
35. obs_def_dew_point_mod.f90: implements a more robust method (based on Bolton's Approximation) for computing dew point.
1. assim_tools/assim_region.f90: new algorithms and methodologies fundamentally replace the need to manually define and then assimilate regions. Think of all those intermediate files that are not needed!
2. filter_server* are no longer needed because async == 3 is no longer supported (again - replaced by MPI).
3. scripts that advance 'ensembles' are all gone - again because of the MPI implementation. The only script now needed is 'advance_model.csh'.
4. smoother/smoother.f90: The standalone smoother program has become a module and the functionality is now part of the filter program.