

The repository has now been updated to include most of the Hawaii
release. The most important changes from Guam implement scalable,
memory-efficient versions of filtering algorithms. This has led to
a major change in the way that ensembles are stored, accessed,
and advanced by models. In particular, a new module called 
ensemble_manager_mod is now used to handle the storage of ensembles
or single model instances. 

The scalable algorithm allows pieces of the model state vector
(referred to as domains or regions) to be assimilated independently
by a single process or by multiple processes. The algorithm 
implemented in Guam by filter can be retrieved by using only a 
single domain. This is, however, unlikely to bitwise reproduce
for large models.

IMPORTANT: When more than one domain is used the algorithm is
fundamentally different! It is extremely unlikely to bitwise
reproduce the single domain results (although it does so for
some simple low-order model tests) in any large model. It is
also conceivable that this algorithm is numerically unstable for
large observation sets in large models (no evidence of this
has come to light but there is no proof of expected stability
so far). 

For those working with low-order models, I believe that everything
has been updated in the repository to allow you to run immediately.
Please note that the namelists have changed and there are new
namelists for ensemble_manager and assim_tools, so update your 
namelist files.





!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
RECOMMENDED PROCEDURE FOR LARGE MODEL KEEPERS
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

WARNING: Nobody has beta-tested these instructions so they 
are unlikely to be sufficient. Call, e-mail, whatever, and
we'll try to synthesize everybody's experience in moving to
Hawaii (that sounds nice, let's move to Hawaii for the winter).

I have not tried to update things or test for the MIT model, WRF, 
GFS, or ROSE. I recommend the following hierarchical implementation
of Hawaii:

1. Modify your path_names files for filter, perfect_model_obs,
create_obs_sequence, and integrate_model to include the new
file ensemble_manager_mod.f90 in directory ensemble_manager.

2. Compile, update with new namelists, and check. Be sure to use
do_parallel = 0 (no attempt to do assimilation with separate
processes) and num_domains = 1 in the assim_tools_nml. Also,
read the documentation on the new namelist for assim_tools and
be sure you are happy with the default options.

3. Test the parallel capability with a single process by setting
num_domains to something greater than 1. Compare results and
complain if they are not similar to your base results.

3a. In order to do parallel assimilation of regions, you will need to create  
an executable called assim_region. This will require creating the
appropriate mkmf and path_names files. Templates for these files can be
found in models/lorenz_96/work in mkmf_assim_region and
path_names_assim_region. You will have to substitute the paths required
for your model_mod for the path for the lorenz_96/model_mod.

4. Create a new model dependent script called assim_region.csh
(see more notes in details below). If you are not able to use
machine dependent scripts for fisher/dart/linux or for
anchorage/bangkok/calgary, create a machine dependent script
assim_filter.csh (see details below). Set do_parallel = 2
and try this out. You can actually run assim_filter.csh as a
foreground process for testing.

5. If you must, create a machine dependent script called
filter_server.csh (see details below). Set async=3 in your
filter_nml and/or perfect_model_nml. Start your filter or
your perfect_models_obs executable and then start the
filter_server as a separate process in the same working 
directory to do an assimilation.

6. Also set do_parallel=3 in assim_tools_nml and repeat the
test in step 5.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
END RECOMMENDED PROCEDURE FOR LARGE MODEL KEEPERS
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!






LIST OF DETAILED HAWAII DIFFERENCES FROM GUAM

1. assim_model_mod.f90

a. the routines advance_state and Aadvance_state are no longer in this
module. advance_state no longer exists and Aadvance_state is now found
in ensemble_manager_mod with revised arguments.

b. Interface adv_1step is now passed through from model_mod.

c. The way in which restart files are done has changed. There are now
two namelist parameters, read_binary_restart_files and 
write_binary_restart_files that independently control whether
read and written files are in binary or formatted. These are not
public and calling modules do not know what format they are writing.
To implement this, subroutines open_restart_read, open_restart_write,
and close_restart are provided. These return an integer handle to a
file but the opening and closing is done in assim_model_mod. 

d. subroutine aget_initial_condition is added to the public interfaces
and reads initial condition time and state vector (functionality
similar to existing get_initial_condition which took assim_model_type
as an argument.

e. The aget_closest_state_time function was enhanced to provide proper
behavior if timesteps are an odd number of seconds.

f. Subroutine write_state_restart and awrite_state_restart arguments 
have been changed so that file_format no longer needs passed. Optional
argument target time has been added so that restart format files with
an extra 'target time' at the front can be read.

g. Subroutine read_state_restart and aread_state_restart arguments 
changed as for write_state_restart.

h. function open_restart_write, open_restart_read, and close_restart
added to allow read, write and destruction of restart format files.

----------------

2. assim_region

New main program in assim_tools module for use in cluster parallel
applications. This program does filter assimilation for a single region
by calling async_assim_region in assim_tools and using info from 
standard files describing obs sequence and region's state.

----------------

3. assim_tools_mod

aa. Added new systematic error correction algorithm and removed old
algorithms that were accessed by confidence_slope and 
prior_spread_correction namelist options in the filter. The new 
namelist in assim_tools controls the systematic error correction and
details of the standard filter algorithms. See the html file for 
assim_tools for details and be sure to update your namelist file.

a. Added two new public interfaces: filter_assim and async_assim_region

b. Error messages now use utilities module appropriately.

c. Capability to do parallel assimilation of separate regions controlled
by two new three new namelist parameters. do_parallel is an integer
that can take values 0, 2, or 3. It controls whether different
assimilation regions are done by the main process (the filter 
program option 0), but executing the command string found in 
parallel_command (option 2), or by making a request to a filter_server
(option 3). num_domains specifies the number of separate regions to
be used for the assimilation. parallel_command contains the command
to be executed for option 2 (normally ./assim_filter.csh or 
qsub ./assim_filter.csh for now).

d. New subroutine filter_assim_region is where all the filtering
takes place for either a single domain or multiple domains. NEED
DETAILS.

e. New subroutine filter_assim sets up domains for calls to 
filter_assim_region. This is also where the break for asynchronous
execution takes place. If domains are not being done in parallel 
(do_parallel is false) then filter_assim_region is called directly 
for each domain. If do_parallel is 2 or 3, then all the arguments
required for each call to filter_assim_region are written to files
and then the shell is used to either launch a process or to 
create a semaphore file that allows other processes to call
assim_region. Assim_region reads in the files for its domain and
calls filter_assim_region as it would have been called directly 
by subroutine call for do_parallel = 0.

f. subroutine async_assim_region is called by assim_region to read
in all the files and load in the obs_sequence before continuing with
a call to filter_assim_region. It also takes the output from
filter_assim_region and puts it in files for going back to the
main filter program.

----------------

4. assim_tool_mod.nml

Several arguments added. Be sure to avoid using the old namelist. 
See assim_tools_mod.f90 for details.

----------------

5. converters

A new directory containing programs required to do various file 
conversion tasks has been included. Details are available from
Alain and Tim.

----------------

6. ensemble_manager_mod

A new module added to manage storage for ensembles. All other
standard modules that access ensembles now do so by taking a 
handle (an ensemble_type) as an argument. Supports either in core
or on disk storage of ensembles. The out of core option only
requires enough storage for 2 ensemble members at any time.
Routines to get a subset of an ensemble (used for assimilating
regions) and to compute ensemble mean and spread are included.
There is a single namelist parameter, in_core. If set to .true., then
the ensemble is stored in memory in an array. If false, a random
access file is used to store the ensemble.

This change has the most far-reaching impact on the DART system
since many routines accessing ensembles required modification.

----------------

7. filter

Technically, the filter program is no longer supported in the 
H-release and is replaced by the parallel_filter.f90. However,
the filter program will remain in the repository. The changes in
the H-release make the filter program consistent with the changes
to the restart file reading and writing that are described in
assim_model. One could run this filter with the untagged repository
version that immediately predates the initial commit of the H-release
(need to have a date for this or possibly tag it as pre-H). 

----------------

8. integrate_model

H-version uses ensemble_manager to handle storage for ensemble
and uses the assim_model_mod revisions to restart file output
and input. 

----------------

9. integrate_model.nml

The default value of the namelist parameters ic_file_name
and ud_file_name are now "temp_ic" and "temp_ud" consistent
with the parallel script defaults.

----------------

10. Location modules: annulus

a. annulus location model added for use with MIT GCM.

----------------

11. location modules: threed_sphere

a. Module now has a namelist with 4 entries: horiz_dist_only,
vert_normalization_pressure, vert_normalization_height, 
vert_normalization_level. horiz_dist_only is a boolean and is
available as a public interface. If it is true, only horizontal
distances on the surface of the sphere are computed by get_dist.
If it is false, then a three dimensional distance is computed
with the ratio of vertical to horizontal distance being 
normalized by one of the three normalization namelist variables.
The module is backward compatible if horiz_dist_only
is set to true.

b.  Which_vert value of -1 is officially recognized as being
a surface observation.

----------------

12. matlab 

a. plot_ens_time_series.m: additional diagnostic output.

b. ReadBinaryObsSeq.m: input format for obs_seq.final changed.

c. ReadObsSeq.m: new function prepared to read all obs sequence
formats.

----------------

13. mkmf

a. Added mkmf.template.ifort.ghotiol, mkmf.template.lf95.ghotiol,
   mkmf.template.pgf90.ghotiol, mkmf.template.gfs

b. Removed comments from: mkmf.template.Intel-7.1.ocotillo,
   mkmf.template.anchorage 

c. Modified mkmf.template.pgi 

----------------

14. models 

All path_names files are modified to include the ensemble_manager_mod.

A. Shell scripts for models. Models that use shell scripts to run in 
background will need a new script, generally called assim_region.csh.
This script is very similar to advance_model.csh but assimilates
individual regions. Examples are available for the bgrid model and
for the lorenz_96 model. Large model users who want to run parallel
will have to create these scripts.

B. wrf: need to get changes from Alain.  Basically it is entirely
different, for the new WRF version I suppose.

C. rose: model added, see Tomoko for any notes.

D. CAM: CAM3 adapted. Comments from Kevin.

E. MITgcm_annulus: Added by Jim Hansen. See documentation. May not
be H compliant?


----------------

15. ncep_obs 

Moisture and surface pressure observation from NCEP BUFR data
are added to the obs_seq. The diagnostic proprgram is also updated.

----------------

16. obs_kind

a. Added a number of additional observation kinds definitions in 
public interface list.

----------------

17. obs_model

a. Subroutine move_ahead now accepts an ensemble_type as an 
argument instead of an ensemble array and an array of times.

b. For low-order models, the new observation window could have
generated negative times for states starting at time 0 days and
0 seconds; this was fixed.

----------------

18. parallel_filter

This is the new main program replacing the filter. See the
documentation for this program in the H-release html file.

----------------

19. perfect_model_obs

a. Modified to use ensemble_manager for the advance of the ensemble
and for storage. 

b. Modified to use the new assim_model_mod restart
read and write options.

----------------

20. random_nr_mod

a. Removed all uses of double precision and replaced with real(r8).
Shouldn't have any impact on our platforms with standard compile
options.

----------------

21. random_seq_mod

a. Removed all uses of double precision and replaced with real(r8).
Shouldn't have any impact on our platforms with standard compile
options.

----------------

22. shell scripts (global, platform specific)

a. Added filter_server_anchorage.csh, filter_server_filter.csh,
assim_filter_anchorage.csh, assim_filter_fisher.csh.
These scripts are platform specific and allow the parallel running
of regions for assimilation (assim_filter...) or allow a persistent
shell process to serve both model advance and regional assimilation
requests (filter_server...). Those running on platforms other than
fisher/dart (linux) and anchorage/bangkok/calgary may need to 
create their own versions of these two new scripts.

b. adv_ens_anchorage.csh: changed location of temporary directory

c. adv_ens_fisher.csh: removed junk for non-existent queuing system

d. BuildWRF: added error exit

e. RunL63, RunL04, Run9var, RunPE2, RunL96_2scale, RunL96,
   RunL84, MakeICs: updated for new assim_model_mod restart file namelist
   and many small changes.

----------------

23. time_manager

Added subroutine interactive_time to input Gregorian calendar
dates from standard input.

----------------
