These scripts are those executed using the async = 3 option in the namelist input.nml.
They are designed for use on a multi-processor machine, and have been tested for
a batch submission environment (PBS/qsub).
The strategy of this set of scripts for doing an assimilation follows.
The functionality of each has been restricted to one domain; a script
is specific to only a machine where the experiment is run, to a model used
in the assimilation, to the filter version, or to the experiment being conducted
using the choices for the previous 3.
Go to cam/model_mod page
The calling tree for these scripts (and fortran executables) is:
SCRIPT | DOMAIN specific | LOCATION |
---|---|---|
job.csh | experiment | experiment central directory where I/O and execution is organized. |
-> qsub filter.csh |
filter version | local disc on a compute node/processor. |
-> filter executable |
||
-> qsub filter_server.csh |
machine | central directory |
-> advance_model.csh |
model | each ens member on a separate *node* using local disc there. |
-> trans_time executable |
||
-> trans_sv_pv executable |
||
-> run_pc.csh |
model | part of model package |
-> mpirun of forecast model |
||
-> trans_pv_sv executable |
||
-> assim_region.csh |
filter version | each region on a separate *processor* using local disc |
-> assim_region executable |
filter version |
The tasks of each script, and communication between them, are summarized in a flow chart . Note the key in the lower right corner. The specific mechanisms for transferring control and moving files will be found in the scripts.
Contents of the files which appear in the script/data file flow charts.
FILE | CONTENTS or PURPOSE |
---|---|
assim_model_state_ic# | the state vectors to be used as initial conditions for the next model advance. Contains the state vector time, the target time, and the state vector. |
assim_model_state_ud# | the updated state vectors returned by the model advance. Contain the state vector time (was the target time) and the state vector for one ensemble member. |
filter_assim_region__in#s | each contains the same region of all of the state vector ensemble members combined into one file, for the several regions to be used in the assimilation. |
filter_assim_region_out#s | same as filter_assim_regions__in#s, but after the corresponding observations have been assimilated. |
filter_ic_old#s | the initial conditions to be used by the filter for the next assimilation of a single obs_seq.out file. There may be one of these, or one for each ensemble member, named filter_ic_old.####, where the #### means a 4 digit number such as 0001. |
filter_ic_new#s | same as filter_ic_new#s, except that it/they are produced at the end of the assimilation, for use by the next assimilation. |
go_advance_model | the semaphore file to tell the models to advance the state vectors. |
go_assim_regions | the semaphore file to tell assim_region to assimilate the observations (in the form of filter_assim_region__in#s). |
go_end_filter | the semaphore file telling all the scripts that filter is done with the current obs_seq.out file. |
input.nml | the filter namelist file, containing the namelists for all the necessary modules of the filter. |
model initial file | such as caminput.nc, provides information about the model which the filter needs, such as state vector size, etc. |
namelists | the forecast model may need namelist(s) to define its advance. |
obs_seq.final | the innovations in observation space which result from the assimilation of all the chosen obs in obs_seq.out. |
obs_seq.out | the set of observations to be assimilated. How the observations are distributed in time defines when the model advances happen. |
Posterior_Diag.nc | the state vector in model space after each assimilation defined by the obs times in obs_seq.out. |
Prior_Diag.nc | the state vector in model space before each assimilation defined by the obs times in obs_seq.out. It results from the previous. |
rm_filter_temp | semaphore file telling filter.csh that the filter_ic_new#s have been successfully copied to the Central directory, so it is safe to remove its temporary directory and exit. |
state shells | CAM has more fields in its initial files that we use in the DART state vector. It's useful to carry these fields along from advance to advance so that they don't need to spin-up as much at the beginning of each advance. trans_sv_pv replaces the state vector fields in these "shells" with the contents of assim_model_state_ic and leaves the other fields alone. |
Organization of output directories created up by job.csh
DIRECTORY | CONTENTS and PURPOSE |
---|---|
Central directory |
(location of scripts and pass-through point for files during execution) (typically named according defining characteristics of a *set* of experiments; resolution, model, obs being assimilated, unique model state variables, etc.) |
Experiment |
(location of subdirectories of output and some diagnostics files. Typically where the obs-space diagnostics are calculated; new_obs_diag) |
Obs_seq subdirectory(s) |
Each holds the obs-space and model-space output from one obs_seq.out file. Typically named according to the date of the obs_seq.out file. new_obs_diag wants to see a 5 character name, i.e. 01_01 for Jan 1. |
DART |
holds the filter restart files (named filter_ic[.#]) created at the end of the filter run for this obs_seq.out. They're used by the next obs_seq.out file to restart. |
CAM |
holds the CAM initial file shells which carry along model fields which are not DART state vector fields (preventing the repeated re-spin-up of those variables) |
CLM |
Same as CAM, but for Community Land Model initial files. |
A typical pathname for a restart file in my case would be: /scratch/cluster/raeder/T21x80/Taper1/01_03/DART/filter_ic | | | DART restart file directory | | Obs_seq (Jan 3) | Experiment (reduced influence of obs above 150 hPa) Central directory (resolution x num_ens_members)
Instructions for setting up a DART-CAM assimilation using these scripts.
&camexp ncdata = 'caminput.nc' caseid = 'whatever_you_want' nsrest = 0 calendar = 'GREGORIAN' inithist = 'ENDOFRUN' / &clmexp finidat = 'clminput.nc' /and not containing ...
> nhtfrq = 4368 > start_ymd = 20020901 > start_tod = 0 > stop_ymd = 20021201 > stop_tod = 0The CAM build-namelist script will use this to make a new namelist with the correct forecast parameters, named 'namelist'.
Use the ensemble size, available compute nodes, and processors/node to figure how many
nodes to request. Make this request in filter_server.csh. For example, suppose there
are 8 nodes
available for long jobs, with 2 processors
per node. Typical ensembles are 80 members, so it's most efficient to request 8 nodes.
This will advance CAM in 5 batches of 16 (1 CAM/processor). Then set the number of domains in
the assimilation to be num_nodes x procs/node (16), since each domain will execute on a
single processor.
Modify and use script 'check_model ens_size' at the end of each model
advance to quickly learn whether all of the model advances succeeded.
Run job.csh as 'nohup ./job.csh >& /dev/null &', to protect the job from being cutoff
by the closure of the window in which it was executed.
Modify and use script 'lslocal user_name' to see what temporary
directories exist on the compute nodes. Use 'rmlocal user_name' to remove those after
the program has crashed and any information desired has been obtained from them.
Otherwise the temporary directories will eventually fill up the local discs.
Modify and use alias 'rmtemp' to remove the temporary files from
the central directory where the experiment is run, before running another experiment.
alias rmtemp 'rm *_ud* *_ic[1-9]* cam_*_temp* c[al]minput_[1-9]*.nc filter_assim_region_* \ *control filter_ic_old* obs_seq.out times'Needless to say, be careful that you don't name files you want to keep in such a way that they'll be deleted by this.
Space requirements (per ensemble member) for several CAM resolutions.
Resolution | filter_ic | CAM initial | CLM initial | Diagnostic |
---|---|---|---|---|
T5 | .16 Mb | .3 Mb | .15 Mb | 1.3 Mb + obs_seq.final |
T21 | 2.5 Mb | 4.5 Mb | 1.4 Mb | 21. Mb + obs_seq.final |
T42 | 10. Mb | 18. Mb | 4.5 Mb | 57. Mb + " |
T85 | 41. Mb | 74. Mb | 15. Mb | 342 Mb + " |