# DART software - Copyright UCAR. This open source software is provided
# by UCAR, "as is", without charge, subject to all terms of use at
# http://www.image.ucar.edu/DAReS/DART/DART_download
#
# DART $Id$

This README describes how to use the scripts in this directory
to set up a CESM (active atmospheric component) assimilation.

This includes changes to CESM and DART scripts which enable 
multiple CAM forecasts and assimilations ("cycles") in a single 
LSF job on yellowstone.  This reduces the time spent waiting 
in the queue by a large percentage.  It also moves the short 
term archiver tasks into a separate, single-task job, which 
saves ~30% of the core hours used in the standard workflow mode 
and saves even more by eliminating most of the copies that are 
in st_archive.csh.  

The "traditional", single-cycle-per-job way of running CESM+DART is to
1) Build the DART executables in ../work, as described elsewhere.
2) Set assimilation parameters in ../work/input.nml according to your needs,
   as described elsewhere.
3) Set set-up parameters in CESM{version}_setup_hybrid and locate the files
   referenced in that script (obs_seq, initial ensemble, ...),
   as described in that script.
4) Run CESM{version}_setup_hybrid interactively to set up the 
   $CASEROOT and $RUNDIR directories and put/modify all the 
   necessary scripts and other files there.
5) Follow directions printed at the end of the CESM_2_1_setup_hybrid run.
6) Run $CASE.submit to submit a single cycle job with 
      CONTINUE_RUN = FALSE
7) Change 
      CONTINUE_RUN = TRUE
      RESUBMIT = as many cycles as desired.
   $CASE.run will resubmit itself to the queue RESUBMIT times.
8) Short- and/or long-term archiving may be done at the end of each cycle,
   as set in env_run.xml when any job finishes.

The new, multicycle way of running CESM+DART uses a different 
workflow starting with 7).  Instead of $CASE.run resubmitting 
itself to the queue RESUBMIT times, the variant script 
$CASE.run_cycles will recursively run itself RESUBMIT times 
within a single job.  RESUBMIT will be decremented as usual
at the end of each assimilation cycle, and will be -1 at the end
of the job.

There is a new submit script as well; $CASE.submit_cycles 
("submit_cycles.csh" in this directory),
which is run interactively in $CASEROOT, just like 
$CASE.submit in the standard CESM workflow.
It can submit any number of multicycle jobs ($CASE.run_cycles), 
each dependent on the previous one.  These are submitted in groups,
separated by single processor archiving jobs, 
to prevent the disk from filling up.  
The user organizes this series of jobs in $CASE.submit_cycles by 3 parameters.
   RESUBMIT = the number of cycles which will fit in the wall clock limit, minus 1
   jobs_loop = the number of multi-cycle jobs which can be run
               before the disk fills up
   archive_loop = the outermost loop; the number of times archive_cycles.csh
                  will be run
So the total number of assimilation cycles will be
($RESUBMIT + 1) * jobs_loop * archive_loop.
So,
  7) Edit $CASE.submit_cycles 
  8) Run $CASE.submit_cycles interactively



Another way to describe the multi-cycle workflow follows.
After the new CESM{version}_setup_cycles and CESM_DART_config scripts 
set up the case, the calling tree of these scripts is
-> $CASE.submit_cycles                  (run this interactively in $CASEROOT)
   -> $CASE.run_cycles #1               (created by CESM{version}_setup_cycles)
      -> the next $CASE.run_cycles      (NOT as a batch job; a recursion)
         ... repeats until RESUBMIT has been reduced to 0.

  [-> $CASE.run_cycles #2,....]         (after previous set of $CASE.run_cycles finishes)
                                        (Resets RESUBMIT to initial value)
   -> archive_cycles.csh                (waits for the series of $CASE.run_cycles)
      archives selected restarts
      archives selected history files
      -> lt_archive.sh -m copy_dirs_hsi

   -> $CASE.run_cycles #($jobs_loop +1)   (waits until the archive_cycles.csh is done)
      ... 


# <next few lines under version control, do not edit>
# $URL$
# $Revision$
# $Date$
