

                  Summary of ESMF-Coupled WRF Experiment

                              Tom Henderson
                             John Michalakes
                                NCAR/MMM
                             27 February 2007

INTRODUCTION

This document describes ESMF-enablement and coupling of WRF through ESMF
to another component that simulates coupling to an ocean model. As a
prelude to availability of an ESMF-enabled implementation of the HYCOM
model, WRF has been coupled through ESMF to a very simple "data-ocean"
component named "SST" via a very simple coupler component named "CPL".
The demonstration, conducted on kraken.navo.hpc.mil, a DoD HPC system
at the NAVO MSRC, was supported by AFWA under UCAR contract FA4600-05-P-0162.  
The demonstration was later repeated on NCAR's bluesky machine (IBM p690) and 
again on NCAR's bluevista machine (IBM p575).  A description of the bluesky 
and bluevista experiments follows.  



EVENT LOOP

Please read the section entitled "NOTES ABOUT THE EVENT LOOP FOR WRF+CPL+SST" 
in file external/io_esmf/README.io_esmf for details about order of operations
in the current event loop (time-stepping loop).  



EXPERIMENT DETAILS

  The "WRF+CPL+SST" experiment was conducted on bluesky on 12 October 2006 at 
which time the following software was installed:  
  xlfrte   8.1.1.6
  bos.mp   5.1.0.58
  xlC.rte   6.0.0.0
  ESMF 2.2.0rp1
WRF source code was installed in
WRFDIR=/ptmp/hender/ESMF2.2.0rp1/WRFV2_20061005_1123_WORK/WRFV2/.  
(Referred to below as "$WRFDIR".)  

  The "WRF+CPL+SST" experiment was repeated on bluevista on 27 February 2007 at 
which time the following software was installed:  
  xlfrte   10.1.0.3
  bos.mp   5.3.0.42
  xlC.rte  8.0.0.5
  ESMF     2.2.2r
WRF source code was installed in
WRFDIR=/ptmp/hender/WRF-LIS/WRFV2_20070214_0906/WRFV2/.

  Additional documentation that describes the SST component and provides 
instructions for building WRF with ESMF can be found in 
external/io_esmf/README.io_esmf.  Briefly, the 
SST component simply reads SST data stored in a file (sstin_d01_000000), sends 
it to WRF via CPL, receives SST data back from WRF via CPL, and verifies that 
data received matches data sent.  Since ESMF coupling is implemented within 
WRF via the WRF I/O and coupling API (WRF IOAPI), the experiment can be 
repeated reading the SST data directly into WRF as a netCDF file simply by 
changing the appropriate I/O format in the WRF "namelist.input" file.  This 
feature is illustrated below.  

  The following "recipe" can be used to reproduce the experiment, if desired.  
All commands are csh.  


1)  On bluesky, set environment variables to build WRF with ESMF using 32-bit 
addressing.  On bluevista, use the default 64-bit addressing.  
Set the $ESMFLIB and $ESMFINC environment variables are to tell the WRF build 
automation where to find an installation of ESMF.  

BLUESKY:  
OBJECT_MODE=32
ESMFLIB=/home/bluesky/hender/esmf/lib/libO/AIX.default.32.default
ESMFINC=/home/bluesky/hender/esmf/mod/modO/AIX.default.32.default

BLUEVISTA:  
OBJECT_MODE=64
ESMFLIB=/home/bluevista/hender/esmf/esmf_2_2_2r/lib/libO/AIX.default.64.mpi.default
ESMFINC=/home/bluevista/hender/esmf/esmf_2_2_2r/mod/modO/AIX.default.64.mpi.default



2)  Set up WRF Registry for WRF+CPL+SST case:  

$WRFDIR/Registry >> mv -f Registry.EM Registry.EM_ORIG
$WRFDIR/Registry >> cp -f Registry.EM_SST Registry.EM



3)  Build WRF from scratch with RSL for use with ESMF:  

$WRFDIR >> echo 4 | configure
$WRFDIR >> compile em_real >&! compile.em_real.4.out

  Verify that executables exist:  

$WRFDIR >> ls -1 main/*exe
main/ndown.exe
main/nup.exe
main/real.exe
main/wrf.exe
main/wrf_SST_ESMF.exe

  Note that "wrf.exe" is the usual stand-alone WRF executable.  
"wrf_SST_ESMF.exe" is the WRF+CPL+SST coupled application.  



4)  Go to run directory, unpack the required SST data, namelists and run 
scripts from WRF_CPL_SST.tar.gz, and verify that all required files are
present:   

$WRFDIR/test/em_esmf_exp >> gunzip WRF_CPL_SST.tar.gz
$WRFDIR/test/em_esmf_exp >> tar xvf WRF_CPL_SST.tar
$WRFDIR/test/em_esmf_exp >> ls -l sstin_d01_000000 namelist.input.jan00.* *.csh
-rw-r--r--   6368 Feb 27 13:16 namelist.input.jan00.ESMFSST
-rw-r--r--   6368 Feb 27 13:16 namelist.input.jan00.NETCDFSST
-rw-r--r--   1286 Feb 27 14:51 real.csh
-rwxr-xr-x    948 Feb 27 14:51 real.lsf.csh
-rw-r--r-- 458064 Oct 12 11:58 sstin_d01_000000
-rw-r--r--   1074 Feb 27 14:51 test4_0.csh
-rwxr-xr-x    732 Feb 27 14:51 test4_0.lsf.csh
-rw-r--r--   1162 Feb 27 14:51 test4_0_ESMFSST.csh
-rw-r--r--    824 Feb 27 14:51 test4_0_ESMFSST.lsf.csh
-rw-r--r--   1190 Feb 27 14:52 test4_0_NETCDFSST_wrfexe.csh
-rw-r--r--    824 Feb 27 14:52 test4_0_NETCDFSST_wrfexe.lsf.csh

  "sstin_d01_000000" contains time-varying SST input data.  
NOTE:  sstin_d01_000000 also contains LANDMASK fields soley for validation 
purposes.  Only the first time-level of LANDMASK is significant.  For
historical reasons, the other time-levels may differ -- they are ignored 
(and should eventually be removed).  

  "test4_0_ESMFSST.csh" is a LoadLeveler batch submission script that runs the 
WRF+CPL+SST coupled system.  It copies "namelist.input.jan00.ESMFSST" to 
"namelist.input" and runs "wrf_SST_ESMF.exe" on four CPUs.
"test4_0_ESMFSST.lsf.csh" does the same thing for LSF.  

  "test4_0_NETCDFSST.csh" is a LoadLeveler batch submission script that runs 
WRF by itself, reading the SST data directly into WRF as a netCDF file.  It 
copies "namelist.input.jan00.NETCDFSST" to "namelist.input" and runs "wrf.exe" 
on four CPUs.  
"test4_0_NETCDFSST.lsf.csh" does the same thing for LSF.  

  Note that namelists namelist.input.jan00.ESMFSST and 
namelist.input.jan00.NETCDFSST differ only in the I/O format used for the 
I/O streams used to "read" and "write" SST data.  Vanilla netCDF I/O is used 
when "io_form" == 2.  ESMFStates are used when "io_form" == 7.  

$WRFDIR/test/em_esmf_exp >> diff namelist.input.jan00.ESMFSST namelist.input.jan00.NETCDFSST
32c32
<  io_form_auxinput5                   = 7,
---
>  io_form_auxinput5                   = 2,
37c37
<  io_form_auxhist5                    = 7,
---
>  io_form_auxhist5                    = 2,

  (Also note that a script is provided to run WRF without SST forcing for 
comparison purposes.  "test4_0.csh" is a LoadLeveler batch submission script 
that runs WRF by itself without SST forcing.  It copies "namelist.input.jan00" 
to "namelist.input" and runs "wrf.exe" on four CPUs.  "test4_0.lsf.csh" does 
the same thing for LSF.)  



5)  Run the wrf real program using the "jan00" data set.  Script "real.csh" 
can be modified to do this if desired (or real.lsf.csh for LSF).  

BLUESKY (LoadLeveler):  
$WRFDIR/test/em_esmf_exp >> llsubmit real.csh
BLUEVISTA (LSF):  
$WRFDIR/test/em_esmf_exp >> bsub < real.lsf.csh

  Verify that real.exe produced the usual WRF input and boundary data files 
"wrfbdy_d01" and "wrfinput_d01".  

$WRFDIR/test/em_esmf_exp >> ls -al wrfbdy_d01 wrfinput_d01
-rw-r--r--   9355944 Feb 27 12:58 wrfbdy_d01
-rw-r--r--   6076408 Feb 27 12:58 wrfinput_d01

Move all other files produced by the real.exe run into a new directory:  

$WRFDIR/test/em_esmf_exp >> mkdir real.out
$WRFDIR/test/em_esmf_exp >> mv PET?.ESMF* namelist.input rsl.*.* real.*.err real.*.out real.out



6)  Run the WRF+CPL+SST test case:  

BLUESKY (LoadLeveler):  
$WRFDIR/test/em_esmf_exp >> llsubmit test4_0_ESMFSST.csh
BLUEVISTA (LSF):  
$WRFDIR/test/em_esmf_exp >> bsub < test4_0_ESMFSST.lsf.csh



7)  Verify that run completed successfully:  

$WRFDIR/test/em_esmf_exp >> tail -1 rsl.out.0000
 d01 2000-01-25_00:00:00 wrf: SUCCESS COMPLETE WRF



8)  Move all files produced by the WRF+CPL+SST run into a new directory:  

$WRFDIR/test/em_esmf_exp >> mkdir test4_0_ESMFSST.out
$WRFDIR/test/em_esmf_exp >> mv PET?.ESMF* namelist.input rsl.*.* test4_0_ESMFSST.*.* wrfout* test4_0_ESMFSST.out



9)  Run the WRF stand-alone test case:  

BLUESKY (LoadLeveler):  
$WRFDIR/test/em_esmf_exp >> llsubmit test4_0_NETCDFSST_wrfexe.csh
BLUEVISTA (LSF):  
$WRFDIR/test/em_esmf_exp >> bsub < test4_0_NETCDFSST_wrfexe.lsf.csh



10)  Verify that run completed successfully:  

$WRFDIR/test/em_esmf_exp >> tail -1 rsl.out.0000
 d01 2000-01-25_00:00:00 wrf: SUCCESS COMPLETE WRF



11)  Move all files produced by the WRF stand-alone run into a new directory:  

$WRFDIR/test/em_esmf_exp >> mkdir test4_0_NETCDFSST.out
$WRFDIR/test/em_esmf_exp >> mv PET?.ESMF* namelist.input rsl.*.* test4_0_NETCDFSST.*.* wrfout* sstout_d01_000000 test4_0_NETCDFSST.out



12)  Verify that both tests produced bitwise-identical history output:  

$WRFDIR/test/em_esmf_exp >> ls -l test4_0_ESMFSST.out/wrfout_d01_2000-01-24_12:00:00 test4_0_NETCDFSST.out/wrfout_d01_2000-01-24_12:00:00
-rw-r--r--   1 hender   ncar       32614704 Oct 12 15:48 test4_0_ESMFSST.out/wrfout_d01_2000-01-24_12:00:00
-rw-r--r--   1 hender   ncar       32614704 Oct 12 15:52 test4_0_NETCDFSST.out/wrfout_d01_2000-01-24_12:00:00

$WRFDIR/test/em_esmf_exp >> cmp -l test4_0_ESMFSST.out/wrfout_d01_2000-01-24_12:00:00 test4_0_NETCDFSST.out/wrfout_d01_2000-01-24_12:00:00 | wc
       0       0       0

