MADIS Data Ingest System

DART project logo

Jump to DART Documentation Main Index
version information for this file:
$Id: MADIS.html 9189 2015-12-07 22:00:54Z nancy $

DATA SOURCES / PROGRAMS / MODULES / NAMELIST / ERRORS / FUTURE PLANS / TERMS OF USE

Overview

The MADIS (Meteorological Assimilation Data Ingest System) service provides access to real-time and archived data of a variety of types, with added Quality Control (QC) and integration of data from a variety of sources.

To convert a series of MADIS data files (where different types of observations are distributed in separate files), one high level view of the workflow is:

  1. convert each madis file, by platform type, into an obs_seq file. one file in, one file out. no time changes. use the shell_scripts/madis_conv.csh script. there are script options for hourly output files, or a single daily output file.
  2. if you aren't using the wrf preprocessing program, you're ready to go.
  3. if you do want to do subsequent wrf preprocessing, you need to:
    1. decide on the windowing. each platform has a different convention and if you're going to put them into the wrf preprocessing you'll need to have the windowing match. use the shell_scripts/windowing.csh script.
    2. the wrf preprocessing takes a list of files and assumes they will all be assimilated at the same time, for superob'ing purposes, so it should match the expected assimilation window when running filter.

DATA SOURCES

http://madis.noaa.gov

There are two satellite wind converter programs; the one in this directory and one in the SSEC directory. The observations distributed here come from NESDIS. The SSEC observations are processed by SSEC itself and will differ from the observations converted here.


PROGRAMS

The programs in the DART/observations/MADIS/ directory extract data from the distribution files and create DART observation sequence (obs_seq) files. Build them in the work directory by running the ./quickbuild.csh script. In addition to the converters, the advance_time and obs_sequence_tool utilities will be built.

There are currently converters for these data types:

ACARS aircraft T,U,V,Q data convert_madis_acars
Marine surface data convert_madis_marine
Mesonet surface data convert_madis_mesonet
Metar data convert_madis_metar
Wind Profiler data convert_madis_profiler
Rawinsonde/Radiosonde data convert_madis_rawin
Satellite Wind data convert_madis_satwnd

Example data files are in the data directory. Example scripts for converting batches of these files are in the shell_scripts directory. These are NOT intended to be turnkey scripts; they will certainly need to be customized for your use. There are comments at the top of the scripts saying what options they include, and should be commented enough to indicate where changes will be likely to need to be made.

Several converters have compile-time choices for outputting various types of moist variables. Check the source code for more details. Some converters also read multiple T/F strings from the console (standard input) to control at run-time what types of observations to convert. Again, check the source code for more details.

Each converter has hard-coded input and output filenames:

convert_madis_acars: acars_input.nc obs_seq.acars
convert_madis_marine: marine_input.nc obs_seq.marine
convert_madis_mesonet: mesonet_input.nc obs_seq.mesonet
convert_madis_metar: metar_input.nc obs_seq.metar
convert_madis_profiler: profiler_input.nc obs_seq.profiler
convert_madis_rawin: rawin_input.nc obs_seq.rawin
convert_madis_satwnd: satwnd_input.nc obs_seq.satwnd

The expected usage pattern is that a script will copy, rename, or make a symbolic link from the actual input file (which often contains a timestamp in the name) to the fixed input name before conversion, and move the output file to an appropriate filename before the next invocation of the converter. If an existing observation sequence file of the same output name is found when the converter is run again, it will open that file and append the next set of observations to it.


KNOWN BUGS

none


FUTURE PLANS

none


Terms of Use

DART software - Copyright 2004 - 2013 UCAR.
This open source software is provided by UCAR, "as is",
without charge, subject to all terms of use at
http://www.image.ucar.edu/DAReS/DART/DART_download

Contact: nancy collins
Revision: $Revision: 9189 $
Source: $URL: https://svn-dares-dart.cgd.ucar.edu/DART/releases/Lanai/observations/MADIS/MADIS.html $
Change Date: $Date: 2015-12-07 15:00:54 -0700 (Mon, 07 Dec 2015) $
Change history:  try "svn log" or "svn diff"