User Tools

Site Tools


fimex:faq

Frequently Asked Questions

Does fimex use file-locks for reading/writing?

It is very common, that NWP (numerical weather prediction) files are updated while the model is running. It is often desirable to be able to read the file already when the model is still running, e.g. to see the first few ours of weather prediction.

  • Fimex does not use file-locks for reading: Graphical interfaces like diana which read via Fimex might keep the file open as long as it does not change, therefor locking the file for too long to allow writers to update the files. Writing additional data to an existing file is usually not a problem, in particularly when the reader re-opens the file when it has updated the time-stamp mtime.
  • Fimex does not implement file-locking for writers generally, though in particularly with the fillWriter interface this situation might change, in particular for netcdf-files

How can I change the units of a variable?

If the original variable e.g. t2m has units-attribute Celsius and you want to have a new netcdf-file with unit Kelvin, you should use the –output.config=cdmWriterConfig.xml and define there the new units for the that variable and make sure you defined the datatype of that variable, too.

If the input-file has the wrong units, e.g. Celsius while the data in fact is Kelvin, you should use the input.config=… and re-define the units-attribute. The data-values won't be changed.

How to solve problems with a short variable?

I have problems with a felt-file with a variable of type 'short'. fimex abborts with

change in scaling-factor for parameter: 31,2,1000 consider using float or double datatpye

What should I do?

  1. While the netcdf-format allows for only 1 scaling-factor per variable, the felt-format can have a scaling-factor per 2d-slice per variable (that are level*time scaling-factors). You should change the type of the variable and its _FillValue attribute from short to float in the file felt2nc_variables.xml.
  2. If you want to save disk-space, and you know that all data of the variable can be subscribed sufficiently with a packed short data-array, use the ncml-configuration as follows:
    <?xml version="1.0" encoding="UTF-8"?>
    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2 ncml-2.2.xsd ">
    <!-- change the type of a variable, repack -->
    <variable name="precipitation_amount" type="short">
       <attribute name="_FillValue" value="-32767" type="short" />
       <attribute name="scale_factor" value="0.1" type="float" />
       <attribute name="add_offset" value="0" type="double" />
    </variable>
    </netcdf>

In this example, the precipitation will be described with an accuracy of 0.1mm and up to 3276.7mm. The ncml-configfile will be applied right before writing with the –ncml.config=….ncml fimex-option.

Vector-rotation

Which vectors are auto-detected for rotation

Vectors are autorotated if one of the following applies:

  • the <spatial_vector direction=“x” counterpart=“y_wind” /> property is set for both vectors, this can be done with ncml, or the different reader-configurations
  • the vector has a known CF-standard-name, e.g.
    • x_wind, y_wind
    • *_x_velocity,*_y_velocity
    • *_x_transport,*_y_transport
    • *_x_displacement,*_y_displacement
    • *_eastward_*, *_northward_*
  • the complete list can be found in the sources after enhanceVectorProperties: https://svn.met.no/viewvc/fimex/trunk/src/coordSys/CF1_xCoordSysBuilder.cc?view=markup

Vector-rotation gives error-message (>= 0.48) and wrong values due to different units

When rotating vectors for spatial interpolations, fimex does not change the scale_factor, add_offset or unit of the vector-components, but requires, that those are equal for both x and y component of the vector. This happens in particular when retrieving wind-data from ECMWF in netcdf, e.g. for ERA data.

A solution is to configure the vector to have the same scale/add_offset, e.g. with ncml (–input.config):

<variable orgName="Uvind" name="x_wind" type="float">
  <attribute name="scale_factor" value="1" type="float" />
  <attribute name="add_offset" value="0" type="float" />
  <remove name="_FillValue"  type="attribute" />
  <attribute name="standard_name" value="x_wind" type="String" />
</variable>

<variable orgName="Vvind" name="y_wind" type="float" >
  <attribute name="scale_factor" value="1" type="float" />
  <attribute name="add_offset" value="0" type="float" />
  <remove name="_FillValue"  type="attribute" />
  <attribute name="standard_name" value="y_wind" type="String" />
</variable>

What does this error-message mean?

CDMException: Felt_File_Error: FeltError: cannot read gridParameters: no Felt_Array2 available

The config-file for felt-files must at least define 1 existing parameter in the corresponding felt-file. Update your config-file (e.g. felt2nc_variables.xml) with the output from the felt-index, which can be read by felt-tools like rfinh.

CDMException: Can't open HDF5 attribute

This error-message is unfortuneatly very misleading. In many cases, the _FillValue type does not correspond to the variables type. Please check your config-file. Unless really required, it's recommended to remove the _FillValue.

How can I read grib-message which are split over several files?

since fimex 0.40.1, it is possible to give multiple optional file-sources:

fimex --input.file=/opdata/lameps/mle/eceps_201211260000_ens00.mars \
      --input.type=grb --input.config=/home/heikok/Programme/MetSis/Fimex/share/etc/cdmGribReaderConfig.xml \
      --input.optional=/opdata/lameps/mle/eceps_201211260000_ens01.mars \
      --input.optional=/opdata/lameps/mle/eceps_201211260000_ens02.mars \
      --output.file=eceps.nc4

How to use fimex as opendap-client?

Install netcdf with integrated dap-support

Prerequisites

  • netcdf4 (4.1.1) from unidata
  • curl library: sudo apt-get install libcurl4-gnutls-dev (on ubuntu hardy)
  • make sure to remove the libnetcdf4-dev package (on ubuntu hardy): apt-get remove libnetcdf4-dev

Build

Configuring, without HDF-file support (can be added if desired). Important: include dap-support via curl; build shared libraries (libtool autodetect dependencies)

./configure --prefix=/usr/local/netcdf4.1.1 --enable-dap --with-curl-config=/usr/bin/curl-config --enable-shared --disable-static
make -k # there might be errors building the man-pages depending on tex-installation
make check
make -k install

Fimex and OpENDAP

Build

./configure --with-netcdf=/usr/local/netcdf4.1.1

Running an extraction from opendap

fimex --input.file=http://thredds.met.no/thredds/dodsC/topaz/dataset-topaz3-arc-myocean-be \
      --input.type=netcdf \
      --output.type=netcdf \
      --output.file=test.nc \
\
      --extract.selectVariables=time \
      --extract.selectVariables=x \
      --extract.selectVariables=y \
      --extract.selectVariables=depth \
      --extract.selectVariables=longitude \
      --extract.selectVariables=latitude \
      --extract.selectVariables=temperature \
\
      --extract.reduceTime.start=2010-10-05 \
      --extract.reduceTime.end=2010-10-07 \
\
      --extract.reduceToBoundingBox.south 55.0 \
      --extract.reduceToBoundingBox.north 80.0 \
      --extract.reduceToBoundingBox.east 20.0 \
      --extract.reduceToBoundingBox.west -5.0

OpENDAP and slicing

OpENDAP servers are often limited on how much data can be read at per request, e.g. thredds allows only 500MB binary data / 50MB ascii data per request before failing with an error like

syntax error, unexpected WORD_STRING, expecting WORD_WORD
context: Error { code = 403; message = "Request too big=19011.0 Mbytes, max=500.0"^;};
terminate called after throwing an instance of 'MetNoFimex::CDMException'
  what():  CDMException: NetCDF: Malformed or inaccessible DAP DATADDS
Aborted

Fimex tries to subdevide requests into one unlimited slice per variable. Unfortunately, OpENDAP often does not know about 'unlimited' dimension and one has to add that manually to fimex, e.g. for an unlimited time dimension: –input.config=unlimitedTime.ncml with contents like:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
		xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2 http://www.unidata.ucar.edu/schemas/netcdf/ncml-2.2.xsd">
<dimension name="time" isUnlimited="true" />
</netcdf>

Authentication

Authentication to opendap servers need to be configured by setup files. Generally, authentication is described in http://docs.opendap.org/index.php/DAP_Clients_-_Authentication . This does not work with older versions of libcurl, e.g. under Ubuntu 12.04. Ubuntu 16.04 works well. (nasa.gov changed the SSL setup as of March '17 and ubuntu 14.04 does not work any longer. Ubuntu 16.04 is fine.)

Example

Authentication to the NASA data available to earthdata users https://urs.earthdata.nasa.gov , i.e. GPM datasets http://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGHHE.03/2016/236/ .

  • Register a user, get a password and enable the needed application on the earthdata web-pages. Follow the instructions on their pages.
  • Make sure you can access the desired data from your browser, e.g. by selecting a small opendap-ASCII dataset.
  • Put the username/password in a file named .netrc, e.g.
    • vi $HOME/.netrc
      machine urs.earthdata.nasa.gov
          login XXXX
          password YYYY

      (Change XXXX with your username and YYYY with your password)

    • make sure the file is only readable by you
      chmod 600 $HOME/.netrc
  • allow fimex/ncdump/opendap-clients to use this .netrc and to store cookies
    • vi $HOME/.dodsrc
      HTTP.COOKIEJAR=/home/heikok/.cookies
      HTTP.NETRC=/home/heikok/.netrc

      (Change /home/heikok with your home directory.)

  • test with an inventory of a opendap file
    ncdump -h http://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGHHE.03/2016/236/3B-HHR-E.MS.MRG.3IMERG.20160823-S000000-E002959.0000.V03E.HDF5

    As you can see here, the data-server is gpm1.gesdisc… while the authentication server is urs.earthdata - this is very confusing but the way it is often implemented. If you don't know the authentication-servers address, you might need to ask the data-provider.

  • test with fimex
    fimex --input.file=http://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGHHE.03/2016/236/3B-HHR-E.MS.MRG.3IMERG.20160823-S000000-E002959.0000.V03E.HDF5 --input.type=netcdf --input.printNcML
fimex/faq.txt · Last modified: 2017-06-09 11:17:52 by heikok