It is very common, that NWP (numerical weather prediction) files are updated while the model is running. It is often desirable to be able to read the file already when the model is still running, e.g. to see the first few ours of weather prediction.
mtime
.
If the original variable e.g. t2m
has units-attribute Celsius and you want to have a new netcdf-file with unit Kelvin, you should use the –output.config=cdmWriterConfig.xml
and define there the new units for the that variable and make sure you defined the datatype of that variable, too.
If the input-file has the wrong units, e.g. Celsius while the data in fact is Kelvin, you should use the input.config=… and re-define the units-attribute. The data-values won't be changed.
I have problems with a felt-file with a variable of type 'short'. fimex abborts with
change in scaling-factor for parameter: 31,2,1000 consider using float or double datatpye
What should I do?
<?xml version="1.0" encoding="UTF-8"?> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2 ncml-2.2.xsd "> <!-- change the type of a variable, repack --> <variable name="precipitation_amount" type="short"> <attribute name="_FillValue" value="-32767" type="short" /> <attribute name="scale_factor" value="0.1" type="float" /> <attribute name="add_offset" value="0" type="double" /> </variable> </netcdf>
In this example, the precipitation will be described with an accuracy of 0.1mm and up to 3276.7mm. The ncml-configfile will be applied right before writing with the –ncml.config=….ncml
fimex-option.
Vectors are autorotated if one of the following applies:
<spatial_vector direction=“x” counterpart=“y_wind” />
property is set for both vectors, this can be done with ncml, or the different reader-configurationsenhanceVectorProperties
: https://svn.met.no/viewvc/fimex/trunk/src/coordSys/CF1_xCoordSysBuilder.cc?view=markupWhen rotating vectors for spatial interpolations, fimex does not change the scale_factor, add_offset or unit of the vector-components, but requires, that those are equal for both x and y component of the vector. This happens in particular when retrieving wind-data from ECMWF in netcdf, e.g. for ERA data.
A solution is to configure the vector to have the same scale/add_offset, e.g. with ncml (–input.config):
<variable orgName="Uvind" name="x_wind" type="float"> <attribute name="scale_factor" value="1" type="float" /> <attribute name="add_offset" value="0" type="float" /> <remove name="_FillValue" type="attribute" /> <attribute name="standard_name" value="x_wind" type="String" /> </variable> <variable orgName="Vvind" name="y_wind" type="float" > <attribute name="scale_factor" value="1" type="float" /> <attribute name="add_offset" value="0" type="float" /> <remove name="_FillValue" type="attribute" /> <attribute name="standard_name" value="y_wind" type="String" /> </variable>
The config-file for felt-files must at least define 1 existing parameter in the corresponding felt-file. Update your config-file (e.g. felt2nc_variables.xml) with the output from the felt-index, which can be read by felt-tools like rfinh
.
This error-message is unfortuneatly very misleading. In many cases, the _FillValue
type does not correspond to the variables type. Please check your config-file. Unless really required, it's recommended to remove the _FillValue
.
since fimex 0.40.1, it is possible to give multiple optional file-sources:
fimex --input.file=/opdata/lameps/mle/eceps_201211260000_ens00.mars \ --input.type=grb --input.config=/home/heikok/Programme/MetSis/Fimex/share/etc/cdmGribReaderConfig.xml \ --input.optional=/opdata/lameps/mle/eceps_201211260000_ens01.mars \ --input.optional=/opdata/lameps/mle/eceps_201211260000_ens02.mars \ --output.file=eceps.nc4
sudo apt-get install libcurl4-gnutls-dev
(on ubuntu hardy)apt-get remove libnetcdf4-dev
Configuring, without HDF-file support (can be added if desired). Important: include dap-support via curl; build shared libraries (libtool autodetect dependencies)
./configure --prefix=/usr/local/netcdf4.1.1 --enable-dap --with-curl-config=/usr/bin/curl-config --enable-shared --disable-static make -k # there might be errors building the man-pages depending on tex-installation make check make -k install
./configure --with-netcdf=/usr/local/netcdf4.1.1
fimex --input.file=http://thredds.met.no/thredds/dodsC/topaz/dataset-topaz3-arc-myocean-be \ --input.type=netcdf \ --output.type=netcdf \ --output.file=test.nc \ \ --extract.selectVariables=time \ --extract.selectVariables=x \ --extract.selectVariables=y \ --extract.selectVariables=depth \ --extract.selectVariables=longitude \ --extract.selectVariables=latitude \ --extract.selectVariables=temperature \ \ --extract.reduceTime.start=2010-10-05 \ --extract.reduceTime.end=2010-10-07 \ \ --extract.reduceToBoundingBox.south 55.0 \ --extract.reduceToBoundingBox.north 80.0 \ --extract.reduceToBoundingBox.east 20.0 \ --extract.reduceToBoundingBox.west -5.0
OpENDAP servers are often limited on how much data can be read at per request, e.g. thredds allows only 500MB binary data / 50MB ascii data per request before failing with an error like
syntax error, unexpected WORD_STRING, expecting WORD_WORD context: Error { code = 403; message = "Request too big=19011.0 Mbytes, max=500.0"^;}; terminate called after throwing an instance of 'MetNoFimex::CDMException' what(): CDMException: NetCDF: Malformed or inaccessible DAP DATADDS Aborted
Fimex tries to subdevide requests into one unlimited slice per variable. Unfortunately, OpENDAP often does not know about 'unlimited' dimension and one has to add that manually to fimex, e.g. for an unlimited time dimension: –input.config=unlimitedTime.ncml
with contents like:
<?xml version="1.0" encoding="UTF-8"?> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2 http://www.unidata.ucar.edu/schemas/netcdf/ncml-2.2.xsd"> <dimension name="time" isUnlimited="true" /> </netcdf>
In theory, it should be possible to specify username and password
$HOME/.dodsrc
and $HOME/.netrc
(see below for an example)https://XXXX:YYYY@thredds.met.no/thredds/dodsC/....nc
Example $HOME/.netrc
:
machine thredds.met.no login XXXX password YYYY
Example $HOME/.dodsrc
(replace YOURHOME
with the value og $HOME
):
HTTP.NETRC=YOURHOME/.netrc
Unfortunately, this does not always seem to work as expected.
Known status of OpenDAP authentication support (applies to fimex
and ncdump
):
/
, at least for Ubuntu 18.04.