/*! @mainpage Fimex User Documentation

Fimex is a the File Interpolation, Manipulation and EXtraction library for
gridded geospatial data. It converts between different, extensible dataformats
(currently netcdf, grib1/2 and felt). It enables you to change the projection
and interpolation of scalar and vector grids. It makes it possible subset the
gridded data and to extract only parts of the files.

Fimex can be used as library called @em Fimex and a command-line program
called @em fimex, which gives access to most but not all functions of the
library.

Fimex is build around the Common Data Model version 1 developped by Unidata
and uses a describes data using the CF-Convention http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.0/cf-conventions.html .
Knowledge of that convention is not required, but will help understanding the config files needed for conversion.

The API of Fimex as included in this document is not stable yet and can
change without warning. The setup-files are considered to be mostly stable.
The fimex-program can thus savely be used. If you want to use the API, please
contact me.

@section setup Setup Files

Detailed information on the differnt configuration files can be found at:

- @ref programDoc
- @ref feltConfigDoc
- @ref ncmlConfiguration
- @ref wdbReaderDoc
- @ref gribWriterDoc
- @ref netcdfWriterDoc
- @ref qualityExtractorDoc

@page programDoc fimex Program Options
@section fimex fimex Program Options

@em fimex is a command-line program. It has the following options:

@verbatim
usage: fimex --input.file  FILENAME [--input.type  INPUT_TYPE]
             --output.file FILENAME [--output.type OUTPUT_TYPE]
             [--input.config CFGFILENAME] [--output.config CFGFILENAME]
             [--extract....]
             [--interpolate....]
             [--timeInterpolate....]

Generic options:
  -h [ --help ]                    help message
  --version                        program version
  --debug                          debug program
  --print-options                  print all options
  -c [ --config ] arg (=fimex.cfg) configuration file

Configurational options:
  --input.file arg                        input file
  --input.type arg                        filetype of input file, e.g. nc, nc4,
                                          ncml, felt, grib1, grib2, wdb
  --input.config arg                      non-standard input configuration
  --input.printNcML                       print NcML description of input file
  --input.printCS                         print CoordinateSystems of input file
  --output.file arg                       output file
  --output.type arg                       filetype of output file, e.g. nc,
                                          nc4, grib1, grib2
  --output.config arg                     non-standard output configuration
  --extract.removeVariable arg            remove variables
  --extract.selectVariables arg           select only those variables
  --extract.reduceDimension.name arg      name of a dimension to reduce
  --extract.reduceDimension.start arg     start position of the dimension to
                                          reduce (>=0)
  --extract.reduceDimension.end arg       end position of the dimension to
                                          reduce
  --extract.reduceTime.start arg          start-time as iso-string
  --extract.reduceTime.end arg            end-time by iso-string
  --extract.reduceVerticalAxis.unit arg   unit of vertical axis to reduce
  --extract.reduceVerticalAxis.start arg  start value of vertical axis
  --extract.reduceVerticalAxis.end arg    end value of the vertical axis
  --extract.reduceToBoundingBox.south arg geographical bounding-box in degree
  --extract.reduceToBoundingBox.north arg geographical bounding-box in degree
  --extract.reduceToBoundingBox.east arg  geographical bounding-box in degree
  --extract.reduceToBoundingBox.west arg  geographical bounding-box in degree
  --extract.printNcML                     print NcML description of extractor
  --extract.printCS                       print CoordinateSystems of extractor
  --qualityExtract.autoConfString arg     configure the quality-assignment
                                          using CF-1.3 status-flag
  --qualityExtract.config arg             configure the quality-assignment with
                                          a xml-config file
  --qualityExtract.printNcML              print NcML description of extractor
  --qualityExtract.printCS                print CoordinateSystems of extractor
  --interpolate.projString arg            proj4 input string describing the new
                                          projection
  --interpolate.method arg                interpolation method, one of
                                          nearestneighbor, bilinear, bicubic,
                                          coord_nearestneighbor, coord_kdtree,
                                          forward_max, forward_mean,
                                          forward_median or forward_sum
  --interpolate.xAxisValues arg           string with values on x-Axis, use ...
                                          to continue, i.e. 10.5,11,...,29.5,
                                          see Fimex::SpatialAxisSpec for full
                                          definition
  --interpolate.yAxisValues arg           string with values on x-Axis, use ...
                                          to continue, i.e. 10.5,11,...,29.5,
                                          see Fimex::SpatialAxisSpec for full
                                          definition
  --interpolate.xAxisUnit arg             unit of x-Axis given as udunits
                                          string, i.e. m or degrees_east
  --interpolate.yAxisUnit arg             unit of y-Axis given as udunits
                                          string, i.e. m or degrees_north
  --interpolate.latitudeName arg          name for auto-generated projection
                                          coordinate latitude
  --interpolate.longitudeName arg         name for auto-generated projection
                                          coordinate longitude
  --interpolate.printNcML                 print NcML description of
                                          interpolator
  --interpolate.printCS                   print CoordinateSystems of
                                          interpolator
  --timeInterpolate.timeSpec arg          specification of times to interpolate
                                          to, see Fimex::TimeSpec for a full
                                          definition
  --timeInterpolate.printNcML             print NcML description of
                                          timeInterpolator
  --timeInterpolate.printCS               print CoordinateSystems of
                                          timeInterpolator
  --ncml.config                           modify/configure with ncml-file
  --ncml.printNcML                        print NcML description after
                                          ncml-configuration
  --ncml.printCS                          print CoordinateSystems after
                                          ncml-configuration

@endverbatim

All the configurational options can be configured using a configuration file which is supplied
using the --config option. All command line options (CLO) will overwrite the config-file. As a rule of
thump, use the CLO for testing and use the config-file for productive usage. The CLOs will be further
explained in @ref fimex_config.

@subsection fimex_config fimex Setup File

@verbinclude test/felt2netcdf.cfg

The @em SpatialAxisSpec used in xAxisValues or yAxisValues for the spatial interpolation
should be formatted as explained in detail in MetNoFimex::SpatialAxisSpec. It allows also
autotuning to the orignal data-values.

The @em TimeSpec string used for the timeInterpolate should be formatted as explained in
detail in MetNoFimex::TimeSpec.

@page feltConfigDoc Configuration files for felt reader
@subsection felt_config Configuration files for felt reader

The xml configuration files are defined by the @em felt2nc_variables.dtd
definition. Since part of this configuration are quite stable, e.g.
the axes (time, level, lat, lon, x, y), other parts change, e.g.
the variables to translate change very often. It is therefore useful
to split the variables from the rest of the configuration via @em xinclude

When writing a new configuration for a new set of felt-files, usually from
a new model, it is wise to group the configuration by

-# time resolution, i.e. one config for 3hourly files, one config for hourly files
-# spatial resolution: fimex doesn't allow different spatial resolutions, but some
   models use coarser resoluton for higher levels
-# vertical levels: it is difficult to have the same parameter with sigma levels
   and with height in m

Grouping can be done in two ways, the first one being faster in operation,
the second is easier to configure/change consistently:

-# write different configuration-files for each group of parameter, stating
   the parameter as well as possible.
-# write one configuration-file for all parameter, keeping the parameters as
   variable as possible. Use a preprocess-step to extract each group.
   Use e.g. @em nyfelt or @em felt2felt as preprocessor

By default, all data is read as @em type="short" data with a scaling factor. While
felt allows for one scaling factor for each timestep, height and parameter,
the CDM allows only for one scaling factor per parameter. When the scaling factor
changes withing height or timestep, @em fimex will fail to read the data as short.
It is therefore useful to read data as @em type="float", which will automatically expand
the scaling factor. If the resulting file is to big, it is possible to
convert to short with one scaling factor and offset using the @ref netcdfWriterConfig.

Before running fimex with a new felt configuration, make sure the file
is valid, e.g. with
@code
xmllint --valid --noout felt2nc_config.xml
@endcode

Unfortuneatly, xinclude and validation don't play well together, since
usual validation happens before the inclusion of external parts. xmllint
uses special options to fix those problem:
@code
xmllint --xinclude --postvalid --noout felt2nc_config.xml
@endcode



Below follows a complete felt-configuration.

@verbinclude share/etc/felt2nc_variables.xml


@page ncmlConfiguration ncml Configuration
@section ncmlConfiguration ncml Configuration

@verbinclude share/etc/ncmlCDMConfig.ncml

Unidata's NetCDF Markup Language (NcML) as described in http://www.unidata.ucar.edu/software/netcdf/ncml/
gives the opportunity to change all information written in the CDM. With the --ncml.config option, the CDM
will be configured immediately after reading a file. It is also possilbe to read in a ncml file with the
--input.file=xxx.ncml option. In this case, the real data must be linked with the 'location' markup.

Input-files can and should be validated against the included ncml-2.2.xsd.

@warning not all features are supported in the current implementation. Missing features are: aggregation,
changing dimension sizes, unlimited dimensions, adding/changing values of variables, groups


@page wdbReader wdb Reader Configuration
@section wdbReader wdb Reader Configuration

@em This feature is still alpha-quality and will change!

Fimex is able to read data from instances of the WDB (http://wdb.sourceforge.net) by using
the WdbCDMReader (or on the command line: input.type=wdb). The usual 'file'-location is in that
case reinterpreted as wdb-connetction string, e.g. 'dbHost=proffdb-devel.met.no;dbPort=5432;dbName=wdb;dbUser=proffread;wciUser=proffwrite'
would connect to the host proffdb-devel on port 5432 and the databse wdb with the user proffread.
Additonal options:
@li refTime use exactly this reference time
@li provider use this provider
@li place use this place

Example:
@code
fimex --input.file='dbHost=proffdb-devel.met.no;dbPort=5432;dbName=wdb;dbUser=proffread;refTime=20110211T000000' --input.type=wdb --input.config=/home/heikok/Programme/MetSis/Fimex/share/etc/proffdb_config.xml --output.file=test.nc --output.type=nc4 --extract.reduceDimension.name=time --extract.reduceDimension.start=1 --extract.reduceDimension.end=3
@endcode


Most of these options, and additional one adding correct metadata can be given in the config file, e.g. in proffdb_config.xml :

@verbinclude share/etc/proffdb_config.xml


@page qualityExtractorDoc quality-extraction Configuration
@section qualityExtractorDoc quality-extraction Configuration

@warning The quality-extraction is still in a very early stage of development. The configuration
and the outcome is very likely to change in further developments. Any feedback is strongly welcome.

@verbinclude share/etc/cdmQualityConfig.xml

In cases where the data should be extracted if certain conditions (qualities) apply, i.e.
the status-flag indicates a properly working instrument, or the sea-surface-temperature is
above 300K, the #MetNoFimex::CDMQualityExtractor allows to add these rules. The cdmQualityConfig.xml
file as shown above gives an example of such an configuration.

@li The variable "bla" will only be set, if "blub" has integer-values between 1 and 6.
@li The variable "air_temperature" will only be extracted for an "altitude" above 1000. The value
1000 is the actual data value in the variable "altitude" without any scaling or unit-conversion applied.

The following use-values can be selected:
@li @c all select all valid values (within valid_max, valid_min or valid_range, without _FillValue)
@li @c highest the highest numerical value found in the data-slice which is valid
@li @c lowest the lowest numerical value fond in the data-slice which is valid
@li @c max:xxx.x all valid-values below or equal xxx.x
@li @c min:xxx.x all valid values above or equal xxx.x

All values which do not match the quality-criteria will be set to the _FillValue of the
variable.



@page gribWriterDoc gribWriter Configuration
@section gribWriterConfig gribWriter Configuration

@verbinclude share/etc/cdmGribWriterConfig.xml

@page netcdfWriterDoc netcdfWriter Configuration
@section netcdfWriterConfig netcdfWriter Configuration

The netcdfWriterConfig gives the opportunity to set some features
explicit only for netcdf-files, i.e. file-format (netcdf3/4) or compression.

It is also possible to add an @ref ncmlConfiguration to the output to change the
internal structure just before writing.

It is also possible ot change units including all value in the netcdfWriterDoc.
Changing the units in the ncmlConfiguration would change the attribute value only, but
not the data.

The CDM resembles a netcdf datastructure. In general, there is
no need to use a configuration for this writer, but it might be useful
in the following cases:

- Output-files are to big, and a change of datatype i.e. from float to short
  is desired
- Different attribute are required for special usages, but the input-configuration
  of the reader shouldn't been changed.
- Different variable or dimension names are required for special usages.

@verbinclude share/etc/cdmWriterConfig.xml

*/
