Differences

This shows you the differences between two versions of the page.

--- fimex:faq [2013-09-16 08:59:27]
heikok
+++ fimex:faq [2024-07-03 07:11:55] (current)
alexanderb opendap auth on jammy
@@ Line 6: / Line 6: @@
   * Fimex does not use file-locks for reading: Graphical interfaces like [[http://diana.met.no|diana]] which read via Fimex might keep the file open as long as it does not change, therefor locking the file for too long to allow writers to update the files. Writing additional data to an existing file is usually not a problem, in particularly when the reader re-opens the file when it has updated the time-stamp ''mtime''.
-  * Fimex does not implement file-locking for writers generally, though in particularly with the fillWriter interface this situation might change, in particular for netcdf-files
+  * Fimex does not implement file-locking for writers generally, though in particularly with the [[http://fimex.met.no/doc/fillWriterDoc.html|fillWriter]] interface this situation might change, in particular for netcdf-files
+===== How can I change the units of a variable? =====
+If the original variable e.g. ''t2m'' has units-attribute Celsius and you want to have a new netcdf-file with unit Kelvin, you should use the ''--output.config=cdmWriterConfig.xml'' and define there the new units for the that variable and make sure you defined the datatype of that variable, too.
+If the input-file has the wrong units, e.g. Celsius while the data in fact is Kelvin, you should use the input.config=... and re-define the units-attribute. The data-values won't be changed.
 ===== How to solve problems with a short variable? =====
@@ Line 26: / Line 34: @@
 In this example, the precipitation will be described with an accuracy of 0.1mm and up to 3276.7mm. The ncml-configfile will be applied right before writing with the ''--ncml.config=....ncml'' fimex-option.
-===== Vector-rotation gives error-message (>= 0.48) and wrong values due to different units =====
+===== Vector-rotation =====
+==== Which vectors are auto-detected for rotation ====
+Vectors are autorotated if one of the following applies:
+  * the ''<spatial_vector direction="x" counterpart="y_wind" />'' property is set for both vectors, this can be done with ncml, or the different reader-configurations
+  * the vector has a known CF-standard-name, e.g.
+    * x_wind, y_wind
+    * *_x_velocity,*_y_velocity
+    * *_x_transport,*_y_transport
+    * *_x_displacement,*_y_displacement
+    * *_eastward_*, *_northward_*
+  * the complete list can be found in the sources after ''enhanceVectorProperties'':  https://svn.met.no/viewvc/fimex/trunk/src/coordSys/CF1_xCoordSysBuilder.cc?view=markup
+==== Vector-rotation gives error-message (>= 0.48) and wrong values due to different units ====
 When rotating vectors for spatial interpolations, fimex does not change the scale_factor, add_offset or unit of the vector-components, but requires, that those are equal for both x and y component of the vector. This happens in particular when retrieving wind-data from ECMWF in netcdf, e.g. for ERA data.
@@ Line 125: / Line 148: @@
 </code>
+=== OpENDAP and slicing ===
+OpENDAP servers are often limited on how much data can be read at per request, e.g. thredds allows only 500MB binary data / 50MB ascii data per request before failing with an error like
+<code>
+syntax error, unexpected WORD_STRING, expecting WORD_WORD
+context: Error { code = 403; message = "Request too big=19011.0 Mbytes, max=500.0"^;};
+terminate called after throwing an instance of 'MetNoFimex::CDMException'
+  what():  CDMException: NetCDF: Malformed or inaccessible DAP DATADDS
+Aborted
+</code>
-===== Why takes linking with ''g++ -g''  ages? =====
+Fimex tries to subdevide requests into one unlimited slice per variable. Unfortunately, OpENDAP often does not know about 'unlimited' dimension and one has to add that manually to fimex, e.g. for an unlimited time dimension: ''--input.config=unlimitedTime.ncml'' with contents like:
+<code>
+<?xml version="1.0" encoding="UTF-8"?>
+<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
+        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+		xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2 http://www.unidata.ucar.edu/schemas/netcdf/ncml-2.2.xsd">
+<dimension name="time" isUnlimited="true" />
+</netcdf>
+</code>
-This happens with older versions of gcc, seen with 3.3 on debian sarge.
-Use ''-gstabs'' or ''-ggdb'' instead of ''-g''. For debugging, this is better suited to c++ code and works with the GNU utils and most other debugger, too. (But unfortunately  not with /valgrind/.
+==== Authentication ====
+In theory, it should be possible to specify username and password
+  - in ''$HOME/.dodsrc'' and ''$HOME/.netrc'' (see below for an example)
+  - as part of the URL for example ''https://XXXX:YYYY@thredds.met.no/thredds/dodsC/....nc''
+Example ''$HOME/.netrc'':
+<code>
+machine thredds.met.no
+    login XXXX
+    password YYYY
+</code>
+Example ''$HOME/.dodsrc'' (replace ''YOURHOME'' with the value og ''$HOME''):
+<code>
+HTTP.NETRC=YOURHOME/.netrc
+HTTP.CREDENTIALS.USERNAME=XXXX
+HTTP.CREDENTIALS.PASSWORD=YYYY
+HTTP.USERNAME=XXXX
+HTTP.PASSWORD=YYYY
+</code>
+Unfortunately, this does not always seem to work as expected.
+Known status of OpenDAP authentication support (applies to ''fimex'' and ''ncdump''):
+  * Specifying the password in the URL does not seem to work when the password contains a ''/'', at least for Ubuntu 18.04.
+  * For Ubuntu 16.04 (xenial, EOL), both URL and netrc seem to work.
+  * For Ubuntu 18.04 (bionic), username and password cannot be read from netrc, but can be specified in the URL.
+  * For Ubuntu 20.04 (focal), username and password can only be read from netrc, but cannot be specified in the URL.
+  * For Ubuntu 22.04 (jammy), the only solution known to us is to add a ''.dodsrc'' file in the working directory of the fimex/ncdump process