fimex:usage

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
fimex:usage [2011-11-03 12:12:30]
heikok 		fimex:usage [2022-11-15 14:13:54] (current)
alexanderb change wording
Line 1: Line 1:
-====== Usage Example ======+====== Usage Examples ======
  
-===== Using fimex as opendap-client =====+===== Converting a felt model output from felt to NetCDF =====
  
-==== Install netcdf with integrated dap-support ====+  - get an overview of times, layers and grid used in the felt-file  
 +  - group the parameters so that all parameters within an group use the same grid, times and the same layers (its possible to have a file with both sigma and pressure layers, but then all sigma-variables should have the same number of sigma layers and all pressure-variables should have the same number of pressure layers. Surface only variable, usually sigma=1000 felt variables, are layer-group of their own) 
 +  - create a felt2nc_variables.xml file for each variable group. Copy parameters from existing examples, or lookup variable and parameter names in the CF standard name documentation. If you have to generate several groups, it might be useful to split the file into axes, global attributes and variables, see i.e. //damocles_felt2nc_variables.xml// If you have two paramters which map to the same standard_name, i.e. make sure to give them different variable names if they are in the same group. Adjust the datatype according to your data. float will always work fine, use short if you know, that the variable even fits as 4-dimensional data into shorts. Use eventually a cdmWriterConfig.xml to process data as float and write the data as short. 
 +  - validate the felt2nc_variables.xml file, i.e. with <code>xmllint --xinclude --postvalid --noout ../../share/etc/felt2nc_variables.xml</code> 
 +  - adjust the projection and area you want to extract in fimex_example.cfg (or use command line arguments to fimex) 
 +  - run <code>fimex -c fimex_example.cfg</code>
  
-=== Prerequisites ===+===== Writing zarr with libnetcdf =====
  
-  * netcdf4 (4.1.1) from unidata +Zarr does not support unlimited dimensions. You may need to set ''unlimited=false'' for some dimensions, e.g. with ncml like 
-  * curl library: ''sudo apt-get install libcurl4-gnutls-dev'' (on ubuntu hardy) +  <?xml version="1.0" encoding="UTF-8"?> 
-  * make sure to remove the libnetcdf4-dev package (on ubuntu hardy): ''apt-get remove libnetcdf4-dev''+  <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" 
 +          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> 
 +    <dimension name="time" length="61" isUnlimited="false" /> 
 +  </netcdf>
  
-=== Build ===+In version 4.8.1, libnetcdf does not support compressed zarr chunks. You probably have to disable compression with a netcdf writer config xml file similar to: 
 +  <?xml version="1.0" encoding="UTF-8"?> 
 +  <cdm_ncwriter_config> 
 +    <default compressionLevel="0" /> 
 +  </cdm_ncwriter_config>
  
-Configuring, without HDF-file support (can be added if desired). Important: include dap-support via curl; build shared libraries (libtool autodetect dependencies)+The fimex command would then be something like 
 +  fimex \ 
 +      --input.file "input.nc" \ 
 +      --input.config "no_unlimited_dim.ncml" \ 
 +      --output.file "file:///path/to/output#mode=nczarr,file" \ 
 +      --output.config "nc_writer_no_compression.xml" \ 
 +      --output.type nc4
  
-<code> +Please note that using libnetcdf to write zarr will still use a lock on libnetcdf function calls, i.e. there will only be one thread at a time reading or writing.
-./configure --prefix=/usr/local/netcdf4.1.1 --enable-dap --with-curl-config=/usr/bin/curl-config --enable-shared --disable-static +
-make -k # there might be errors building the man-pages depending on tex-installation +
-make check +
-make -k install +
-</code>+
  
-==== Fimex and OpENDAP ====+Datasets that were written like this do not seem to be readable with the same url (observed with netcdf 4.8.1), but reading appears to work with ''mode=zarr,file'' (i.e., not ''nczarr'').
  
-=== Build === +See the [[https://docs.unidata.ucar.edu/nug/current/nczarr_head.html|libnetcdf zarr documentation]] for more details on zarr support in libnetcdf.
- +
-<code> +
-./configure --with-netcdf=/usr/local/netcdf4.1.1 +
-</code> +
- +
-=== Running an extraction from opendap === +
- +
-<code> +
-fimex --input.file=http://thredds.met.no/thredds/dodsC/topaz/dataset-topaz3-arc-myocean-be \ +
-      --input.type=netcdf \ +
-      --output.type=netcdf \ +
-      --output.file=test.nc \ +
-\ +
-      --extract.selectVariables=time \ +
-      --extract.selectVariables=x \ +
-      --extract.selectVariables=y \ +
-      --extract.selectVariables=depth \ +
-      --extract.selectVariables=longitude \ +
-      --extract.selectVariables=latitude \ +
-      --extract.selectVariables=temperature \ +
-\ +
-      --extract.reduceTime.start=2010-10-05 \ +
-      --extract.reduceTime.end=2010-10-07 \ +
-\ +
-      --extract.reduceToBoundingBox.south 55.0 \ +
-      --extract.reduceToBoundingBox.north 80.0 \ +
-      --extract.reduceToBoundingBox.east 20.0 \ +
-      --extract.reduceToBoundingBox.west -5.0 +
- +
-</code> +
- +
-===== Converting a model output from felt to NetCDF ===== +
- +
-  - get an overview of times, layers and grid used in the felt-file  +
-  - group the parameters so that all parameters within an group use the same grid, times and the same layers (its possible to have a file with both sigma and pressure layers, but then all sigma-variables should have the same number of sigma layers and all pressure-variables should have the same number of pressure layers. Surface only variable, usually sigma=1000 felt variables, are layer-group of their own) +
-  - create a felt2nc_variables.xml file for each variable group. Copy parameters from existing examples, or lookup variable and parameter names in the CF standard name documentation. If you have to generate several groups, it might be useful to split the file into axes, global attributes and variables, see i.e. //damocles_felt2nc_variables.xml// If you have two paramters which map to the same standard_name, i.e. make sure to give them different variable names if they are in the same group. Adjust the datatype according to your data. float will always work fine, use short if you know, that the variable even fits as 4-dimensional data into shorts. Use eventually a cdmWriterConfig.xml to process data as float and write the data as short. +
-  - validate the felt2nc_variables.xml file, i.e. with <code>xmllint --xinclude --postvalid --noout ../../share/etc/felt2nc_variables.xml</code> +
-  - adjust the projection and area you want to extract in fimex_example.cfg (or use command line arguments to fimex) +
-  - run <code>fimex -c fimex_example.cfg</code>+
  
  • fimex/usage.1320322350.txt.gz
  • Last modified: 2022-05-31 09:23:16
  • (external edit)