Differences

This shows you the differences between two versions of the page.

--- metamod:minutes2009_12_03 [2009-12-08 14:56:48]
egils
+++ metamod:minutes2009_12_03 [2009-12-08 15:16:19]
egils
@@ Line 14: / Line 14: @@
 </code>
 Calling this url will extract the metadata from the files named by ''filename'' and add the metadata to the ''dataset''. The ''dirkey'' is used as a password and needs to be set up when setting up the ''dataset''. The data-files need be uploaded to the final data-store (ftp-server) before triggering that task. This task will only harvest metadata, but not move any data.
+===== Discussion =====
 Comments by --- //[[egil.storen@met.no|Egil Støren]] 2009/12/08 14:34//:
-This solution (with no restrictions on file names belonging to a given dataset) gives us some problems regarding THREDDS catalogs. Up to now, we have used the datasetScan element in the THREDDS catalog to create a dataset entry within THREDDS corresponding to all files in a directory. We can continue to use this method if all files in a dataset belongs to the same directory. Then the consequences will not be wery difficult to cope with: We can contunue to use the same method to create THREDDS catalogs, and few changes need to be done in the web interface for creating new datasets. On the other hand, if several datasets are to share the same directory, we have to do major changes both in THREDDS catalog generation and in the web interface for creating new datasets. If files in the same dataset are allowed to be situated in different directories, the situation will be even worse. So my queston is: Can we assume a one to one correspondence between dataset and lowest level directory in the data repository?
+This solution (with no restrictions on file names belonging to a given dataset) gives us some problems regarding THREDDS catalogs. Up to now, we have used the datasetScan element in the THREDDS catalog to create a dataset entry within THREDDS corresponding to all files in a directory. We can continue to use this method if all files in a dataset belongs to the same directory. Then the consequences will not be wery difficult to cope with: We can contunue to use the same method to create THREDDS catalogs, and few changes need to be done in the web interface for creating new datasets.
+On the other hand, if several datasets are to share the same directory, we have to do major changes both in THREDDS catalog generation and in the web interface for creating new datasets. If files in the same dataset are allowed to be situated in different directories, the situation will be even worse.
+So my question is: //**Can we assume a one to one correspondence between dataset and lowest level directory in the data repository**//?
+If we can not assume this correspondence, the next best alternative will be, for each dataset, to identify a regular expression (or wildcard expression) that will correspond to all files in the dataset. This expression would have to be entered by the data provider in the web interface when creating the dataset. This expression could then be used in the THREDDS catalog as a filter element inside the datasetScan element.
 ===== Timeline =====