metamod:minutes2008_10_23

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision 		Previous revision
metamod:minutes2008_10_23 [2008-10-23 09:30:57]
heikok created 		metamod:minutes2008_10_23 [2022-05-31 09:29:32] (current)
Line 1: Line 1:
-====== Minutes 2008-10-23 ======+====== How to manage different input metadata formats? ======
  
-Heiko, Egil+Discussion between Heiko and Egil 2008-10-23
  
 ===== Metadata-Storage ===== ===== Metadata-Storage =====
  
-We have currently to places to store metadata:+We have currently two places to store metadata:
  
   - XML-files: These are the files as received from the (meta-)data provider in the original metadata-standard. Currently, providers are: //digest_nc.pl (reading from nc-cf-1.0 files), quest, oai-pmh harvest//   - XML-files: These are the files as received from the (meta-)data provider in the original metadata-standard. Currently, providers are: //digest_nc.pl (reading from nc-cf-1.0 files), quest, oai-pmh harvest//
   - SQL-database: The SQL database keeps a normalized and indexed view of the XML files. The SQL-database has a known set of supported metadata-names, e.g. //institution, variable, datacollection_period, abstract//. Those can be found in the table //MetadataType//. The normalized metadata in the SQL-database can be searched through the search-module, and can be exported to other formats in the //oai-pmh// module (currently, conversion to DIF).   - SQL-database: The SQL database keeps a normalized and indexed view of the XML files. The SQL-database has a known set of supported metadata-names, e.g. //institution, variable, datacollection_period, abstract//. Those can be found in the table //MetadataType//. The normalized metadata in the SQL-database can be searched through the search-module, and can be exported to other formats in the //oai-pmh// module (currently, conversion to DIF).
 +
 +==== Problem description ====
 +
 +Currently we receive metadata as attributes in netCDF files and from forms in the //quest// module. In the near future, we will also recieve metadata as DIF XML from the //harvest// module. We can also expect other XML metadata formats (e.g. WMO XML profile). This situation raise three conserns:
 +
 +  - We need a general way to manage different metadata formats as input to the database.
 +  - We need to transform the received metadata so that the same information is stored in the SQL database in the same way. We do not want to miss datasets when we search the database only because the search items were tagged differently in the source XML files. So we need to normalize the metadata.
 +  - We also need to keep the matadata as we received them in their original XML formats.
  
 ==== Interaction with the different modules ==== ==== Interaction with the different modules ====
Line 14: Line 22:
 Here is the state as planned for Metamod 2.1, not everything exists yet! Here is the state as planned for Metamod 2.1, not everything exists yet!
  
-  * search: read from database +  * search: read from database //PHP// 
-  * base: read XML-files, **normalize**, write to SQL-database (import_dataset.pl) +  * base: read XML-files, **normalize**, write to SQL-database (import_dataset.pl) //Perl// 
-  * quest: write to XML-files, read old parameters from SQL-database **this will change the metadata-format to our internal format** +  * quest: write to XML-files, read old parameters from SQL-database **this will change the metadata-format to our internal format** //PHP// 
-  * upload: write to XML-files (digest_nc.pl), eventually edit metadata via //quest// +  * upload: write to XML-files (digest_nc.pl) //Perl//, eventually edit metadata via //quest// 
-  * pmh: read from SQL-database +  * pmh: read from SQL-database //PHP// 
-  * harvest: write to XML-files+  * harvest: write to XML-files //PHP//
      
 ==== Outstanding problems ==== ==== Outstanding problems ====
  • metamod/minutes2008_10_23.1224754257.txt.gz
  • Last modified: 2022-05-31 09:23:19
  • (external edit)