searchdata.xml
METAMOD was initially developed to support scientific projects with data management. Development has focused on a cost effective system where several projects or operational services can share the same backend while maintaining a custom built user interface specific for each project. One METAMOD implementation may support many METAMOD instances. The first instance always own the database and some maintenance and configuration aspects are only available through this instance.
METAMOD was developed to support data management focusing on:
The runtime environment is generated during installation of the software (see the README file coming with the software for details). Basically the following procedure is used to set up a METAMOD instance:
master_config.txt
under app/<appname>
update_target.pl
found in source code using the user owning the web servermaster_config.txt
create_and_load_all.sh
located within the runtime source code tree directory init
prepare_runtime_env.sh
located within runtime source code tree root directorymetamodInit.sh
located in the runtime source code tree root directory
If changes are made to master_config.txt
, repeat steps 1, 2, 4 and 5 and remember that all scripts need to be run as the user owning the web server. In step 5 use e.g. sudo -u wwwuser ./metamodInit.sh restart
or the two step procedure of first stopping and then starting the server.
The visual appearance of METAMOD is determined by style sheets and contents of the master_config.txt
file. Style sheets are located in the directory named htdocs
within the runtime source code directory of METAMOD. Appearance of most of the web pages, including buttons can be changed by changing the corresponding style elements within the following style sheets:
htdocs/style.css
- Main style sheet determining the appearance of main menus and containers.htdocs/sch/style.css
- Style sheet used by the search interface.htdocs/upl/style.css
- Style sheet used by the upload interface.htdocs/qst/style.css
- Style sheet used by the quest interface.
METAMOD has two main menu containers. One top left (available within all modules) and one left (mainly in the search module and the upload). The top left menu is set up using the APP_MENU
tag of the master_config.txt
.
The search module menu is configured using the SEARCH_CATEGORY_SEQUENCE
of the master_config.txt
. This is configured using numbers (defined within the file searchdata.xml
) where the following mapping apply by default1):
In addition the keyword Fulltext
can be used.
In order to add new search categories the following sequence is required:
searchdata.xml
(located within directory app/<appname>/staticdata
)import_searchdata.pl
which is located in the runtime directory <appname>/scripts
with searchdata.xml
as inputIf several METAMOD instances are using the same database, this has to be done in the METAMOD instance that actually own the database. If the database has been running for a while, reinitialisation may take some time during which the METAMOD instances using the database will be unavailable.
The XMD-files used within METAMOD has a tag that identifies the metadata standard used. Currently the following standards are supported:
The native format of METAMOD is MM2 and this is connected to the datamodel used in the database. As of version 2.13 work is ongoing to support a new format named MMD which makes translation to DIF and ISO easier. For a new format to be supported, modifications of the software is required.
The best solution is to use the administration interface (require login and appropriate user rights) and the “Dataset manager”. Within this interface datasets can be deactivated. they will still be in the database, but will be invisible. Sometimes conflicts between datasets may arise (due to naming conventions) which may require a full reinitialisation of the database. If a dataset is to be permanently removed, remove the XML and XMD files from the webrun directory and follow the steps under How do I reinitialise the database?. This require access to the webrun directory at the server running METAMOD.
Unused or obsolete metadata files may be removed from the XML directory. Any warnings caused by this will be removed after full reinitialisation of the database.
- A new editor has been developed, but not fully integrated yet.
The preferred solution is to use the metadata editor included in the software. This is launched using the “File upload” menu and the option “Dataset administration”. Login and appropriate privileges are required.
Sometimes, the solution above is insufficient. Under such conditions, direct modification of metadata and import of these from the webrun directory using import_datasets
may be an option. This option should be used with care!
Visualisation and transformation of datasets are configured through metadata.
For OGC WMS to be activated for a dataset a section similar to the one below is required.
<wmsInfo xmlns:w="http://www.met.no/schema/metamod/ncWmsSetup"> <w:ncWmsSetup url="%THREDDS_DATAREF%" aggregate_url="http://thredds.nersc.no/thredds/wms/normap/arctic25km_seaice_aggr" xsi:schemaLocation="http://www.met.no/schema/metamod/ncWmsSetup ncWmsSetup.xsd "> <w:displayArea crs="EPSG:32661" left="-1000000" right="5000000" bottom="-1000000" top="5000000"/> <w:layer name="NORSEX" style="BOXFILL/rainbow"/> <w:layer name="NASA_Team" style="BOXFILL/rainbow"/> <w:layer name="Bristol" style="BOXFILL/rainbow"/> <w:layer name="Bootstrap" style="BOXFILL/rainbow"/> </w:ncWmsSetup> </wmsInfo>
This section describes the initial view to present and the layers that are available. If the number of layers should be truncated, add maxlayers=“0”
to ncWmsSetup
'. In the displayArea field, coordinate need to be provided in the coordinates of the map projection defined (using EPSG codes).
To enable transformation of a gridded dataset, add dataref_OPENDAP
to MM2 metadata or edit e.g. dif2MM2.xslt to reflect this variable.
If the dataset is not a gridded dataset, but e.g. a timeseries, addition of a MM2 field named timeseries
will remove the transform button and add Timeseries and ASCII buttons which are used to visualise a timeseries or download it as ASCII formatted data. The format of the timeseries field is of the form
<metadata name="timeseries">time,sea_ice_extent</metadata>
Where the first variable is used as the X axis. The others (up to two) will be plotted on the Y axis.
Use the following recipe (information in <> to be substituted as appropriate):
This is possible provided the user has the correct priveliges. XML-files in the XML-directory may be edited manually and inserted in the database utilising the script import_dataset.pl
. This can also be used to include metadata harvested from an external system.
Open Archive Initiative Protocol for metadata harvesting has been supported for a time. Support for ISO23950/Z39.50 (SRU implementation) is currently under implementation and is expected in the next release.
METAMOD use an internal metadata standard that is mapped to Dublin Core, GCMD DIF and ISO191152).
Information and schemata of the metadata-files used since Metamod 2.1 can be found under XML format for dataset descriptions.
The metadata structure is configurable, so this question must relate to a specific application of the database. The answer below is based on the applications implemented at met.no, which corresponds more or less to the EXAMPLE application available as part of the SVN source tree.
The METAMODPMH module contain a translation from the database to DIF XML. For the time being, this translation is implemented in a PHP script oaidp-config.php. This script contains an array which defines the translation:
$key_conversion = array( '!DS_name 1', 'Entry_ID', '', 'title', 'Entry_Title', '', 'PI_name', 'Data_Set_Citation Dataset_Creator', '', 'title', 'Data_Set_Citation Dataset_Title', '', 'institution', 'Data_Set_Citation Dataset_Publisher', '', 'dataref', 'Data_Set_Citation Online_Resource', '', 'variable', '*Parameters Category', 'EARTH SCIENCE', 'variable 1', 'Parameters Topic', '', 'variable 2', 'Parameters Term', '', 'variable 3', 'Parameters Variable_Level_1', '', 'variable', 'Parameters Detailed_Variable', '', 'topiccategory', 'ISO_Topic_Category', '', 'keywords', 'Keyword', '', 'datacollection_period 1', 'Temporal_Coverage Start_Date', '', 'datacollection_period 2', 'Temporal_Coverage Stop_Date', '', 'southernmost_latitude', 'Spatial_Coverage Southernmost_Latitude', '', 'northernmost_latitude', 'Spatial_Coverage Northernmost_Latitude', '', 'westernmost_longitude', 'Spatial_Coverage Westernmost_Longitude', '', 'easternmost_longitude', 'Spatial_Coverage Easternmost_Longitude', '', 'area 1', '*Location Location_Category', '', 'area 2', 'Location Location_Type', '', 'area 3', 'Location Location_Subregion1', '', 'area 4', 'Location Detailed_Location', '', 'latitude_resolution 1', 'Data_Resolution Latitude_Resolution', '', 'longitude_resolution 1', 'Data_Resolution Longitude_Resolution', '', '!DS_ownertag 1', 'Project Short_Name', '', 'distribution_statement', 'Access_Constraints', '', 'institution', 'Originating_Center', '', '', 'Data_Center Data_Center_Name Short_Name', 'met.no', '', 'Data_Center Data_Center_Name Long_Name', 'Norwegian Meteorological Institute', '', 'Data_Center Data_Center_URL', 'http://met.no/', '', 'Data_Center Personnel Role', 'DATA CENTER CONTACT', '', 'Data_Center Personnel First_Name', 'Egil', '', 'Data_Center Personnel Last_Name', 'Støren', '', 'Data_Center Personnel Phone', '+4722963000', '', 'Data_Center Personnel Contact_Address Address', "Norwegian Meteorological Institute\nP.O. Box 43\nBlindern", '', 'Data_Center Personnel Contact_Address City', 'Oslo', '', 'Data_Center Personnel Contact_Address Postal_Code', 'N-0313', '', 'Data_Center Personnel Contact_Address Country', 'Norway', 'references', 'Reference', '', 'abstract', 'Summary', '', '', 'Metadata_Name', 'CEOS IDN DIF', '', 'Metadata_Version', '9.7', '!DS_datestamp', 'Last_DIF_Revision_Date', '', '', 'Private', 'False', );
In this array, triples of three consecutive elements (starting with elements 0, 1 and 2 on the first line) constitute the translation of one metadata item in the database to one XML element in the DIF.
Each triple comprise the following components:
If these three components has, for example, the following contenet:
'institution', 'Data_Set_Citation Dataset_Publisher', '',
and “METNO Norwegian Meteorological Institute” is the value in the database found for the 'institution' metadata type, then the following text will be part of the DIF XML:
<DIF ...> ... <Data_Set_Citation> ... <Dataset_Publisher>METNO Norwegian Meteorological Institute</Dataset_Publisher> ... </Data_Set_Citation> ... </DIF>
The sequence of triples in the array define the sequence of the elements in the DIF XML. If possible, one triple will use the same higher level XML elements as the previous triple.
These points constitute the general rule, but some special rules apply:
METAMOD is essentially agnostic concerning file formats. However, as mentioned below, METAMOD has extended support for data uploaded as NetCDF/CF following the requirements of METAMOD.
Basically METAMOD can relate to any file as long as the metadata are created manually and uploaded, but metadata can be extracted automatically from NetCDF/CF files.
METAMOD has extended support for NetCDF/CF files where conformance checks and automatic extraction of metadata are supported. Some relevant information concerning NetCDF/CF can be found here:
Common Data Language provides an ASCII representation of NetCDF files. METAMOD do support direct upload of CDL files and will perform the same conformance checks in CDL as on NetCDF/CF files, as well as conversion to NetCDF/CF.
The following global attributes are required in NetCDF/CF files uploaded through METAMOD for proper stewardship. Please note that this is a minimum requirement and is subject to change (some proposed changes are indicated below).
Name | Purpose | Status | Comment |
---|---|---|---|
title | A short description of the data set | Active | |
abstract | A short summary of the data collection activity and data set. This element may alternatively be provided as the global attribute “comment” in a NetCDF file. | Active | |
topiccategory | A blank separated list of topic keywords describing the dataset. biota climatologyMeteorologyAtmosphere environment geoscientificinformation imageryBaseMapsEarthCover inlandWaters oceans society | Active | |
keywords | A blank separated list of keywords describing the dataset. | Active, TBR | Controlled vocabulary |
gcmd_keywords | Newline separated list of GCMD scientific keywords describing the various variables. This will be used to categorize the datasets according to the “Topics and variables” menu selection in the metadata search facility. If proper standard names have been used for the variables, data will be mapped for search under Topics and variables even without the gcmd_keywords attribute. | Active | Controlled vocabulary |
activity_type | Comma separated list of activity types. Current vocabulary includes Moored instrument Cruise Aircraft Model run Land station Ice station Submersible Float Space borne instrument Other | Active | Controlled vocabulary, should be harmonised with other existing vocabularies and simplified |
operational_status | Status of the dataset, being one of Experimental Pre operational Operational Scientific | Active | Controlled vocabulary |
Conventions | The metadata convention used, should be CF-1.x where x=1, 2, or 3. | Active | Controlled vocabulary |
product_name | A product name of the dataset. | Active | Controlled vocabulary |
history | Modification history of the dataset. Should be of the form: 2007-05-12 creation 2007-06-10 revision and separated by newlines. | Active | |
area | Area name describing the geographical area being studied.If several area names are used, separate them using comma. Arctic Ocean Barents Sea Beufort Sea Chukchi Sea Denmark Strait Sea East Siberian Sea Fram Strait Greenland Sea Iceland Sea Kara Sea Laptev Sea Nordic Seas Northern Hemisphere White Sea | Active | Controlled vocabulary, should be harmonised with other existing vocabularies. These keywords are mapped to GCMD geographical keywords currently. |
southernmost_latitude | Elements to describe a geographicalbounding box for the data. Should be a floating point value (decimal degrees). | Active | |
northernmost_latitude | Elements to describe a geographicalbounding box for the data. Should be a floating point value (decimal degrees). | Active | |
westernmost_longitude | Elements to describe a geographicalbounding box for the data. Should be a floating point value (decimal degrees) with positive values towards the East. | Active | |
easternmost_longitude | Elements to describe a geographicalbounding box for the data. Should be a floating point value (decimal degrees) with positive values towards the East. | Active | |
start_date | Start date and time of the dataset in the form 2007-06-12 12:30:00 UTC | Active | |
stop_date | Stop date and time of the dataset in the form 2007-06-12 12:30:00 UTC | Active | |
institution | Name of the institution responsible for the dataset. | Active | Controlled vocabulary |
PI_name | Name of the person responsible for the data set. | Active | |
PI_address | Address to the person responsible for the dataset. | Proposed | |
contact | email address to responsible user support or principal investigator. If the email address of the principal investigator is used, the variable PI_name should be set accordingly. | Active | |
citation | Information on how to cite the information properly. | Proposed | |
distribution_statement | A distribution statement describing the conditions for distributing and access to the dataset. Free Restricted to <project> until <YYYY-MM-DD> No access until <YYYY-MM-DD> | Active | Controlled vocabulary |
use_statement | A use statement describing conditions for using the dataset.Free Cite properly No commercial use, cite properly PIC/CC0 PIC/CCby | Proposed | Controlled vocabulary, or link to CC license or similar. |
project_name | Name of the project within which the data were collected. Several project dependencies can be indicated using comma separated project names. | Active | Controlled vocabulary |
One possible cause can be that there is a conflict between users. Please remember that all scripts and processes has to be run under the user that owns the web server (e.g. www-user or www-data). If this is not the case, try removing the webrun
directory and reinstall everything using sudo -u <user> …
.
Another possible cause is that the XML-information is inconsistent/incomplete. If metadata files has been added manually, check that they are consistent (e.g. using xmllint or similar tools) prior to import. Furthermore, in the current version each XML-file has to be accompanied by a XMD-file providing additional information on data sources and metadata organisation. If this is absent, metadata are not processed for import. Check the log in the webrun directory for details.
One possible cause can be that the dataset tag is not correct. Check the contents of the master_config.txt
and especially elements DATASET_TAGS
, UPLOAD_OWNERTAG
, QUEST_OWNERTAG
, and PMH_EXPORT_TAGS
for consistency.
See Why is METAMOD not including the XML-information located in the webrun directory? as well.