wdb:manuals:wdb_metadata

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision 		Previous revision
wdb:manuals:wdb_metadata [2012-04-04 12:58:16]
michaeloa created 		wdb:manuals:wdb_metadata [2022-05-31 09:29:32] (current)
Line 1: Line 1:
 ====== WDB Metadata User Manual ====== ====== WDB Metadata User Manual ======
  
-NOTE: This document is not yet completed.+NOTE: This manual is incomplete.
  
  
Line 10: Line 10:
 This is the User's Manual for WDB Metadata. This is the User's Manual for WDB Metadata.
  
----- +===== Introduction =====
- +
-**Table of Contents** +
- +
-[[#cha:metadata_introduction|1. Introduction]][[#sec:about_manual|About this Manual]][[#sec:intended_usage|Intended Usage]][[#sec:metadata_outline|Outline]][[#cha:key_concepts|2. Key Concepts]][[#sec:data_dimensions|WDB Metadata Dimensions]][[#sec:namespaces|WDB Namespaces]][[#cha:data_provider|3. Data Provider]][[#sec:data_provider|Definition]][[#sec:data_provider_browse|Browse Data Providers]][[#sec:data_provider_add|Add Data Providers]][[#sec:data_provider_group|Data Provider Groups]][[#cha:place_definition|4. Place (Geographic Location)]][[#sec:place_definition|Definition]][[#sec:place_definition_browse|Browse Place Definitions]][[#sec:place_definition_add|Add place Definitions]][[#sec:srid|Data Provider Groups]][[#sec:place|Place (Geographic Location)]][[#sec:place:addplacedef|Adding a Place Definition]] +
- +
-===== Chapter�1.�Introduction ===== +
- +
-**Table of Contents** +
- +
-[[#sec:about_manual|About this Manual]][[#sec:intended_usage|Intended Usage]][[#sec:metadata_outline|Outline]]+
  
 The WDB system contains a lot of metadata that is used to identify and describe the values stored in the database. The WDB system administrator must have a good understanding of this metadata in order to maintain the system effectively. The WDB system contains a lot of metadata that is used to identify and describe the values stored in the database. The WDB system administrator must have a good understanding of this metadata in order to maintain the system effectively.
  
-===== About this Manual =====+==== About this Manual ====
  
 This manual is intended for the system administrator of the WDB system. It describes how to maintain the metadata in the WDB system. It is assumed that the reader has some familiarity with SQL and weather data. This manual is intended for the system administrator of the WDB system. It describes how to maintain the metadata in the WDB system. It is assumed that the reader has some familiarity with SQL and weather data.
Line 30: Line 20:
 For a more elaborate description of the vision and system architecture of the WDB system, see the WDB Developer's Manual. For a more elaborate description of the vision and system architecture of the WDB system, see the WDB Developer's Manual.
  
-===== Intended Usage =====+==== Intended Usage ====
  
 The metadata in WDB is used to describe the values stored in the system. By carefully and correctly maintaining this metadata, it is possible to view stored data from several different points of view, while maintaining the consistency of the stored data. The metadata in WDB is used to describe the values stored in the system. By carefully and correctly maintaining this metadata, it is possible to view stored data from several different points of view, while maintaining the consistency of the stored data.
  
-===== Outline =====+==== Outline ====
  
 This document is structured as follows: This document is structured as follows:
Line 47: Line 37:
   * Known bugs and limitations   * Known bugs and limitations
  
-===== Chapter�2.�Key Concepts ===== +===== Key Concepts =====
- +
-**Table of Contents** +
- +
-[[#sec:data_dimensions|WDB Metadata Dimensions]][[#sec:namespaces|WDB Namespaces]]+
  
 This section defines and explains some of the key concepts of the WDB metadata system. This section defines and explains some of the key concepts of the WDB metadata system.
  
-===== WDB Metadata Dimensions =====+==== WDB Metadata Dimensions ====
  
 WDB is a database system that stores weather data items. Each data item in WDB could be an observation, a forecast, an analysis, etc. Each item consists of a value and a number of dimensions that describe the value. WDB is a database system that stores weather data items. Each data item in WDB could be an observation, a forecast, an analysis, etc. Each item consists of a value and a number of dimensions that describe the value.
Line 71: Line 57:
 Of these dimensions, data version is simply a number sequence and will not be discussed further here. The other dimensions will be discussed in greater detail in the following chapters. Of these dimensions, data version is simply a number sequence and will not be discussed further here. The other dimensions will be discussed in greater detail in the following chapters.
  
-===== WDB Namespaces =====+==== WDB Namespaces ====
  
 There are a large number of different naming conventions that exist in the meteorological world. Each of these naming conventions may be valid in its own context, but there is also typically a large overlap (for instance, a meteorological station may be identified by several different indicators or index numbers). WDB provides name spaces as a mechanism to handle this kind of issue. There are a large number of different naming conventions that exist in the meteorological world. Each of these naming conventions may be valid in its own context, but there is also typically a large overlap (for instance, a meteorological station may be identified by several different indicators or index numbers). WDB provides name spaces as a mechanism to handle this kind of issue.
Line 81: Line 67:
 A namespace in WDB is usually identified by a numerical ID. The default namespace uses ID 0; the test namespace (utilized for the WDB testing framework) is defined by the ID 999. The namespaces from 1 to 254 are reserved for usage by WMO centers (we use the same ID code as the WMO numbers). A namespace in WDB is usually identified by a numerical ID. The default namespace uses ID 0; the test namespace (utilized for the WDB testing framework) is defined by the ID 999. The namespaces from 1 to 254 are reserved for usage by WMO centers (we use the same ID code as the WMO numbers).
  
-===== Chapter�3.�Data Provider ===== +===== Data Provider =====
- +
-**Table of Contents** +
- +
-[[#sec:data_provider|Definition]][[#sec:data_provider_browse|Browse Data Providers]][[#sec:data_provider_add|Add Data Providers]][[#sec:data_provider_group|Data Provider Groups]]+
  
 This section defines the metadata for data providers. This section defines the metadata for data providers.
  
-===== Definition =====+==== Definition ====
  
 The Data Provider identifies the source of the data; literally, //the entity that provides the data//. Where multiple sources could be identified as the source of the data, the entity that can be identified closest to the creation of the data item at the time of loading is usually used. The Data Provider identifies the source of the data; literally, //the entity that provides the data//. Where multiple sources could be identified as the source of the data, the entity that can be identified closest to the creation of the data item at the time of loading is usually used.
Line 99: Line 81:
 DataProviderNames exist within a data provider namespace. A namespace can be defined by the WDB administrator, in order to permit the user or an application to retrieve data in an accustomed language or code set. The default namespace of WDB is the DataProviderNameSpaceId 0, and is always based on English language names and international standard codes. The data provider namespace being used in a querying session can be defined by the user when starting up the session. DataProviderNames exist within a data provider namespace. A namespace can be defined by the WDB administrator, in order to permit the user or an application to retrieve data in an accustomed language or code set. The default namespace of WDB is the DataProviderNameSpaceId 0, and is always based on English language names and international standard codes. The data provider namespace being used in a querying session can be defined by the user when starting up the session.
  
-===== Browse Data Providers =====+==== Browse Data Providers ====
  
 To retrieve all of the DataProviderNames (excluding data provider groups) that are currently stored in the database for the currently specified namespace, the following wci.browse function call could be used: To retrieve all of the DataProviderNames (excluding data provider groups) that are currently stored in the database for the currently specified namespace, the following wci.browse function call could be used:
Line 109: Line 91:
   SELECT * FROM wci.getDataProvider( NULL );   SELECT * FROM wci.getDataProvider( NULL );
  
-===== Add Data Providers =====+==== Add Data Providers ====
  
 To add a new data provider to the database, the wci.adddataprovider function is used. The following function call adds the dataprovider gribload to the database. To add a new data provider to the database, the wci.adddataprovider function is used. The following function call adds the dataprovider gribload to the database.
Line 136: Line 118:
 The fifth parameter is a comment field. Ideally, it contains a few brief lines of text that describe and explain the data provider entity. The fifth parameter is a comment field. Ideally, it contains a few brief lines of text that describe and explain the data provider entity.
  
-===== Data Provider Groups =====+==== Data Provider Groups ====
  
 Data provider groups permit the user a convenient way to retrieve a set of data providers without having to specify each Data provider groups permit the user a convenient way to retrieve a set of data providers without having to specify each
Line 148: Line 130:
   SELECT * FROM wci.getDataProvider( NULL );   SELECT * FROM wci.getDataProvider( NULL );
  
-===== Chapter�4.�Place (Geographic Location) ===== +===== Place (Geographic Location) =====
- +
-**Table of Contents** +
- +
-[[#sec:place_definition|Definition]][[#sec:place_definition_browse|Browse Place Definitions]][[#sec:place_definition_add|Add place Definitions]][[#sec:srid|Data Provider Groups]][[#sec:place|Place (Geographic Location)]][[#sec:place:addplacedef|Adding a Place Definition]]+
  
 This section defines the metadata for place definitions. This section defines the metadata for place definitions.
  
-===== Definition =====+==== Definition ====
  
 The place (geographic location) of a data item is the position of the item on the earth in a 2D space. In WDB, the geographic location is by default specified using longitude and latitude in a WGS84 coordinate system (though this can be changed when the database is set up; consult your system administrator). The geographic dimension is specified using a geometry object and can be either a point or a polygon. The place (geographic location) of a data item is the position of the item on the earth in a 2D space. In WDB, the geographic location is by default specified using longitude and latitude in a WGS84 coordinate system (though this can be changed when the database is set up; consult your system administrator). The geographic dimension is specified using a geometry object and can be either a point or a polygon.
Line 166: Line 144:
 Place Definitions are always defined using a spatial reference ID (SRID); defined using PROJ.4 definition strings. WDB is set up with the default SRIDs that come with the Postgis package; additional SRIDs as required by new data types are added as needed in WDB meta data packages. Place Definitions are always defined using a spatial reference ID (SRID); defined using PROJ.4 definition strings. WDB is set up with the default SRIDs that come with the Postgis package; additional SRIDs as required by new data types are added as needed in WDB meta data packages.
  
-===== Browse Place Definitions =====+==== Browse Place Definitions ====
  
 To retrieve all of the Place Definition Names that are currently stored in the database for the currently specified namespace, the following wci.browse function call could be used: To retrieve all of the Place Definition Names that are currently stored in the database for the currently specified namespace, the following wci.browse function call could be used:
Line 180: Line 158:
   SELECT * FROM wci.getsrid( NULL );   SELECT * FROM wci.getsrid( NULL );
  
-===== Add place Definitions =====+==== Add place Definitions ====
  
-There are various differing functions to add a place definition to the database, depending on the type of place definition to be added. The function wci.addplacepoint is used for adding point data, while wci.addplaceregulargrid is used to define regular grids as follows.+There are various differing functions to add a place definition to the database, depending on the type of place definition to be added. The function wci.addplacepoint is used for adding point data, while wci.addplaceregulargrid is used to define regular grids as follows:
  
-  select wci.addplacepoint( 'oslo', +  select wci.addplacepoint('oslo', 
-                            'POINT(10.7564 59.9494)' );+                           st_geomfromtext('POINT(10.7464 59.9111)',4030));
  
-The first parameter in the function is the place name, while the second is the geographical position of the place definition in the default coordinate system of the WDB instance.+The first parameter in the function is the place name, while the second is the geographical position of the place definition in the default coordinate system of the WDB instance (as a binary geometry). Note that the SRID provided must be the same as the default PostGIS SRID used to build WDB (by default, this tends to be PostGIS ID 4030).
  
   select wci.addplaceregulargrid( 'ecmwf 0.5 grid',   select wci.addplaceregulargrid( 'ecmwf 0.5 grid',
Line 201: Line 179:
  
 The fifth parameter is a comment field. Ideally, it contains a few brief lines of text that describe and explain the data provider entity. The fifth parameter is a comment field. Ideally, it contains a few brief lines of text that describe and explain the data provider entity.
 +==== Adding a Place Definition ====
  
-===== Data Provider Groups =====+=== Adding a Place Definition Point ===
  
-Data provider groups permit the user a convenient way to retrieve a set of data providers without having to specify each+=== Adding a Place Definition Grid ===
  
-retrieve all of the DataProviderNames (excluding data provider groups) that are currently stored in the database for the currently specified namespace, the following wci.browse function call could be used:+==== Spatial Reference ID ==== 
  
-select wci.adddataprovider( 'demload', 'computer system', 'grid', '1 day', 'Data received from WDB demLoad (unidentified)' ); select wci.adddataprovidertogroup( 'demload', 'wdb');+All SRIDs must be defined with "+no_defs". Not including the "+no_defs" parameter in the PROJ.4 transformation string makes the SRID dependent on the /usr/share/proj/proj_def.dat, essentially breaking the integrity of the metadata in the database. 
 +==== Setting a Place Name ====
  
-To retrieve all of the DataProviderNames (including data provider groups) that the database currently has the capacity to store and display in the currently specified namespace, the following wci.info function call could be used: 
  
-  SELECT * FROM wci.getDataProvider( NULL ); 
  
-Spatial Reference ID (SRID) The Spatial Reference ID in the database is used for conversion between the different projections that we encounter data in. The database is set up to use the "standard" WGS84 longitude-latitude projection as its base projection. 
  
-===== Place (Geographic Location) ===== 
  
-==== Adding a Place Definition ====+===== Parameters ===== 
 + 
 +This section defines the metadata for parameters.  
 + 
 +==== Definition ==== 
 + 
 +The Parameter in WDB identifies the characteristic or measurable factor of the value being parameterized. Parameters provide a definitive description of what the data represents, including spatial and temporal properties. The parameter names are based around the NetCDF climate and forecast (CF) metadata conventions. Unfortunately, CF standard names are not well suited for the purposes of WDB, as they lack uniqueness; consequently, the WDB parameter system is only based on, rather than a direct adaption of the CF metadata convention. Every CF standard name should be possible to map directly to a WDB parameter, but the converse may not always be the case. 
 + 
 +Parameter names exist with a parameter namespace. A namespace can be defined by the WDB administrator, in order to permit the user or an application to retrieve data in an accustomed language or code set. The default namespace of WDB is the ParameterNameSpaceId 0, and is always based on English language names and maps CF standard names. The parameter namespace being used in a querying session can be defined by the user when starting up the session. 
 + 
 +==== Browse Parameters ==== 
 + 
 +To retrieve all of the value parameters that are currently stored in the database for the currently specified namespace, the following wci.browse function call could be used: 
 + 
 +  SELECT * FROM wci.browse( NULL::wci.valueparameter ); 
 + 
 +For the level parameters, use  
 + 
 +  SELECT * FROM wci.browse( NULL::wci.levelparameter ); 
 + 
 +To retrieve all of the ValueParameterNames and LevelParameterNames that the database currently has the capacity to store and display in the currently specified namespace, the following wci.info function call could be used: 
 + 
 +  SELECT * FROM wci.getparameter( NULL ); 
 + 
 +==== Add Parameters ==== 
 + 
 +To add a parameter to the database, the wci.addvalueparameter function is used. The default parameter name structure is constructed based on the [[http://cf-pcmdi.llnl.gov/documents/cf-standard-names/guidelines|Guidelines for Constructing CF Standard Names]]. The function call for adding a parameter is as follows: 
 + 
 +  select wci.addparameter( 'standard-name', 
 +            'surface', 
 +            'component', 
 +    'medium', 
 +    'process', 
 +    'condition', 
 +            'methods', 
 +    'unit of measure' )  
 + 
 +This gives a default parameter in the default namespace of: 
 + 
 +  single-word-surface component standard-name //at// multi-word-surface //in// medium //due to// process //assuming// condition [methods] 
 + 
 +Standard name is one of the CF standard names. 
 + 
 +[[http://cf-pcmdi.llnl.gov/documents/cf-standard-names/guidelines#surface|Surface]] is defined as a function of a horizontal position. A new surface is added to the database using the wci.addcfsurface function, for example: 
 + 
 +  select wci.addcfsurface( 'sea level', 'MSL - mean sea level' ) 
 + 
 +[[http://cf-pcmdi.llnl.gov/documents/cf-standard-names/guidelines#component|Component]] defines the spatial component of the parameter. A new component is added to the database using the wci.addcfcomponent function, for example: 
 + 
 +  select wci.addcfcomponent( 'upward', 'Upward component' ); 
 + 
 +[[http://cf-pcmdi.llnl.gov/documents/cf-standard-names/guidelines#medium|Medium]] indicated the local medium or layer of the parameter. A new medium is added to the database using the wci.addcfmedium function. Example: 
 + 
 +  select wci.addcfmedium( 'atmosphere layer', 'Atmosphere layer medium' ) 
 + 
 +[[http://cf-pcmdi.llnl.gov/documents/cf-standard-names/guidelines#process|Process]] specifies a physical process. A new process is added to the database using the wci.addcfprocess function. Example: 
 + 
 +  select wci.addcfprocess( 'large scale precipitation', 'Due to large scale precipitation' ) 
 + 
 +[[http://cf-pcmdi.llnl.gov/documents/cf-standard-names/guidelines#condition|Condition]] indicates special circumstances of the parameter. A new condition is added to the database using the wci.addcfcondition function. Example: 
 + 
 +  select wci.addcfcondition( 'deep snow', 'Assumption of deep snow' ) 
 + 
 +Methods indicate the calculations used for the parameter. A new method is added to the database using the wci.addcfmethods function. Example: 
 + 
 +  select wci.addcfmethods('maximum within days', 'Maximum value', 'max' ); 
 + 
 +Unit of measure is the standard unit of measure used by the parameter. WDB uses SI units defined using context-sensitive [[http://unitsofmeasure.org/|UCUM]]. 
 + 
 +The default (canonical) parameter name is constructed using the various components described above. 
 + 
 +For examples of adding new parameters to WDB, see the [[https://github.com/wdb/wdb/blob/master/etc/metadata/wdb_parameter.in.sql|wdb_parameters]] install files. 
 + 
 +==== Set Parameter Names in Namespace ==== 
 + 
 +The canonical parameter name is valid for the default (0) parameter name space only. To set the parameter name in other namespaces, the wci.setparametername function should be used. Example: 
 + 
 +  select wci.setparametername( 'air temperature', 'TEMP' ); 
 + 
 +This sets a parameter name 'TEMP' in the currently defined namespace, which is equivalent to the canonical parameter 'air temperature'. 
 + 
 +If you are satisfied with using CF-like parameters, then the function wci.copyParameterNameSpace( 0 ) can be used to copy all of the parameters in the default namespace into the currently defined parameter namespace.  
 + 
 +  select wci.copyParameterNameSpace( 0 )
  
-Adding a Place Definition Point Grid Spatial Reference ID Setting a Place Name PlaceName+Parameters created using this method are similar to the CF standard name, except that the short form of the CF methods precedes the CF standard name in order to generate a more natural language parameter description. Thus 'air temperature [maximum over days]' becomes 'max air temperature'.
  
 +For an example of adapting and adding parameters to a private parameter namespace, see the [[https://github.com/metno/wdb-metadata/blob/master/etc/wdb_parameter.in.sql|Met.no WDB metadata files]].
  • wdb/manuals/wdb_metadata.1333544296.txt.gz
  • Last modified: 2022-05-31 09:23:29
  • (external edit)