Differences

This shows you the differences between two versions of the page.

--- noresm:norstoreesg [2014-04-24 12:20:29]
ingo.bethke@gmail.com created
+++ noresm:norstoreesg [2022-05-31 09:29:32] (current)
@@ Line 1: / Line 1: @@
-====== Data sharing via NorStore's Earth System Grid data portal ======
-===== Basic use =====
+====== Simple online sharing ======
-All NorStore users of the project ns2345k have the possibility to share their data via NorStore's ESG data portal [[http://noresg.norstore.no|http://noresg.norstore.no]].
+===== Share data online =====
+For simple data sharing, use the script **/projects/NS2345K/tools/wwwpublish**:
+<file>
+Usage:
+   wwwpublish <absolute path to data directory> <dataset name>
+   or
+   wwwpublish <absolute path to data file> <dataset name>
+Example:
+   wwwpublish /projects/NS2345K/noresm/thredds/CORE2/AMOC CORE2-AMOC
+   or
+   wwwpublish /projects/NS2345K/noresm/noresm2cmor.tbz noresm2cmor
+Purpose:
+   Publish data online at http://ns2345k.web.nird.no
+Description:
+   The input path must be absolute and start with /projects/NS2345K.
+   Only regular files are considered. Symbolic links are skipped.
+   All date files must have unique names as the folder stucture is
+   flattened during publication.
+   Reading permissions of the data files have to be open to everyone.
+   However, reading permissions of the folders containing the files
+   can be closed.
+   <dataset name> should be one word and not contain any special
+   characters other than "-" and "_".
+</file>
+The script provides following:
+  *local path to a folder that contains hard-links to the data files
+  *URL that points to the online-catalogue where the shared files are exposed
+  *download-URL to a wget-script that can be used to download the entire dataset (note that execution permissions have to be changed prior to running the script)
+NOTE: The sharing method uses hard-links. This has the advantages that no additional disk space is used and that the sharing happens instantaneous. A removal of the hard-links will unshare the data, but will not affect the original files.
+===== Unshare data =====
+If you don't want to share a dataset anymore, use the script **/projects/NS2345K/tools/wwwunpublish**:
+<file>
+Usage: wwwunpublish <dataset name>
+Example: wwwunpublish CORE2-AMOC
+Purpose: Unpublish a dataset from NorStore's www-server
+</file>
+NOTE: Unsharing data will not affect the original data.
+====== Data sharing via the Norwegian ESG data portal ======
+====IMPORTANT: The portal is currently offline for maintenance and its operation is expected to be resumed sometime in October. For an alternative data sharing method see previous section "Simple data sharing". ====
+[[http://noresg.norstore.no|{{ :noresm:noresg.png?nolink&512 |}}]]
+===== Simplified publishing (unofficial ESG publication)=====
+All users of the NorStore project ns2345k have the possibility to share their data via NorStore's ESG data portal [[http://noresg.norstore.no|http://noresg.norstore.no]].
 Requirements are:
-  * the data needs to reside in /projects/ns2345k
+  * the data has to reside in /projects/NS2345K (same as /norstore_osl/projects/NS2345K)
-  * only netcdf files are considered
   * all files must be regular (i.e., no symbolic links)
-  * file names must all be unique (because folder hierarchy is flattened)
+  * file names must all be unique (folder hierarchy is flattened during publication)
-  * reading permissions of data files must be open to everyone (does not apply to folders)
+  * reading permissions of data files must be open to everyone (does not apply to their parent folders)
+Note that the data sharing solution uses hard links to avoid additional use of disk resources. The original data is not touched/modified.
+==== Publish a simple dataset to NorStore's ESG data portal ====
+For simple data sharing with ESG support, use the script **/projects/NS2345K/tools/esgpublish**:
+<file>
+Usage: esgpublish <absolute path to data directory> <dataset name> [unlisted]
+       or
+       esgpublish <absolute path to data file> <dataset name> [unlisted]
+Example: esgpublish /projects/NS2345K/noresm/thredds/CORE2/AMOC CORE2-AMOC
+         or
+         esgpublish /projects/NS2345K/noresm/noresm2cmor.tbz noresm2cmor unlisted
+Purpose: Publish data via NorStore's online data portal http://noresg.norstore.no
+Description: The input path must be absolute and start with /projects/NS2345K or
+             /norstore_osl/projects/NS2345K.
+             Only regular files (i.e., no symbolic links) are considered.
+             All date files must have unique names as the folder stucture is
+             flattened during publication.
+             Reading permissions of the data files have to be open to everyone,
+             but reading permissions of the folders containing the files can be
+             closed.
+             <dataset name> should be one word and not contain any special
+             characters other than "-" and "_".
+             If a third argument is specified and set to "unlisted" then the
+             dataset will be served through the file server but not published
+             to the portal.
+</file>
+==== Remove a simple dataset from NorStore's ESG data portal ====
+To unpublish a dataset that previously has been published with esgpublish, use the script **/projects/NS2345K/tools/esgunpublish**:
+<file>
+Usage: esgunpublish <dataset name>
+Example: esgunpublish CORE2-AMOC
+Purpose: Unpublish a dataset from NorStore's online data portal http://noresg.norstore.no
+</file>
-==== Publishing a dataset to NorStore's ESG data portal ====
+===== Standard publishing (official ESG publication) =====
+Several projects (e.g, CMIP, GeoMIP, etc) require special post-processing and ESG publishing procedures.
-==== Removing a dataset from NorStore's ESG data portal ====
+Please contact Ingo Bethke (ingo.bethke[at]uni.no) or Alf Grini (alfg[at]met.no) for assistance.