Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
noresm:norstoreesg [2014-04-24 12:57:12] ingo.bethke@gmail.com |
noresm:norstoreesg [2022-05-31 09:29:32] (current) |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== Data sharing via NorStore' | ||
| - | ===== Basic use ===== | + | ====== Simple online sharing ====== |
| + | |||
| + | ===== Share data online ===== | ||
| + | |||
| + | For simple data sharing, | ||
| + | < | ||
| + | Usage: | ||
| + | | ||
| + | or | ||
| + | | ||
| + | |||
| + | Example: | ||
| + | | ||
| + | or | ||
| + | | ||
| + | |||
| + | Purpose: | ||
| + | | ||
| + | |||
| + | Description: | ||
| + | The input path must be absolute and start with / | ||
| + | |||
| + | Only regular files are considered. Symbolic links are skipped. | ||
| + | |||
| + | All date files must have unique names as the folder stucture is | ||
| + | | ||
| + | |||
| + | | ||
| + | | ||
| + | can be closed. | ||
| + | |||
| + | < | ||
| + | | ||
| + | </ | ||
| + | |||
| + | The script provides following: | ||
| + | *local path to a folder that contains hard-links to the data files | ||
| + | *URL that points to the online-catalogue where the shared files are exposed | ||
| + | *download-URL to a wget-script that can be used to download the entire dataset (note that execution permissions have to be changed prior to running the script) | ||
| + | |||
| + | NOTE: The sharing method uses hard-links. This has the advantages that no additional disk space is used and that the sharing happens instantaneous. A removal of the hard-links will unshare the data, but will not affect the original files. | ||
| + | |||
| + | ===== Unshare data ===== | ||
| + | |||
| + | If you don't want to share a dataset anymore, use the script **/ | ||
| + | < | ||
| + | Usage: wwwunpublish <dataset name> | ||
| + | |||
| + | Example: wwwunpublish CORE2-AMOC | ||
| + | |||
| + | Purpose: Unpublish a dataset from NorStore' | ||
| + | </ | ||
| + | |||
| + | NOTE: Unsharing data will not affect the original data. | ||
| + | |||
| + | ====== Data sharing via the Norwegian ESG data portal ====== | ||
| + | |||
| + | ====IMPORTANT: | ||
| + | |||
| + | |||
| + | [[http:// | ||
| + | |||
| + | ===== Simplified publishing (unofficial ESG publication)===== | ||
| All users of the NorStore project ns2345k have the possibility to share their data via NorStore' | All users of the NorStore project ns2345k have the possibility to share their data via NorStore' | ||
| Requirements are: | Requirements are: | ||
| - | * the data needs to reside in /projects/ns2345k | + | * the data has to reside in /projects/NS2345K (same as / |
| - | * only netcdf files are considered | + | |
| * all files must be regular (i.e., no symbolic links) | * all files must be regular (i.e., no symbolic links) | ||
| * file names must all be unique (folder hierarchy is flattened during publication) | * file names must all be unique (folder hierarchy is flattened during publication) | ||
| - | * reading permissions of data files must be open to everyone (does not apply to folders) | + | * reading permissions of data files must be open to everyone (does not apply to their parent |
| Note that the data sharing solution uses hard links to avoid additional use of disk resources. The original data is not touched/ | Note that the data sharing solution uses hard links to avoid additional use of disk resources. The original data is not touched/ | ||
| - | ==== Publishing | + | ==== Publish |
| - | For simple data sharing, use the script / | + | For simple data sharing |
| < | < | ||
| - | Usage: esgpublish <path to data directory> | + | Usage: esgpublish <absolute |
| + | or | ||
| + | | ||
| Example: esgpublish / | Example: esgpublish / | ||
| - | + | or | |
| + | | ||
| Purpose: Publish data via NorStore' | Purpose: Publish data via NorStore' | ||
| - | Description: | + | Description: |
| + | / | ||
| - | Only regular files (i.e., no symbolic links) | + | Only regular files (i.e., no symbolic links) are considered. |
| - | considered. | + | |
| | | ||
| All date files must have unique names as the folder stucture is | All date files must have unique names as the folder stucture is | ||
| Line 36: | Line 100: | ||
| | | ||
| | | ||
| - | < | + | < |
| | | ||
| - | + | ||
| - | Change history: 2014.04.24 first version of esgpublish | + | If a third argument is specified and set to " |
| + | | ||
| + | to the portal. | ||
| </ | </ | ||
| - | ==== Removing | + | ==== Remove |
| - | To unpublish a dataset that previously has been published with esgpublish, use the script / | + | To unpublish a dataset that previously has been published with esgpublish, use the script |
| < | < | ||
| Usage: esgunpublish <dataset name> | Usage: esgunpublish <dataset name> | ||
| Line 51: | Line 117: | ||
| Purpose: Unpublish a dataset from NorStore' | Purpose: Unpublish a dataset from NorStore' | ||
| - | |||
| - | Change history: 2014.04.24 first version of esgunpublish | ||
| </ | </ | ||
| - | ===== Advanced use ===== | + | ===== Standard publishing (official ESG publication) |
| Several projects (e.g, CMIP, GeoMIP, etc) require special post-processing and ESG publishing procedures. | Several projects (e.g, CMIP, GeoMIP, etc) require special post-processing and ESG publishing procedures. | ||
| Please contact Ingo Bethke (ingo.bethke[at]uni.no) or Alf Grini (alfg[at]met.no) for assistance. | Please contact Ingo Bethke (ingo.bethke[at]uni.no) or Alf Grini (alfg[at]met.no) for assistance. | ||
| - | |||
| - | |||