NorStore Tape Storage

Basic use

Instructions for basic use of the tape resources are found on NorStore's homepage.

Advanced use

disk2tape

A collection of high-level tape tools is available in /projects/NS2345K/tools.

Usage: noresm2tape <path on disk> <path on tape> <chunk size> <replication flag>
Example: noresm2tape /scratch/ingo/mycase /tape/NS2345K/cases/mycase auto replica
Purpose: Copies NorESM case output from disk to tape.
Description: <path on disk> and <path on tape> must be full path names (see example).
<chunk size> must be one of: auto, all, 1, 10, 100, or 1000 (numbers
indicate simulation years per chunk)
<replication flag> must be either 'replica' - resulting in two tape copies -
or 'noreplica'.
The output folder <path on tape> will be created - with parent
directories if necessary - and must not exist before running this
script.
The input data is staged in form of tar-chuncks in
/scratch/ingo/noresm2tape. The tar-chunks are removed from
scratch after successfull transfer to tape.
Checksums for all input files are computed and stored in a separate
checksum file. The checksums are used to verify the tar-chunks.

disk2tape

disk2tape is a script for copying a folder from NorStore's disk area to NorStore's tape resource. In contrast to noresm2tape, the data folder can have any structure and content.

IMPORTANT: It is highly recommended to run the script in background using the nohup command, e.g., “nohup /projects/NS2345K/tools/disk2tape <path on disk> <path on tape> &”. Make sure not to forget the “&” at the end of the line. After submitting the script with nohup, it is safe to log out. You will receive a notification email when the transfer to tape is completed.

Run /projects/NS2345K/tools/disk2tape -h to print detailed instructions:

Usage: disk2tape <path on disk> <path on tape> [none|gzip|bzip2] [replicate]
Example: disk2tape /projects/NS2345K/hirlam /tape/NS2345K/hirlam
Purpose: Copies a folder from disk to tape.
Description: <path on disk> must be the absolute path to the input folder.
<path on tape> will be created - with parent directories if
necessary - and must not exist before running this script.
gzip or bzip2 compression will be performed if the third argument is
set to "gzip" or "bzip2", respectively. The default is no compression.
If the forth argument is set to "replicate" then a second copy of
the data will be stored on another physical tape medium.
The path of the additional copy differs from the path of the first
copy by that /tape is replaced with /replica.
The input data is staged in form of tar-chuncks in
/scratch/$USER/`basename $0`. The tar-chunks are removed from
scratch after success transfer to tape.
The size of a single tar-chunk is about 15Gb if no compression is
applied. If compression is applied, the size of the tar-chunks can
be up to 60Gb as a compression factor 4 is assumed.
Checksums for all input files are computed and stored in a separate
checksum file. The checksums are used to verify the tar-chunks.

tape2disk

tape2disk is a script for retrieving a folder that had been put on tape with either noresm2tape or disk2tape. All tar-archieves found in the top-level folder are unpacked while tar-files that are stored within tar-files are not inflated.

Run /projects/NS2345K/tools/tape2disk -h to print detailed instructions:

Usage: /projects/NS2345K/tools/tape2disk <path on tape> <path on disk>
Example: /projects/NS2345K/tools/tape2disk /tape/NS2345K/hirlam /scratch/ingo/hirlam
Purpose: Retrieves a folder from tape and unpacks all tar-archieves contained in it.
Description: <path on disk> will be created if it does not exist.

listontape

listontape is a script list all files that are archived in a tape folder.

Run /projects/NS2345K/tools/listontape -h to print detailed instructions:

Usage: /projects/NS2345K/tools/listontape <path to data-directory on tape>
or
/projects/NS2345K/tools/listontape <path to checksum file on tape>
Example: /projects/NS2345K/tools/listontape /tape/NS2345K/hirham_nobackup/BCM
or
/projects/NS2345K/tools/listontape /tape/NS2345K/hirham_nobackup/BCM/BCM.md5.tar
Purpose: Lists the names of files that are archived in a given directory on tape.
Description: The script relies on the existence of a checksum file, with extension
md5.tar, that is stored in the archive directory.
If exactly one checksum file exists then directory path is sufficient
as input argument. However, if two or more checksum file exist then
the absolute path to the checksum file has to be provided.