NorStore Tape Storage
Basic use
Instructions for basic use of the tape resources are found on NorStore's homepage.
Advanced use
A collection of high-level tape tools is available in /projects/NS2345K/tools.
noresm2tape
noresm2tape copies the output of a NorESM case from NorStore's disk area to NorStore's tape resource. The output folder should be organised in the standard CCSM way (e.g., atmospheric output is expected in 'atm/hist'). If that is not the case, please use the command disk2tape instead.
IMPORTANT: It is highly recommended to run the script in background using the nohup command, e.g.,
“nohup /projects/NS2345K/tools/noresm2tape <path on disk> <path on tape> &
”. Make sure not to forget the “&” at the end of the line. After submitting the script with nohup, it is safe to log out. You will receive a notification email when the transfer to tape is completed.
Run /projects/NS2345K/tools/noresm2tape -h
to print detailed instructions:
Usage: noresm2tape <path on disk> <path on tape> <chunk size> <replication flag> Example: noresm2tape /scratch/ingo/mycase /tape/NS2345K/cases/mycase auto replica Purpose: Copies NorESM case output from disk to tape. Description: <path on disk> and <path on tape> must be full path names (see example). <chunk size> must be one of: auto, all, 1, 10, 100, or 1000 (numbers indicate simulation years per chunk) <replication flag> must be either 'replica' - resulting in two tape copies - or 'noreplica'. The output folder <path on tape> will be created - with parent directories if necessary - and must not exist before running this script. The input data is staged in form of tar-chuncks in /scratch/ingo/noresm2tape. The tar-chunks are removed from scratch after successfull transfer to tape. Checksums for all input files are computed and stored in a separate checksum file. The checksums are used to verify the tar-chunks.
disk2tape
disk2tape copies a folder from NorStore's disk area to NorStore's tape resource. In contrast to noresm2tape, the data folder can have any structure and content.
IMPORTANT: It is highly recommended to run the script in background using the nohup command, e.g.,
“nohup /projects/NS2345K/tools/disk2tape <path on disk> <path on tape> &
”. Make sure not to forget the “&” at the end of the line. After submitting the script with nohup, it is safe to log out. You will receive a notification email when the transfer to tape is completed.
Run /projects/NS2345K/tools/disk2tape -h
to print detailed instructions:
Usage: disk2tape <path on disk> <path on tape> [none|gzip|bzip2] [replicate] Example: disk2tape /projects/NS2345K/hirlam /tape/NS2345K/hirlam Purpose: Copies a folder from disk to tape. Description: <path on disk> must be the absolute path to the input folder. <path on tape> will be created - with parent directories if necessary - and must not exist before running this script. gzip or bzip2 compression will be performed if the third argument is set to "gzip" or "bzip2", respectively. The default is no compression. If the forth argument is set to "replicate" then a second copy of the data will be stored on another physical tape medium. The path of the additional copy differs from the path of the first copy by that /tape is replaced with /replica. The input data is staged in form of tar-chuncks in /scratch/$USER/`basename $0`. The tar-chunks are removed from scratch after success transfer to tape. The size of a single tar-chunk is about 15Gb if no compression is applied. If compression is applied, the size of the tar-chunks can be up to 60Gb as a compression factor 4 is assumed. Checksums for all input files are computed and stored in a separate checksum file. The checksums are used to verify the tar-chunks.
tape2disk
tape2disk copies a folder that had been put on tape with either noresm2tape or disk2tape. All tar-archieves found in the top-level folder are unpacked while tar-files stored within tar-files are not inflated.
Run /projects/NS2345K/tools/tape2disk -h
to print detailed instructions:
Usage: /projects/NS2345K/tools/tape2disk <path on tape> <path on disk> Example: /projects/NS2345K/tools/tape2disk /tape/NS2345K/hirlam /scratch/ingo/hirlam Purpose: Retrieves a folder from tape and unpacks all tar-archieves contained in it. Description: <path on disk> will be created if it does not exist.
listontape
listontape lists all files that are archived in a tape folder.
Run /projects/NS2345K/tools/listontape -h
to print detailed instructions:
Usage: /projects/NS2345K/tools/listontape <path to data-directory on tape> or /projects/NS2345K/tools/listontape <path to checksum file on tape> Example: /projects/NS2345K/tools/listontape /tape/NS2345K/hirham_nobackup/BCM or /projects/NS2345K/tools/listontape /tape/NS2345K/hirham_nobackup/BCM/BCM.md5.tar Purpose: Lists the names of files that are archived in a given directory on tape. Description: The script relies on the existence of a checksum file, with extension md5.tar, that is stored in the archive directory. If exactly one checksum file exists then directory path is sufficient as input argument. However, if two or more checksum file exist then the absolute path to the checksum file has to be provided.