====== NorStore Tape Storage ======

===== Basic use =====

Instructions for basic use of the tape resources are found on NorStore's  [[https://www.norstore.no/services/tape-storage|homepage]].

===== Advanced use =====

A collection of high-level tape tools is available in **/projects/NS2345K/tools**.

==== noresm2tape ====

noresm2tape copies the output of a NorESM case from NorStore's disk area to NorStore's tape resource. The output folder should be organised in the standard CCSM way (e.g., atmospheric output is expected in 'atm/hist'). If that is not the case, please use the command disk2tape instead.

**IMPORTANT:** It is highly recommended to run the script in background using the nohup command, e.g., 
 "''nohup /projects/NS2345K/tools/noresm2tape <path on disk> <path on tape> &''". Make sure not to forget the "&" at the end of the line. After submitting the script with nohup, it is safe to log out. You will receive a notification email when the transfer to tape is completed.   


Run ''/projects/NS2345K/tools/noresm2tape -h'' to print detailed instructions:
<file>

Usage: noresm2tape <path on disk> <path on tape> <chunk size> <replication flag>
 
Example: noresm2tape /scratch/ingo/mycase /tape/NS2345K/cases/mycase auto replica
 
Purpose: Copies NorESM case output from disk to tape. 

Description: <path on disk> and <path on tape> must be full path names (see example). 
             
             <chunk size> must be one of: auto, all, 1, 10, 100, or 1000 (numbers 
             indicate simulation years per chunk)
          
             <replication flag> must be either 'replica' - resulting in two tape copies -
             or 'noreplica'.  
                          
             The output folder <path on tape> will be created - with parent  
             directories if necessary - and must not exist before running this 
             script. 

             The input data is staged in form of tar-chuncks in 
             /scratch/ingo/noresm2tape. The tar-chunks are removed from 
             scratch after successfull transfer to tape. 

             Checksums for all input files are computed and stored in a separate 
             checksum file. The checksums are used to verify the tar-chunks.  
</file>


==== disk2tape ====

disk2tape copies a folder from NorStore's disk area to NorStore's tape resource. In contrast to noresm2tape, the data folder can have any structure and content. 

**IMPORTANT:** It is highly recommended to run the script in background using the nohup command, e.g., 
 "''nohup /projects/NS2345K/tools/disk2tape <path on disk> <path on tape> &''". Make sure not to forget the "&" at the end of the line. After submitting the script with nohup, it is safe to log out. You will receive a notification email when the transfer to tape is completed.   

Run ''/projects/NS2345K/tools/disk2tape -h'' to print detailed instructions:
<file>
Usage: disk2tape <path on disk> <path on tape> [none|gzip|bzip2] [replicate] 
 
Example: disk2tape /projects/NS2345K/hirlam /tape/NS2345K/hirlam 
 
Purpose: Copies a folder from disk to tape. 

Description: <path on disk> must be the absolute path to the input folder. 
              
             <path on tape> will be created - with parent directories if 
             necessary - and must not exist before running this script. 
             
             gzip or bzip2 compression will be performed if the third argument is 
             set to "gzip" or "bzip2", respectively. The default is no compression.
                 
             If the forth argument is set to "replicate" then a second copy of 
             the data will be stored on another physical tape medium. 
             The path of the additional copy differs from the path of the first 
             copy by that /tape is replaced with /replica. 

             The input data is staged in form of tar-chuncks in 
             /scratch/$USER/`basename $0`. The tar-chunks are removed from 
             scratch after success transfer to tape. 

             The size of a single tar-chunk is about 15Gb if no compression is 
             applied. If compression is applied, the size of the tar-chunks can 
             be up to 60Gb as a compression factor 4 is assumed.  

             Checksums for all input files are computed and stored in a separate 
             checksum file. The checksums are used to verify the tar-chunks.  
</file>

==== tape2disk ====

tape2disk copies a folder that had been put on tape with either noresm2tape or disk2tape. All tar-archieves found in the top-level folder are unpacked while tar-files stored within tar-files are not inflated.  

Run ''/projects/NS2345K/tools/tape2disk -h'' to print detailed instructions:
<file>

Usage: /projects/NS2345K/tools/tape2disk <path on tape> <path on disk> 
 
Example: /projects/NS2345K/tools/tape2disk /tape/NS2345K/hirlam /scratch/ingo/hirlam 
 
Purpose: Retrieves a folder from tape and unpacks all tar-archieves contained in it. 

Description: <path on disk> will be created if it does not exist. 

</file>

==== listontape ====

listontape lists all files that are archived in a tape folder. 

Run ''/projects/NS2345K/tools/listontape -h'' to print detailed instructions:
<file>

Usage: /projects/NS2345K/tools/listontape <path to data-directory on tape> 
        or 
       /projects/NS2345K/tools/listontape <path to checksum file on tape>

Example: /projects/NS2345K/tools/listontape /tape/NS2345K/hirham_nobackup/BCM 
          or 
         /projects/NS2345K/tools/listontape /tape/NS2345K/hirham_nobackup/BCM/BCM.md5.tar
  
Purpose: Lists the names of files that are archived in a given directory on tape.  

Description: The script relies on the existence of a checksum file, with extension 
             md5.tar, that is stored in the archive directory. 

             If exactly one checksum file exists then directory path is sufficient 
             as input argument. However, if two or more checksum file exist then 
             the absolute path to the checksum file has to be provided.  

</file>