noresm:norstoretape

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision 		Previous revision
noresm:norstoretape [2014-04-16 09:26:20]
ingo.bethke@gmail.com created 		noresm:norstoretape [2022-05-31 09:29:32] (current)
Line 3: Line 3:
 ===== Basic use ===== ===== Basic use =====
  
-Instructions for basic use of the tape resources are found on NorStore's homepage [[https://www.norstore.no/services/tape-storage|homepage]].+Instructions for basic use of the tape resources are found on NorStore's  [[https://www.norstore.no/services/tape-storage|homepage]].
  
-===== High-level tools =====+===== Advanced use =====
  
-A collection of high-level tape tools, available in /projects/NS2345K/TapeTools, can be download {{:noresm:tapetools_v20140416.tgz|here}}.+A collection of high-level tape tools is available in **/projects/NS2345K/tools**.
  
-==== CopyToTape ====+==== noresm2tape ====
  
-CopyToTape is a script for copying a folder from NorStore's disk area to NorStore's tape resource. +noresm2tape copies the output of a NorESM case from NorStore's disk area to NorStore's tape resource. The output folder should be organised in the standard CCSM way (e.g., atmospheric output is expected in 'atm/hist'). If that is not the case, please use the command disk2tape instead.
  
-Run ''CopyToTape -h'' to print detailed instructions:+**IMPORTANT:** It is highly recommended to run the script in background using the nohup command, e.g.,  
 + "''nohup /projects/NS2345K/tools/noresm2tape <path on disk> <path on tape> &''". Make sure not to forget the "&" at the end of the line. After submitting the script with nohup, it is safe to log out. You will receive a notification email when the transfer to tape is completed.    
 + 
 + 
 +Run ''/projects/NS2345K/tools/noresm2tape -h'' to print detailed instructions:
 <file> <file>
-Usage: CopyToTape <path on disk> <path on tape> [none|gzip|bzip2]+ 
 +Usage: noresm2tape <path on disk> <path on tape> <chunk size> <replication flag>
    
-Example: CopyToTape /projects/NS2345K/hirlam_nobackup /tape/NS2345K/hirlam +Example: noresm2tape /scratch/ingo/mycase /tape/NS2345K/cases/mycase auto replica
    
-Purpose: Copies a folder from disk to tape. +Purpose: Copies NorESM case output from disk to tape. 
  
-Description: <path on disk> must be the absolute path to the input folder.  +Description: <path on disk> and <path on tape> must be full path names (see example).  
-              +              
 +             <chunk size> must be one of: auto, all, 1, 10, 100, or 1000 (numbers  
 +             indicate simulation years per chunk) 
 +           
 +             <replication flag> must be either 'replica' - resulting in two tape copies - 
 +             or 'noreplica'.   
 +                          
              The output folder <path on tape> will be created - with parent                The output folder <path on tape> will be created - with parent  
              directories if necessary - and must not exist before running this               directories if necessary - and must not exist before running this 
Line 28: Line 39:
  
              The input data is staged in form of tar-chuncks in               The input data is staged in form of tar-chuncks in 
-             /scratch/ingo/copytotape. The tar-chunks are removed from scratch  +             /scratch/ingo/noresm2tape. The tar-chunks are removed from  
-             after success transfer to tape. +             scratch after successfull transfer to tape.  
 + 
 +             Checksums for all input files are computed and stored in a separate  
 +             checksum file. The checksums are used to verify the tar-chunks.   
 +</file> 
 + 
 + 
 +==== disk2tape ==== 
 + 
 +disk2tape copies a folder from NorStore's disk area to NorStore's tape resource. In contrast to noresm2tape, the data folder can have any structure and content.  
 + 
 +**IMPORTANT:** It is highly recommended to run the script in background using the nohup command, e.g.,  
 + "''nohup /projects/NS2345K/tools/disk2tape <path on disk> <path on tape> &''". Make sure not to forget the "&" at the end of the line. After submitting the script with nohup, it is safe to log out. You will receive a notification email when the transfer to tape is completed.    
 + 
 +Run ''/projects/NS2345K/tools/disk2tape -h'' to print detailed instructions: 
 +<file> 
 +Usage: disk2tape <path on disk> <path on tape> [none|gzip|bzip2] [replicate]  
 +  
 +Example: disk2tape /projects/NS2345K/hirlam /tape/NS2345K/hirlam  
 +  
 +Purpose: Copies a folder from disk to tape.  
 + 
 +Description: <path on disk> must be the absolute path to the input folder.  
 +               
 +             <path on tape> will be created - with parent directories if  
 +             necessary - and must not exist before running this script.  
 +              
 +             gzip or bzip2 compression will be performed if the third argument is  
 +             set to "gzip" or "bzip2", respectively. The default is no compression. 
 +                  
 +             If the forth argument is set to "replicate" then a second copy of  
 +             the data will be stored on another physical tape medium.  
 +             The path of the additional copy differs from the path of the first  
 +             copy by that /tape is replaced with /replica.  
 + 
 +             The input data is staged in form of tar-chuncks in  
 +             /scratch/$USER/`basename $0`. The tar-chunks are removed from  
 +             scratch after success transfer to tape. 
  
              The size of a single tar-chunk is about 15Gb if no compression is               The size of a single tar-chunk is about 15Gb if no compression is 
Line 37: Line 85:
              Checksums for all input files are computed and stored in a separate               Checksums for all input files are computed and stored in a separate 
              checksum file. The checksums are used to verify the tar-chunks.                checksum file. The checksums are used to verify the tar-chunks.  
 +</file>
 +
 +==== tape2disk ====
 +
 +tape2disk copies a folder that had been put on tape with either noresm2tape or disk2tape. All tar-archieves found in the top-level folder are unpacked while tar-files stored within tar-files are not inflated.  
 +
 +Run ''/projects/NS2345K/tools/tape2disk -h'' to print detailed instructions:
 +<file>
 +
 +Usage: /projects/NS2345K/tools/tape2disk <path on tape> <path on disk> 
 + 
 +Example: /projects/NS2345K/tools/tape2disk /tape/NS2345K/hirlam /scratch/ingo/hirlam 
 + 
 +Purpose: Retrieves a folder from tape and unpacks all tar-archieves contained in it. 
 +
 +Description: <path on disk> will be created if it does not exist. 
  
-Change history: 2014.04.16 first version of CopyToTape 
 </file> </file>
  
 +==== listontape ====
 +
 +listontape lists all files that are archived in a tape folder. 
 +
 +Run ''/projects/NS2345K/tools/listontape -h'' to print detailed instructions:
 +<file>
 +
 +Usage: /projects/NS2345K/tools/listontape <path to data-directory on tape> 
 +        or 
 +       /projects/NS2345K/tools/listontape <path to checksum file on tape>
 +
 +Example: /projects/NS2345K/tools/listontape /tape/NS2345K/hirham_nobackup/BCM 
 +          or 
 +         /projects/NS2345K/tools/listontape /tape/NS2345K/hirham_nobackup/BCM/BCM.md5.tar
 +  
 +Purpose: Lists the names of files that are archived in a given directory on tape.  
 +
 +Description: The script relies on the existence of a checksum file, with extension 
 +             md5.tar, that is stored in the archive directory. 
 +
 +             If exactly one checksum file exists then directory path is sufficient 
 +             as input argument. However, if two or more checksum file exist then 
 +             the absolute path to the checksum file has to be provided.  
 +
 +</file>
  • noresm/norstoretape.1397640380.txt.gz
  • Last modified: 2022-05-31 09:23:24
  • (external edit)