noresm:norstoretape

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
noresm:norstoretape [2014-04-25 08:38:11]
ingo.bethke@gmail.com 		noresm:norstoretape [2022-05-31 09:29:32] (current)
Line 3: Line 3:
 ===== Basic use ===== ===== Basic use =====
  
-Instructions for basic use of the tape resources are found on NorStore's homepage [[https://www.norstore.no/services/tape-storage|homepage]].+Instructions for basic use of the tape resources are found on NorStore's  [[https://www.norstore.no/services/tape-storage|homepage]].
  
 ===== Advanced use ===== ===== Advanced use =====
  
-A collection of high-level tape tools, available in /projects/NS2345K/tools.+A collection of high-level tape tools is available in **/projects/NS2345K/tools**.
  
-==== disk2tape ====+==== noresm2tape ====
  
-disk2tape is a script for copying a folder from NorStore's disk area to NorStore's tape resource. +noresm2tape copies the output of a NorESM case from NorStore's disk area to NorStore's tape resource. The output folder should be organised in the standard CCSM way (e.g., atmospheric output is expected in 'atm/hist'). If that is not the case, please use the command disk2tape instead.
  
-Run ''disk2tape -h'' to print detailed instructions:+**IMPORTANT:** It is highly recommended to run the script in background using the nohup command, e.g.,  
 + "''nohup /projects/NS2345K/tools/noresm2tape <path on disk> <path on tape> &''". Make sure not to forget the "&" at the end of the line. After submitting the script with nohup, it is safe to log out. You will receive a notification email when the transfer to tape is completed.    
 + 
 + 
 +Run ''/projects/NS2345K/tools/noresm2tape -h'' to print detailed instructions:
 <file> <file>
  
-Usage: disk2tape <path on disk> <path on tape> [none|gzip|bzip2]+Usage: noresm2tape <path on disk> <path on tape> <chunk size> <replication flag>
    
-Example: disk2tape /projects/NS2345K/hirlam_nobackup /tape/NS2345K/hirlam +Example: noresm2tape /scratch/ingo/mycase /tape/NS2345K/cases/mycase auto replica
    
-Purpose: Copies a folder from disk to tape. +Purpose: Copies NorESM case output from disk to tape. 
  
-Description: <path on disk> must be the absolute path to the input folder.  +Description: <path on disk> and <path on tape> must be full path names (see example).  
-              +              
 +             <chunk size> must be one of: auto, all, 1, 10, 100, or 1000 (numbers  
 +             indicate simulation years per chunk) 
 +           
 +             <replication flag> must be either 'replica' - resulting in two tape copies - 
 +             or 'noreplica'.   
 +                          
              The output folder <path on tape> will be created - with parent                The output folder <path on tape> will be created - with parent  
              directories if necessary - and must not exist before running this               directories if necessary - and must not exist before running this 
Line 29: Line 39:
  
              The input data is staged in form of tar-chuncks in               The input data is staged in form of tar-chuncks in 
-             /scratch/ingo/copytotape. The tar-chunks are removed from scratch  +             /scratch/ingo/noresm2tape. The tar-chunks are removed from  
-             after success transfer to tape. +             scratch after successfull transfer to tape.  
 + 
 +             Checksums for all input files are computed and stored in a separate  
 +             checksum file. The checksums are used to verify the tar-chunks.   
 +</file> 
 + 
 + 
 +==== disk2tape ==== 
 + 
 +disk2tape copies a folder from NorStore's disk area to NorStore's tape resource. In contrast to noresm2tape, the data folder can have any structure and content.  
 + 
 +**IMPORTANT:** It is highly recommended to run the script in background using the nohup command, e.g.,  
 + "''nohup /projects/NS2345K/tools/disk2tape <path on disk> <path on tape> &''". Make sure not to forget the "&" at the end of the line. After submitting the script with nohup, it is safe to log out. You will receive a notification email when the transfer to tape is completed.    
 + 
 +Run ''/projects/NS2345K/tools/disk2tape -h'' to print detailed instructions: 
 +<file> 
 +Usage: disk2tape <path on disk> <path on tape> [none|gzip|bzip2] [replicate]  
 +  
 +Example: disk2tape /projects/NS2345K/hirlam /tape/NS2345K/hirlam  
 +  
 +Purpose: Copies a folder from disk to tape.  
 + 
 +Description: <path on disk> must be the absolute path to the input folder.  
 +               
 +             <path on tape> will be created - with parent directories if  
 +             necessary - and must not exist before running this script.  
 +              
 +             gzip or bzip2 compression will be performed if the third argument is  
 +             set to "gzip" or "bzip2", respectively. The default is no compression. 
 +                  
 +             If the forth argument is set to "replicate" then a second copy of  
 +             the data will be stored on another physical tape medium.  
 +             The path of the additional copy differs from the path of the first  
 +             copy by that /tape is replaced with /replica.  
 + 
 +             The input data is staged in form of tar-chuncks in  
 +             /scratch/$USER/`basename $0`. The tar-chunks are removed from  
 +             scratch after success transfer to tape. 
  
              The size of a single tar-chunk is about 15Gb if no compression is               The size of a single tar-chunk is about 15Gb if no compression is 
Line 38: Line 85:
              Checksums for all input files are computed and stored in a separate               Checksums for all input files are computed and stored in a separate 
              checksum file. The checksums are used to verify the tar-chunks.                checksum file. The checksums are used to verify the tar-chunks.  
- 
-Change history: 2014.04.23 removed top level directory from tar tree  
-                2014.04.22 fixed bug in scratch cleaning  
-                2014.04.16 first version of disk2tape 
 </file> </file>
  
 ==== tape2disk ==== ==== tape2disk ====
  
-tape2disk is a script for retrieve a folder from NorStore's tape area. All tar-archieves are unpacked.  +tape2disk copies a folder that had been put on tape with either noresm2tape or disk2tape. All tar-archieves found in the top-level folder are unpacked while tar-files stored within tar-files are not inflated.  
  
-Run ''tape2disk -h'' to print detailed instructions:+Run ''/projects/NS2345K/tools/tape2disk -h'' to print detailed instructions:
 <file> <file>
  
-Usage: tape2disk <path on tape> <path on disk> +Usage: /projects/NS2345K/tools/tape2disk <path on tape> <path on disk> 
    
-Example: tape2disk /tape/NS2345K/hirlam /scratch/ingo/hirlam +Example: /projects/NS2345K/tools/tape2disk /tape/NS2345K/hirlam /scratch/ingo/hirlam 
    
-Purpose: Retrieves a folder from tape and unpacks tar-archieves. +Purpose: Retrieves a folder from tape and unpacks all tar-archieves contained in it. 
  
 Description: <path on disk> will be created if it does not exist.  Description: <path on disk> will be created if it does not exist. 
- 
-Change history: 2014.04.25 first version of tape2disk  
  
 </file> </file>
Line 65: Line 106:
 ==== listontape ==== ==== listontape ====
  
-listontape is a script list all files that are archived in a tape folder. +listontape lists all files that are archived in a tape folder. 
  
-Run ''listontape -h'' to print detailed instructions:+Run ''/projects/NS2345K/tools/listontape -h'' to print detailed instructions:
 <file> <file>
  
-Usage: listontape <path to data-directory on tape> +Usage: /projects/NS2345K/tools/listontape <path to data-directory on tape> 
         or          or 
-       listontape <path to checksum file on tape>+       /projects/NS2345K/tools/listontape <path to checksum file on tape>
  
-Example: listontape /tape/NS2345K/hirham_nobackup/BCM +Example: /projects/NS2345K/tools/listontape /tape/NS2345K/hirham_nobackup/BCM 
           or            or 
-         listontape /tape/NS2345K/hirham_nobackup/BCM/BCM.md5.tar+         /projects/NS2345K/tools/listontape /tape/NS2345K/hirham_nobackup/BCM/BCM.md5.tar
      
 Purpose: Lists the names of files that are archived in a given directory on tape.   Purpose: Lists the names of files that are archived in a given directory on tape.  
Line 86: Line 127:
              as input argument. However, if two or more checksum file exist then               as input argument. However, if two or more checksum file exist then 
              the absolute path to the checksum file has to be provided.                the absolute path to the checksum file has to be provided.  
- 
-Change history: 2014.04.22 first version of listontape 
  
 </file> </file>
  • noresm/norstoretape.1398415091.txt.gz
  • Last modified: 2022-05-31 09:23:24
  • (external edit)