Earth System Model eValuation Tool (ESMValTool)

ESMValTool extension for NorESM (NorESMValTool) covers

• NorESM cmor-ization tools
• ESMValTool
• observational data
• CMIP5 model data
• namelists and scripts customised for NorESM
• wrapper scripts for easy use

and is available at

• local: login.nird.sigma2.no:/projects/NS2345K/NorESMValTool
• web: https://github.com/NorwegianClimateCentre/NorESMValTool.git

On Norstore/Nird, run <span style="font-family:monospace;font-size:125%">/projects/NS2345K/NorESMValTool/install.sh</span>

This will create your personal setup in <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool</span>:

 data/		 - input and derived data (symbolic link to project area)
    clim/           - climatologies derived from original input data 
    cmor/	    - cmor output of new experiments  
    model/	    - CMIP5 model output
    obs/	    - obs. data installed with ESMValTool scripts
    rawobs/	    - obs. data as downloaded from internet 
    regrid/	    - data regridded by ESMValTool
    work/	    - work directory of ESMValTool
 mods/		 - source mods for ESMValTool 
    namelists/      - customised namelists for Norstore/NorESM
 scripts/        - wrapper scripts for cmor and ESMValTool 
    cmorize	    - cmor wrapper script for NorESM output 
    esmval	    - ESMValTool wrapper script
 tools/          - noresm2cmor and ESMValTool reside here
    noresm2cmor/    - cmorization tools 
    ESMValTool/	    - original ESMValTool installation 

<span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/scripts/cmorize <span style="color:blue">case folder</span> <span style="color:green">start year</span> <span style="color:orange">end year</span> </span>

NOTE: Cmor-ization on NIRD requires intel's shared libraries. To make them available, add following line to your ~/.bashrc and log out and in again <br><span style="font-family:monospace;font-size:125%"> source /opt/intel/compilers_and_libraries/linux/bin/compilervars.sh -arch intel64 -platform linux </span>

<span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/scripts/cmorize <span style="color:blue">/projects/NS2345K/www/cmor/sampledata/N20TRAERCN_f19_g16_01</span> <span style="color:green">2000</span> <span style="color:orange"> 2000</span> </span>

The output can be customised in <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/tools/noresm2cmor/namelists/noresm2cmor_NorESM_GENERIC_template.nml</span> </span>

New variables need to be defined in the corresponding GENERIC_* tables in <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/tools/noresm2cmor/tools/noresm2cmor/tables</span>

The cmor-ized output is written to the folder <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/data/cmor/<span style="color:blue">case name</span>.<span style="color:green">start year</span>-<span style="color:orange">end year</span></span> e.g. <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/data/cmor/<span style="color:blue">N20TRAERCN_f19_g16_01</span>.<span style="color:green">2000</span>-<span style="color:orange">2000</span></span>

NOTE: The cmor-ized output is stored in Norstore/Nird's project area. The path <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/data/cmor</span> is a symbolic link to <span style="font-family:monospace;font-size:125%">/projects/NS2345K/NorESMValTool/data/cmor</span>.

ESMValTool takes one namelist as input argument. Each namelist contains instructions for performing a specific diagnostic.

A collection of ported namelists is available in <span style="font-family:monospace;font-size:125%"> $HOME/NorESMValTool/mods/namelist</span>. Path information to observational and simulation data is hardcoded in the namelists and need to be edited in order to apply ESMValTool on new simulation output.

More namelists provided by the official ESMValTool repository are available in <span style="font-family:monospace;font-size:125%"> $HOME/NorESMValTool/tools/ESMValTool/nml</span>.

These namelists cannot be readily used with the ESMValTool on Norstore/Nird but have to be ported first. The specifications of observational and model data in the namelists have to be adapted to the data available in the system. We recommend to first copy the namelist to <span style="font-family:monospace;font-size:125%"> $HOME/NorESMValTool/mods/namelist</span> and edit it there.

Note that in many cases new input data has to be made available first.

Change directory to <span style="font-family:monospace;font-size:125%"> $HOME/NorESMValTool/tools/ESMValTool</span>.

Execute <br><span style="font-family:monospace;font-size:125%"> ./main.py <path to input namelist> </span><br> e.g. <br><span style="font-family:monospace;font-size:125%"> ./main.py ../../mods/namelists/namelist_SouthernOcean_norstore.xml </span>

NOTE: ESMValTool requires NCL version 6.4. To make this version available, add following lines to your ~/.bash_profile and log out and in again <span style="font-family:monospace;font-size:125%"> <br>export NCARG_ROOT=/opt/ncl64 <br>export PATH=/opt/ncl64/bin/:${PATH} </span>

<span style="font-family:monospace;font-size:125%"> $HOME/NorESMValTool/scripts/esmval <span style="color:blue">cmor folder</span> <span style="color:green;font-family:monospace">ESMValTool namelist</span> </span>

<span style="font-family:monospace;font-size:125%"> $HOME/NorESMValTool/scripts/esmval <span style="color:blue">$HOME/NorESMValTool/data/cmor/N20TRAERCN_f19_g16_01.2000-2000</span> <span style="color:green">$HOME/NorESMValTool/mods/namelist/namelist_MyDiag.xml</span></span>

The <span style="font-family:monospace;font-size:125%">esmval</span> script replaces <span style="font-family:monospace;font-size:125%">MODELTAG</span> with the path to the cmorized output in the customised ESMValTool namelist that has been selected (e.g., <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/mods/namelists/namelist_MyDiag.xml</span>).

It then calls ESMValTool main script <span style="font-family:monospace;font-size:125%">main.py</span> with updated namelist as argument.

NOTE: If called outside of wrapper script, <span style="font-family:monospace;font-size:125%">main.py</span> requires the modules python2/2.7 and ncl on Norstore and must be run from <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/tools/ESMValTool</span>

Plots are stored in home directory in <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/plots/</span>

Regridded input data and derived climatologies are stored in project area in <span style="font-family:monospace;font-size:125%">/norstore_osl/projects/NS2345K/NorESMValTool/data</span>

The portrait diagram is one ESMValTool's key features, but unfortunately also the most complex diagnostic performance-wise. A dedicated wrapper script has therefore been created on NIRD in order to make the creation of portrait diagrams as easy as possible. The script performs all the necessary steps, including CMORization.

For setup and user instructions, see

<span style="font-family:monospace;font-size:125%">/projects/NS2345K/NorESMValTool/portrait_diagram/README</span>

An 'extension' to the portrait diagram diagnostic allowing the use of ground based (non-gridded) observations has also been created.

For setup and user instructions, see

<span style="font-family:monospace;font-size:125%">/projects/NS2345K/NorESMValTool/PerfmetricsGroundObs/README</span>

Run <span style="font-family:monospace;font-size:125%">"umask u=rxw,g=rxw,o=rx"</span>. This will ensure that newly created folders and files can be modified by members of the NorESM project.

If you do not already have a personal ESGF account, then create one at e.g. https://esgf-data.dkrz.de (Safari does not work properly, while Firefox just works fine) and sign up for the group CMIP5_Research.

Additional CMIP5 model output should be stored in $HOME/NorESMValTool/data/model/CMIP5 (which is a symbolic link to the project folder /projects/NS2345K/NorESMValTool/data/model/CMIP5). Each model has its own sub-folder that matches the models acronym.

Modify the variable entry in this template and then paste everything into the address field of your browser:

http://esgf-data.dkrz.de/esg-search/wget?
download_structure=realm,experiment,variable,time_frequency,model,ensemble
&project=CMIP5
&experiment=historical
&time_frequency=mon
&variable=ua
&limit=10000
&distrib=true
&replica=true

The browser will generate a wget-script and try to store it on your local computer. This particular example will prepare a download script for monthly ua output of all available models for the CMIP5 historical experiment.

Alternatively, one can use the ESGF portal functionality to generate the download script.

Use the scp command to copy the wget-script into <span style="font-family:monospace;font-size:125%">login.nird.sigma2.no:/projects/NS2345K/NorESMValTool/data/model/CMIP5/download/</span>.

Log on to login.nird.sigma2.no and cd to <span style="font-family:monospace;font-size:125%">/projects/NS2345K/NorESMValTool/data/model/CMIP5/download/</span>.

Fix execution permission of wget script with “chmod +x wget*”. Run command <span style="font-family:monospace;font-size:125%">"umask u=rxw,g=rxw,o=rx"</span>, which will ensure that the permissions of the downloaded data will be correct.

Execute wget-script with “-n” (i.e. dummy download option) option to update certificates to ~/.esg/ directory. The script will then prompt for your ESGF OpenID and password. Once done, you will be able to run the download script without prompts for username/password.

Run download script in background with “nohup wget-XXX.sh >& wget-XXX.log &” (replace XXX to match the name of your download script).

Use the mv command to move the output of model YYY to folder <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/data/model/CMIP5/YYY</span> (replace YYY with actual model name).

At the bottom of this page are three tables that list the ESMValTool input data that is available on NorStore. Always update these tables after installing new model/observation data.

Detailed instructions for the installation of observational data are provided in section 6.2.2 of the ESMValTool user guide.

Observational products that are served through the Earth System Grid (use Firefox if Safari does not work) via the projects obs4mips and ana4mips follow the CMIP format standard and can be readily downloaded to sub-folders in <span style="font-family:monospace;font-size:125%">/projects/NS2345K/NorESMValTool/data/obs</span>. In principal, the download scripts can be generated in the same manner as for the model output (see previous section), with the one difference that the project specifier “CMIP5” has to be changed to “obs4mips” or “ana4mips”. Alternatively, one can use the ESGF portal functionality to generate the download script.

Observational products that do not follow the CMIP standard should instead be stored in sub-folders in <span style="font-family:monospace;font-size:125%">/projects/NS2345K/NorESMValTool/data/rawobs</span>. In the next step, the dataset has to be reformatted using the corresponding conversion script (one for each dataset) in <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/tools/ESMValTool/reformat_scripts/obs</span>.

The reformat scripts provided by ESMValTool are sometimes outdated and do not always work out of the box. Hence it is likely that minor modifications are required to make them working.

Next, change directory to <span style="font-family:monospace;font-size:125%">$HOME/NorESMValTool/tools/ESMValTool/</span> and execute <span style="font-family:monospace;font-size:125%"> "python main.py reformat_scripts/obs/script.ncl"</span>, where <span style="font-family:monospace;font-size:125%">script.ncl</span> should be replaced with the name of the reformatting script.

Make sure to run <span style="font-family:monospace;font-size:125%">"umask u=rxw,g=rxw,o=rx"</span> before downloading and reformatting.

Always update the tables at the bottom of the wiki page after installing new observation data.

A number of namelists that reproduce a selection of figures from the GMD paper (Eyring et al. 2016) have been ported to NIRD and are 'ready to run'. These namelists, with corresponding GMD figures, can be found in the table below. Some of the namelists also produce additional figures related to the GMD figures - these are listed under “By-products” in the table below. The NIRD compatible namelists are all labeled *_nird.xml, and can be found in

<span style="font-family:monospace;font-size:125%">/projects/NS2345K/NorESMValTool/mods/namelists/</span>

See above under “Running ESMValTool” for launching instructions.

• login.nird.sigma2.no:/projects/NS2345K/NorESMValTool/README
• login.nird.sigma2.no:/projects/NS2345K/noresm2cmor/README
• ESMValTool home page: www.esmvaltool.org
• ESMValTool user guide: https://github.com/ESMValGroup/ESMValTool/raw/master/doc/ESMValTool_Users_Guide.pdf
• GMD paper (Eyring et al. 2016): www.geosci-model-dev.net/9/1747/2016
• ESMValTool setup for NorESM (10 May 2017) - Ingo Bethke
• ESMValTool: Implementation and use (08 Sep 2017 - Kristian Ingvaldsen

A detailed description of all supported datasets is presented in table S9 in section 6.6.2 (page 28) of the ESMValTool user guide.

A list of supported variables with explanation is presented in table S8 in section 3.3 (page 17) of the ESMValTool user guide.