aerocom:data_submission

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
aerocom:data_submission [2013-12-12 16:38:30]
michaels created
aerocom:data_submission [2022-05-31 09:29:31] (current)
Line 1: Line 1:
-HTAP-AeroCom  data submission requirements+ATTENTION THIS WIKI PAGE IS NO LONGER UPDATED - PLEASE GO TO [[http://aerocom.met.no/|aerocom.met.no]]FOR LATEST INFO
  
-==== SUBMISSION OF DATA PROCEDURE ==== 
-Any submission of data to the AeroCom database requires to follow the following steps: 1) [[#FORMAT|format data]] to agreed netCDF htap standards, cf compliant  2) [[#ADJUST|adjust filenames]]  3) [[#CHECK|check format]] with cf compliance checker, correct and check again  4) [[#OBTAIN|obtain account]] on htap server to access ftp and/or aerocom-users server 5) [[#TRANSFER|transfer data]] to the ftp server at the Norwegian Meteorological Institute 6) [[#NOTIFY|notify metno]] that data have been submitted  7) [[#LOG|log submission]] in [[htap2-file-submission|log file]] on htap wiki page 8) [[#UPDATE|update errors]] on submission log on htap wiki page  
  
-<<Anchor(FORMAT)>>+AeroCom  data submission requirements
  
-=== 1) FORMAT DATA === +====== SUBMISSION OF DATA PROCEDURE ====== 
-Each variable should be stored in ONE cf-compliant netCDF filewhich includes all time steps for a given year.  The list of variables to be output is provided in a spreadsheet per experiment and a description is provided in the experiment specifications document on the same wiki page.+Any submission of data to the AeroCom database requires to follow the following steps:  
 +  [[#FORMAT DATA|format data]] to agreed netCDF htap standardscf compliant   
 +  - [[#ADJUST FILENAMES |adjust filenames]] 
 +  - [[#CHECK FORMAT|check format]] with cf compliance checker, correct and check again 
 +  - [[#OBTAIN ACCOUNT|obtain account]] on htap server to access aerocom-users server  
 +  - [[#TRANSFER DATA|transfer data]] to the aerocom-users.met.no at the Norwegian Meteorological Institute  
 +  - [[#NOTIFY MET.NO|notify metno]] that data have been submitted   
 +  - [[#LOG FILE SUBMISSION|log submission]] in [[aerocom:modelupdates|log file]] on htap wiki page  
 +  - [[#UPDATES|update errors]] on submission log on htap wiki page 
  
-CF-compliance implies that sufficient attributes, coordinates and coordinate describing variables are included in each file to describe metadata needed to understand grid representation, units and origin of the data. See excel htap protocol file for variable names, standard names (comming end of September 2013). 
  
-Formatting can be achieved with cmor tools (http://www-pcmdi.llnl.gov/software-portal/cmor/and nco tools (http://nco.sourceforge.net/nco.html). For using cmor tools it is required to obtain the htap-2 cmor tables (to be provided shortly).+===== 1) format data ===== 
 +Each variable should be stored in ONE cf-compliant NetCDF file, which includes all time steps for a given year The list of variables to be output is provided in a spreadsheet per experiment and a description is provided in the experiment specifications document on the same wiki page.
  
-An example file as used for the previous AeroCom experiment is found here: http://aerocom.met.no/download/CMOR/FILE-EXAMPLE/ (htap example files will follow soon )+CF-compliance implies that sufficient attributes, coordinates and coordinate describing variables are included in each file to describe metadata needed to understand grid representation, units and origin of the data
  
-<<Anchor(ADJUST)>>+See also frequently asked questions on CF: [[https://github.com/graybealski/cf-conventions-work/blob/master/FAQ.md|URL-link]]
  
-==== 2) ADJUST FILENAMES ==== 
-The obligatory Filename for using the cf checker and the submission of data is 
  
-"aerocom_<!ModelName>_<!ExperimentName>_<!VariableName>_<!VerticalCoordinateType>_<Period>_<Frequency>.nc"+===== 2) adjust filenames ===== 
 +The obligatory filename for using the cf checker and the submission of data is:\\ 
 +**aerocom3_<ModelName>-<MeteoConfigSpecifier>_<ExperimentName>-<PerturbationName>_<VariableName>_<VerticalCoordinateType>_<Period>_<Frequency>.nc**
  
 +Details about new file naming can be found here:
 +[[https://docs.google.com/spreadsheets/d/1NiHLVTDsBo0JEBSnnDECNI2ojUnCVlxuy2PFrsRJW38/edit?usp=sharing | AeroCom experiments diagnostics sheets 28.2.2019]]
  
-<!ModelName> => can be chosen such that Model Name, Model version and possibly the institution can be identified.  No underscores (_) are allowed in <!ModelName>, use (-) instead. Restrict <!ModelName> to max 20 characters.+Be aware of updates \\
  
-<!ExperimentName> => see  Experiment Specifications +versions will have a date attached.\\
  
-<!VariableName> => see output variable names in excel sheet+Examples: 
 +  * aerocom3_GOCARTv7.2_AP3-CTRL_od550aer_Column_2010_monthly.nc 
 +  * aerocom3_GISS-MATRIX-met2010_AP3-ACRI-AFRICA_od550aer_Column_2010_daily.nc 
 +  * aerocom3_GOCARTv7.2-met2010_AP3-CTRL-od550aer_Column_1850_monthly.nc 
 +  * aerocom3_GISS-MATRIX-fSST_AP3-CTRL-od550aer_Column_1850_monthly.nc
  
-<!VerticalCoordinateType> ="Surface", "Column", "!ModelLevel", "!SurfaceAtStations", "!ModelLevelAtStations"+OLD (aerocom phase3)  
 +**aerocom3_<ModelName>_<ExperimentName>_<VariableName>_<VerticalCoordinateType>_<Period>_<Frequency>.nc**
  
-<Period> =eg "2008", "2010"+OLD (aerocom phase 2) aerocom naming convention is also fine for now:  
 +**aerocom_<ModelName>_<ExperimentName>_<VariableName>_<Period>_<Frequency>.nc**
  
-<Frequency> => "timeinvariant","hourly", "daily", "monthly", "sat1000", "sat1330", "sat2200", "sat0130"+  * <ModelName> => can be chosen such that Model Name, Model version and possibly the institution can be identified.  No underscores (_) are allowed in <!ModelName>, use (-) instead. Restrict <ModelName> to max 20 characters. 
 +  * <MeteoConfigSpecifiers> => "met2010" (default, nudged to 2010 meteo) / "fSST" (fixed 2010 SST monthly fields, not nudged) / "CPL" (coupled AOGCM model, not nudged) A MeteoSpecifier has to be present 
 +  * <ExperimentName> => see  Experiment Specifications  
 +  * <PerturbationName> see experiment descriptions on aerocom wiki, if required for the experiment, use exact case sensitive name from description 
 +  * <VariableName> => see output variable names in excel sheet 
 +  * <VerticalCoordinateType> => "Surface", "Column", "ModelLevel", "SurfaceAtStations", "ModelLevelAtStations" 
 +  * <Period> => eg "2008", "2010" 
 +  * <Frequency> => "timeinvariant","hourly", "3hourly", "daily", "monthly", "sat1000", "sat1330", "sat2200", "sat0130"
  
-<<Anchor(CHECK)>> 
  
-==== 3) CHECK FORMAT ==== +===== 3) check format ===== 
-Typical files must be checked prior to submission via the cf-compliance-checker.  URL http://aerocom-test.met.no/upload (will be changed to http://cfchecker.met.no shortly)+Typical files must be checked prior to submission via cf-compliance-checker. [[http://cfconventions.org | cfconventions.org]]  maintains a list of working and frequently updated [[http://cfconventions.org/compliance-checker.html|CF checkers]]. Some come as a script, and to others one can upload a file
  
-Typical files? A check shall comprise at least one file of each VerticalCoordinateType and each Frequency and each experimentbest varying the variables!+For the upload checker you want to check a typical but small file. To achieve thisyou can cut out the first time steps with the nco command: 
 +ncks -O -d time,0,10 sourcefile.nc filetest.nc
  
-<<Anchor(OBTAIN)>>+Per default, a check shall comprise at least one file of each VerticalCoordinateType and each Frequency and each experiment, best varying the variables!
  
-==== 4) OBTAIN ACCOUNT ON AeroCom SERVER ==== 
-An account at Met.No is needed to submit data via ftp. A data server (aerocom-users.met.no) allows users also to access the data via ssh. Please contact jan.griesfeller@met.no for password and user name. You will then be notified further about the procedure. A public ssh-key has to be sent to Jan Griesfeller to activate the account. 
  
-<<Anchor(TRANSFER)>>+===== 4obtain account =====
  
-==== 5) TRANSFER DATA TO MET.NO ==== +...for METNO aerocom servers and WIKI documentation server 
-After obtaining an account, data can then be uploaded in two ways:+
  
-via ftp.aerocom.met.no: make your directory under /incoming/ and put files thereOverall size limit of ca 500 GB ( limit for all users)  +An account at Met.No is needed to submit data a data server (aerocom-users.met.no) via ssh and edit [[aerocom:|AeroCom wiki pages]]. Please contact jan.griesfeller@met.no and/or anna.benedictow@met.no for password and user name. You will then be notified further about the procedure. A public ssh-key has to be sent to jan and/or anna to activate the account.
  
-via aerocom-users.met.no: make your directory under /media/scratch/incoming/ and put files there. Overall size limit of ca 500 GB ( limit for all users)    
  
-Very large submissions (> 10 GBrequire prior coordination with Met.No/Jan Griesfeller&Michael Schulz via email, in particular in times of simultaneous heavy upload activity.+===== 5transfer data ===== 
 +After obtaining an account, data can then be uploaded to the aerocom user server aerocom-users.met.no: 
 +  * Create a directory under /media/scratch/incoming/AEROCOM_AUTO_UPLOAD/<ModelName>/renamed/\\ Use exactly the same model name as used for file names. Attention to lower&upper case!!\\ Put files in that renamed directory. Overall size limit of ca 6 TB (limit for all users)    
 +  * upload via ftp.aerocom.met.no has been discontinued due to security concerns. 
 +  
  
-<<Anchor(NOTIFY)>>+Large submissions (> 100 GB) may require extra attention (use rsync). They might require prior coordination with Met.No/Anna Benedictow/Jan Griesfeller/Michael Schulz via email, in particular in times of simultaneous heavy upload activity. 
  
-==== 6) NOTIFY MET.NO ==== +On aerocom-users.met.no the directory /media/scratch/incoming/AEROCOM-P3-AUTO-UPLOAD allows for automatic incorporation into the AeroCom database and workupUploaded files are processed automatically by the AeroCom tools and transfered into the AeroCom phase III data directory. (Still in test phase send also e-mail (ref[[#NOTIFY MET.NO|notify metno]]))
-When files are uploaded send email to jan.griesfeller@met.no AND michael.schulz@met.no, so that the files can be transferred from the ftp disk to the database. When the data are included in the database the files appear on aerocom-users.met.no under /metno/aerocom/users/aerocom1/<!ModelName>.+
  
-After the data are transferred to the databasedata are removed from ftp server or aerocom-users:scratch server.+If correct in format and with correct file namesresults uploaded here will be processed over night and appear after a day as image catalogue on http://aerocom.met.no/cgi-bin/aerocom/surfobs_annualrs.pl?PROJECT=AEROCOM&MODELLIST=AEROCOM-Phase-III
  
  
-<<Anchor(LOG)>>+===== 6) notify met.no ===== 
 +When files are uploaded send email to anna.benedictow@met.no and jan.griesfeller@met.no and michael.schulz@met.no (AND coordinator of project), so that the files can be transferred from aerocom-users.met.no to the database. When the data are included in the database the files appear on aerocom-users.met.no under /metno/aerocom-users-database/PROJECT(AEROCOM-PHASE-III)/<ModelName>_<exp>/.
  
-==== 7) LOG FILE SUBMISSION ==== +After the data are transferred to the databasedata are removed from aerocom-users:scratch server.
-FOR EACH SUBMISSION: Put an entry into the htap wiki page [[modelupdates|Log file]], describing what has been submitted. This is also obligatory if changes or updates (even partial) to the original submission are made. We also request modelers to record on the wiki information on known bugsor any other relevant information needed for analysis of the model results by others.+
  
-<<Anchor(UPDATE)>> 
  
-==== 7) UPDATES / Error corrections ====+===== 7) submit a log file ====
 +FOR EACH SUBMISSION: Put an entry into the htap wiki page [[modelupdates|Log file]], describing what has been submitted. This is also obligatory if changes or updates (even partial) to the original submission are made. We also request modellers to record on the wiki information on known bugs, or any other relevant information needed for analysis of the model results by others.
  
-In case of updates of files, the model version identifier should also contain information on the resubmission of model results. Additional files which do not change an earlier submission may contain the same version number as an older submission. Corrections and a model rerun should be accompanied by a new model version identifier. Old still valid files may be merged upon request with new data. Old versions of model submissions will be temporally moved to a separate directory by Met.No, and where possible, deleted after some time. An exception will be made for versions that have been used in publications. Modellers are responsible for communication of major updates to their models (e-mail and log-files, see below). Model analysers are responsible for frequently verifying changes in model submissions (e.g. before submitting a paper). 
  
-<<Anchor(SUBMIT)>>+===== 8) updates ===== 
 + 
 +In case of updates of files, the model version identifier should also contain information on the resubmission of model results. Additional files which do not change an earlier submission may contain the same version number as an older submission. Corrections and a model rerun should be accompanied by a new model version identifier. Old still valid files may be merged upon request with new data. Old versions of model submissions will be temporally moved to a separate directory by Met.No, and where possible, deleted after some time. An exception will be made for versions that have been used in publications. Modellers are responsible for communication of major updates to their models (e-mail and log-files, see below). Model analysers are responsible for frequently verifying changes in model submissions (e.g. before submitting a paper).
  
-==== 9) SUBMIT MODEL DESCRIPTION DOCUMENT ==== +       
-FOR EACH MODEL:  We request that for each participating model, a model description document be submitted under [[WP2.4|Work Package 2.4]].  On the [[WP2.4]] wiki page, a Word document template for this model description is available to be downloaded, filled in, and reposted as an attachment to that page (or emailed to keating.terry@epa.gov for posting). We request that this documentation be filled out when the data is submitted to facilitate analysis of the data.       +
  • aerocom/data_submission.1386866310.txt.gz
  • Last modified: 2022-05-31 09:23:10
  • (external edit)