This page is hosted by:
This page is hosted by:
AeroCom data submission requirements
Any submission of data to the AeroCom database requires to follow the following steps: 1) format data to agreed netCDF htap standards, cf compliant 2) adjust filenames 3) check format with cf compliance checker, correct and check again 4) obtain account on htap server to access ftp and/or aerocom-users server 5) transfer data to the ftp server at the Norwegian Meteorological Institute 6) notify metno that data have been submitted 7) log submission in log file on htap wiki page 8) update errors on submission log on htap wiki page
Each variable should be stored in ONE cf-compliant NetCDF file, which includes all time steps for a given year. The list of variables to be output is provided in a spreadsheet per experiment and a description is provided in the experiment specifications document on the same wiki page.
CF-compliance implies that sufficient attributes, coordinates and coordinate describing variables are included in each file to describe metadata needed to understand grid representation, units and origin of the data. See excel htap protocol file for variable names, standard names (http://iek8wikis.iek.fz-juelich.de/HTAPWiki/WP2.2#A1._HTAP2_experiments_and_output_variables).
See also frequently asked questions on CF: URL-link
Formatting can be achieved with cmor tools (http://www-pcmdi.llnl.gov/software-portal/cmor/) and nco tools (URL-link). For using cmor tools it is required to obtain the htap-2 cmor tables: URL-link to htap wiki with link to CMOR tables).
An example file as used for the previous AeroCom experiment is found here: http://aerocom.met.no/download/CMOR/FILE-EXAMPLE/
The obligatory Filename for using the cf checker and the submission of data is aerocom3_<ModelName>_<ExperimentName>_<VariableName>_<VerticalCoordinateType>_<Period>_<Frequency>.nc
OLD aerocom naming convention is also fine for now: aerocom_<ModelName>_<ExperimentName>_<VariableName>_<Period>_<Frequency>.nc
Typical files must be checked prior to submission via a cf-compliance-checker. cfconventions.org maintains a list of working and frequently updated CF checkers. Some come as a script, and to others one can upload a file.
For the upload checker you want to check a typical but small file. To achieve this, you can cut out the first time steps with the nco command: ncks -O -d time,0,10 sourcefile.nc filetest.nc
Per default, a check shall comprise at least one file of each VerticalCoordinateType and each Frequency and each experiment, best varying the variables!
…for METNO aerocom servers and WIKI documentation server
An account at Met.No is needed to submit data via ftp and edit AeroCom wiki pages. A data server (aerocom-users.met.no) allows users also to access the data via ssh. Please contact firstname.lastname@example.org and/or email@example.com for password and user name. You will then be notified further about the procedure. A public ssh-key has to be sent to jan and/or anna to activate the account.
After obtaining an account, data can then be uploaded in two ways:
Large submissions (> 100 GB) may require extra attention (use rsync, scftp, not simple ftp clients). They might require prior coordination with Met.No/Anna Benedictow/Jan Griesfeller/Michael Schulz via email, in particular in times of simultaneous heavy upload activity.
On aerocom-users.met.no the directory /media/scratch/incoming/AEROCOM-P3-AUTO-UPLOAD allows for automatic incorporation into the AeroCom database and workup. Uploaded files are processed automatically by the AeroCom tools and transfered into the AeroCom phase III data directory. (Still in test phase - send also e-mail (ref. notify metno))
If correct in format and with correct file names, results uploaded here will be processed over night and appear after a day as image catalogue on http://aerocom.met.no/cgi-bin/aerocom/surfobs_annualrs.pl?PROJECT=AEROCOM&MODELLIST=AEROCOM-Phase-III
When files are uploaded send email to firstname.lastname@example.org and email@example.com and firstname.lastname@example.org (AND coordinator of project), so that the files can be transferred from the ftp disk or aerocom-users to the database. When the data are included in the database the files appear on aerocom-users.met.no under /metno/aerocom-users-database/PROJECT(AEROCOM-PHASE-III)/<ModelName>_<exp>/.
After the data are transferred to the database, data are removed from ftp server or aerocom-users:scratch server.
FOR EACH SUBMISSION: Put an entry into the htap wiki page Log file, describing what has been submitted. This is also obligatory if changes or updates (even partial) to the original submission are made. We also request modelers to record on the wiki information on known bugs, or any other relevant information needed for analysis of the model results by others.
In case of updates of files, the model version identifier should also contain information on the resubmission of model results. Additional files which do not change an earlier submission may contain the same version number as an older submission. Corrections and a model rerun should be accompanied by a new model version identifier. Old still valid files may be merged upon request with new data. Old versions of model submissions will be temporally moved to a separate directory by Met.No, and where possible, deleted after some time. An exception will be made for versions that have been used in publications. Modellers are responsible for communication of major updates to their models (e-mail and log-files, see below). Model analysers are responsible for frequently verifying changes in model submissions (e.g. before submitting a paper).