Differences
This shows you the differences between two versions of the page.
Next revision | Previous revision | ||
aerocom:data_submission [2013-12-12 16:38:30] michaels created |
aerocom:data_submission [2022-05-31 09:29:31] (current) |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | HTAP-2 / AeroCom | + | ATTENTION |
- | ==== SUBMISSION OF DATA PROCEDURE ==== | ||
- | Any submission of data to the AeroCom database requires to follow the following steps: 1) [[# | ||
- | << | + | AeroCom |
- | === 1) FORMAT | + | ====== SUBMISSION OF DATA PROCEDURE ====== |
- | Each variable should be stored in ONE cf-compliant | + | Any submission of data to the AeroCom database requires to follow the following steps: |
+ | | ||
+ | - [[#ADJUST FILENAMES |adjust filenames]] | ||
+ | - [[#CHECK FORMAT|check format]] with cf compliance checker, correct | ||
+ | - [[#OBTAIN ACCOUNT|obtain account]] | ||
+ | - [[#TRANSFER DATA|transfer data]] to the aerocom-users.met.no at the Norwegian Meteorological Institute | ||
+ | - [[#NOTIFY MET.NO|notify metno]] that data have been submitted | ||
+ | - [[#LOG FILE SUBMISSION|log submission]] in [[aerocom: | ||
+ | - [[# | ||
- | CF-compliance implies that sufficient attributes, coordinates and coordinate describing variables are included in each file to describe metadata needed to understand grid representation, | ||
- | Formatting can be achieved with cmor tools (http://www-pcmdi.llnl.gov/ | + | ===== 1) format data ===== |
+ | Each variable should | ||
- | An example | + | CF-compliance implies that sufficient attributes, coordinates and coordinate describing variables are included in each file to describe metadata needed to understand grid representation, |
- | << | + | See also frequently asked questions on CF: [[https:// |
- | ==== 2) ADJUST FILENAMES ==== | ||
- | The obligatory Filename for using the cf checker and the submission of data is | ||
- | " | + | ===== 2) adjust filenames ===== |
+ | The obligatory filename for using the cf checker and the submission of data is:\\ | ||
+ | **aerocom3_< | ||
+ | Details about new file naming can be found here: | ||
+ | [[https:// | ||
- | <!ModelName> | + | Be aware of updates |
- | < | + | versions will have a date attached.\\ |
- | < | + | Examples: |
+ | * aerocom3_GOCARTv7.2_AP3-CTRL_od550aer_Column_2010_monthly.nc | ||
+ | * aerocom3_GISS-MATRIX-met2010_AP3-ACRI-AFRICA_od550aer_Column_2010_daily.nc | ||
+ | * aerocom3_GOCARTv7.2-met2010_AP3-CTRL-od550aer_Column_1850_monthly.nc | ||
+ | * aerocom3_GISS-MATRIX-fSST_AP3-CTRL-od550aer_Column_1850_monthly.nc | ||
- | <!VerticalCoordinateType> | + | OLD (aerocom phase3) |
+ | **aerocom3_< | ||
- | < | + | OLD (aerocom phase 2) aerocom naming convention is also fine for now: |
+ | **aerocom_< | ||
- | < | + | * < |
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
- | << | ||
- | ==== 3) CHECK FORMAT | + | ===== 3) check format ===== |
- | Typical files must be checked prior to submission via the cf-compliance-checker. | + | Typical files must be checked prior to submission via a cf-compliance-checker. |
- | Typical files? A check shall comprise at least one file of each VerticalCoordinateType and each Frequency and each experiment, best varying | + | For the upload checker you want to check a typical but small file. To achieve this, you can cut out the first time steps with the nco command: |
+ | ncks -O -d time,0,10 sourcefile.nc filetest.nc | ||
- | << | + | Per default, a check shall comprise at least one file of each VerticalCoordinateType and each Frequency and each experiment, best varying the variables! |
- | ==== 4) OBTAIN ACCOUNT ON AeroCom SERVER ==== | ||
- | An account at Met.No is needed to submit data via ftp. A data server (aerocom-users.met.no) allows users also to access the data via ssh. Please contact jan.griesfeller@met.no for password and user name. You will then be notified further about the procedure. A public ssh-key has to be sent to Jan Griesfeller to activate the account. | ||
- | << | + | ===== 4) obtain account ===== |
- | ==== 5) TRANSFER DATA TO MET.NO ==== | + | ...for METNO aerocom servers and WIKI documentation server |
- | After obtaining an account, data can then be uploaded in two ways: | + | |
- | via ftp.aerocom.met.no: | + | An account at Met.No is needed to submit data a data server (aerocom-users.met.no) via ssh and edit [[aerocom:|AeroCom wiki pages]]. Please contact jan.griesfeller@met.no |
- | via aerocom-users.met.no: | ||
- | Very large submissions (> 10 GB) require prior coordination with Met.No/Jan Griesfeller&Michael Schulz via email, | + | ===== 5) transfer data ===== |
+ | After obtaining an account, data can then be uploaded to the aerocom user server aerocom-users.met.no: | ||
+ | * Create a directory under /media/ | ||
+ | * upload | ||
+ | | ||
- | << | + | Large submissions |
- | ==== 6) NOTIFY MET.NO ==== | + | On aerocom-users.met.no the directory / |
- | When files are uploaded send email to jan.griesfeller@met.no | + | |
- | After the data are transferred to the database, data are removed from ftp server or aerocom-users: | + | If correct in format and with correct file names, results uploaded here will be processed over night and appear after a day as image catalogue on http://aerocom.met.no/cgi-bin/ |
- | << | + | ===== 6) notify met.no ===== |
+ | When files are uploaded send email to anna.benedictow@met.no and jan.griesfeller@met.no and michael.schulz@met.no | ||
- | ==== 7) LOG FILE SUBMISSION ==== | + | After the data are transferred |
- | FOR EACH SUBMISSION: Put an entry into the htap wiki page [[modelupdates|Log file]], describing what has been submitted. This is also obligatory if changes or updates (even partial) to the original submission | + | |
- | << | ||
- | ==== 7) UPDATES / Error corrections | + | ===== 7) submit a log file ===== |
+ | FOR EACH SUBMISSION: Put an entry into the htap wiki page [[modelupdates|Log file]], describing what has been submitted. This is also obligatory if changes or updates (even partial) to the original submission are made. We also request modellers to record on the wiki information on known bugs, or any other relevant information needed for analysis of the model results by others. | ||
- | In case of updates of files, the model version identifier should also contain information on the resubmission of model results. Additional files which do not change an earlier submission may contain the same version number as an older submission. Corrections and a model rerun should be accompanied by a new model version identifier. Old still valid files may be merged upon request with new data. Old versions of model submissions will be temporally moved to a separate directory by Met.No, and where possible, deleted after some time. An exception will be made for versions that have been used in publications. Modellers are responsible for communication of major updates to their models (e-mail and log-files, see below). Model analysers are responsible for frequently verifying changes in model submissions (e.g. before submitting a paper). | ||
- | << | + | ===== 8) updates ===== |
+ | |||
+ | In case of updates of files, the model version identifier should also contain information on the resubmission of model results. Additional files which do not change an earlier submission may contain the same version number as an older submission. Corrections and a model rerun should be accompanied by a new model version identifier. Old still valid files may be merged upon request with new data. Old versions of model submissions will be temporally moved to a separate directory by Met.No, and where possible, deleted after some time. An exception will be made for versions that have been used in publications. Modellers are responsible for communication of major updates to their models (e-mail and log-files, see below). Model analysers are responsible for frequently verifying changes in model submissions | ||
- | ==== 9) SUBMIT MODEL DESCRIPTION DOCUMENT ==== | + | |
- | FOR EACH MODEL: | + |