Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
aerocom:data_submission [2015-08-10 11:47:20] michaels |
aerocom:data_submission [2022-05-31 09:29:31] (current) |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| + | ATTENTION - THIS WIKI PAGE IS NO LONGER UPDATED - PLEASE GO TO [[http:// | ||
| + | |||
| + | |||
| AeroCom | AeroCom | ||
| - | ==== SUBMISSION OF DATA PROCEDURE ==== | + | ====== SUBMISSION OF DATA PROCEDURE |
| - | Any submission of data to the AeroCom database requires to follow the following steps: | + | Any submission of data to the AeroCom database requires to follow the following steps: |
| + | - [[#FORMAT DATA|format data]] to agreed netCDF htap standards, cf compliant | ||
| + | - [[#ADJUST FILENAMES |adjust filenames]] | ||
| + | - [[#CHECK FORMAT|check format]] with cf compliance checker, correct and check again | ||
| + | - [[#OBTAIN ACCOUNT|obtain account]] on htap server to access aerocom-users server | ||
| + | - [[#TRANSFER DATA|transfer data]] to the aerocom-users.met.no | ||
| + | - [[#NOTIFY MET.NO|notify metno]] that data have been submitted | ||
| + | - [[#LOG FILE SUBMISSION|log submission]] in [[aerocom: | ||
| + | - [[# | ||
| - | === 1) FORMAT DATA === | + | ===== 1) format data ===== |
| - | Each variable should be stored in ONE cf-compliant | + | Each variable should be stored in ONE cf-compliant |
| - | CF-compliance implies that sufficient attributes, coordinates and coordinate describing variables are included in each file to describe metadata needed to understand grid representation, | + | CF-compliance implies that sufficient attributes, coordinates and coordinate describing variables are included in each file to describe metadata needed to understand grid representation, |
| See also frequently asked questions on CF: [[https:// | See also frequently asked questions on CF: [[https:// | ||
| - | Formatting can be achieved with cmor tools (http:// | ||
| - | An example file as used for the previous AeroCom experiment | + | ===== 2) adjust filenames ===== |
| + | The obligatory filename | ||
| + | **aerocom3_< | ||
| + | Details about new file naming can be found here: | ||
| + | [[https:// | ||
| + | Be aware of updates ! \\ | ||
| - | === 2) ADJUST FILENAMES === | + | versions will have a date attached.\\ |
| - | The obligatory Filename for using the cf checker and the submission of data is | + | |
| - | " | + | Examples: |
| + | * aerocom3_GOCARTv7.2_AP3-CTRL_od550aer_Column_2010_monthly.nc | ||
| + | * aerocom3_GISS-MATRIX-met2010_AP3-ACRI-AFRICA_od550aer_Column_2010_daily.nc | ||
| + | * aerocom3_GOCARTv7.2-met2010_AP3-CTRL-od550aer_Column_1850_monthly.nc | ||
| + | * aerocom3_GISS-MATRIX-fSST_AP3-CTRL-od550aer_Column_1850_monthly.nc | ||
| - | OLD aerocom | + | OLD (aerocom |
| + | **aerocom3_< | ||
| - | "aerocom_< | + | OLD (aerocom phase 2) aerocom naming convention is also fine for now: |
| + | **aerocom_< | ||
| + | * < | ||
| + | * < | ||
| + | * < | ||
| + | * < | ||
| + | * < | ||
| + | * < | ||
| + | * < | ||
| + | * < | ||
| - | < | ||
| - | < | + | ===== 3) check format ===== |
| + | Typical files must be checked prior to submission via a cf-compliance-checker. [[http:// | ||
| - | < | + | For the upload checker you want to check a typical but small file. To achieve this, you can cut out the first time steps with the nco command: |
| + | ncks -O -d time,0,10 sourcefile.nc filetest.nc | ||
| - | <VerticalCoordinateType> => " | + | Per default, a check shall comprise at least one file of each VerticalCoordinateType |
| - | < | ||
| - | < | + | ===== 4) obtain account |
| - | + | ||
| - | + | ||
| - | === 3) CHECK FORMAT | + | |
| - | Typical files must be checked prior to submission via the cf-compliance-checker. | + | |
| - | + | ||
| - | If you want to check a typical file and rather want to use a smaller file, you can cut out the first time steps with an nco command: | + | |
| - | ncks -O -d time,0,10 file.nc filetest.nc | + | |
| - | + | ||
| - | Read also there the help section. Note: Errors indicate section in the original | + | |
| - | [[http:// | + | |
| - | + | ||
| - | Typical files? A check shall comprise at least one file of each VerticalCoordinateType and each Frequency and each experiment, best varying the variables! | + | |
| - | + | ||
| - | + | ||
| - | === 4) OBTAIN ACCOUNT | + | |
| ...for METNO aerocom servers and WIKI documentation server | ...for METNO aerocom servers and WIKI documentation server | ||
| - | An account at Met.No is needed to submit data via ftp and edit [[aerocom: | + | An account at Met.No is needed to submit data a data server (aerocom-users.met.no) |
| - | + | ||
| - | + | ||
| - | === 5) TRANSFER DATA === | + | |
| - | After obtaining an account, data can then be uploaded in two ways: | + | |
| - | + | ||
| - | Preferred path: via aerocom-users.met.no: | + | |
| - | Use exactly the same model name as used for file names. Attention to lower& | + | |
| - | + | ||
| - | Put files in that renamed directory. Overall size limit of ca 500 GB ( limit for all users) | + | |
| - | + | ||
| - | via ftp.aerocom.met.no: | + | |
| - | Large submissions (> 10 GB) may require extra attention (use rsync, scftp, not simple ftp clients). They might require prior coordination with Met.No/Anna Benedictow/ | ||
| - | The directory / | + | ===== 5) transfer data ===== |
| + | After obtaining an account, data can then be uploaded to the aerocom user server aerocom-users.met.no: | ||
| + | * Create a directory | ||
| + | * upload via ftp.aerocom.met.no has been discontinued due to security concerns. | ||
| + | | ||
| - | If correct in format and with correct filenames, results uploaded here will be processed over night and appear after a day as image catalogue on http:// | + | Large submissions (> 100 GB) may require extra attention (use rsync). They might require prior coordination |
| + | On aerocom-users.met.no the directory / | ||
| - | === 6) NOTIFY MET.NO === | + | If correct in format |
| - | When files are uploaded send email to jan.griesfeller@met.no AND michael.schulz@met.no | + | |
| - | After the data are transferred to the database, data are removed from ftp server or aerocom-users: | ||
| + | ===== 6) notify met.no ===== | ||
| + | When files are uploaded send email to anna.benedictow@met.no and jan.griesfeller@met.no and michael.schulz@met.no (AND coordinator of project), so that the files can be transferred from aerocom-users.met.no to the database. When the data are included in the database the files appear on aerocom-users.met.no under / | ||
| + | After the data are transferred to the database, data are removed from aerocom-users: | ||
| - | === 7) LOG FILE SUBMISSION | + | ===== 7) submit a log file ===== |
| - | FOR EACH SUBMISSION: Put an entry into the htap wiki page [[modelupdates|Log file]], describing what has been submitted. This is also obligatory if changes or updates (even partial) to the original submission are made. We also request | + | FOR EACH SUBMISSION: Put an entry into the htap wiki page [[modelupdates|Log file]], describing what has been submitted. This is also obligatory if changes or updates (even partial) to the original submission are made. We also request |
| - | === 7) UPDATES | + | ===== 8) updates ===== |
| In case of updates of files, the model version identifier should also contain information on the resubmission of model results. Additional files which do not change an earlier submission may contain the same version number as an older submission. Corrections and a model rerun should be accompanied by a new model version identifier. Old still valid files may be merged upon request with new data. Old versions of model submissions will be temporally moved to a separate directory by Met.No, and where possible, deleted after some time. An exception will be made for versions that have been used in publications. Modellers are responsible for communication of major updates to their models (e-mail and log-files, see below). Model analysers are responsible for frequently verifying changes in model submissions (e.g. before submitting a paper). | In case of updates of files, the model version identifier should also contain information on the resubmission of model results. Additional files which do not change an earlier submission may contain the same version number as an older submission. Corrections and a model rerun should be accompanied by a new model version identifier. Old still valid files may be merged upon request with new data. Old versions of model submissions will be temporally moved to a separate directory by Met.No, and where possible, deleted after some time. An exception will be made for versions that have been used in publications. Modellers are responsible for communication of major updates to their models (e-mail and log-files, see below). Model analysers are responsible for frequently verifying changes in model submissions (e.g. before submitting a paper). | ||