This is an old revision of the document!
Qc2 Prototype
A protptype has been developed to tackle an example Qc2 problem: the redistribution of accumulated 24 hour rainfall measurements. Details of the problem were earlier described in presentations made at ECSN 2007 and during an internal status meeting (see slides below). The report documents the development of the prototype and also identifies a number of specification, design and operational issues which need to be resolved in order to establish an overall design for the Qc2 system.
The report is provided here and feedback on any of the questions raised is most welcome. Please enter your comments under the appropriate sub-heading or make a new section.
Open Issues
Communication of Qc2 with Qc1 daemons
Description
When Qc2 updates a value in the kvalobs database all interested processes need also to be informed. In the prototype this is achieved by sending a message directly to the kvServiced that new data is available. Other options are to have the communication handled through kvManagerd, or to use an independent system (even a different architecture).
Feedback
…
Qc2 Task Management
Description
The Qc2 prototype runs each task sequentially and the process is controlled by an external configuration file. Are any more sophisticated options required? For example: (i)monitoring of database contents and running algorithms when a data availability criteria is met, (ii)utilisation of a work queue concept as in Qc1, (iii)or the capability to run Qc2 algorithms in parallel?.
Feedback
…
Useinfo
Description
The initial plan is not to change the set up of useinfo flags as defined in the kvalobs libraries. The Qc2 controls and functions will change the controlinfo and the useinfo will change as a result of that. The impact on the useinfo will be tested and only then will any need for modification to the kvalobs library handling the useinfo be identified. The changes though to turn on the qc2 flags inside the kvalobs lib will require regression testing of the kvalobs Qc1 system and downstream useinfo settings. The issue is that Qc2 will manage the controlinfo but the impact on the useinfo is unknown.
Feedback
…
End User Interface
Description
The current plan is to have all of the Qc2 algorithms as compiled code. With user control limited to defining and submitting configuration files. Even though the large data management aspects of Qc2 favours compiled code over the use of a scripting language (e.g. like the use of Perl in Qc1) are there requirements for such interfaces for any particular tasks? A Perl, or Python, etc option could be included.
Feedback
…
Interaction with HQC
Description
At the minimum this occurs by Qc2 setting flags which are brought to the attention of HQC. Additional interaction can be implemented by special Qc2 incident lists and also plots of the interpolated fields (including parameters such as standard deviation) hotspots indicate areas where the Qc2 algorithms may have produced unreliable results.
Feedback
…
Model Output
Description
The model output, e.g, interpolations generated during a control will be stored in netCDF files for future re-use and comparison. Other formats or another database could also be used: just needs to be defined. A switch may be made to using HDF5 since it supports the definition of user defined data-types.
Feedback
…
Lessons Learned from Qc1
Description
The Qc2 prototype reuses components from Qc1: kvManagerd, kvQabased, and the database connection and CORBA architecture in general. Are there any components which are recommended to be redeveloped in conjunction with Qc2?
Feedback
…
Ongoing Tasks
Description
Please see details in the Report-
Feedback
…
Miscellaneous
Feedback
…