Qc2 Prototype
A prototype has been developed to tackle an example Qc2 problem: the redistribution of accumulated 24 hour rainfall measurements. Details of the problem were earlier described in presentations made at ECSN 2007 and during an internal status meeting (see slides below).
Slides: Presentation NORDKLIM-OBS 20080911
The report documents the development of the prototype and also identifies a number of specification, design and operational issues which need to be resolved in order to establish an overall design for the Qc2 system.
The report is provided above and feedback on any of the questions raised is most welcome. Please enter your comments under the appropriate sub-heading or make a new section.
Open Issues
See the slides from 2008-11-21 for further resolution on the open issues below. The issues below will be updated anon to reflect the discussion on 2008-11-21 … TBD.
Communication of Qc2 with Qc1 daemons
Description
When Qc2 updates a value in the kvalobs database all interested processes need also to be informed. In the prototype this is achieved by sending a message directly to the kvServiced that new data is available. Other options are to have the communication handled through kvManagerd, or to use an independent system (even a different architecture).
Feedback
[2008-08-06] This needs a speedy solution. Please discuss the issue directly with Børge Moe and Vegard Bønes. Gabriel.
[2008-08-28] excerpts from that discussion
Preferably Qc2 is built without any dependencies on Qc1, other than using flags previously written by Qc1 into the kvalobs database as filters for the Qc2 processing.
The issue of communication between Qc2 and Qc1 only arises because of the requirement that updates to the kvalobs database are notified to all services using kvalobs data, and because Qc2 could do this by piggybacking on the Qc1 solution, either:
- (1)send messages directly to kvServiced
- (2)communicate with- and delegate responsibility to- kvManagerd
failing this
- (3) we use a completey Qc1 independnet solution (e.g. kvQc2Serviced or something)
Solution (1) is the simplest but there is a concern that the work load cannot be handled without the mediation of kvManagerd, therefore (2) is an option. I have not seen any hard evidence for such a concern but I have also not studied closely the kvServiced↔kvManagerd architecture. Or should we go for a (3)?
To find out a bit more, I would like to run some Qc2 experiments on seca at the same time that Qc1 is being tested and see what happens when kvServiced is subjected to extra messages from Qc2. paule
[2008 Week 43] Decision is to adopt solution (1) and send messages directly to kvServiced. BM & PE
Qc2 Task Management
Description
The Qc2 prototype runs each task sequentially and the process is controlled by an external configuration file. Are any more sophisticated options required? For example: (i)monitoring of database contents and running algorithms when a data availability criteria is met, (ii)utilisation of a work queue concept as in Qc1, (iii)or the capability to run Qc2 algorithms in parallel?
Feedback
…
Useinfo
Description
The initial plan is not to change the set up of useinfo flags as defined in the kvalobs libraries. The Qc2 controls and functions will change the controlinfo and the useinfo will change as a result of that. The impact on the useinfo will be tested and only then will any need for modification to the kvalobs library handling the useinfo be identified. The changes though to turn on the qc2 flags inside the kvalobs lib will require regression testing of the kvalobs Qc1 system and downstream useinfo settings. The issue is that Qc2 will manage the controlinfo but the impact on the useinfo is unknown.
Feedback
[2008-08-06] This is strongly linked to the uppermost issue. Regression needs should influence on how QC1-QC2 communication is resolved. Gabriel.
End User Interface
Description
The current plan is to have all of the Qc2 algorithms as compiled code. With user control limited to defining and submitting configuration files. Even though the large data management aspects of Qc2 favours compiled code over the use of a scripting language (e.g. like the use of Perl in Qc1) are there requirements for such interfaces for any particular tasks? A Perl, or Python, etc option could be included.
Feedback
…
Interaction with HQC
Description
At the minimum this occurs by Qc2 setting flags which are brought to the attention of HQC. Additional interaction can be implemented by special Qc2 incident lists and also plots of the interpolated fields (including parameters such as standard deviation) hotspots indicate areas where the Qc2 algorithms may have produced unreliable results.
Feedback
…
Model Output
Description
The model output, e.g, interpolations generated during a control will be stored in netCDF files for future re-use and comparison. Other formats or another database could also be used: just needs to be defined. A switch may be made to using HDF5 since it supports the definition of user defined data-types.
Feedback
…
Lessons Learned from Qc1
Description
The Qc2 prototype reuses components from Qc1: kvManagerd, kvQabased, and the database connection and CORBA architecture in general. Are there any components which are recommended to be redeveloped in conjunction with Qc2?
Feedback
…
Ongoing Tasks
Description
Please see details in the Report.
Feedback
…
Miscellaneous
Feedback
…