Note: a number of Qc2 useinfo changes are coded in lib/kvalobs/kvDataFlag.cc but are all commented out and inactive. For development and reflectedin the examples provided below, the Qc2 items are all turned on. Please see below for details.
A general guide to setting values in Qc2 Control Flags is described in the following set of slides from Lars. The current algorithms and proposals for new algorithms are based on this:
Controlflag Setting: | fd=7 |
---|---|
Condition: | The algorithm is successful. |
Cfailed: | Updated to include algorithm name and type of interpolation used to generate the correction |
(Also possible to use fd=A, fd=B, … for other methods of redistribution, but such information can also always be kept in cfailed.) |
Examples
An example of the redistribution of accumulated precipitation. Where successful estimation of corrected values occurs, the controlinfo(12), fd is set to 7. The useinfo is then updated in accordance with the updated routine above.
BEFORE Qc2 stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------------- 53950 | 2007-12-07 06:00:00 | 9 | 110 | 2007-12-07 12:10:30 | 302 | 0 | 0 | 9 | 1140000000001000 | 7020400000000001 | QC1-2-72.b12:1,QC1-2-72.c12:1 53950 | 2007-12-08 06:00:00 | -32767 | 110 | 2007-12-09 00:36:39 | 302 | 0 | 0 | -32767 | 1000003000002000 | 7899900000000000 | QC1-7-110:1 53950 | 2007-12-09 06:00:00 | -32767 | 110 | 2007-12-10 00:40:02 | 302 | 0 | 0 | -32767 | 1000003000002000 | 7899900000000000 | QC1-7-110:1 53950 | 2007-12-10 06:00:00 | 1.5 | 110 | 2007-12-10 11:25:20 | 302 | 0 | 0 | 1.5 | 1140000000002000 | 7330900000000001 | QC1-2-72.b12:1,QC1-2-72.c12:1,QC1-7-110:1 (4 rows) AFTER Qc2 kvalobs=# select * from data where stationid=53950 and obstime>'2007-12-07' and obstime<'2007-12-11' and paramid=110; stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------------------------------------------------------------------- 53950 | 2007-12-07 06:00:00 | 9 | 110 | 2007-12-07 12:10:30 | 302 | 0 | 0 | 9 | 1140000000001000 | 7020400000000001 | QC1-2-72.b12:1,QC1-2-72.c12:1 53950 | 2007-12-08 06:00:00 | -32767 | 110 | 2007-12-09 00:36:39 | 302 | 0 | 0 | 0.1 | 1000003000007000 | 5899900000000000 | QC1-7-110:1 Qc2 Redis corrected was:-32767 53950 | 2007-12-09 06:00:00 | -32767 | 110 | 2007-12-10 00:40:02 | 302 | 0 | 0 | 0 | 1000003000007000 | 5899900000000000 | QC1-7-110:1 Qc2 Redis corrected was:-32767 53950 | 2007-12-10 06:00:00 | 1.5 | 110 | 2007-12-10 11:25:20 | 302 | 0 | 0 | 1.4 | 1140000000007000 | 5336900000000001 | QC1-2-72.b12:1,QC1-2-72.c12:1,QC1-7-110:1 Qc2 Redis corrected was:1.5 (4 rows)
NOTES:
Interpolation of single missing temperature values using the average of TAN and TAX corresponding to the same time interval.
Controlflag Setting: | ftime=1 |
---|---|
Condition: | The algorithm is successful |
Cfailed: | Algorithm method recorded. |
Examples
Sample data before Qc2 59680|2008-03-19 05:00:00|-1.7 |211|2008-03-19 04:54:05|311|0|0|-1.7|1111100000000010|7000000000000000| 59680|2008-03-19 06:00:00|-32767|211|2008-03-19 06:32:48|311|0|0|-1.7|1000601000000007|3891900000000021|QC1-4-211:1,hqc 59680|2008-03-19 07:00:00|-0.6 |211|2008-03-19 06:54:00|311|0|0|-0.6|1110100000000010|7000000000000000| After the Qc2 algorithm the row with the missing line is updated as follows: 59680|2020-03-19 06:00:00|-32767|211|2020-03-19 06:32:48|311|0|0|-0.7|1000601100000007|1891900000000021|QC1-4-211:1,hqc Qc2 UnitT corrected was:-1.7
Discussion (a summary of inputs from many different people, identities removed since this is the public wiki)
Scheduling
Whereas the default operation is that Qc2 shall not overwrite HQC results the need that this is ever considered shall be minimised by always having Qc2 before HQC. Even so there may be situations that new data is available and Qc2 is being rerun that Qc2 may overwrite HQC. When this case arises the rules to be followed have to specified.
Also in the case when Qc2 is delayed this may happen:
In a Space Check an independent estimate of a parameter measured at a station is derived from a set of neighbouring values. Comparison with the estimate is used to assign a confidence level to the original parameter observation.
The estimate can be considered to be a “model” value and therefore the higher setting of the fnum controlflag (7-A) may be used to indicate the performance of this quality control. If the Space Check result is that there is agreement between the observed measurement and the model value then fnum is set to 1 as usual.
Controlflag Setting: | fnum=0-6 | fnum=7-A |
---|---|---|
Condition: | As already defined for numerical model runs. | Qc2 Values For Space Check |
Cfailed: | Specific details of the SpaceCheck performed, e.g. type of check and interpolation method etc. | |
–oo0oo– | ||
fnum=7 | Controlled. Deviation between observed value and SpaceCheck estimate higher than high test value | |
fnum=8 | Controlled. Deviation between observed value and SpaceCheck estimate lower than low test value | |
fnum=9 | Controlled. Deviation between observed value and SpaceCheck estimate higher than highest test value | |
fnum=A | Controlled. Deviation between observed value and SpaceCheck estimate lower than lowest test value |
fnum [0-6] are set by Qc1-4 based on comparison with numerical model values. Qc1-4 will run before the Qc2-SpaceCheck.
Open issues:
Another type of Space Check is to determine the variabilty in the nearest neighbour field. If the variability is high then spatial algorithms are either not applied or the confidence parameters are lowered to mark any given result or check (e.g. use ftime=2 rather than ftime=1).
Future Qc2 checks will involve comparison with radar, satellite data etc. How will this eventuality be flagged? For weather analysis fw is available (although possibly now proposed for other uses as well) and for climatological controls (e.g. comparison with expected monthly statistics) fclim is available.
Controlflag Setting: | ftime=1 |
---|---|
Condition: | The algorithm is succesful |
cfailed: | Algorithm applied is recorded |
–oo0oo– |
General assumption is that only actually replace a missing value (be it a single point or a set of points) if there is good confidence in the corrected estimates, e.g. the time interpolation and spatial interpolation agree. ftime=2 and ftime=3 are also available for use as defined in the kvalobs original specification.
The Space Check and Time Interpolation methods may also identify outliers, e.g. Dip test. The existing specification for controlinfo(3), fs can capture such events from Qc2 checks.
TBD
This specification is taken from Slide 3 of Flags in Qc2.
fw=0 | Not controlled |
---|---|
fw=1 | Controlled, found OK |
fw=2 | Controlled. Slightly suspect value, not corrected (changed) |
fw=3 | Controlled. Highly suspect value, not corrected (changed) |
fw=4 | Controlled. Erronous value, not corrected (new) |
fw=5 | Controlled. Erronous value, corrected automatically (new) |
- | - |
fw=8 | Outlier, rejected (new) |
The kvalobs library has a built in function which sets useinfo based on controlinfo settings.
<key> bool setUseFlags(const kvControlInfo& cinfo); </key>
that is located in … src/lib/kvalobs/kvDataFlag.cc
The setUseFlags already anticipates the setting of user flags in response to QC2 controls. However, most of the logic is commented out in the operational code, as indicated below:
bool kvControlInfo::qc2dDone() const { return false;//flag( f_fs ) or flag( f_ftime ) or flag( f_fw ) or flag( f_fstat ); } ... bool kvControlInfo::qc2mDone() const { return false;//flag( f_fclim ) or flag( f_fd ); } ... ui[2]= 9; //if ( cinfo.qc1Done() or cinfo.qc2Done() or cinfo.hqcDone() ) ... ui[ 3 ] = 9; //if ( cinfo.qc1Done() or cinfo.qc2Done() or cinfo.hqcDone() ) ... ui[ 4 ] = 9; // NB: After useinfo[2] //if ( cinfo.qc1Done() or cinfo.qc2Done() or cinfo.hqcDone() )
Implications:
In the ProcessUnitT example below the effect of turning on the Qc2 useinfo is visible; i.e.
bool kvControlInfo::qc2dDone() const { return flag( f_fs ) or flag( f_ftime ) or flag( f_fw ) or flag( f_fstat ); } ... bool kvControlInfo::qc2mDone() const { return flag( f_fclim ) or flag( f_fd ); } ... ui[2]= 9; if ( cinfo.qc1Done() or cinfo.qc2Done() or cinfo.hqcDone() ) ... ui[ 3 ] = 9; if ( cinfo.qc1Done() or cinfo.qc2Done() or cinfo.hqcDone() ) ... ui[ 4 ] = 9; // NB: After useinfo[2] if ( cinfo.qc1Done() or cinfo.qc2Done() or cinfo.hqcDone() )
Generate a sample Qc2 controlinfo value and observe what codes the above generates for useinfo. (Please suggest additional test cases)
Qc2 Flag or Specific Algorithm | Controlinfo | Resulting useinfo |
---|---|---|
RR24 Redistribution | [1000001000007000] | [5896900000000000] |
RR24 Redistribution | [1140001000007000] | [5896900000000001] |
ftime from TAN TAX interpolation | [1000600100000000] | [5033700000000001] |
ftime | [1000600200000000] | [5033700000000001] |
ftime | [1000600300000000] | [5033700000000001] |
fnum | [1100700000000000] | [7000000000000001] |
fnum | [1100800000000000] | [7000000000000001] |
fnum | [1100900000000000] | [7000000000000001] |
fnum | [1100A00000000000] | [7000000000000001] |
fclim | [1111000000010000] | [5000000000000000] |
fclim | [1111000000020000] | [5010500000000001] |
fclim | [1111000000030000] | [5033500000000001] |
fw | [1111000010000000] | [5000000000000000] |
fw | [1111000020000000] | [5010500000000001] |
fw | [1111000030000000] | [5033500000000001] |
fw | [1111000040000000] | [5000000000000001] |
fw | [1111000050000000] | [5000000000000001] |
fw | [1111000080000000] | [5000000000000001] |
control (no Qc2) | [1111000000000000] | [5000000000000000] |
control (no Qc2) | [1111000000000000] | [5000000000000000] |
control (no Qc2) | [1000001000000000] | [9899900000000000] |
Furthermore, here is an example of Qc1 Regression tests.