Algorithm Configuration Options
The program_options configuration file currently controls the algorithm selected by the value of AlgoCode set.
On 2009/07/08 20:54 the possible values for AlgoCode are:
| AlgoCode | Description | Status |
|---|---|---|
| 1 | Redistribute accumulated precipitation. | OK, but not released |
| 2 | Generate variability parameters. | Development. |
| 3 | General interpolation in both time and space and estimate a modelled value | Development |
| 4 | Temporal interpolation for “one-point” gaps (fill in missing values). Performs the update based on min and max recorded at the end of the hour. | OK, but not released |
| 5 | Perform only an interpolation. | For testing and quality control. |
| 6 | SpaceCheck: Set a quality control flag based on a comparison with neighbours. Configure for any implemented interpolation method. | Development |
| 7 | FlagTester: built in method to support the analysis of flag values. | OK, but not released |
| 8 | StressTester: method to set Qc2 running forever: the one algorithm repeats, e.g. continuous interpolation roots. | OK, but not released |
| 9 | SingleMinMaxAverage | kvqc2_1.0.1 For a given paramid and corresponding Max paramid and Min paramid calculates a correction for an single missing value as average of max and min. If no max or min available or specified then reverts to simple linear. |
| 10 | SingleLinear | kvqc2_1.0.1 Replaces a single missing value with a linear interpolated value. If specified checks that the correction lies within the available max … min range, if not sets to nearest of max or min. Run for any paramid and optional max, min specified in the configuration file. |
| 11 | Null call … does nothing. | For testing only. |
| other | Other values are handled gracefully and no algorithm initiated. | OK |
Temporal interpolation for "one-point" gaps
[ALGOCODE:4]
Algorithm location in kvalobs distribution: src/kvQc2/algoithms/ProcessUnitT
Currently configured for hourly temperature values:
- Locates a single missing value
- Checks if data available immediately before and after the interval
- If available generates first estimate from linear interpolation
- Checks for the Max and Min temperature for the missing value from the measurements registered in the hour afterwards
- Performs a second estimate as the average of 0.5*(MaxT + MinT)
TBD … add logic to propose best model value … some example data:
T-Corrected: The current supplied corrected value (typically a model value and/or HQC check)
Linear: Linear interpolation of the proceeding and following value.
T-Tan-Tax: The mean of the Min and Max values for the same hour (recorded in the following hour) is another possible corrected value.
T-neighbour: For reference, the value of the nearest neighbour.
| T0 | T1 | T2 | T-Corrected | Linear | T-Tan-Tax | T-neighbour | TAN | TAX | CFAILED |
|---|---|---|---|---|---|---|---|---|---|
| -0.3 | -32767 | -1.7 | -1.2 | -1 | -1.65 | 0.2322 | -2.1 | -1.2 | QC1-4-211:1,hqc |
| 0.1 | -32767 | -0.1 | -0.1 | 0 | -0.1 | -1.68 | -0.3 | 0.1 | QC1-4-211:1,hqc |
| -8.5 | -32767 | -6.1 | -7.5 | -7.3 | -6.85 | -16.0474 | -7.7 | -6 | QC1-4-211:1,hqc |
| -8.4 | -32767 | -8.6 | -8.5 | -8.5 | -8.55 | -10 | -8.6 | -8.5 | QC1-4-211:1,hqc |
| -6.4 | -32767 | -7.3 | -7 | -6.85 | -7.05 | 6.4144 | -7.3 | -6.8 | QC1-4-211:1,hqc |
| -5.9 | -32767 | -6.4 | -6.2 | -6.15 | -6.1 | 5.4016 | -6.4 | -5.8 | QC1-4-211:1,hqc |
| -16.7 | -32767 | -17.2 | -17 | -16.95 | -17.15 | -1.48 | -17.5 | -16.8 | QC1-4-211:1,hqc |
| -0.7 | -32767 | -2.7 | -1.2 | -1.7 | -1.9 | 0 | -2.7 | -1.1 | QC1-4-211:1,hqc |
| -3 | -32767 | -2.8 | -2.8 | -2.9 | -2.9 | -0.7224 | -3.3 | -2.5 | QC1-4-211:1,hqc |
| 3.4 | -32767 | 3.4 | 3.3 | 3.4 | 3 | 3.26372 | 2.5 | 3.5 | QC1-4-211:1,hqc |
| -4.6 | -32767 | -4.1 | -2.7 | -4.35 | -4.15 | -13.26 | -4.3 | -4 | QC1-4-211:1 |
| 2.7 | -32767 | 1.6 | 2 | 2.15 | 1.35 | 5.194 | 0.9 | 1.8 | QC1-4-211:1,hqc |
| 5.4 | -32767 | 4.2 | 4.3 | 4.8 | 4.3 | 5.7732 | 3.6 | 5 | QC1-4-211:1 |
| 4.9 | -32767 | 5 | 3.4 | 4.95 | 5.1 | 5.47888 | 4.8 | 5.4 | QC1-4-211:1 |
| 3.2 | -32767 | 4.6 | 3.7 | 3.9 | 4.1 | 4.27896 | 3.6 | 4.6 | QC1-4-211:1 |
| -11.6 | -32767 | -9 | -10 | -10.3 | -9.55 | -10 | -10.2 | -8.9 | QC1-4-211:1,hqc,hqc |
| -9.5 | -32767 | -9.5 | -9.5 | -9.5 | -9.65 | -5.74 | -9.8 | -9.5 | QC1-4-211:1,hqc |
| 0.4 | -32767 | 1.4 | 1.2 | 0.9 | 1.8 | 0.50176 | 1.2 | 2.4 | QC1-4-211:1,hqc |
Given that the corrected value must lie between TAN and TAX, the bold values indicate possible actual values (other values are out of range). The model values are sometimes very different, but note (for example):
| -8.4 | -32767 | -8.6 | -8.5 | -8.5 | -8.55 | -10 | -8.6 | -8.5 | QC1-4-211:1,hqc |
Since TAN=-8.6, and TAX=-8.5, the expected value should lie between these values …but -8.5 may still be closer to the value than the mean -8.55!!!! Suggest only to use mean(TAX,TAN) when the corrected value is observed to be outside possible range ??? — Yes this is the current implementation
if ( Tseries[1].corrected() < MinT.begin()→original() || Tseries[1].corrected() > MaxT.begin()→original() ) { / Update if correction is out of TAN TAX range!
But needs testing and debugging !!
Some results comparing the method to the corrections already in place for missing data for March 2008:
Some thoughts on control flag settings
===== General interpolation in both time and space and estimate a modelled value =====
[ALGOCODE:3]
This is a generalisation of processUnitT and identifies a gap of arbitrary length in a time series and identifies valid points before and after the gap up until the next gap on either side. This provides a small time series (with missing values) upon which the most appropriate temporal (and spatial … taking into account neighbours) may be applied.
<code>src/kvQc2/algorithms/Process4D.cc</code>