Single Missing Point

First Algorithm

The first single missing point algorithm addressed hourly temperature and provided a corrected value by taking the average of the maximum and minimum observed temperature for that hour recorded at the following time stamp (as described here).

Test data has been taken from March 2008. On each day for all stations the “original” hourly temperature has been set to a missing value -32767 for the times 05:00, 11:00, 17:00 and 23:00 UT and the original value is kept for reference. The results of the algorithm are then compared to the original data:

Good results are obtained. Note the points at -99.9 arise due to this value being present in the original data record (and they are not a result of the algorithm).

Generalisation

The algorithm has now been generalised (so-called SIngleMissingPoint):

• The paramids for the value to be corrected and the minimum and maximum values are now specified in the configuration file. This means the same algorithm may be applied to different parameters as appropriate (each case still to be tested).
• If no Max and Min value are available, or if these are just not declared in the config file, the algorithm performs a linear interpolation instead. (in future, the interpolation method can also be selected, e.g. when an akima routine has been finaliused etc).

Running the new algorithm on the same data as above, the algorithm first tries to do a Max and Min average. If this is not possible a linear interpolation is performed across the gap. First results:

It is clear that in the above graph there are additional points that also show good agreement with the original values. There are also many original values that are unphysical (e.g. high temperatures in March and filtering on flags has also to be applid so that these cases are not included). Examples of high temperatures in the database:

     18020 | 2008-03-10 03:00:00 |    58.78 |     211 | 2008-03-10 06:07:09 |      4 | 0      |     0 |       2.8 | 1000600000000610 | 7033700000000002 | QC1-0-211:1,QC1-4-211:1
     18020 | 2008-03-10 04:00:00 |     57.4 |     211 | 2008-03-10 06:07:09 |      4 | 0      |     0 |       2.9 | 1000600000000610 | 7033700000000002 | QC1-0-211:1,QC1-4-211:1
     18020 | 2008-03-10 05:00:00 |    56.02 |     211 | 2008-03-10 06:07:09 |      4 | 0      |     0 |       3.1 | 1000600000000610 | 7033700000000002 | QC1-0-211:1,QC1-4-211:1

Ok, lets filter out results where control(4)=6 … fnum=6. This gives the following improved result, but the algorithm still appears to act on some temperatures that are too high and should not be corrected by interpolating between the times before and after, that are also too high!?!

Configuration File Used:

#[Specific Runs]
#Last 24hours .....
#Daily Run ....
# /disk1/KVALOBS/Qc2Config .... $KVALOBS

# Performs simple linear interpolation for a single point replacement.
AlgoCode=9
#InterpCode=9   # 9
RunAtHour=4 
RunAtMinute=58   

#[Time Range]
Start_YYYY=2036  #NB in the test data I set all 2008->2036 for convenience ...
Start_MM=2
Start_DD=28
Start_hh=0
Start_mm=0
Start_ss=0

End_YYYY=2036
End_MM=4
End_DD=1
End_hh=0
End_mm=0
End_ss=0

#[Time Step]
#Step_hh=12 

#[Specific Data Type and Paramters ids etc.]
ParamId=211
MaxParamId=215
MinParamId=213
#InterpolationDistance=50.0
# Only write back the result if not prviously controlled
W_fhqc=0

#Flag to set if value is corrected
S_ftime=1
change_fmis=3->1
change_fmis=0->4
#Flag to check if the algorithm shall be applied ...
A_ftime=1   #
A_fnum=6    # This is to cut out the data with large values