Feedback on this page (LA):
- The size of gap is essential for allowing interpolation.
- The second derivative of the curve (f“(x)) the last 2-3 hours before the gap should influence on the size of an acceptable gap for interpolation; and also influence on whether to interpolate the first two hours of a larger gap or not. E.g.: If f”(x)>0 the last 2-3 hours before a 10 hours gap (graph 4),
- Skeptical concerning interpolation of the first two hours. But if the the gap is only 5 hours suggest to accept the interpolation.
- If f“(x) is changing sign the last hour before the gap (or second hour after the gap) suggest a two hours interpolation of a larger gap.
- Concerning flagging, the acceptable size of a gap will become important for using ftime=1 or ftime=2.
GAP INTERPOLATION
The Akima Spline interpolates based on the gradients between points of the nearest neighbours in the time series. At least five neighbours are required (this may be satisfied by having two to three points on either side of a gap). The values of more distant neighbours appear to have little effect on the finally interpolated data.
Advantage is taken of this behaviour in the proposed variable length gap algorithm, as described here:
- For a given station an arbitrary length time series of data is selected.
- Points in the time series that satisfy a given quality condition are used to generate the Akima Spline. (in the first version of the algorithm only data where useinfo(2)=0 , any other criteria may also be set).
- The time series is then parsed for missing values. If a value is missing but it is sufficiently close to enough good neigbours (exact details of this to be decided - please see discussion below) then it can be replaced with the value from the Akima spline.
- Note: this method has a good performance because very few actual interpolations runs using Akima are required. Input is needed though on determining the logic to decide when to use an interpolation and when not too (again, please read on).
- The method may be further developed by also performing a spatial cross check with neighbours at the same time for a missing point (this would reuse existing Qc2 code).
Using Akima For Variable Length Gaps
An example illustrating the performance of the Akima interpolation.
Original Data
Simulated Gaps
For this example we generate our own gaps so that the results may also be compared to the original values.
Akima Fit
Akima Fit also indicating the interpolated data points (triangles) and original values (red circles)
Comparison of data
| Hour | … | 21 | 22 | 23 | 24 | 25 | … | 38 | 39 | 40 | … | 48 | 49 | 50 | 51 | … | 97 | 98 | 99 | 100 | 101 | 102 | 103 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Original Value | … | -0.2 | 0.1 | -0.4 | 1 | 0.7 | … | 3.8 | 4.5 | 4 | … | 2.7 | 2.5 | 2.4 | 2.5 | … | 1.8 | 1.7 | 1.5 | 1.6 | 1.4 | 1.4 | 1.8 |
| Akima | … | -0.6 | -0.6 | -0.4 | 0.0 | 0.3 | … | 4.1 | 3.8 | 3.7 | … | 2.7 | 2.6 | 2.6 | 2.6 | … | 1.7 | 1.5 | 1.4 | 1.4 | 1.4 | 1.6 | 1.8 |
Some Thoughts about when to fit and when not to fit
The criteria to decide if to input an interpolation may be one or a combination of the following:
- Length of the gap
- Distance of a missing point from the nearest neighbour
- Distance of a missing point from the second nearest neighbour
- etc. …
- Other criteria
The following sketch illustrates these points:
Think Tank
Legend:
| G | Good Point |
|---|---|
| X | Missing Point |
| I | Interpolated Point |
| Time Series | Gap ⇐ 4 hours | Two Neighbours in the same direction ⇐ 2 hours |
|---|---|---|
| GGGXXXXXXGGG | GGGXXXXXXGGG | GGGIIXXIIGG |
| GGGXXXXGXXXXGGG | GGGIIIIGIIIIGGG | GGGIIXXGXXIIGGG |
etc. …
| Which cases do we want to apply long interpolations to? |
|---|
FLAGS and CFAILED values to SET
In the current implementation the same flag settings are used as is the case for the Single Linear Interpolation (ftime=1) and CFAILED is set to QC2d-2-A, the extra string (“LONG GAPS” or similar) is temporary and just helps my testing.
In these examples the algorithm is over-writing values corrected by HQC. This would not be the default option, but is useful in testing with old data because one finds many gaps if one triggers the algorithm based on original data. Specification details are very welcome here!
67560 | 2008-04-18 06:00:00 | 2.1 | 215 | 2008-04-18 05:48:32 | 330 | 0 | 0 | 2.1 | 1111000000000010 | 7100000300000000 |
67560 | 2008-04-18 07:00:00 | 2.8 | 215 | 2008-04-18 06:49:28 | 330 | 0 | 0 | 2.8 | 1111000000000010 | 7100000300000000 |
67560 | 2008-04-18 08:00:00 | -32767 | 215 | 2008-04-18 08:30:18 | 330 | 0 | 0 | 3.8 | 0000001100000005 | 2892900000000041 | hqc,QC2d-2-A,XLONG GAPS
67560 | 2008-04-18 09:00:00 | -32767 | 215 | 2008-04-18 09:30:38 | 330 | 0 | 0 | 4.8 | 0000001100000005 | 2892900000000041 | hqc,QC2d-2-A,XLONG GAPS
67560 | 2008-04-18 10:00:00 | 5.6 | 215 | 2008-04-18 10:11:32 | 330 | 0 | 0 | 5.6 | 1110000000000010 | 7000000000000000 |
67560 | 2008-04-18 11:00:00 | 5.9 | 215 | 2008-04-18 10:48:32 | 330 | 0 | 0 | 5.9 | 1111000000000010 | 7100000300000000 |
Note the same algorithm will also do the single points as well
12320 | 2008-03-23 02:00:00 | -13.9 | 211 | 2008-03-23 02:07:03 | 330 | 0 | 0 | -13.9 | 1111100000000010 | 7000000000000000 |
12320 | 2008-03-23 03:00:00 | -32767 | 211 | 2008-03-23 03:31:13 | 330 | 0 | 0 | -14.6 | 1000601100000007 | 1892900000000052 | QC1-4-211:1,hqc,QC2d-2-A,LONG GAPS
12320 | 2008-03-23 04:00:00 | -15.1 | 211 | 2008-03-23 04:07:04 | 330 | 0 | 0 | -15.1 | 1110100000000010 | 7000000000000000 |
Here is the data history for the above point:
version | stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed | modificationtime ----------+-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------+---------------------------- 11929428 | 12320 | 2008-03-23 03:00:00 | -32767 | 211 | 2008-03-23 03:31:13 | 330 | 0 | 0 | -14.5 | 1000601000000007 | 3891900000000051 | QC1-4-211:1,hqc | 2011-02-27 23:44:23.996873 51889231 | 12320 | 2008-03-23 03:00:00 | | 211 | 2008-03-23 03:31:13 | 330 | 0 | 0 | | | | | 2011-07-04 20:34:59.17184 51889232 | 12320 | 2008-03-23 03:00:00 | -32767 | 211 | 2008-03-23 03:31:13 | 330 | 0 | 0 | -14.6 | 1000601100000007 | 1892900000000052 | QC1-4-211:1,hqc,QC2d-2-A,LONG GAPS | 2011-07-04 20:34:59.254227 (3 rows)