GAP INTERPOLATION

The Akima Spline interpolates based on the gradients between points of the nearest neighbours in the time series. At least five neighbours are required (this may be satisfied by having two to three points on either side of a gap). The values of more distant neighbours appear to have little effect on the finally interpolated data.

Advantage is taken of this behaviour in the proposed variable length gap algorithm, as described here:

For a given station an arbitrary length time series of data is selected.

Points in the time series that satisfy a given quality condition are used to generate the Akima Spline. (in the first version of the algorithm only data where useinfo(2)=0 , any other criteria may also be set).

The time series is then parsed for missing values. If a value is missing but it is sufficiently close to enough good neigbours (exact details of this to be decided - please see discussion below) then it can be replaced with the value from the Akima spline.

Using Akima For Variable Length Gaps

An example illustrating the performance of the Akima interpolation.

Original Data

Simulated Gaps

For this example we generate our own gaps so that the results may also be compared to the original values.

Akima Fit

Akima Fit also indicating the interpolated data points (triangles) and original values (red circles)

Comparison of data

Hour	...	21	22	23	24	25	...	38	39	40	...	48	49	50	51	...	97	98	99	100	101	102	103
Original Value	...	-0.2	0.1	-0.4	1	0.7	...	3.8	4.5	4	...	2.7	2.5	2.4	2.5	...	1.8	1.7	1.5	1.6	1.4	1.4	1.8
Akima	...	-0.6	-0.6	-0.4	0.0	0.3	...	4.1	3.8	3.7	...	2.7	2.6	2.6	2.6	...	1.7	1.5	1.4	1.4	1.4	1.6	1.8

Some Thoughts about when to fit and when not to fit

The criteria to decide if to input an interpolation may be one or a combination of the following:

Length of the gap

Distance of a missing point from the nearest neighbour

Distance of a missing point from the second nearest neighbour

etc. ...

Other criteria

The following sketch illustrates these points:

Think Tank

Legend:

G	Good Point
X	Missing Point
I	Interpolated Point
Time Series	Gap <= 4 hours	Two Neighbours in the same direction <= 2 hours
GGGXXXXXXGGG	GGGXXXXXXGGG	GGGIIXXIIGG
GGGXXXXGXXXXGGG	GGGIIIIGIIIIGGG	GGGIIXXGXXIIGGG

etc. ...

Which cases do we want to apply long interpolations to?

FLAGS and CFAILED values to SET

In the current implementation the same flag settings are used as is the case for the Single Linear Interpolation (ftime=1) and CFAILED is set to QC2d-2-A, the extra string (“LONG GAPS” or similar) is temporary and just helps my testing.

In these examples the algorithm is over-writing values corrected by HQC. This would not be the default option, but is useful in testing with old data because one finds many gaps if one triggers the algorithm based on original data. Specification details are very welcome here !

67560 | 2008-04-18 06:00:00 | 2.1 | 215 | 2008-04-18 05:48:32 | 330 | 0 | 0 | 2.1 | 1111000000000010 | 7100000300000000 |
67560 | 2008-04-18 07:00:00 | 2.8 | 215 | 2008-04-18 06:49:28 | 330 | 0 | 0 | 2.8 | 1111000000000010 | 7100000300000000 |
67560 | 2008-04-18 08:00:00 | -32767 | 215 | 2008-04-18 08:30:18 | 330 | 0 | 0 | 3.8 | 0000001100000005 | 2892900000000041 | hqc,QC2d-2-A,XLONG GAPS
67560 | 2008-04-18 09:00:00 | -32767 | 215 | 2008-04-18 09:30:38 | 330 | 0 | 0 | 4.8 | 0000001100000005 | 2892900000000041 | hqc,QC2d-2-A,XLONG GAPS
67560 | 2008-04-18 10:00:00 | 5.6 | 215 | 2008-04-18 10:11:32 | 330 | 0 | 0 | 5.6 | 1110000000000010 | 7000000000000000 |
67560 | 2008-04-18 11:00:00 | 5.9 | 215 | 2008-04-18 10:48:32 | 330 | 0 | 0 | 5.9 | 1111000000000010 | 7100000300000000 |

Note the same algorithm will also do the single points as well

12320 | 2008-03-23 02:00:00 | -13.9 | 211 | 2008-03-23 02:07:03 | 330 | 0 | 0 | -13.9 | 1111100000000010 | 7000000000000000 |
12320 | 2008-03-23 03:00:00 | -32767 | 211 | 2008-03-23 03:31:13 | 330 | 0 | 0 | -14.6 | 1000601100000007 | 1892900000000052 | QC1-4-211:1,hqc,QC2d-2-A,LONG GAPS
12320 | 2008-03-23 04:00:00 | -15.1 | 211 | 2008-03-23 04:07:04 | 330 | 0 | 0 | -15.1 | 1110100000000010 | 7000000000000000 |

Here is the data history for the above point:

version | stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed | modificationtime
----------+-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------+----------------------------
11929428 | 12320 | 2008-03-23 03:00:00 | -32767 | 211 | 2008-03-23 03:31:13 | 330 | 0 | 0 | -14.5 | 1000601000000007 | 3891900000000051 | QC1-4-211:1,hqc | 2011-02-27 23:44:23.996873
51889231 | 12320 | 2008-03-23 03:00:00 | | 211 | 2008-03-23 03:31:13 | 330 | 0 | 0 | | | | | 2011-07-04 20:34:59.17184
51889232 | 12320 | 2008-03-23 03:00:00 | -32767 | 211 | 2008-03-23 03:31:13 | 330 | 0 | 0 | -14.6 | 1000601100000007 | 1892900000000052 | QC1-4-211:1,hqc,QC2d-2-A,LONG GAPS | 2011-07-04 20:34:59.254227
(3 rows)