====== Akima Tests ====== The following graphs illustrate comparisons between an Akima Interpolation and Linear Interpolation for single missing points. The title gives the time of the missing value, which is located at Hours=3 on each graph. The linear interpolation is the square on the straight line segment. The real data are displayed as blue squares. The curve is the Akima Spline and the corrected value that would be provided by the Akima Interpolation is indicated by the triangle. {{kvalobs:kvoss:system:qc2:test:algorithms:o2.dat.jpg|}} {{kvalobs:kvoss:system:qc2:test:algorithms:o3.dat.jpg|}} {{kvalobs:kvoss:system:qc2:test:algorithms:o4.dat.jpg|}} {{kvalobs:kvoss:system:qc2:test:algorithms:o5.dat.jpg|}} ====== StationID = 59800; Start Time=20080706-06 ====== Investigation with some longer time intervals. * Point selected at Hour=33, **Actual Original Value = 15.9** degrees Celsius. * Neighbours: (32,15.2) (36,17.1) => **Linear Interpolation = 16.2** (16.15) * The point is treated as a missing point for the following Akima interpolations. **Key:** ^Blue Squares: |Original data | ^Red circles: |Points used in the interpolation | ^Triangle: |The Akima Point | ^Curve: |The Akima Fit | {{kvalobs:kvoss:system:qc2:test:algorithms:ALL.jpg?300x300|}} {{kvalobs:kvoss:system:qc2:test:algorithms:8_12.jpg?300x300|}} {{kvalobs:kvoss:system:qc2:test:algorithms:3_5.jpg?300x300|}} {{kvalobs:kvoss:system:qc2:test:algorithms:2_3.jpg?300x300|}} {{kvalobs:kvoss:system:qc2:test:algorithms:3_2.jpg?300x300|}} ====== StationID = 18700; Start Time=20080706-06; Example 1 ====== * Actual original = 10.1 Degrees Celsius * {{kvalobs:kvoss:system:qc2:test:algorithms:18700-a_all.jpg|}} * {{kvalobs:kvoss:system:qc2:test:algorithms:18700-a.jpg|}} * {{kvalobs:kvoss:system:qc2:test:algorithms:18700-a_8_2.jpg|}} * Square below is the linear interpolation: * {{kvalobs:kvoss:system:qc2:test:algorithms:18700-a_3_2_l.jpg|}} ====== StationID = 18700; Start Time=20080706-06; Example 2 ====== * {{kvalobs:kvoss:system:qc2:test:algorithms:18700-b_all.jpg|}} * {{kvalobs:kvoss:system:qc2:test:algorithms:18700-b_4_4.jpg|}} * {{kvalobs:kvoss:system:qc2:test:algorithms:18700-b_2_3.jpg|}} ====== Using Akima in the DipTest Algorithm ====== stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------ 99990 | 2027-07-08 12:00:00 | -30.8 | 211 | 2007-07-08 12:03:18 | 1 | 0 | 0 | -30.8 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-08 13:00:00 | -28.1 | 211 | 2007-07-08 13:03:26 | 1 | 0 | 0 | -28.1 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-08 14:00:00 | -27.8 | 211 | 2007-07-08 14:03:27 | 1 | 0 | 0 | -27.8 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-08 15:00:00 | -32 | 211 | 2007-07-08 15:03:35 | 1 | 0 | 0 | -32 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-08 16:00:00 | -31.8 | 211 | 2007-07-08 16:03:39 | 1 | 0 | 0 | -31.8 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-08 17:00:00 | -24.1 | 211 | 2007-07-08 17:03:20 | 1 | 0 | 0 | -31 | 1119000000100010 | 5033300000000001 | QC1-3a-211:1,QC2d-1-A,Linear=-30.8 99990 | 2027-07-08 18:00:00 | -29.8 | 211 | 2007-07-08 18:03:25 | 1 | 0 | 0 | -29.8 | 1114000000100010 | 5010300000000001 | QC2d-1 99990 | 2027-07-08 19:00:00 | -26.8 | 211 | 2007-07-08 19:03:31 | 1 | 0 | 0 | -26.8 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-08 20:00:00 | -25.6 | 211 | 2007-07-08 20:03:12 | 1 | 0 | 0 | -25.6 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-08 21:00:00 | -26.8 | 211 | 2007-07-08 21:03:18 | 1 | 0 | 0 | -26.8 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-08 22:00:00 | -26.6 | 211 | 2007-07-08 22:02:43 | 1 | 0 | 0 | -26.6 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-08 23:00:00 | -25.5 | 211 | 2007-07-08 23:03:28 | 1 | 0 | 0 | -25.5 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-09 17:00:00 | -31.3 | 211 | 2007-07-09 17:02:50 | 1 | 0 | 0 | -31.3 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-09 18:00:00 | -29.3 | 211 | 2007-07-09 18:02:55 | 1 | 0 | 0 | -29.3 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-09 19:00:00 | -30.6 | 211 | 2007-07-09 19:03:01 | 1 | 0 | 0 | -30.6 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-09 20:00:00 | -28.8 | 211 | 2007-07-09 20:03:20 | 1 | 0 | 0 | -28.8 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-09 21:00:00 | -25.1 | 211 | 2007-07-09 21:02:25 | 1 | 0 | 0 | -25.1 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-09 22:00:00 | -32.7 | 211 | 2007-07-09 22:03:39 | 1 | 0 | 0 | -28 | 1119000000100010 | 5033300000000001 | QC1-3a-211:1,QC2d-1-A,Linear=-28.5 99990 | 2027-07-09 23:00:00 | -32 | 211 | 2007-07-09 23:04:18 | 1 | 0 | 0 | -32 | 1114000000100010 | 5010300000000001 | QC2d-1 99990 | 2027-07-10 00:00:00 | -32 | 211 | 2007-07-10 00:03:19 | 1 | 0 | 0 | -32 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-10 01:00:00 | -34.7 | 211 | 2007-07-10 01:03:28 | 1 | 0 | 0 | -34.7 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-10 02:00:00 | -34.7 | 211 | 2007-07-10 02:03:53 | 1 | 0 | 0 | -34.7 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-10 03:00:00 | -31.3 | 211 | 2007-07-10 03:03:55 | 1 | 0 | 0 | -31.3 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-10 04:00:00 | -30.3 | 211 | 2007-07-10 04:04:01 | 1 | 0 | 0 | -30.3 | 1111000000100010 | 7000000000000000 | 99990 | 2027-07-10 05:00:00 | -31 | 211 | 2007-07-10 05:04:32 | 1 | 0 | 0 | -31 | 1111000000100010 | 7000000000000000 | paule@pc3029:~/TESTDATA$ ====== Some results from dev-vm101 ====== stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+---------------------------+--------+--------+-------+-----------+------------------+------------------+-------------------- 90800 | 2011-03-23 12:00:00 | 29.4 | 90 | 2011-03-23 11:56:04.26427 | 330 | 0 | 0 | 29.4 | 0101000000000000 | 7000000000000000 | 90800 | 2011-03-23 13:00:00 | 16.1 | 90 | 2011-03-23 12:56:07 | 330 | 0 | 0 | 31.4 | 0119000000000000 | 5033300000000001 | QC1-3a-90,QC2d-1-L 90800 | 2011-03-23 14:00:00 | 33.4 | 90 | 2011-03-23 13:56:05 | 330 | 0 | 0 | 33.4 | 0114000000000000 | 5010300000000001 | QC1-3a-90,QC2d-1 (3 rows) Note: in this first case for StationID=90800 Linear Interpolation is used because there is a missing neighbour and therefore not enough points for Akima Interpolation: stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+-------------------- 90800 | 2011-03-23 08:00:00 | 22.5 | 90 | 2011-03-23 07:56:01.165423 | 330 | 0 | 0 | 22.5 | 0111000000000000 | 7000000000000000 | 90800 | 2011-03-23 09:00:00 | 25.5 | 90 | 2011-03-23 08:56:04.61153 | 330 | 0 | 0 | 25.5 | 0111000000000000 | 7000000000000000 | 90800 | 2011-03-23 10:00:00 | 27 | 90 | 2011-03-23 09:56:06.744562 | 330 | 0 | 0 | 27 | 0111000000000000 | 7000000000000000 | 90800 | 2011-03-23 11:00:00 | -32767 | 90 | 2011-03-23 17:19:33 | 330 | 0 | 0 | -32767 | 0000003000000000 | 9999900000000000 | 90800 | 2011-03-23 12:00:00 | 29.4 | 90 | 2011-03-23 11:56:04.26427 | 330 | 0 | 0 | 29.4 | 0101000000000000 | 7000000000000000 | 90800 | 2011-03-23 13:00:00 | 16.1 | 90 | 2011-03-23 12:56:07 | 330 | 0 | 0 | 31.4 | 0119000000000000 | 5033300000000001 | QC1-3a-90,QC2d-1-L 90800 | 2011-03-23 14:00:00 | 33.4 | 90 | 2011-03-23 13:56:05 | 330 | 0 | 0 | 33.4 | 0114000000000000 | 5010300000000001 | QC1-3a-90,QC2d-1 90800 | 2011-03-23 15:00:00 | 33.3 | 90 | 2011-03-23 14:56:06.771527 | 330 | 0 | 0 | 33.3 | 0111000000000000 | 7000000000000000 | 90800 | 2011-03-23 16:00:00 | 30.1 | 90 | 2011-03-23 15:56:02.758994 | 330 | 0 | 0 | 30.1 | 0111000000000000 | 7000000000000000 | 90800 | 2011-03-23 17:00:00 | 26.1 | 90 | 2011-03-23 16:56:06.907794 | 330 | 0 | 0 | 26.1 | 0111000000000000 | 7000000000000000 | 90800 | 2011-03-23 18:00:00 | 23.4 | 90 | 2011-03-23 17:57:09.594429 | 330 | 0 | 0 | 23.4 | 0111000000000000 | 7000000000000000 | (11 rows) Next an Akima interpolation is applied but note that the Linear value is also recorded in CFAILED for reference (NB only for testing). stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+--------------------------------------- 79600 | 2011-03-23 15:00:00 | 19.3 | 90 | 2011-03-23 14:53:09.732768 | 311 | 0 | 0 | 19.3 | 0101000000000000 | 7000000000000000 | 79600 | 2011-03-23 16:00:00 | 36.4 | 90 | 2011-03-23 15:53:15 | 311 | 0 | 0 | 15.2 | 0409000000000000 | 5033300000000002 | QC1-1-90,QC1-3a-90,QC2d-1-A,Linear=15 79600 | 2011-03-23 17:00:00 | 10.8 | 90 | 2011-03-23 16:53:10 | 311 | 0 | 0 | 10.8 | 0404000000000000 | 5020300000000002 | QC1-3a-90,QC2d-1 (3 rows) The following example is incorrect implementation. The Akima interpolkation gives a -ve value for Vindhastighet that is always positive! stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+---------------------------------------------- 25830 | 2011-03-23 16:00:00 | 0.6 | 81 | 2011-03-23 15:50:54.548297 | 330 | 0 | 0 | 0.6 | 0111500000000000 | 7000000000000001 | QC1-4-81,QC1-4-81 25830 | 2011-03-23 17:00:00 | 11.8 | 81 | 2011-03-23 16:50:51 | 330 | 0 | 0 | -1.5 | 0119100000000000 | 5033300000000001 | QC1-3a-81,QC1-3a-81,QC2d-1-A,Linear=1.1 25830 | 2011-03-23 18:00:00 | 1.6 | 81 | 2011-03-23 17:50:50 | 330 | 0 | 0 | 1.6 | 0114100000000000 | 5010300000000001 | QC1-3a-81,QC1-4-81,QC1-3a-81,QC1-4-81,QC2d-1 (3 rows) The case is even more interesting. Qc1 identifies a Dip at Hour=17:00 where the value is 11.8. If one ignores the points at Hours 16 and 18 then 11.8 doe snot look like a Dip. {{kvalobs:kvoss:system:qc2:test:algorithms:o6.dat.jpg|}} stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+----------------------------------------- 39100 | 2011-03-24 00:00:00 | 12.4 | 90 | 2011-03-23 23:52:01.558744 | 330 | 0 | 0 | 12.4 | 0111000000000000 | 7000000000000000 | 39100 | 2011-03-24 01:00:00 | 25 | 90 | 2011-03-24 00:52:03 | 330 | 0 | 0 | 11.4 | 0219000000000000 | 5033300000000002 | QC1-1-90,QC1-3a-90,QC2d-1-A,Linear=11.4 39100 | 2011-03-24 02:00:00 | 10.4 | 90 | 2011-03-24 01:52:02 | 330 | 0 | 0 | 10.4 | 0214000000000000 | 5010300000000002 | QC1-3a-90,QC2d-1 (3 rows) stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+---------------------------+--------+--------+-------+-----------+------------------+------------------+----------------------------------------- 94500 | 2011-03-24 01:00:00 | 24.5 | 90 | 2011-03-24 00:52:38.94146 | 330 | 0 | 0 | 24.5 | 0111000000000000 | 7000000000000000 | 94500 | 2011-03-24 02:00:00 | 41.9 | 90 | 2011-03-24 01:52:38 | 330 | 0 | 0 | 23.7 | 0419000000000000 | 5033300000000002 | QC1-1-90,QC1-3a-90,QC2d-1-A,Linear=23.5 94500 | 2011-03-24 03:00:00 | 22.5 | 90 | 2011-03-24 02:52:38 | 330 | 0 | 0 | 22.5 | 0414000000000000 | 5020300000000002 | QC1-3a-90,QC2d-1 (3 rows) stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------- 85040 | 2011-03-24 04:00:00 | 8.3 | 90 | 2011-03-24 03:51:40.216838 | 330 | 0 | 0 | 8.3 | 0111000000000000 | 7000000000000000 | 85040 | 2011-03-24 05:00:00 | 24.5 | 90 | 2011-03-24 04:51:37 | 330 | 0 | 0 | 8.1 | 0119000000000000 | 5033300000000001 | QC1-3a-90,QC2d-1-A,Linear=8.8 85040 | 2011-03-24 06:00:00 | 9.2 | 90 | 2011-03-24 05:51:40 | 330 | 0 | 0 | 9.2 | 0114000000000000 | 5010300000000001 | QC1-3a-90,QC2d-1 (3 rows) stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+--------------------------------------- 85450 | 2011-03-25 02:00:00 | 5.2 | 90 | 2011-03-25 01:54:18.460609 | 311 | 0 | 0 | 5.2 | 0101000000000000 | 7000000000000000 | 85450 | 2011-03-25 03:00:00 | 32.5 | 90 | 2011-03-25 02:54:06 | 311 | 0 | 0 | 10.9 | 0409000000000000 | 5033300000000002 | QC1-1-90,QC1-3a-90,QC2d-1-A,Linear=11 85450 | 2011-03-25 04:00:00 | 16.8 | 90 | 2011-03-25 03:54:15 | 311 | 0 | 0 | 16.8 | 0404000000000000 | 5020300000000002 | QC1-3a-90,QC2d-1 (3 rows) stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+---------------------------------------- 79600 | 2011-03-25 21:00:00 | 3.2 | 90 | 2011-03-25 20:53:22.679937 | 311 | 0 | 0 | 3.2 | 0101000000000000 | 7000000000000000 | 79600 | 2011-03-25 22:00:00 | 28.6 | 90 | 2011-03-25 21:53:10 | 311 | 0 | 0 | 3.9 | 0409000000000000 | 5033300000000002 | QC1-1-90,QC1-3a-90,QC2d-1-A,Linear=3.9 79600 | 2011-03-25 23:00:00 | 4.6 | 90 | 2011-03-25 22:53:21 | 311 | 0 | 0 | 4.6 | 0404000000000000 | 5020300000000002 | QC1-3a-90,QC2d-1 (3 rows) Finally ... another example where linear is adopted rather than Akima. stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed -----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+-------------------- 94500 | 2011-03-25 09:00:00 | 15.8 | 90 | 2011-03-25 08:52:39.871496 | 330 | 0 | 0 | 15.8 | 0111000000000000 | 7000000000000000 | 94500 | 2011-03-25 10:00:00 | 15.6 | 90 | 2011-03-25 09:52:37.959374 | 330 | 0 | 0 | 15.6 | 0111000000000000 | 7000000000000000 | 94500 | 2011-03-25 11:00:00 | -32767 | 90 | 2011-03-25 11:40:48 | 330 | 0 | 0 | -32767 | 0000003000000000 | 9999900000000000 | 94500 | 2011-03-25 12:00:00 | 15.6 | 90 | 2011-03-25 11:52:40.90737 | 330 | 0 | 0 | 15.6 | 0110000000000000 | 7000000000000000 | 94500 | 2011-03-25 13:00:00 | 16.6 | 90 | 2011-03-25 12:52:42.714296 | 330 | 0 | 0 | 16.6 | 0111000000000000 | 7000000000000000 | 94500 | 2011-03-25 14:00:00 | 36 | 90 | 2011-03-25 13:52:39 | 330 | 0 | 0 | 16.7 | 0119000000000000 | 5033300000000001 | QC1-3a-90,QC2d-1-L 94500 | 2011-03-25 15:00:00 | 16.8 | 90 | 2011-03-25 14:52:38 | 330 | 0 | 0 | 16.8 | 0114000000000000 | 5010300000000001 | QC1-3a-90,QC2d-1 94500 | 2011-03-25 16:00:00 | 18.5 | 90 | 2011-03-25 15:52:34.325543 | 330 | 0 | 0 | 18.5 | 0111000000000000 | 7000000000000000 | 94500 | 2011-03-25 17:00:00 | 18.5 | 90 | 2011-03-25 16:52:40.451525 | 330 | 0 | 0 | 18.5 | 0111000000000000 | 7000000000000000 | 94500 | 2011-03-25 18:00:00 | 18.4 | 90 | 2011-03-25 17:52:40.31381 | 330 | 0 | 0 | 18.4 | 0111000000000000 | 7000000000000000 | 94500 | 2011-03-25 19:00:00 | 17.4 | 90 | 2011-03-25 18:52:41.345007 | 330 | 0 | 0 | 17.4 | 0111000000000000 | 7000000000000000 |