This is an old revision of the document!
Single Missing Point Revised
The tests at singlemissingpoint confirmed the basic performance of the interpolation. This page now looks at the revised specification and the flag handling that this implies.
Example Scenarios:
fmis=1
50540 | 2008-03-30 14:00:00 | 6.4 | 211 | 2008-03-30 13:54:58 | 330 | 0 | 0 | 6.4 | 1110100000000010 | 7000000000000000 | 50540 | 2008-03-30 15:00:00 | -32767 | 211 | 2008-03-30 15:30:04 | 3 | 0 | 0 | -32767 | 1000501000000000 | 7890700000000001 | QC1-4-211:1,QC1-4-211:1 50540 | 2008-03-30 15:00:00 | 7.3 | 211 | 2008-03-30 14:54:58 | 330 | 0 | 0 | 7.3 | 1110100000100010 | 7000000000000000 |
fmis=2 : no single point examples found
85891 | 2008-03-30 09:00:00 | 2.3 | 211 | 2008-03-30 08:55:28 | 330 | 0 | 0 | -32766 | 100060200000061A | 3038900000000FF2 | QC1-0-211:1,QC1-4-211:1, hqc
fmis=3
44630 | 2008-03-30 14:00:00 | 6 | 211 | 2008-03-30 14:02:52 | 3 | 0 | 0 | 6 | 1001100000000000 | 7000000000000000 | 44630 | 2008-03-30 14:00:00 | -32767 | 211 | 2008-03-30 14:02:53 | 3 | 0 | 10 | -32767 | 0000003000000000 | 9899900000000000 | 44630 | 2008-03-30 14:00:00 | 6.1 | 211 | 2008-03-30 14:02:52 | 3 | 1 | 10 | 6.1 | 0000000000000000 | 9999900000000000
fmis=4 no examples found for paramid=211, but here is something for 213:
33890 | 2006-12-17 23:00:00 | -48.2 | 213 | 2006-12-18 11:17:15 | 3 | 0 | 0 | -32766 | 15120040000000AA | 3138300400000032 | QC1-1-213:1,QC1-3a-213:1,QC1-9-213:1,hqc
TODO: check kvalobs db on other servers …
The revised algorithm also has to recheck values in case the neighbours have changed. In general terms the logic is as follows:
NB The algorithm in the end has been implemented by checking the neighbours first and then working through the ftime options, … but the result is the same.
Test Run
The following is a first test with the updated algorithm.
| Step 1 | This is just setting up some test data … only one point for now! |
|---|
kvalobs=# DELETE from DATA where stationid=76992 and obstime between '2010-07-29' and '2010-07-30' and paramid=211;
DELETE 25
kvalobs=# COPY data FROM '/metno/kvalobs/TESTDATA/kvqc2_1.0.4/DB-Input.dat' USING DELIMITERS ';' ;
COPY 25
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------
76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | -32767 | 0000003000000000 | 9899900000000000 |
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
| Step 2 | Run the algorithm to provide an interpolated point for the missing value |
|---|
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | 2 | 0000001100000000 | 6894900000000000 | QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
| Step 3 | Change the the corrected value |
|---|
kvalobs=# UPDATE data SET corrected=10 WHERE stationid=76992 AND obstime BETWEEN '2010-07-29' AND '2010-07-30' AND paramid=211 AND original=-32767;
UPDATE 1
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | 10 | 0000001100000000 | 6894900000000000 | QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
| Step 4 | Rerun the algorithm … it interpolates again and sees the new value is different, the new value is inserted |
|---|
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------
76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | 2 | 0000001100000000 | 6894900000000000 | QC2d-2 QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
| Step 5 | Now update the neighbour so that it no longer shall be used for an interpolation |
|---|
kvalobs=# UPDATE data SET useinfo='7010000000000000' WHERE stationid=76992 and obstime='2010-07-29 14:00:00' and paramid=211;
UPDATE 1
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------
76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7010000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | 2 | 0000001100000000 | 6894900000000000 | QC2d-2 QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
| Step 6 | Rerun the algorithm … this time it should update corrected as a missing value once more … |
|---|
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+--------------------------
76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7010000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | -32767 | 0000003100000000 | 6899900000000000 | QC2d-2 QC2d-2 QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
Notes
- Check behaviour of CFAILED … is this desired?
Test Run 2
The changes in this run:
- latest version of kvalobs lib installed and qc2 rebuilt
- a chnage in the neighbour value is included (rather than a change in neighbour flag value)
kvalobs=# DELETE from DATA where stationid=76992 and obstime between '2010-07-29' and '2010-07-30' and paramid=211;
DELETE 25
kvalobs=# COPY data FROM '/metno/kvalobs/TESTDATA/kvqc2_1.0.4/DB-Input.dat' USING DELIMITERS ';' ;
COPY 25
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------
76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | -32767 | 0000003000000000 | 9899900000000000 |
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | 2 | 0000001100000000 | 6894900000000000 | QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
kvalobs=# UPDATE data SET corrected=-1 WHERE stationid=76992 AND obstime='2010-07-29 14:00:00' AND paramid=211;
UPDATE 1
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7000000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | 2 | 0000001100000000 | 6894900000000000 | QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7000000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | 2 | 0000001100000000 | 6894900000000000 | QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
kvalobs=# UPDATE data SET original=-1 WHERE stationid=76992 AND obstime='2010-07-29 14:00:00' AND paramid=211;
UPDATE 1
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
76992 | 2010-07-29 14:00:00 | -1 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7000000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | 2 | 0000001100000000 | 6894900000000000 | QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------
76992 | 2010-07-29 14:00:00 | -1 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7000000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | 0.5 | 0000001100000000 | 6894900000000000 | QC2d-2 QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
kvalobs=# UPDATE data SET useinfo='7010000000000000' WHERE stationid=76992 and obstime='2010-07-29 14:00:00' and paramid=211;
UPDATE 1
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------
76992 | 2010-07-29 14:00:00 | -1 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7010000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | 0.5 | 0000001100000000 | 6894900000000000 | QC2d-2 QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)
kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+--------------------------
76992 | 2010-07-29 14:00:00 | -1 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7010000000000000 |
76992 | 2010-07-29 15:00:00 | -32767 | 211 | 2010-07-29 16:30:04 | 11 | 0 | 0 | -32767 | 0000003100000000 | 6899900000000000 | QC2d-2 QC2d-2 QC2d-2
76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 |
(3 rows)