Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
kvalobs:kvoss:system:qc2:test:algorithms:singlemissingpoint2 [2010-09-13 21:50:35] paule |
kvalobs:kvoss:system:qc2:test:algorithms:singlemissingpoint2 [2022-05-31 09:29:32] (current) |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== Single Missing Point Revised ====== | ====== Single Missing Point Revised ====== | ||
- | The tests at [[kvoss: | + | The tests at [[kvoss: |
Example Scenarios: | Example Scenarios: | ||
Line 34: | Line 34: | ||
The revised algorithm also has to recheck values in case the neighbours have changed. In general terms the logic is as follows: | The revised algorithm also has to recheck values in case the neighbours have changed. In general terms the logic is as follows: | ||
- | {{: | + | {{kvalobs: |
+ | |||
+ | **NB** The algorithm in the end has been implemented by checking the neighbours first and then working through the ftime options, ... but the result is the same. | ||
+ | |||
+ | ====== Test Run ====== | ||
+ | |||
+ | The following is a first test with the updated algorithm. | ||
+ | |||
+ | ^Step 1 | This is just setting up some test data ... only one point for now!| | ||
+ | < | ||
+ | kvalobs=# DELETE from DATA where stationid=76992 and obstime between ' | ||
+ | DELETE 25 | ||
+ | kvalobs=# COPY data FROM '/ | ||
+ | COPY 25 | ||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+--------- | ||
+ | 76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | </ | ||
+ | |||
+ | ^Step 2 | Run the algorithm to provide an interpolated point for the missing value| | ||
+ | |||
+ | < | ||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------- | ||
+ | 76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | </ | ||
+ | |||
+ | ^Step 3 | Change the the corrected value| | ||
+ | |||
+ | < | ||
+ | kvalobs=# UPDATE data SET corrected=10 WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | UPDATE 1 | ||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | stationid | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------- | ||
+ | 76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | </ | ||
+ | |||
+ | |||
+ | ^Step 4 | Rerun the algorithm ... it interpolates again and sees the new value is different, the new value is inserted| | ||
+ | |||
+ | < | ||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------ | ||
+ | 76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | </ | ||
+ | |||
+ | ^Step 5 | Now update the neighbour so that it no longer shall be used for an interpolation| | ||
+ | |||
+ | < | ||
+ | kvalobs=# UPDATE data SET useinfo=' | ||
+ | UPDATE 1 | ||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------ | ||
+ | 76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7010000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | </ | ||
+ | |||
+ | ^Step 6 | Rerun the algorithm ... this time it should update corrected as a missing value once more ...| | ||
+ | |||
+ | < | ||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+-------------------------- | ||
+ | 76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7010000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | </ | ||
+ | |||
+ | **Notes** | ||
+ | |||
+ | * Check behaviour of CFAILED ... is this desired? | ||
+ | |||
+ | ====== Test Run 2 ====== | ||
+ | |||
+ | The changes in this run: | ||
+ | |||
+ | * latest version of kvalobs lib installed and qc2 rebuilt | ||
+ | * a change in the neighbour value is included (rather than a change in neighbour flag value) | ||
+ | |||
+ | < | ||
+ | kvalobs=# DELETE from DATA where stationid=76992 and obstime between ' | ||
+ | DELETE 25 | ||
+ | kvalobs=# COPY data FROM '/ | ||
+ | COPY 25 | ||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+--------- | ||
+ | 76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | |||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------- | ||
+ | 76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | |||
+ | kvalobs=# UPDATE data SET corrected=-1 WHERE stationid=76992 AND obstime=' | ||
+ | UPDATE 1 | ||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------- | ||
+ | 76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7000000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | |||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------- | ||
+ | 76992 | 2010-07-29 14:00:00 | 2 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7000000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | |||
+ | kvalobs=# UPDATE data SET original=-1 WHERE stationid=76992 AND obstime=' | ||
+ | UPDATE 1 | ||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------- | ||
+ | 76992 | 2010-07-29 14:00:00 | -1 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7000000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | |||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------ | ||
+ | 76992 | 2010-07-29 14:00:00 | -1 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7000000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | |||
+ | kvalobs=# UPDATE data SET useinfo=' | ||
+ | UPDATE 1 | ||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------ | ||
+ | 76992 | 2010-07-29 14:00:00 | -1 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7010000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | |||
+ | kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+-------------------------- | ||
+ | 76992 | 2010-07-29 14:00:00 | -1 | 211 | 2010-07-29 14:06:19 | 11 | 0 | 0 | -1 | 0111000000100000 | 7010000000000000 | | ||
+ | 76992 | 2010-07-29 15:00:00 | | ||
+ | 76992 | 2010-07-29 16:00:00 | 2 | 211 | 2010-07-29 16:06:19 | 11 | 0 | 0 | 2 | 0111000000100000 | 7000000000000000 | | ||
+ | (3 rows) | ||
+ | |||
+ | </ | ||
+ | |||
+ | ====== Test Run 3 ====== | ||
+ | |||
+ | This works the same as the above ... but also includes a -32766 example. Here are the test {{kvalobs: | ||
+ | |||
+ | < | ||
+ | COPY data FROM '/ | ||
+ | SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN ' | ||
+ | |||
+ | ### Run ALgorithm... | ||
+ | |||
+ | SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN ' | ||
+ | UPDATE data SET corrected=5, | ||
+ | UPDATE data SET corrected=7.2, | ||
+ | SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN ' | ||
+ | |||
+ | ### Run ALgorithm... | ||
+ | |||
+ | SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN ' | ||
+ | UPDATE data SET useinfo=' | ||
+ | UPDATE data SET useinfo=' | ||
+ | SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN ' | ||
+ | |||
+ | ### Run ALgorithm... | ||
+ | |||
+ | SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN ' | ||
+ | |||
+ | DELETE from DATA where stationid=87120 and obstime BETWEEN ' | ||
+ | </ | ||
+ | |||
+ | |||
+ | ====== Interesting case? ====== | ||
+ | |||
+ | Time Series | ||
+ | < | ||
+ | kvalobs=# select * from data where paramid=213 and stationid=54710 and obstime between ' | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------------------------- | ||
+ | 54710 | 2010-11-20 00:00:00 | -23.8 | 213 | 2010-11-19 23:56:31 | 330 | 0 | 0 | -23.8 | 0311000000000010 | 7010100000000001 | QC1-1-213 | ||
+ | 54710 | 2010-11-20 01:00:00 | -24.2 | 213 | 2010-11-20 00:56:36 | 330 | 0 | 0 | -24.2 | 0311000000000010 | 7010100000000001 | QC1-1-213 | ||
+ | 54710 | 2010-11-20 02:00:00 | -24.8 | 213 | 2010-11-20 01:56:33 | 330 | 0 | 0 | -24.8 | 0511000000000010 | 7000000000000001 | QC1-1-213 | ||
+ | 54710 | 2010-11-20 03:00:00 | -24.6 | 213 | 2010-11-20 02:59:18 | 330 | 0 | 0 | -24.8 | 0511004100000090 | 5031100000000001 | QC1-1-213, | ||
+ | 54710 | 2010-11-20 04:00:00 | -24.8 | 213 | 2010-11-20 03:56:34 | 330 | 0 | 0 | -24.8 | 0510000000000010 | 7000000000000001 | QC1-1-213 | ||
+ | 54710 | 2010-11-20 05:00:00 | -22.1 | 213 | 2010-11-20 04:56:31 | 330 | 0 | 0 | -22.1 | 0311000000000010 | 7010100000000001 | QC1-1-213 | ||
+ | 54710 | 2010-11-20 06:00:00 | -21.3 | 213 | 2010-11-20 05:58:51 | 330 | 0 | 0 | -21.3 | 0311000000000010 | 7010100000000001 | QC1-1-213 | ||
+ | 54710 | 2010-11-20 07:00:00 | -21.6 | 213 | 2010-11-20 06:56:38 | 330 | 0 | 0 | -21.6 | 0311000000000010 | 7010100000000001 | QC1-1-213 | ||
+ | 54710 | 2010-11-20 08:00:00 | -21.5 | 213 | 2010-11-20 07:56:34 | 330 | 0 | 0 | -21.5 | 0311000000000010 | 7010100000000001 | QC1-1-213 | ||
+ | (9 rows) | ||
+ | |||
+ | </ | ||
+ | |||
+ | Modiification history: | ||
+ | |||
+ | |||
+ | < | ||
+ | kvalobs=# select * from data_history where paramid=213 and stationid=54710 and obstime=' | ||
+ | version | ||
+ | -----------+-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------------------------+---------------------------- | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | (9 rows) | ||
+ | </ |