kvalobs:kvoss:system:qc2:test:algorithms:singlemissingpoint2

LDAP: couldn't connect to LDAP server

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
kvalobs:kvoss:system:qc2:test:algorithms:singlemissingpoint2 [2010-09-15 23:00:43]
paule 		kvalobs:kvoss:system:qc2:test:algorithms:singlemissingpoint2 [2022-05-31 09:29:32] (current)
Line 1: Line 1:
 ====== Single Missing Point Revised ====== ====== Single Missing Point Revised ======
  
-The tests at [[kvoss:system:qc2:test:algorithms:singlemissingpoint]] confirmed the basic performance of the interpolation. This page now looks at the {{:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.doc|revised specification}} and the flag handling that this implies.+The tests at [[kvoss:system:qc2:test:algorithms:singlemissingpoint]] confirmed the basic performance of the interpolation. This page now looks at the   {{kvalobs:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.pdf|revised specification}} and the flag handling that this implies.
  
 Example Scenarios: Example Scenarios:
Line 34: Line 34:
 The revised algorithm also has to recheck values in case the neighbours have changed. In general terms the logic is as follows: The revised algorithm also has to recheck values in case the neighbours have changed. In general terms the logic is as follows:
  
-{{:kvoss:system:qc2:test:algorithms:revisedsinglemissingpoint.jpg?600x400|}}+{{kvalobs:kvoss:system:qc2:test:algorithms:revisedsinglemissingpoint.jpg?600x400|}} 
 + 
 +**NB** The algorithm in the end has been implemented by checking the neighbours first and then working through the ftime options, ... but the result is the same.
  
 ====== Test Run ====== ====== Test Run ======
Line 72: Line 74:
 kvalobs=# UPDATE data SET corrected=10 WHERE stationid=76992 AND  obstime BETWEEN '2010-07-29' AND '2010-07-30' AND paramid=211 AND original=-32767; kvalobs=# UPDATE data SET corrected=10 WHERE stationid=76992 AND  obstime BETWEEN '2010-07-29' AND '2010-07-30' AND paramid=211 AND original=-32767;
 UPDATE 1 UPDATE 1
-kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211; stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;  
 +stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  
 -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------- -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
      76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 |       76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
Line 93: Line 96:
 </code> </code>
  
 +^Step 5 | Now update the neighbour so that it no longer shall be used for an interpolation|
 +
 +<code>
 kvalobs=# UPDATE data SET useinfo='7010000000000000' WHERE stationid=76992 and  obstime='2010-07-29 14:00:00' and paramid=211; kvalobs=# UPDATE data SET useinfo='7010000000000000' WHERE stationid=76992 and  obstime='2010-07-29 14:00:00' and paramid=211;
 UPDATE 1 UPDATE 1
Line 102: Line 108:
      76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 |       76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 (3 rows) (3 rows)
 +</code>
  
 +^Step 6 | Rerun the algorithm ... this time it should update corrected as a missing value once more ...|
 +
 +<code>
 kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211; kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
  stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |         cfailed            stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |         cfailed          
Line 110: Line 120:
      76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 |       76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 (3 rows) (3 rows)
 +</code>
 +
 +**Notes**
 +
 +  * Check behaviour of CFAILED ... is this desired?
 +
 +====== Test Run 2 ======
 +
 +The changes in this run:
 +
 +  * latest version of kvalobs lib installed and qc2 rebuilt
 +  * a change in the neighbour value is included (rather than a change in neighbour flag value)
 +
 +<code>
 +kvalobs=# DELETE from DATA where  stationid=76992 and  obstime between '2010-07-29' and '2010-07-30' and paramid=211;
 +DELETE 25
 +kvalobs=# COPY data FROM '/metno/kvalobs/TESTDATA/kvqc2_1.0.4/DB-Input.dat' USING DELIMITERS ';' ;
 +COPY 25
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed 
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------
 +     76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |    -32767 | 0000003000000000 | 9899900000000000 | 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
 +     76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |         2 | 0000001100000000 | 6894900000000000 |  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# UPDATE data SET corrected=-1 WHERE stationid=76992 AND  obstime='2010-07-29 14:00:00' AND paramid=211;
 +UPDATE 1
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
 +     76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |         2 | 0000001100000000 | 6894900000000000 |  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
 +     76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |         2 | 0000001100000000 | 6894900000000000 |  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# UPDATE data SET original=-1 WHERE stationid=76992 AND  obstime='2010-07-29 14:00:00' AND paramid=211;
 +UPDATE 1
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
 +     76992 | 2010-07-29 14:00:00 |       -1 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |         2 | 0000001100000000 | 6894900000000000 |  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |     cfailed      
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------
 +     76992 | 2010-07-29 14:00:00 |       -1 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |       0.5 | 0000001100000000 | 6894900000000000 |  QC2d-2  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# UPDATE data SET useinfo='7010000000000000' WHERE stationid=76992 and  obstime='2010-07-29 14:00:00' and paramid=211;
 +UPDATE 1
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |     cfailed      
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------
 +     76992 | 2010-07-29 14:00:00 |       -1 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7010000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |       0.5 | 0000001100000000 | 6894900000000000 |  QC2d-2  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |         cfailed          
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+--------------------------
 +     76992 | 2010-07-29 14:00:00 |       -1 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7010000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |    -32767 | 0000003100000000 | 6899900000000000 |  QC2d-2  QC2d-2  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +</code>
 +
 +====== Test Run 3 ======
 +
 +This works the same as the above ... but also includes a -32766 example. Here are the test {{kvalobs:kvoss:system:qc2:test:algorithms:qc2-211-singlepointtest.db.dat.gz|data}} and the associated sql commands:
 +
 +<code>
 +COPY data FROM '/metno/kvalobs/TESTDATA/kvqc2_1.0.4/Qc2-211-SinglePointTest.db.dat' USING DELIMITERS ';' ;
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +
 +### Run ALgorithm...
 +
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +UPDATE data SET corrected=5, original=5 WHERE stationid=87120 AND  obstime='2010-09-17 09:00:00' AND paramid=211;
 +UPDATE data SET corrected=7.2, original=7.2 WHERE stationid=87120 AND  obstime='2010-09-16 11:00:00' AND paramid=211;
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +
 +### Run ALgorithm...
 +
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +UPDATE data SET useinfo='7010000000000000' WHERE stationid=87120 and  obstime='2010-09-17 09:00:00' and paramid=211;
 +UPDATE data SET useinfo='7010000000000000' WHERE stationid=87120 and  obstime='2010-09-16 11:00:00' and paramid=211;
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +
 +### Run ALgorithm...
 +
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +
 +DELETE from DATA where  stationid=87120 and  obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' and paramid=211;
 +</code>
 +
 +
 +====== Interesting case? ======
 +
 +Time Series
 +<code>
 +kvalobs=# select * from data where paramid=213 and stationid=54710 and obstime between '2010-11-20 00:00:00' and '2010-11-20 08:00:00';
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |          cfailed           
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------------------------
 +     54710 | 2010-11-20 00:00:00 |    -23.8 |     213 | 2010-11-19 23:56:31 |    330 | 0      |     0 |     -23.8 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +     54710 | 2010-11-20 01:00:00 |    -24.2 |     213 | 2010-11-20 00:56:36 |    330 | 0      |     0 |     -24.2 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +     54710 | 2010-11-20 02:00:00 |    -24.8 |     213 | 2010-11-20 01:56:33 |    330 | 0      |     0 |     -24.8 | 0511000000000010 | 7000000000000001 | QC1-1-213
 +     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:59:18 |    330 | 0      |     0 |     -24.8 | 0511004100000090 | 5031100000000001 | QC1-1-213,QC1-9-211,QC2d-2
 +     54710 | 2010-11-20 04:00:00 |    -24.8 |     213 | 2010-11-20 03:56:34 |    330 | 0      |     0 |     -24.8 | 0510000000000010 | 7000000000000001 | QC1-1-213
 +     54710 | 2010-11-20 05:00:00 |    -22.1 |     213 | 2010-11-20 04:56:31 |    330 | 0      |     0 |     -22.1 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +     54710 | 2010-11-20 06:00:00 |    -21.3 |     213 | 2010-11-20 05:58:51 |    330 | 0      |     0 |     -21.3 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +     54710 | 2010-11-20 07:00:00 |    -21.6 |     213 | 2010-11-20 06:56:38 |    330 | 0      |     0 |     -21.6 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +     54710 | 2010-11-20 08:00:00 |    -21.5 |     213 | 2010-11-20 07:56:34 |    330 | 0      |     0 |     -21.5 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +(9 rows)
 +
 +</code>
 +
 +Modiification history:
 +
 +
 +<code>
 +kvalobs=# select * from data_history where paramid=213 and stationid=54710 and obstime='2010-11-20 03:00:00' order by modificationtime;
 +  version  | stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |          cfailed           |      modificationtime      
 +-----------+-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------------------------+----------------------------
 + 100031462 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:58:43 |    330 | 0      |     0 |     -24.6 | 0000000000000000 | 9999900000000000 |                            | 2010-11-20 02:58:43.706604
 + 100032803 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:58:43 |    330 | 0      |     0 |    -32766 | 0511002000000090 | 7038100000000001 | QC1-1-213,QC1-9-211        | 2010-11-20 02:59:08.283732
 + 100032987 |     54710 | 2010-11-20 03:00:00 |          |     213 | 2010-11-20 02:58:43 |    330 | 0      |     0 |           |                  |                  |                            | 2010-11-20 02:59:11.801073
 + 100032988 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:59:11 |    330 | 0      |     0 |     -24.6 | 0000000000000000 | 9999900000000000 |                            | 2010-11-20 02:59:11.804903
 + 100033675 |     54710 | 2010-11-20 03:00:00 |          |     213 | 2010-11-20 02:59:11 |    330 | 0      |     0 |           |                  |                  |                            | 2010-11-20 02:59:18.788327
 + 100033676 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:59:18 |    330 | 0      |     0 |     -24.6 | 0000000000000000 | 9999900000000000 |                            | 2010-11-20 02:59:18.78926
 + 100037281 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:59:18 |    330 | 0      |     0 |    -32766 | 0511002000000090 | 7038100000000001 | QC1-1-213,QC1-9-211        | 2010-11-20 03:00:40.907203
 + 100071281 |     54710 | 2010-11-20 03:00:00 |          |     213 | 2010-11-20 02:59:18 |    330 | 0      |     0 |           |                  |                  |                            | 2010-11-20 06:40:02.10392
 + 100071282 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:59:18 |    330 | 0      |     0 |     -24.8 | 0511004100000090 | 5031100000000001 | QC1-1-213,QC1-9-211,QC2d-2 | 2010-11-20 06:40:02.149274
 +(9 rows)
 </code> </code>
  • kvalobs/kvoss/system/qc2/test/algorithms/singlemissingpoint2.1284591643.txt.gz
  • Last modified: 2022-05-31 09:23:19
  • (external edit)