kvalobs:kvoss:system:qc2:test:algorithms:singlemissingpoint2

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
kvalobs:kvoss:system:qc2:test:algorithms:singlemissingpoint2 [2010-09-15 23:05:22]
paule 		kvalobs:kvoss:system:qc2:test:algorithms:singlemissingpoint2 [2022-05-31 09:29:32] (current)
Line 1: Line 1:
 ====== Single Missing Point Revised ====== ====== Single Missing Point Revised ======
  
-The tests at [[kvoss:system:qc2:test:algorithms:singlemissingpoint]] confirmed the basic performance of the interpolation. This page now looks at the {{:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.doc|revised specification}} and the flag handling that this implies.+The tests at [[kvoss:system:qc2:test:algorithms:singlemissingpoint]] confirmed the basic performance of the interpolation. This page now looks at the   {{kvalobs:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.pdf|revised specification}} and the flag handling that this implies.
  
 Example Scenarios: Example Scenarios:
Line 34: Line 34:
 The revised algorithm also has to recheck values in case the neighbours have changed. In general terms the logic is as follows: The revised algorithm also has to recheck values in case the neighbours have changed. In general terms the logic is as follows:
  
-{{:kvoss:system:qc2:test:algorithms:revisedsinglemissingpoint.jpg?600x400|}}+{{kvalobs:kvoss:system:qc2:test:algorithms:revisedsinglemissingpoint.jpg?600x400|}} 
 + 
 +**NB** The algorithm in the end has been implemented by checking the neighbours first and then working through the ftime options, ... but the result is the same.
  
 ====== Test Run ====== ====== Test Run ======
Line 72: Line 74:
 kvalobs=# UPDATE data SET corrected=10 WHERE stationid=76992 AND  obstime BETWEEN '2010-07-29' AND '2010-07-30' AND paramid=211 AND original=-32767; kvalobs=# UPDATE data SET corrected=10 WHERE stationid=76992 AND  obstime BETWEEN '2010-07-29' AND '2010-07-30' AND paramid=211 AND original=-32767;
 UPDATE 1 UPDATE 1
-kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211; stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;  
 +stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  
 -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------- -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
      76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 |       76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
Line 122: Line 125:
  
   * Check behaviour of CFAILED ... is this desired?   * Check behaviour of CFAILED ... is this desired?
 +
 +====== Test Run 2 ======
 +
 +The changes in this run:
 +
 +  * latest version of kvalobs lib installed and qc2 rebuilt
 +  * a change in the neighbour value is included (rather than a change in neighbour flag value)
 +
 +<code>
 +kvalobs=# DELETE from DATA where  stationid=76992 and  obstime between '2010-07-29' and '2010-07-30' and paramid=211;
 +DELETE 25
 +kvalobs=# COPY data FROM '/metno/kvalobs/TESTDATA/kvqc2_1.0.4/DB-Input.dat' USING DELIMITERS ';' ;
 +COPY 25
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed 
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------
 +     76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |    -32767 | 0000003000000000 | 9899900000000000 | 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
 +     76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |         2 | 0000001100000000 | 6894900000000000 |  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# UPDATE data SET corrected=-1 WHERE stationid=76992 AND  obstime='2010-07-29 14:00:00' AND paramid=211;
 +UPDATE 1
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
 +     76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |         2 | 0000001100000000 | 6894900000000000 |  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
 +     76992 | 2010-07-29 14:00:00 |        2 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |         2 | 0000001100000000 | 6894900000000000 |  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# UPDATE data SET original=-1 WHERE stationid=76992 AND  obstime='2010-07-29 14:00:00' AND paramid=211;
 +UPDATE 1
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      | cfailed  
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
 +     76992 | 2010-07-29 14:00:00 |       -1 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |         2 | 0000001100000000 | 6894900000000000 |  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |     cfailed      
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------
 +     76992 | 2010-07-29 14:00:00 |       -1 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7000000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |       0.5 | 0000001100000000 | 6894900000000000 |  QC2d-2  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# UPDATE data SET useinfo='7010000000000000' WHERE stationid=76992 and  obstime='2010-07-29 14:00:00' and paramid=211;
 +UPDATE 1
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |     cfailed      
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------
 +     76992 | 2010-07-29 14:00:00 |       -1 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7010000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |       0.5 | 0000001100000000 | 6894900000000000 |  QC2d-2  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +kvalobs=# SELECT * FROM data WHERE stationid=76992 AND obstime BETWEEN '2010-07-29 14:00:00' AND '2010-07-29 16:00:00' AND paramid=211;
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |         cfailed          
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+--------------------------
 +     76992 | 2010-07-29 14:00:00 |       -1 |     211 | 2010-07-29 14:06:19 |     11 | 0      |     0 |        -1 | 0111000000100000 | 7010000000000000 | 
 +     76992 | 2010-07-29 15:00:00 |   -32767 |     211 | 2010-07-29 16:30:04 |     11 | 0      |     0 |    -32767 | 0000003100000000 | 6899900000000000 |  QC2d-2  QC2d-2  QC2d-2 
 +     76992 | 2010-07-29 16:00:00 |        2 |     211 | 2010-07-29 16:06:19 |     11 | 0      |     0 |         2 | 0111000000100000 | 7000000000000000 | 
 +(3 rows)
 +
 +</code>
 +
 +====== Test Run 3 ======
 +
 +This works the same as the above ... but also includes a -32766 example. Here are the test {{kvalobs:kvoss:system:qc2:test:algorithms:qc2-211-singlepointtest.db.dat.gz|data}} and the associated sql commands:
 +
 +<code>
 +COPY data FROM '/metno/kvalobs/TESTDATA/kvqc2_1.0.4/Qc2-211-SinglePointTest.db.dat' USING DELIMITERS ';' ;
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +
 +### Run ALgorithm...
 +
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +UPDATE data SET corrected=5, original=5 WHERE stationid=87120 AND  obstime='2010-09-17 09:00:00' AND paramid=211;
 +UPDATE data SET corrected=7.2, original=7.2 WHERE stationid=87120 AND  obstime='2010-09-16 11:00:00' AND paramid=211;
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +
 +### Run ALgorithm...
 +
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +UPDATE data SET useinfo='7010000000000000' WHERE stationid=87120 and  obstime='2010-09-17 09:00:00' and paramid=211;
 +UPDATE data SET useinfo='7010000000000000' WHERE stationid=87120 and  obstime='2010-09-16 11:00:00' and paramid=211;
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +
 +### Run ALgorithm...
 +
 +SELECT * FROM data WHERE stationid=87120 AND obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' AND paramid=211;
 +
 +DELETE from DATA where  stationid=87120 and  obstime BETWEEN '2010-09-16 06:00:00' AND '2010-09-17 16:00:00' and paramid=211;
 +</code>
 +
 +
 +====== Interesting case? ======
 +
 +Time Series
 +<code>
 +kvalobs=# select * from data where paramid=213 and stationid=54710 and obstime between '2010-11-20 00:00:00' and '2010-11-20 08:00:00';
 + stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |          cfailed           
 +-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------------------------
 +     54710 | 2010-11-20 00:00:00 |    -23.8 |     213 | 2010-11-19 23:56:31 |    330 | 0      |     0 |     -23.8 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +     54710 | 2010-11-20 01:00:00 |    -24.2 |     213 | 2010-11-20 00:56:36 |    330 | 0      |     0 |     -24.2 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +     54710 | 2010-11-20 02:00:00 |    -24.8 |     213 | 2010-11-20 01:56:33 |    330 | 0      |     0 |     -24.8 | 0511000000000010 | 7000000000000001 | QC1-1-213
 +     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:59:18 |    330 | 0      |     0 |     -24.8 | 0511004100000090 | 5031100000000001 | QC1-1-213,QC1-9-211,QC2d-2
 +     54710 | 2010-11-20 04:00:00 |    -24.8 |     213 | 2010-11-20 03:56:34 |    330 | 0      |     0 |     -24.8 | 0510000000000010 | 7000000000000001 | QC1-1-213
 +     54710 | 2010-11-20 05:00:00 |    -22.1 |     213 | 2010-11-20 04:56:31 |    330 | 0      |     0 |     -22.1 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +     54710 | 2010-11-20 06:00:00 |    -21.3 |     213 | 2010-11-20 05:58:51 |    330 | 0      |     0 |     -21.3 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +     54710 | 2010-11-20 07:00:00 |    -21.6 |     213 | 2010-11-20 06:56:38 |    330 | 0      |     0 |     -21.6 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +     54710 | 2010-11-20 08:00:00 |    -21.5 |     213 | 2010-11-20 07:56:34 |    330 | 0      |     0 |     -21.5 | 0311000000000010 | 7010100000000001 | QC1-1-213
 +(9 rows)
 +
 +</code>
 +
 +Modiification history:
 +
 +
 +<code>
 +kvalobs=# select * from data_history where paramid=213 and stationid=54710 and obstime='2010-11-20 03:00:00' order by modificationtime;
 +  version  | stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |          cfailed           |      modificationtime      
 +-----------+-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------------------------+----------------------------
 + 100031462 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:58:43 |    330 | 0      |     0 |     -24.6 | 0000000000000000 | 9999900000000000 |                            | 2010-11-20 02:58:43.706604
 + 100032803 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:58:43 |    330 | 0      |     0 |    -32766 | 0511002000000090 | 7038100000000001 | QC1-1-213,QC1-9-211        | 2010-11-20 02:59:08.283732
 + 100032987 |     54710 | 2010-11-20 03:00:00 |          |     213 | 2010-11-20 02:58:43 |    330 | 0      |     0 |           |                  |                  |                            | 2010-11-20 02:59:11.801073
 + 100032988 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:59:11 |    330 | 0      |     0 |     -24.6 | 0000000000000000 | 9999900000000000 |                            | 2010-11-20 02:59:11.804903
 + 100033675 |     54710 | 2010-11-20 03:00:00 |          |     213 | 2010-11-20 02:59:11 |    330 | 0      |     0 |           |                  |                  |                            | 2010-11-20 02:59:18.788327
 + 100033676 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:59:18 |    330 | 0      |     0 |     -24.6 | 0000000000000000 | 9999900000000000 |                            | 2010-11-20 02:59:18.78926
 + 100037281 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:59:18 |    330 | 0      |     0 |    -32766 | 0511002000000090 | 7038100000000001 | QC1-1-213,QC1-9-211        | 2010-11-20 03:00:40.907203
 + 100071281 |     54710 | 2010-11-20 03:00:00 |          |     213 | 2010-11-20 02:59:18 |    330 | 0      |     0 |           |                  |                  |                            | 2010-11-20 06:40:02.10392
 + 100071282 |     54710 | 2010-11-20 03:00:00 |    -24.6 |     213 | 2010-11-20 02:59:18 |    330 | 0      |     0 |     -24.8 | 0511004100000090 | 5031100000000001 | QC1-1-213,QC1-9-211,QC2d-2 | 2010-11-20 06:40:02.149274
 +(9 rows)
 +</code>
  • kvalobs/kvoss/system/qc2/test/algorithms/singlemissingpoint2.1284591922.txt.gz
  • Last modified: 2022-05-31 09:23:19
  • (external edit)