Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
kvalobs:kvoss:system:qc2:test:algorithms:diptest [2011-04-11 12:40:27] paule |
kvalobs:kvoss:system:qc2:test:algorithms:diptest [2022-05-31 09:29:32] (current) |
||
---|---|---|---|
Line 3: | Line 3: | ||
Output of various tests. The latest is always posted at the top of the page. | Output of various tests. The latest is always posted at the top of the page. | ||
- | * ... | + | {{kvalobs: |
- | * ... | + | |
- | * ... | + | |
- | * ... | + | |
* Also see the application of [[https:// | * Also see the application of [[https:// | ||
====== Simulated Test Data after Flag 9.7 Updates ====== | ====== Simulated Test Data after Flag 9.7 Updates ====== | ||
+ | |||
+ | **Review** | ||
+ | |||
+ | The last results of testing this algorithm are provided below (subsections labeled " | ||
+ | |||
+ | * The results of the DipTest with regards the flags set at the final step and the interpolation performed are in general OK but the applicability of the DipTest to all of the parameters originally proposed should be checked. | ||
+ | * For example, review of the results sets suggests that paramid=104 should not be checked and should be removed from the paramid listing. (Difficult to correctly handle the periods where the bucket is manually emptied in the time series). | ||
+ | * For some examples the flagging apears not to be correctly set prior to the DipTest (e.g. useinfo not set according to teh controlinfo), | ||
+ | |||
+ | **Results** | ||
The following data was taken from 2008 and then modified to create and emphasise some dips. Controlflags may also have been modified to ensure that the DipTest ran - this simulation is just to get some more DipTest results to look at, but simulating data may result in impossible flag settings etc ... so please be wary. | The following data was taken from 2008 and then modified to create and emphasise some dips. Controlflags may also have been modified to ensure that the DipTest ran - this simulation is just to get some more DipTest results to look at, but simulating data may result in impossible flag settings etc ... so please be wary. | ||
+ | |||
+ | The Dip Test updates the outlier datum and also the following point. In the following ten examples the previuous values and control flags for the two updated points are included (the two lines with compressed format). | ||
< | < | ||
Line 143: | Line 153: | ||
16610 | 2018-09-08 12:00:00 | 903.3 | 176 | 2018-09-08 11:54:44 | 3 | 0 | 0 | 903.3 | 1101000000000000 | 7000000000000000 | | 16610 | 2018-09-08 12:00:00 | 903.3 | 176 | 2018-09-08 11:54:44 | 3 | 0 | 0 | 903.3 | 1101000000000000 | 7000000000000000 | | ||
(7 rows) | (7 rows) | ||
+ | </ | ||
+ | Same data as above but some of the original useinfo changed to check the library update for the setUseFlags: | ||
+ | < | ||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+--------------------------------- | ||
+ | 18230 | 2018-09-09 00:00:00 | 2.5 | 87 | 2018-09-08 23:57:51 | 330 | 0 | 0 | 2.5 | 1101000000000000 | 7000000000000000 | | ||
+ | 18230 | 2018-09-09 01:00:00 | 3.3 | 87 | 2018-09-09 00:56:45 | 330 | 0 | 0 | 3.3 | 1101000000000000 | 7000000000000000 | | ||
+ | 18230 | 2018-09-09 02:00:00 | 4.3 | 87 | 2018-09-09 01:56:30 | 330 | 0 | 0 | 4.3 | 1101000000000000 | 7000000000000000 | | ||
+ | 18230 | 2018-09-09 03:00:00 | 18 | 87 | 2018-09-09 02:56:33 | 330 | 0 | 0 | 4.3 | 1109000000000000 | 5033300000000001 | QC1-3a-87: | ||
+ | 18230 | 2018-09-09 04:00:00 | 4 | 87 | 2018-09-09 03:57:17 | 330 | 0 | 0 | 4 | 1104000000000000 | 5000000000000001 | QC2d-1 | ||
+ | 18230 | 2018-09-09 05:00:00 | 4.1 | 87 | 2018-09-09 04:56:28 | 330 | 0 | 0 | 4.1 | 1102000000000000 | 7010300000000001 | QC1-3a-87:1 | ||
+ | 18230 | 2018-09-09 06:00:00 | 4 | 87 | 2018-09-09 05:56:22 | 330 | 0 | 0 | 4 | 1101000000000000 | 7000000000000000 | | ||
+ | (7 rows) | ||
+ | |||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+--------------------------------- | ||
+ | 17280 | 2018-09-04 08:00:00 | 5.9 | 87 | 2018-09-04 08:01:50 | 330 | 0 | 0 | 5.9 | 1101000000000000 | 7000000000000000 | | ||
+ | 17280 | 2018-09-04 09:00:00 | 3.3 | 87 | 2018-09-04 09:01:46 | 330 | 0 | 0 | 3.3 | 1101000000000000 | 7000000000000000 | | ||
+ | 17280 | 2018-09-04 10:00:00 | 3 | 87 | 2018-09-04 10:01:49 | 330 | 0 | 0 | 3 | 1101000000000000 | 7000000000000000 | | ||
+ | 17280 | 2018-09-04 11:00:00 | 24.3 | 87 | 2018-09-04 11:01:51 | 330 | 0 | 0 | 3.3 | 1109000000000000 | 5033300000000001 | QC1-3a-87: | ||
+ | 17280 | 2018-09-04 12:00:00 | 4.4 | 87 | 2018-09-04 12:01:53 | 330 | 0 | 0 | 4.4 | 1104000000000000 | 5000000000000001 | QC2d-1 | ||
+ | 17280 | 2018-09-04 13:00:00 | 13.8 | 87 | 2018-09-04 13:01:51 | 330 | 0 | 0 | 13.8 | 1101000000000000 | 7000000000000000 | | ||
+ | 17280 | 2018-09-04 14:00:00 | 16.1 | 87 | 2018-09-04 14:01:50 | 330 | 0 | 0 | 16.1 | 1101000000000000 | 7000000000000000 | | ||
+ | (7 rows) | ||
+ | |||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------------------------------- | ||
+ | 88690 | 2018-09-23 05:00:00 | 21.82 | 90 | 2018-09-23 04:56:49 | 3 | 0 | 0 | 21.82 | 1101000000000000 | 7000000000000000 | | ||
+ | 88690 | 2018-09-23 06:00:00 | 16.95 | 90 | 2018-09-23 05:56:51 | 3 | 0 | 0 | 16.95 | 1101000000000000 | 7000000000000000 | | ||
+ | 88690 | 2018-09-23 07:00:00 | 16.64 | 90 | 2018-09-23 06:56:57 | 3 | 0 | 0 | 16.64 | 1101000000000000 | 7000000000000000 | | ||
+ | 88690 | 2018-09-23 08:00:00 | 81.15 | 90 | 2018-09-23 07:56:58 | 3 | 0 | 0 | 15.7 | 1109000000000000 | 5033300000000001 | QC1-3a-90: | ||
+ | 88690 | 2018-09-23 09:00:00 | 14.91 | 90 | 2018-09-23 08:56:50 | 3 | 0 | 0 | 14.91 | 1104000000000000 | 5000000000000001 | QC2d-1 | ||
+ | 88690 | 2018-09-23 10:00:00 | 22.75 | 90 | 2018-09-23 09:56:57 | 3 | 0 | 0 | 22.75 | 1101000000000000 | 7000000000000000 | | ||
+ | 88690 | 2018-09-23 11:00:00 | 17.88 | 90 | 2018-09-23 10:56:50 | 3 | 0 | 0 | 17.88 | 1101000000000000 | 7000000000000000 | | ||
+ | (7 rows) | ||
+ | |||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------------------------------- | ||
+ | 88690 | 2018-09-22 02:00:00 | 18.35 | 90 | 2018-09-22 01:56:50 | 3 | 0 | 0 | 18.35 | 1101000000000000 | 7000000000000000 | | ||
+ | 88690 | 2018-09-22 03:00:00 | 22.65 | 90 | 2018-09-22 02:56:49 | 3 | 0 | 0 | 22.65 | 1101000000000000 | 7000000000000000 | | ||
+ | 88690 | 2018-09-22 04:00:00 | 27.93 | 90 | 2018-09-22 04:56:59 | 3 | 0 | 0 | 27.93 | 1101000000000000 | 7100000400000000 | | ||
+ | 88690 | 2018-09-22 05:00:00 | 2.84 | 90 | 2018-09-22 04:56:59 | 3 | 0 | 0 | 30.3 | 1109000000000000 | 5033300000000001 | QC1-3a-90: | ||
+ | 88690 | 2018-09-22 06:00:00 | 28.27 | 90 | 2018-09-22 06:56:50 | 3 | 0 | 0 | 28.27 | 1104000000000000 | 5100000400000001 | QC2d-1 | ||
+ | 88690 | 2018-09-22 07:00:00 | 10.24 | 90 | 2018-09-22 06:56:50 | 3 | 0 | 0 | 10.24 | 1101000000000000 | 7000000000000000 | | ||
+ | 88690 | 2018-09-22 08:00:00 | 7.17 | 90 | 2018-09-22 07:56:50 | 3 | 0 | 0 | 7.17 | 1101000000000000 | 7000000000000000 | | ||
+ | (7 rows) | ||
+ | |||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------ | ||
+ | 12320 | 2018-09-25 17:00:00 | 381.7 | 104 | 2018-09-25 16:53:47 | 330 | 0 | 0 | 381.7 | 1101000000100100 | 7000000000000000 | | ||
+ | 12320 | 2018-09-25 18:00:00 | 381.7 | 104 | 2018-09-25 17:54:15 | 330 | 0 | 0 | 381.7 | 1101000000100100 | 7000000000000000 | | ||
+ | 12320 | 2018-09-25 19:00:00 | 381.6 | 104 | 2018-09-25 18:53:50 | 330 | 0 | 0 | 381.6 | 1101000000100100 | 7000000000000000 | | ||
+ | 12320 | 2018-09-25 20:00:00 | 284.6 | 104 | 2018-09-25 19:54:35 | 330 | 0 | 0 | 381.6 | 1109000000100100 | 5033300000000001 | QC1-3a-104: | ||
+ | 12320 | 2018-09-25 21:00:00 | 381.6 | 104 | 2018-09-25 20:54:44 | 330 | 0 | 0 | 381.6 | 1104000000100100 | 5000000000000001 | QC1-3a-104: | ||
+ | 12320 | 2018-09-25 22:00:00 | 381.6 | 104 | 2018-09-25 21:53:48 | 330 | 0 | 0 | 381.6 | 1101000000100100 | 7000000000000000 | | ||
+ | 12320 | 2018-09-25 23:00:00 | 381.6 | 104 | 2018-09-25 22:53:46 | 330 | 0 | 0 | 381.6 | 1101000000100100 | 7000000000000000 | | ||
+ | (7 rows) | ||
+ | |||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------------------- | ||
+ | 18500 | 2018-09-02 05:00:00 | 346.2 | 104 | 2018-09-02 04:50:10 | 342 | 0 | 0 | 346.2 | 1101000000100100 | 7000000000000000 | | ||
+ | 18500 | 2018-09-02 06:00:00 | 346.1 | 104 | 2018-09-02 05:50:10 | 342 | 0 | 0 | 346.1 | 1101000000100100 | 7000000000000000 | | ||
+ | 18500 | 2018-09-02 07:00:00 | 346.3 | 104 | 2018-09-02 06:50:04 | 342 | 0 | 0 | 346.3 | 1101000000100100 | 7000000000000000 | | ||
+ | 18500 | 2018-09-02 08:00:00 | 13.4 | 104 | 2018-09-02 07:50:05 | 342 | 0 | 0 | 337.8 | 1109000000100100 | 5033300000000001 | QC1-3a-104: | ||
+ | 18500 | 2018-09-02 09:00:00 | 313.4 | 104 | 2018-09-02 09:19:46 | 342 | 0 | 0 | 313.4 | 1104000000100100 | 5000000000000001 | QC2d-1 | ||
+ | 18500 | 2018-09-02 10:00:00 | 113.4 | 104 | 2018-09-02 09:50:09 | 342 | 0 | 0 | 113.4 | 1101000000100100 | 7000000000000000 | | ||
+ | 18500 | 2018-09-02 11:00:00 | 113.4 | 104 | 2018-09-02 10:50:06 | 342 | 0 | 0 | 113.4 | 1101000000100100 | 7000000000000000 | | ||
+ | (7 rows) | ||
+ | |||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+--------------------------------- | ||
+ | 46510 | 2018-09-11 23:00:00 | 54 | 123 | 2018-09-11 22:55:22 | 3 | 0 | 0 | 54 | 1101000000000000 | 7000000000000000 | | ||
+ | 46510 | 2018-09-12 00:00:00 | 58 | 123 | 2018-09-11 23:55:22 | 3 | 0 | 0 | 58 | 1101000000000000 | 7000000000000000 | | ||
+ | 46510 | 2018-09-12 01:00:00 | 60 | 123 | 2018-09-12 00:55:21 | 3 | 0 | 0 | 60 | 1101000000000000 | 7000000000000000 | | ||
+ | 46510 | 2018-09-12 02:00:00 | 0 | 123 | 2018-09-12 01:55:22 | 3 | 0 | 0 | 60 | 1109000000000000 | 5033300000000001 | QC1-3a-123: | ||
+ | 46510 | 2018-09-12 03:00:00 | 60 | 123 | 2018-09-12 02:55:23 | 3 | 0 | 0 | 60 | 1104000000000000 | 5000000000000001 | QC2d-1 | ||
+ | 46510 | 2018-09-12 04:00:00 | 60 | 123 | 2018-09-12 03:59:22 | 3 | 0 | 0 | 60 | 1101000000000000 | 7000000000000000 | | ||
+ | 46510 | 2018-09-12 05:00:00 | 60 | 123 | 2018-09-12 04:55:21 | 3 | 0 | 0 | 60 | 1101000000000000 | 7000000000000000 | | ||
+ | (7 rows) | ||
+ | |||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------- | ||
+ | 17000 | 2018-09-25 06:00:00 | | ||
+ | 17000 | 2018-09-25 07:00:00 | | ||
+ | 17000 | 2018-09-25 08:00:00 | | ||
+ | 17000 | 2018-09-25 09:00:00 | 690.6 | 172 | 2018-09-25 09:02:24 | 3 | 0 | 0 | 1036.3 | 1109000000000000 | 5033300000000001 | QC1-3a-172: | ||
+ | 17000 | 2018-09-25 10:00:00 | | ||
+ | 17000 | 2018-09-25 11:00:00 | | ||
+ | 17000 | 2018-09-25 12:00:00 | 1036 | 172 | 2018-09-25 12:02:25 | 3 | 0 | 0 | 1036 | 1201000000000000 | 7010100000000001 | QC1-1-172:1 | ||
+ | (7 rows) | ||
+ | |||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------- | ||
+ | 17000 | 2018-09-11 13:00:00 | 1022 | 172 | 2018-09-11 13:01:42 | 3 | 0 | 0 | 1022 | 1101000000000000 | 7000000000000000 | | ||
+ | 17000 | 2018-09-11 14:00:00 | 1022 | 172 | 2018-09-11 14:01:42 | 3 | 0 | 0 | 1022 | 1101000000000000 | 7000000000000000 | | ||
+ | 17000 | 2018-09-11 15:00:00 | | ||
+ | 17000 | 2018-09-11 16:00:00 | | ||
+ | 17000 | 2018-09-11 17:00:00 | | ||
+ | 17000 | 2018-09-11 18:00:00 | 1023 | 172 | 2018-09-11 18:02:24 | 3 | 0 | 0 | 1023 | 1101000000000000 | 7000000000000000 | | ||
+ | 17000 | 2018-09-11 19:00:00 | | ||
+ | (7 rows) | ||
+ | |||
+ | | ||
+ | -----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------------------------------------------- | ||
+ | 16610 | 2018-09-08 06:00:00 | 902.9 | 176 | 2018-09-08 05:54:44 | 3 | 0 | 0 | 902.9 | 1101000000000000 | 7000000000000000 | | ||
+ | 16610 | 2018-09-08 07:00:00 | 903 | 176 | 2018-09-08 06:54:45 | 3 | 0 | 0 | 903 | 1101000000000000 | 7000000000000000 | | ||
+ | 16610 | 2018-09-08 08:00:00 | 903.1 | 176 | 2018-09-08 07:54:43 | 3 | 0 | 0 | 903.1 | 1101000000000000 | 7000000000000000 | | ||
+ | 16610 | 2018-09-08 09:00:00 | | ||
+ | 16610 | 2018-09-08 10:00:00 | 903.2 | 176 | 2018-09-08 09:54:43 | 3 | 0 | 0 | 903.2 | 1404000000000000 | 5020300000000002 | QC1-3a-176: | ||
+ | 16610 | 2018-09-08 11:00:00 | 903.2 | 176 | 2018-09-08 10:54:44 | 3 | 0 | 0 | 903.2 | 1101000000000000 | 7000000000000000 | | ||
+ | 16610 | 2018-09-08 12:00:00 | 903.3 | 176 | 2018-09-08 11:54:44 | 3 | 0 | 0 | 903.3 | 1101000000000000 | 7000000000000000 | | ||
+ | (7 rows) | ||
</ | </ | ||
+ | |||
+ | ====== Flag Document Version 9.7 verification ====== | ||
+ | |||
+ | The main change in the flag document is: | ||
+ | |||
+ | ^Previous ^ Version 9.7 ^ | ||
+ | |useinfo(2) settes til 2 hvis fagg=3 eller %%((%%fr=4 eller fr=5) og fcombi !=1,2) eller (fcc=3, fcc=4, fcc=6, fcc=7, fcp=3, fcp=4, fcp=6, fcp=7, fs=3, fw=4, fw=5, fpos=3, fstat=2 eller fd=3) | useinfo(2) settes til 2 hvis fagg=3 eller %%((%%fr=4 eller fr=5) og fcombi !=1,2) eller (fcc=3, fcc=4, fcc=6, fcc=7, fcp=3, fcp=4, fcp=6, fcp=7, fs=3, fw=4, fw=5, fpos=3, fstat=2 eller fd=3) | | ||
+ | | useinfo(2) settes til 1 hvis fagg=2, fr=2, fr=3, fcc=2, fcp=2, fs=2**, fs=4, fs=5**, fs=7, fw=2, fw=3, fclim=2 eller fcombi=2 | useinfo(2) settes til 1 hvis fagg=2, fr=2, fr=3, fcc=2, fcp=2, fs=2, fs=7, fw=2, fw=3, fclim=2 eller fcombi=2 | | ||
+ | |useinfo(2) settes til 0 hvis fagg=1, fr=1, fcc=1, fcp=1, fs=1, fw=1, fpos=1, fstat=1, fclim=1, fd=1 eller fcombi=1 | useinfo(2) settes til 0 hvis fagg=1, fr=1, fcc=1, fcp=1, fs=1**, fs=4**, fw=1, fpos=1, fstat=1, fclim=1, fd=1 eller fcombi=1. | | ||
+ | |||
+ | exact diff here: https:// | ||
+ | |||
+ | https:// | ||
+ | |||
+ | The example below illustrates useinfo(2) being reset to 0 when fs is set to 4. | ||
+ | |||
+ | < | ||
+ | kvalobs=# select * from data_history where paramid=87 and stationid=18230 and obstime=' | ||
+ | |||
+ | |||
+ | | ||
+ | ----------+-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------+---------------------------- | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | (7 rows) | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
====== MARCH 2011 (200803-200808) ====== | ====== MARCH 2011 (200803-200808) ====== |