kvalobs:kvoss:system:qc2:user:version:kvqc2d-1.1.2

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
kvalobs:kvoss:system:qc2:user:version:kvqc2d-1.1.2 [2011-05-08 19:52:45]
paule 		kvalobs:kvoss:system:qc2:user:version:kvqc2d-1.1.2 [2022-05-31 09:29:32] (current)
Line 1: Line 1:
- 
-.... in preparation ... FIXME 
- 
 ====== kvqc2d Version: 1.1.2 ====== ====== kvqc2d Version: 1.1.2 ======
  
Line 16: Line 13:
  
 ^ Algorithm(+) ^Description  ^ ^ Algorithm(+) ^Description  ^
-| SingleLinear | As in previous release kvqc2d_1.1.0: {{:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.pdf| specification}}|  +| SingleLinear | As in previous release kvqc2d_1.1.0: {{kvalobs:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.pdf| specification}}|  
 | Redistribute | As in previous release kvqc2d_1.1.0: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:rarr_24 |  | Redistribute | As in previous release kvqc2d_1.1.0: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:rarr_24 | 
 | DipTest | New algorithm: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:diptest03 | | DipTest | New algorithm: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:diptest03 |
Line 22: Line 19:
 (+) This is the name to specify in the configuration file. (+) This is the name to specify in the configuration file.
  
-**Example Configuration Files**+**Install Configuration Files and Cron Jobs**
  
 For this release all the configuration files need to be re-installed since the use of AlgoCode="a number" is now replaced with Algorithm="the algorithm name". The changes are illustrated in the example configuration files below. For this release all the configuration files need to be re-installed since the use of AlgoCode="a number" is now replaced with Algorithm="the algorithm name". The changes are illustrated in the example configuration files below.
Line 30: Line 27:
 This includes a tar file of all of the required contents for easier deployment This includes a tar file of all of the required contents for easier deployment
  
-To install new configuration files and setup the associated cronjob:+To install new configuration files and setup the associated cronjobs: 
 + 
 +**New cfg files**
 <code> <code>
-# after logging into the operational machine as the kvalobs user+# log into the operational machine as the kvalobs user 
 + 
 +# stop the existing kvqc2d 
 +$ kvstop -l 
 + 
 +#setup new configuration
  
 $ cd /etc/kvalobs/Qc2Config $ cd /etc/kvalobs/Qc2Config
-$ rm ./*.cfg +# remove the old files (-i is suggested below so that the user is prompted to confirm removal and verify that only the desired files are removed). 
-$ rm ./*.hold+$ rm -i ./*.cfg 
 +$ rm -i ./*.hold  
 +# fetch the new files, e.g.:
 $ wget https://svn.met.no/kvoss/kvqc2d/branches/kvqc2d-1.1.2/src/Reference/cfg.tar.gz $ wget https://svn.met.no/kvoss/kvqc2d/branches/kvqc2d-1.1.2/src/Reference/cfg.tar.gz
 $ tar xvzf cfg.tar.gz $ tar xvzf cfg.tar.gz
Line 43: Line 49:
 # The configuration files are now in place. # The configuration files are now in place.
  
----------------------------------------------+# restart kvqc2d when ready:
  
 +$ kvstart
 +
 +</code>
 +
 +**cron jobs**
 +<code>
 # In order to set up the crontab for the weekly redistribution run # In order to set up the crontab for the weekly redistribution run
  
 +$ cd /etc/kvalobs/Qc2Config
 +# The scripts are present in the cfg.tar.gz archive and should be present if the new cfg file installation steps above have been followed.
 $ chmod +x Redistribution_14days_off_cron.sh $ chmod +x Redistribution_14days_off_cron.sh
 $ chmod +x Redistribution_14days_on_cron.sh $ chmod +x Redistribution_14days_on_cron.sh
Line 63: Line 77:
    
 Notes: the new scripts in cron renames "Redistribution_14days.hold" to "Redistribution_14days.cfg" every Wednesday and the sets it back to "Redistribution_14days.hold" 24 hours later. Notes: the new scripts in cron renames "Redistribution_14days.hold" to "Redistribution_14days.cfg" every Wednesday and the sets it back to "Redistribution_14days.hold" 24 hours later.
 +
 +The Redistribution algorithm will then run automatically to the following schedule:
 +   * typically daily run at **06:35** looking at the last 4 days 
 +   * **01:20** Wednesday Night/Thursday Morning run to check the last 14 days
 +
 +Previously the management of the Redistribution configuration files (i.e. hold -> cfg -> hold) has been handled manually.
  
 **Example Configuration Files** **Example Configuration Files**
  
 ^SingleLinear_v32^ ^SingleLinear_v32^
- 
-*NB* Different/simpler checks applied to the neighbours (CF version 1.0.5) 
  
 <code> <code>
Line 188: Line 206:
 93   15.0 93   15.0
 94   15.0 94   15.0
-104   100 
 105   3.0 105   3.0
 113   15.0 113   15.0
Line 209: Line 226:
 </code> </code>
  
 +**Note:** Previously "104   100" was also an entry in the above file.
 +
 +====== Notes on first production run ======
 +
 +   * Released into operations on 2011-05-18 circa 09:38 UTC
 +
 +Algorithms ran as expected:
 +
 +^DipTest^
 +
 +First result (note __only one__ Dip was identified):
 +
 +<code>
 +kvalobs=# select * from data where stationid=79600 and obstime between '2011-05-18 00:00:00' and '2011-05-19 00:00:00' and paramid=90;
 + stationid |       obstime       | original | paramid |           tbtime           | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |           cfailed          
 +-----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+-----------------------------
 +     79600 | 2011-05-18 15:00:00 |      2.9 |      90 | 2011-05-18 14:53:12.296014 |    311 | 0      |     0 |       2.9 | 0101000000000000 | 7000000000000000 |
 +     79600 | 2011-05-18 16:00:00 |      3.1 |      90 | 2011-05-18 15:53:08.861966 |    311 | 0      |     0 |       3.1 | 0101000000000000 | 7000000000000000 |
 +     79600 | 2011-05-18 17:00:00 |      3.5 |      90 | 2011-05-18 16:53:21.316684 |    311 | 0      |     0 |       3.5 | 0101000000000000 | 7000000000000000 |
 +     79600 | 2011-05-18 18:00:00 |       22 |      90 | 2011-05-18 17:53:13        |    311 | 0      |     0 |       3.3 | 0409000000000000 | 5033300000000002 | QC1-1-90,QC1-3a-90,QC2d-1-A
 +     79600 | 2011-05-18 19:00:00 |      3.1 |      90 | 2011-05-18 18:53:08        |    311 | 0      |     0 |       3.1 | 0404000000000001 | 1000000000000022 | QC1-3a-90,QC2d-1,hqc
 +     79600 | 2011-05-18 20:00:00 |      3.5 |      90 | 2011-05-18 19:53:09.543286 |    311 | 0      |     0 |       3.5 | 0101000000000000 | 7000000000000000 |
 +     79600 | 2011-05-18 21:00:00 |      2.4 |      90 | 2011-05-18 20:53:16.250386 |    311 | 0      |     0 |       2.4 | 0101000000000000 | 7000000000000000 |
 +     79600 | 2011-05-18 22:00:00 |        2 |      90 | 2011-05-18 21:53:10.393342 |    311 | 0      |     0 |         2 | 0101000000000000 </code>
 +     
 +     
 +**Propagated downstream?**
 +
 +In this link the data appears as 'x' ...
 +
 +[[http://klapp/kdvhpub/production/Report?ct=text/html&sh=0&del=space&ddel=comma&no=0&nod=blank&nod=&sflag=1&flag=9&qa=5&co=NO&la=no&co=MX&co=US&dup=V&batch=Henter+rapport...&st=1&re=8&fd=18.05.2011&td=19.05.2011&w=1&p=FG_1&s=79600]]
 +
 +But when resetting the limit qa=6 ...
 +
 +[[http://klapp/kdvhpub/production/Report?ct=text/html&sh=0&del=space&ddel=comma&no=0&nod=blank&nod=&sflag=1&flag=9&qa=6&co=NO&la=no&co=MX&co=US&dup=V&batch=Henter+rapport...&st=1&re=8&fd=18.05.2011&td=19.05.2011&w=1&p=FG_1&s=79600]]
 +
 +the data is visible downstream.
 +
 +**...more DipTest data from later runs are included hereafter as they are spotted:** 
 +     
 +<code>
 +     15270 | 2011-05-21 02:00:00 |     20.1 |      81 | 2011-05-21 01:51:31.275198 |    330 | 0      |     0 |      20.1 | 0411400000000000 | 7020100000000002 | QC1-1-81,QC1-4-81
 +     15270 | 2011-05-21 03:00:00 |     18.5 |      81 | 2011-05-21 02:51:34.196886 |    330 | 0      |     0 |      18.5 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
 +     15270 | 2011-05-21 04:00:00 |      2.7 |      81 | 2011-05-21 03:51:30        |    330 | 0      |     0 |      17.8 | 0119100000000010 | 5033300000000001 | QC1-3a-81,QC2d-1-L
 +     15270 | 2011-05-21 05:00:00 |       17 |      81 | 2011-05-21 04:51:33        |    330 | 0      |     0 |        17 | 0114100000000010 | 5000000000000001 | QC1-1-81,QC1-3a-81,QC1-4-81,QC2d-1
 +     15270 | 2011-05-21 06:00:00 |     17.2 |      81 | 2011-05-21 05:51:26.594347 |    330 | 0      |     0 |      17.2 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
 +     15270 | 2011-05-21 07:00:00 |     19.4 |      81 | 2011-05-21 06:51:32.539021 |    330 | 0      |     0 |      19.4 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
 +</code>     
 +
 +^Single Linear^
 +
 +<code>
 +     39100 | 2011-05-18 11:00:00 |      9.9 |     213 | 2011-05-18 10:51:44.559729 |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 | 
 +     39100 | 2011-05-18 12:00:00 |      9.9 |     213 | 2011-05-18 11:51:49.58822  |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 | 
 +     39100 | 2011-05-18 13:00:00 |     10.2 |     213 | 2011-05-18 12:51:45.981462 |    330 | 0      |     0 |      10.2 | 0111000000000000 | 7000000000000000 | 
 +     39100 | 2011-05-18 14:00:00 |     9999 |     213 | 2011-05-18 13:51:43        |    330 | 0      |     0 |      10.1 | 0700001100000000 | 5892900000000001 | QC1-1-213,QC1-1-213x,QC2d-2
 +     39100 | 2011-05-18 15:00:00 |       10 |     213 | 2011-05-18 14:51:49.52721  |    330 | 0      |     0 |        10 | 0110000000000000 | 7000000000000000 | 
 +     39100 | 2011-05-18 16:00:00 |      9.9 |     213 | 2011-05-18 15:51:41.77344  |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 | 
 +     39100 | 2011-05-18 17:00:00 |      9.8 |     213 | 2011-05-18 16:51:46.538881 |    330 | 0      |     0 |       9.8 | 0111000000000000 | 7000000000000000 | 
 +</code>
 +
 +
 +^Redistribution^
 +
 +The algorithm ran but no redistribution took for place for the last 4 days. Results will be checked again after the next runs ... and here they are:
 +
 +<code>
 +     48780 | 2011-05-11 06:00:00 |       -1 |     110 | 2011-05-11 05:54:59.95515  |    302 | 0      |     0 |        -1 | 0110000000003000 | 7220900000000000 | QC1-7-110
 +     48780 | 2011-05-12 06:00:00 |      1.9 |     110 | 2011-05-12 05:49:59.465124 |    302 | 0      |     0 |       1.9 | 0110000000001000 | 7000000000000000 | 
 +     48780 | 2011-05-13 06:00:00 |   -32767 |     110 | 2011-05-14 00:30:23        |    302 | 0      |     0 |       3.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_50120_48500_50150_50080_47820_47890_48450_46450_51130,QC2-redist
 +     48780 | 2011-05-14 06:00:00 |   -32767 |     110 | 2011-05-15 00:31:09        |    302 | 0      |     0 |      11.7 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
 +     48780 | 2011-05-15 06:00:00 |   -32767 |     110 | 2011-05-16 00:31:27        |    302 | 0      |     0 |       9.3 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
 +     48780 | 2011-05-16 06:00:00 |   -32767 |     110 | 2011-05-17 00:30:31        |    302 | 0      |     0 |      12.8 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
 +     48780 | 2011-05-17 06:00:00 |   -32767 |     110 | 2011-05-18 00:31:26        |    302 | 0      |     0 |       5.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
 +     48780 | 2011-05-18 06:00:00 |     54.8 |     110 | 2011-05-18 09:22:21        |    302 | 0      |     0 |      12.2 | 0140004000007000 | 5336900000000001 | QC1-2-72.b12,QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
 +     48780 | 2011-05-19 06:00:00 |      8.2 |     110 | 2011-05-19 05:06:08.533353 |    302 | 0      |     0 |       8.2 | 0110000000001000 | 7000000000000000 | 
 +</code>
 +
 +^crontab control of weekly, two week redistribution^
 +
 +The crontab called the scripts successfully but the timing was wrong so the wrong script was called resulting in no algorithm running (but no other impact).
 +
 +Problem identified as the wrong time in the crontab (hours and minutes mixed up ... pilot error!)
 +
 +The erroneous entry:
 +
 +<code>
 +# kvqc2d
 +23 0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh
 +23 0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
 +</code>
 +
 +shall be changed to
 +
 +<code>
 +# kvqc2d
 +25 23 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh
 +25 23 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
 +</code>
 +
 +and installed on Friday (i.e. day=5) so that the cron runs next week. The belated 14 day run is now ready to go for this week.
  
 +These are real time notes on 2011-05-19
  
 +**Update** For 2011-05-25 (Wednesday) to 26 (Thursday) : the cron cycling worked OK.
  • kvalobs/kvoss/system/qc2/user/version/kvqc2d-1.1.2.1304884365.txt.gz
  • Last modified: 2022-05-31 09:23:19
  • (external edit)