kvalobs:kvoss:system:qc2:user:version:kvqc2d-1.1.2

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
kvalobs:kvoss:system:qc2:user:version:kvqc2d-1.1.2 [2011-05-08 19:41:06]
paule 		kvalobs:kvoss:system:qc2:user:version:kvqc2d-1.1.2 [2022-05-31 09:29:32] (current)
Line 1: Line 1:
- 
-.... in preparation ... FIXME 
- 
 ====== kvqc2d Version: 1.1.2 ====== ====== kvqc2d Version: 1.1.2 ======
  
Line 16: Line 13:
  
 ^ Algorithm(+) ^Description  ^ ^ Algorithm(+) ^Description  ^
-| SingleLinear | As in previous release kvqc2d_1.1.0: {{:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.pdf| specification}}|  +| SingleLinear | As in previous release kvqc2d_1.1.0: {{kvalobs:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.pdf| specification}}|  
 | Redistribute | As in previous release kvqc2d_1.1.0: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:rarr_24 |  | Redistribute | As in previous release kvqc2d_1.1.0: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:rarr_24 | 
 | DipTest | New algorithm: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:diptest03 | | DipTest | New algorithm: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:diptest03 |
Line 22: Line 19:
 (+) This is the name to specify in the configuration file. (+) This is the name to specify in the configuration file.
  
-**Example Configuration Files**+**Install Configuration Files and Cron Jobs**
  
 For this release all the configuration files need to be re-installed since the use of AlgoCode="a number" is now replaced with Algorithm="the algorithm name". The changes are illustrated in the example configuration files below. For this release all the configuration files need to be re-installed since the use of AlgoCode="a number" is now replaced with Algorithm="the algorithm name". The changes are illustrated in the example configuration files below.
Line 30: Line 27:
 This includes a tar file of all of the required contents for easier deployment This includes a tar file of all of the required contents for easier deployment
  
-To install new configuration files and setup the associated cronjob:+To install new configuration files and setup the associated cronjobs: 
 + 
 +**New cfg files**
 <code> <code>
-# after logging into the operational machine as the kvalobs user+# log into the operational machine as the kvalobs user 
 + 
 +# stop the existing kvqc2d 
 +$ kvstop -l 
 + 
 +#setup new configuration
  
 $ cd /etc/kvalobs/Qc2Config $ cd /etc/kvalobs/Qc2Config
-$ rm ./*.cfg +# remove the old files (-i is suggested below so that the user is prompted to confirm removal and verify that only the desired files are removed). 
-$ rm ./*.hold+$ rm -i ./*.cfg 
 +$ rm -i ./*.hold  
 +# fetch the new files, e.g.:
 $ wget https://svn.met.no/kvoss/kvqc2d/branches/kvqc2d-1.1.2/src/Reference/cfg.tar.gz $ wget https://svn.met.no/kvoss/kvqc2d/branches/kvqc2d-1.1.2/src/Reference/cfg.tar.gz
 $ tar xvzf cfg.tar.gz $ tar xvzf cfg.tar.gz
Line 43: Line 49:
 # The configuration files are now in place. # The configuration files are now in place.
  
----------------------------------------------+# restart kvqc2d when ready:
  
 +$ kvstart
 +
 +</code>
 +
 +**cron jobs**
 +<code>
 # In order to set up the crontab for the weekly redistribution run # In order to set up the crontab for the weekly redistribution run
  
 +$ cd /etc/kvalobs/Qc2Config
 +# The scripts are present in the cfg.tar.gz archive and should be present if the new cfg file installation steps above have been followed.
 $ chmod +x Redistribution_14days_off_cron.sh $ chmod +x Redistribution_14days_off_cron.sh
 $ chmod +x Redistribution_14days_on_cron.sh $ chmod +x Redistribution_14days_on_cron.sh
Line 57: Line 71:
 # insert the following lines: # insert the following lines:
  
-# 23 0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh +23 0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh 
-# 23 0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh+23 0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
  
 </code> </code>
 + 
 +Notes: the new scripts in cron renames "Redistribution_14days.hold" to "Redistribution_14days.cfg" every Wednesday and the sets it back to "Redistribution_14days.hold" 24 hours later.
  
 +The Redistribution algorithm will then run automatically to the following schedule:
 +   * typically daily run at **06:35** looking at the last 4 days 
 +   * **01:20** Wednesday Night/Thursday Morning run to check the last 14 days
  
 +Previously the management of the Redistribution configuration files (i.e. hold -> cfg -> hold) has been handled manually.
  
 **Example Configuration Files** **Example Configuration Files**
  
 ^SingleLinear_v32^ ^SingleLinear_v32^
- 
-*NB* Different/simpler checks applied to the neighbours (CF version 1.0.5) 
  
 <code> <code>
-# AlgCode 10 corresponds to https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/algorithms/SingleLinear.cc 
 # Performs simple linear interpolation for a single point replacement. # Performs simple linear interpolation for a single point replacement.
-# If the result is outside an available max min range, the nearest of the +Algorithm=SingleLinear 
-# max or min values are used. +RunAtHour=6 
-AlgoCode=10 +RunAtMinute=50    
-# Time to run the algorithm +Last_NDays=1
-RunAtHour=16 +
-RunAtMinute=47 +
-# Check data from the last three days +
-Last_NDays=40+
  
 #[Specific Data Type and Paramters ids etc.] #[Specific Data Type and Paramters ids etc.]
-ParamId=211 +ParamId=104 
-# MaxParamId=215   # Optional  !!!! The Max and Min options are no longer required (to be modified). +# Only write back the result if not prviously controlled
-# MinParamId=213   # Optional +
-# Only write back the result if not previously controlled+
 W_fhqc=0 W_fhqc=0
  
Line 94: Line 105:
 change_fmis=0->4 change_fmis=0->4
 change_fmis=2->4 change_fmis=2->4
- +#Flag to check if the algorithm shall be applied ... 
-#If there is already a 1 in place then we do not want run the algorithm again! +U_2=0
-U_2=0+
 </code> </code>
  
Line 103: Line 113:
 <code> <code>
 # Algorithm configuration for redistribution of accumulated precipitation # Algorithm configuration for redistribution of accumulated precipitation
-AlgoCode=1+Algorithm=Redistribute
 InterpCode=2 InterpCode=2
-RunAtHour=23+RunAtHour=6
 RunAtMinute=35 RunAtMinute=35
-Last_NDays=14+Last_NDays=4
  
 #[Time Range] The time window for when to apply the algorithm #[Time Range] The time window for when to apply the algorithm
Line 138: Line 148:
 # #
 # Only write data it it has not been analysed by HQC # Only write data it it has not been analysed by HQC
 +# W_fhqc=9  # 9 set here because of the test data being used ...
 W_fhqc=0 W_fhqc=0
- 
-# Added for a special test run 
-# CfailedString="any text" 
- 
  
 # Condition for running the algorithm # Condition for running the algorithm
Line 158: Line 165:
 </code> </code>
  
-**Notes** 
  
-   * This algorithm will redistribute data flagged with fd=2 (A_fd)  and use neighbours with fd=1 for interpolation (I_fd) and set fd=7 (S_fd) for the new corrected data. But fhqc must be 0 for data to be written back. +^DipTest^
-   * To re-run the algorithm on the same data again set A_fd=7 and keep W_fhqc=0, i.e. do not re-write any decision by HQC.+
  
 +The DipTest configuration comprises two files:
  
 +**DipTest.cfg**
 +<code>
 +# DIP-TEST
 +Algorithm=DipTest
 +RunAtHour=5 
 +RunAtMinute=5
  
 +#[Time Range]
 +Last_NDays=1
  
-The schedule for the algorithm is as follows: +#[Specific Data Type and Paramters ids etc.] 
-   * **06:35** perform a run looking at the last 4 days - the last 4 days is selected since from the first run on murky (see above) about 30 redistributions were made of which 15 were over 3 days and 10 over 4 days. +ParValFilename="/etc/kvalobs/Qc2Config/DipTestParameters.txt" 
-   * **01:20** Wednesday Night/Thursday Morning run to check the last 14 days (Note: Could we run every night for the last two weeks to pick up any changes?)+ 
 +# Only write data it it has not been analysed by HQC 
 +# W_fhqc=9  # 9 set here because of the test data being used ... 
 +W_fhqc=0 
 + 
 +#Flag required for neighbour to be used for an interpolation 
 +U_2=0 
 +</code> 
 + 
 +That works through the parameters and corresponding delta values specified in **DipTestParameters.txt** as below: 
 +<code> 
 +1   9.0 
 +2   60.0 
 +81   10.0 
 +85   10.0 
 +86   12.5 
 +87   10.0 
 +88   12.5 
 +89   15.0 
 +90   12.5 
 +91   12.5 
 +92   15.0 
 +93   15.0 
 +94   15.0 
 +105   3.0 
 +113   15.0 
 +123   60.0 
 +172   12.0 
 +174   12.0 
 +175   12.0 
 +176   12.0 
 +177   12.0 
 +197   1000.0 
 +198   500.0 
 +199   500.0 
 +200   1000.0 
 +211   7.5 
 +212   7.5 
 +213   7.5 
 +215   7.5 
 +221   15.0 
 +222   15.0 
 +</code> 
 + 
 +**Note:** Previously "104   100" was also an entry in the above file. 
 + 
 +====== Notes on first production run ====== 
 + 
 +   * Released into operations on 2011-05-18 circa 09:38 UTC 
 + 
 +Algorithms ran as expected: 
 + 
 +^DipTest^ 
 + 
 +First result (note __only one__ Dip was identified): 
 + 
 +<code> 
 +kvalobs=# select * from data where stationid=79600 and obstime between '2011-05-18 00:00:00' and '2011-05-19 00:00:00' and paramid=90; 
 + stationid |       obstime       | original | paramid |           tbtime           | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |           cfailed           
 +-----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+----------------------------- 
 +     79600 | 2011-05-18 15:00:00 |      2.9 |      90 | 2011-05-18 14:53:12.296014 |    311 | 0      |     0 |       2.9 | 0101000000000000 | 7000000000000000 | 
 +     79600 | 2011-05-18 16:00:00 |      3.1 |      90 | 2011-05-18 15:53:08.861966 |    311 | 0      |     0 |       3.1 | 0101000000000000 | 7000000000000000 | 
 +     79600 | 2011-05-18 17:00:00 |      3.5 |      90 | 2011-05-18 16:53:21.316684 |    311 | 0      |     0 |       3.5 | 0101000000000000 | 7000000000000000 | 
 +     79600 | 2011-05-18 18:00:00 |       22 |      90 | 2011-05-18 17:53:13        |    311 | 0      |     0 |       3.3 | 0409000000000000 | 5033300000000002 | QC1-1-90,QC1-3a-90,QC2d-1-A 
 +     79600 | 2011-05-18 19:00:00 |      3.1 |      90 | 2011-05-18 18:53:08        |    311 | 0      |     0 |       3.1 | 0404000000000001 | 1000000000000022 | QC1-3a-90,QC2d-1,hqc 
 +     79600 | 2011-05-18 20:00:00 |      3.5 |      90 | 2011-05-18 19:53:09.543286 |    311 | 0      |     0 |       3.5 | 0101000000000000 | 7000000000000000 | 
 +     79600 | 2011-05-18 21:00:00 |      2.4 |      90 | 2011-05-18 20:53:16.250386 |    311 | 0      |     0 |       2.4 | 0101000000000000 | 7000000000000000 | 
 +     79600 | 2011-05-18 22:00:00 |        2 |      90 | 2011-05-18 21:53:10.393342 |    311 | 0      |     0 |         2 | 0101000000000000 </code> 
 +      
 +      
 +**Propagated downstream?** 
 + 
 +In this link the data appears as 'x' ... 
 + 
 +[[http://klapp/kdvhpub/production/Report?ct=text/html&sh=0&del=space&ddel=comma&no=0&nod=blank&nod=&sflag=1&flag=9&qa=5&co=NO&la=no&co=MX&co=US&dup=V&batch=Henter+rapport...&st=1&re=8&fd=18.05.2011&td=19.05.2011&w=1&p=FG_1&s=79600]] 
 + 
 +But when resetting the limit qa=6 ... 
 + 
 +[[http://klapp/kdvhpub/production/Report?ct=text/html&sh=0&del=space&ddel=comma&no=0&nod=blank&nod=&sflag=1&flag=9&qa=6&co=NO&la=no&co=MX&co=US&dup=V&batch=Henter+rapport...&st=1&re=8&fd=18.05.2011&td=19.05.2011&w=1&p=FG_1&s=79600]] 
 + 
 +the data is visible downstream. 
 + 
 +**...more DipTest data from later runs are included hereafter as they are spotted:**  
 +      
 +<code> 
 +     15270 | 2011-05-21 02:00:00 |     20.1 |      81 | 2011-05-21 01:51:31.275198 |    330 | 0      |     0 |      20.1 | 0411400000000000 | 7020100000000002 | QC1-1-81,QC1-4-81 
 +     15270 | 2011-05-21 03:00:00 |     18.5 |      81 | 2011-05-21 02:51:34.196886 |    330 | 0      |     0 |      18.5 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81 
 +     15270 | 2011-05-21 04:00:00 |      2.7 |      81 | 2011-05-21 03:51:30        |    330 | 0      |     0 |      17.8 | 0119100000000010 | 5033300000000001 | QC1-3a-81,QC2d-1-L 
 +     15270 | 2011-05-21 05:00:00 |       17 |      81 | 2011-05-21 04:51:33        |    330 | 0      |     0 |        17 | 0114100000000010 | 5000000000000001 | QC1-1-81,QC1-3a-81,QC1-4-81,QC2d-1 
 +     15270 | 2011-05-21 06:00:00 |     17.2 |      81 | 2011-05-21 05:51:26.594347 |    330 | 0      |     0 |      17.2 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81 
 +     15270 | 2011-05-21 07:00:00 |     19.4 |      81 | 2011-05-21 06:51:32.539021 |    330 | 0      |     0 |      19.4 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81 
 +</code>      
 + 
 +^Single Linear^ 
 + 
 +<code> 
 +     39100 | 2011-05-18 11:00:00 |      9.9 |     213 | 2011-05-18 10:51:44.559729 |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 |  
 +     39100 | 2011-05-18 12:00:00 |      9.9 |     213 | 2011-05-18 11:51:49.58822  |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 |  
 +     39100 | 2011-05-18 13:00:00 |     10.2 |     213 | 2011-05-18 12:51:45.981462 |    330 | 0      |     0 |      10.2 | 0111000000000000 | 7000000000000000 |  
 +     39100 | 2011-05-18 14:00:00 |     9999 |     213 | 2011-05-18 13:51:43        |    330 | 0      |     0 |      10.1 | 0700001100000000 | 5892900000000001 | QC1-1-213,QC1-1-213x,QC2d-2 
 +     39100 | 2011-05-18 15:00:00 |       10 |     213 | 2011-05-18 14:51:49.52721  |    330 | 0      |     0 |        10 | 0110000000000000 | 7000000000000000 |  
 +     39100 | 2011-05-18 16:00:00 |      9.9 |     213 | 2011-05-18 15:51:41.77344  |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 |  
 +     39100 | 2011-05-18 17:00:00 |      9.8 |     213 | 2011-05-18 16:51:46.538881 |    330 | 0      |     0 |       9.8 | 0111000000000000 | 7000000000000000 |  
 +</code> 
 + 
 + 
 +^Redistribution^ 
 + 
 +The algorithm ran but no redistribution took for place for the last 4 days. Results will be checked again after the next runs ... and here they are: 
 + 
 +<code> 
 +     48780 | 2011-05-11 06:00:00 |       -1 |     110 | 2011-05-11 05:54:59.95515  |    302 | 0      |     0 |        -1 | 0110000000003000 | 7220900000000000 | QC1-7-110 
 +     48780 | 2011-05-12 06:00:00 |      1.9 |     110 | 2011-05-12 05:49:59.465124 |    302 | 0      |     0 |       1.9 | 0110000000001000 | 7000000000000000 |  
 +     48780 | 2011-05-13 06:00:00 |   -32767 |     110 | 2011-05-14 00:30:23        |    302 | 0      |     0 |       3.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_50120_48500_50150_50080_47820_47890_48450_46450_51130,QC2-redist 
 +     48780 | 2011-05-14 06:00:00 |   -32767 |     110 | 2011-05-15 00:31:09        |    302 | 0      |     0 |      11.7 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist 
 +     48780 | 2011-05-15 06:00:00 |   -32767 |     110 | 2011-05-16 00:31:27        |    302 | 0      |     0 |       9.3 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist 
 +     48780 | 2011-05-16 06:00:00 |   -32767 |     110 | 2011-05-17 00:30:31        |    302 | 0      |     0 |      12.8 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist 
 +     48780 | 2011-05-17 06:00:00 |   -32767 |     110 | 2011-05-18 00:31:26        |    302 | 0      |     0 |       5.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist 
 +     48780 | 2011-05-18 06:00:00 |     54.8 |     110 | 2011-05-18 09:22:21        |    302 | 0      |     0 |      12.2 | 0140004000007000 | 5336900000000001 | QC1-2-72.b12,QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist 
 +     48780 | 2011-05-19 06:00:00 |      8.2 |     110 | 2011-05-19 05:06:08.533353 |    302 | 0      |     0 |       8.2 | 0110000000001000 | 7000000000000000 |  
 +</code> 
 + 
 +^crontab control of weekly, two week redistribution^ 
 + 
 +The crontab called the scripts successfully but the timing was wrong so the wrong script was called resulting in no algorithm running (but no other impact). 
 + 
 +Problem identified as the wrong time in the crontab (hours and minutes mixed up ... pilot error!) 
 + 
 +The erroneous entry: 
 + 
 +<code> 
 +# kvqc2d 
 +23 0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh 
 +23 0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh 
 +</code> 
 + 
 +shall be changed to 
 + 
 +<code> 
 +# kvqc2d 
 +25 23 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh 
 +25 23 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh 
 +</code>
  
 +and installed on Friday (i.e. day=5) so that the cron runs next week. The belated 14 day run is now ready to go for this week.
  
 +These are real time notes on 2011-05-19
  
 +**Update** For 2011-05-25 (Wednesday) to 26 (Thursday) : the cron cycling worked OK.
  • kvalobs/kvoss/system/qc2/user/version/kvqc2d-1.1.2.1304883666.txt.gz
  • Last modified: 2022-05-31 09:23:19
  • (external edit)