Differences

This shows you the differences between two versions of the page.

--- kvalobs:kvoss:system:qc2:user:version:kvqc2d-1.1.2 [2011-05-08 18:31:42]
paule
+++ kvalobs:kvoss:system:qc2:user:version:kvqc2d-1.1.2 [2022-05-31 09:29:32] (current)
@@ Line 1: / Line 1: @@
-.... in preparation ... FIXME
 ====== kvqc2d Version: 1.1.2 ======
@@ Line 16: / Line 13: @@
 ^ Algorithm(+) ^Description  ^
-| SingleLinear | As in previous release kvqc2d_1.1.0: {{:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.pdf| specification}}|
+| SingleLinear | As in previous release kvqc2d_1.1.0: {{kvalobs:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.pdf| specification}}|
 | Redistribute | As in previous release kvqc2d_1.1.0: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:rarr_24 |
 | DipTest | New algorithm: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:diptest03 |
@@ Line 22: / Line 19: @@
 (+) This is the name to specify in the configuration file.
-**Example Configuration Files**
+**Install Configuration Files and Cron Jobs**
 For this release all the configuration files need to be re-installed since the use of AlgoCode="a number" is now replaced with Algorithm="the algorithm name". The changes are illustrated in the example configuration files below.
@@ Line 30: / Line 27: @@
 This includes a tar file of all of the required contents for easier deployment
-To install new configuration files:
+To install new configuration files and setup the associated cronjobs:
+**New cfg files**
 <code>
-# after logging into the operational machine as the kvalobs user
+# log into the operational machine as the kvalobs user
+# stop the existing kvqc2d
+$ kvstop -l
+#setup new configuration
 $ cd /etc/kvalobs/Qc2Config
-$ rm ./*.cfg
+# remove the old files (-i is suggested below so that the user is prompted to confirm removal and verify that only the desired files are removed).
-$ rm ./*.hold
+$ rm -i ./*.cfg
+$ rm -i ./*.hold
+# fetch the new files, e.g.:
 $ wget https://svn.met.no/kvoss/kvqc2d/branches/kvqc2d-1.1.2/src/Reference/cfg.tar.gz
 $ tar xvzf cfg.tar.gz
 $ rm cfg.tar.gz
+# The configuration files are now in place.
+# restart kvqc2d when ready:
+$ kvstart
+</code>
+**cron jobs**
+<code>
+# In order to set up the crontab for the weekly redistribution run
+$ cd /etc/kvalobs/Qc2Config
+# The scripts are present in the cfg.tar.gz archive and should be present if the new cfg file installation steps above have been followed.
 $ chmod +x Redistribution_14days_off_cron.sh
 $ chmod +x Redistribution_14days_on_cron.sh
-$
+# next edit the crontab and insert the following two lines
+# i.e. as user kvalobs:
+$ crontab -e
+# insert the following lines:
+0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh
+0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
 </code>
+Notes: the new scripts in cron renames "Redistribution_14days.hold" to "Redistribution_14days.cfg" every Wednesday and the sets it back to "Redistribution_14days.hold" 24 hours later.
+The Redistribution algorithm will then run automatically to the following schedule:
+   * typically daily run at **06:35** looking at the last 4 days
+   * **01:20** Wednesday Night/Thursday Morning run to check the last 14 days
+Previously the management of the Redistribution configuration files (i.e. hold -> cfg -> hold) has been handled manually.
 **Example Configuration Files**
 ^SingleLinear_v32^
-*NB* Different/simpler checks applied to the neighbours (CF version 1.0.5)
 <code>
-# AlgCode 10 corresponds to https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/algorithms/SingleLinear.cc
 # Performs simple linear interpolation for a single point replacement.
-# If the result is outside an available max min range, the nearest of the
+Algorithm=SingleLinear
-# max or min values are used.
+RunAtHour=6
-AlgoCode=10
+RunAtMinute=50
-# Time to run the algorithm
+Last_NDays=1
-RunAtHour=16
-RunAtMinute=47
-# Check data from the last three days
-Last_NDays=40
 #[Specific Data Type and Paramters ids etc.]
-ParamId=211
+ParamId=104
-# MaxParamId=215   # Optional  !!!! The Max and Min options are no longer required (to be modified).
+# Only write back the result if not prviously controlled
-# MinParamId=213   # Optional
-# Only write back the result if not previously controlled
 W_fhqc=0
@@ Line 79: / Line 105: @@
 change_fmis=0->4
 change_fmis=2->4
+#Flag to check if the algorithm shall be applied ...
-#If there is already a 1 in place then we do not want run the algorithm again!
+U_2=0
-U_2=0
 </code>
@@ Line 88: / Line 113: @@
 <code>
 # Algorithm configuration for redistribution of accumulated precipitation
-AlgoCode=1
+Algorithm=Redistribute
 InterpCode=2
-RunAtHour=23
+RunAtHour=6
 RunAtMinute=35
-Last_NDays=14
+Last_NDays=4
 #[Time Range] The time window for when to apply the algorithm
@@ Line 123: / Line 148: @@
 #
 # Only write data it it has not been analysed by HQC
+# W_fhqc=9  # 9 set here because of the test data being used ...
 W_fhqc=0
-# Added for a special test run
-# CfailedString="any text"
 # Condition for running the algorithm
@@ Line 143: / Line 165: @@
 </code>
-**Notes**
-   * This algorithm will redistribute data flagged with fd=2 (A_fd)  and use neighbours with fd=1 for interpolation (I_fd) and set fd=7 (S_fd) for the new corrected data. But fhqc must be 0 for data to be written back.
+^DipTest^
-   * To re-run the algorithm on the same data again set A_fd=7 and keep W_fhqc=0, i.e. do not re-write any decision by HQC.
+The DipTest configuration comprises two files:
+**DipTest.cfg**
+<code>
+# DIP-TEST
+Algorithm=DipTest
+RunAtHour=5
+RunAtMinute=5
+#[Time Range]
+Last_NDays=1
-The schedule for the algorithm is as follows:
+#[Specific Data Type and Paramters ids etc.]
-   * **06:35** perform a run looking at the last 4 days - the last 4 days is selected since from the first run on murky (see above) about 30 redistributions were made of which 15 were over 3 days and 10 over 4 days.
+ParValFilename="/etc/kvalobs/Qc2Config/DipTestParameters.txt"
-   * **01:20** Wednesday Night/Thursday Morning run to check the last 14 days (Note: Could we run every night for the last two weeks to pick up any changes?)
+# Only write data it it has not been analysed by HQC
+# W_fhqc=9  # 9 set here because of the test data being used ...
+W_fhqc=0
+#Flag required for neighbour to be used for an interpolation
+U_2=0
+</code>
+That works through the parameters and corresponding delta values specified in **DipTestParameters.txt** as below:
+<code>
+   9.0
+   60.0
+   10.0
+   10.0
+   12.5
+   10.0
+   12.5
+   15.0
+   12.5
+   12.5
+   15.0
+   15.0
+   15.0
+   3.0
+   15.0
+   60.0
+   12.0
+   12.0
+   12.0
+   12.0
+   12.0
+   1000.0
+   500.0
+   500.0
+   1000.0
+   7.5
+   7.5
+   7.5
+   7.5
+   15.0
+   15.0
+</code>
+**Note:** Previously "104   100" was also an entry in the above file.
+====== Notes on first production run ======
+   * Released into operations on 2011-05-18 circa 09:38 UTC
+Algorithms ran as expected:
+^DipTest^
+First result (note __only one__ Dip was identified):
+<code>
+kvalobs=# select * from data where stationid=79600 and obstime between '2011-05-18 00:00:00' and '2011-05-19 00:00:00' and paramid=90;
+ stationid |       obstime       | original | paramid |           tbtime           | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |           cfailed
+-----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+-----------------------------
+| 2011-05-18 15:00:00 |      2.9 |      90 | 2011-05-18 14:53:12.296014 |    311 | 0      |     0 |       2.9 | 0101000000000000 | 7000000000000000 |
+| 2011-05-18 16:00:00 |      3.1 |      90 | 2011-05-18 15:53:08.861966 |    311 | 0      |     0 |       3.1 | 0101000000000000 | 7000000000000000 |
+| 2011-05-18 17:00:00 |      3.5 |      90 | 2011-05-18 16:53:21.316684 |    311 | 0      |     0 |       3.5 | 0101000000000000 | 7000000000000000 |
+| 2011-05-18 18:00:00 |       22 |      90 | 2011-05-18 17:53:13        |    311 | 0      |     0 |       3.3 | 0409000000000000 | 5033300000000002 | QC1-1-90,QC1-3a-90,QC2d-1-A
+| 2011-05-18 19:00:00 |      3.1 |      90 | 2011-05-18 18:53:08        |    311 | 0      |     0 |       3.1 | 0404000000000001 | 1000000000000022 | QC1-3a-90,QC2d-1,hqc
+| 2011-05-18 20:00:00 |      3.5 |      90 | 2011-05-18 19:53:09.543286 |    311 | 0      |     0 |       3.5 | 0101000000000000 | 7000000000000000 |
+| 2011-05-18 21:00:00 |      2.4 |      90 | 2011-05-18 20:53:16.250386 |    311 | 0      |     0 |       2.4 | 0101000000000000 | 7000000000000000 |
+| 2011-05-18 22:00:00 |        2 |      90 | 2011-05-18 21:53:10.393342 |    311 | 0      |     0 |         2 | 0101000000000000 </code>
+**Propagated downstream?**
+In this link the data appears as 'x' ...
+[[http://klapp/kdvhpub/production/Report?ct=text/html&sh=0&del=space&ddel=comma&no=0&nod=blank&nod=&sflag=1&flag=9&qa=5&co=NO&la=no&co=MX&co=US&dup=V&batch=Henter+rapport...&st=1&re=8&fd=18.05.2011&td=19.05.2011&w=1&p=FG_1&s=79600]]
+But when resetting the limit qa=6 ...
+[[http://klapp/kdvhpub/production/Report?ct=text/html&sh=0&del=space&ddel=comma&no=0&nod=blank&nod=&sflag=1&flag=9&qa=6&co=NO&la=no&co=MX&co=US&dup=V&batch=Henter+rapport...&st=1&re=8&fd=18.05.2011&td=19.05.2011&w=1&p=FG_1&s=79600]]
+the data is visible downstream.
+**...more DipTest data from later runs are included hereafter as they are spotted:**
+<code>
+| 2011-05-21 02:00:00 |     20.1 |      81 | 2011-05-21 01:51:31.275198 |    330 | 0      |     0 |      20.1 | 0411400000000000 | 7020100000000002 | QC1-1-81,QC1-4-81
+| 2011-05-21 03:00:00 |     18.5 |      81 | 2011-05-21 02:51:34.196886 |    330 | 0      |     0 |      18.5 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
+| 2011-05-21 04:00:00 |      2.7 |      81 | 2011-05-21 03:51:30        |    330 | 0      |     0 |      17.8 | 0119100000000010 | 5033300000000001 | QC1-3a-81,QC2d-1-L
+| 2011-05-21 05:00:00 |       17 |      81 | 2011-05-21 04:51:33        |    330 | 0      |     0 |        17 | 0114100000000010 | 5000000000000001 | QC1-1-81,QC1-3a-81,QC1-4-81,QC2d-1
+| 2011-05-21 06:00:00 |     17.2 |      81 | 2011-05-21 05:51:26.594347 |    330 | 0      |     0 |      17.2 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
+| 2011-05-21 07:00:00 |     19.4 |      81 | 2011-05-21 06:51:32.539021 |    330 | 0      |     0 |      19.4 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
+</code>
+^Single Linear^
+<code>
+| 2011-05-18 11:00:00 |      9.9 |     213 | 2011-05-18 10:51:44.559729 |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 |
+| 2011-05-18 12:00:00 |      9.9 |     213 | 2011-05-18 11:51:49.58822  |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 |
+| 2011-05-18 13:00:00 |     10.2 |     213 | 2011-05-18 12:51:45.981462 |    330 | 0      |     0 |      10.2 | 0111000000000000 | 7000000000000000 |
+| 2011-05-18 14:00:00 |     9999 |     213 | 2011-05-18 13:51:43        |    330 | 0      |     0 |      10.1 | 0700001100000000 | 5892900000000001 | QC1-1-213,QC1-1-213x,QC2d-2
+| 2011-05-18 15:00:00 |       10 |     213 | 2011-05-18 14:51:49.52721  |    330 | 0      |     0 |        10 | 0110000000000000 | 7000000000000000 |
+| 2011-05-18 16:00:00 |      9.9 |     213 | 2011-05-18 15:51:41.77344  |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 |
+| 2011-05-18 17:00:00 |      9.8 |     213 | 2011-05-18 16:51:46.538881 |    330 | 0      |     0 |       9.8 | 0111000000000000 | 7000000000000000 |
+</code>
+^Redistribution^
+The algorithm ran but no redistribution took for place for the last 4 days. Results will be checked again after the next runs ... and here they are:
+<code>
+| 2011-05-11 06:00:00 |       -1 |     110 | 2011-05-11 05:54:59.95515  |    302 | 0      |     0 |        -1 | 0110000000003000 | 7220900000000000 | QC1-7-110
+| 2011-05-12 06:00:00 |      1.9 |     110 | 2011-05-12 05:49:59.465124 |    302 | 0      |     0 |       1.9 | 0110000000001000 | 7000000000000000 |
+| 2011-05-13 06:00:00 |   -32767 |     110 | 2011-05-14 00:30:23        |    302 | 0      |     0 |       3.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_50120_48500_50150_50080_47820_47890_48450_46450_51130,QC2-redist
+| 2011-05-14 06:00:00 |   -32767 |     110 | 2011-05-15 00:31:09        |    302 | 0      |     0 |      11.7 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
+| 2011-05-15 06:00:00 |   -32767 |     110 | 2011-05-16 00:31:27        |    302 | 0      |     0 |       9.3 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
+| 2011-05-16 06:00:00 |   -32767 |     110 | 2011-05-17 00:30:31        |    302 | 0      |     0 |      12.8 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
+| 2011-05-17 06:00:00 |   -32767 |     110 | 2011-05-18 00:31:26        |    302 | 0      |     0 |       5.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
+| 2011-05-18 06:00:00 |     54.8 |     110 | 2011-05-18 09:22:21        |    302 | 0      |     0 |      12.2 | 0140004000007000 | 5336900000000001 | QC1-2-72.b12,QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
+| 2011-05-19 06:00:00 |      8.2 |     110 | 2011-05-19 05:06:08.533353 |    302 | 0      |     0 |       8.2 | 0110000000001000 | 7000000000000000 |
+</code>
+^crontab control of weekly, two week redistribution^
+The crontab called the scripts successfully but the timing was wrong so the wrong script was called resulting in no algorithm running (but no other impact).
+Problem identified as the wrong time in the crontab (hours and minutes mixed up ... pilot error!)
+The erroneous entry:
+<code>
+# kvqc2d
+0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh
+0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
+</code>
+shall be changed to
+<code>
+# kvqc2d
+23 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh
+23 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
+</code>
+and installed on Friday (i.e. day=5) so that the cron runs next week. The belated 14 day run is now ready to go for this week.
+These are real time notes on 2011-05-19
+**Update** For 2011-05-25 (Wednesday) to 26 (Thursday) : the cron cycling worked OK.