Differences

This shows you the differences between two versions of the page.

--- kvalobs:kvoss:system:qc2:user:version:kvqc2d-1.1.2 [2011-05-08 14:20:00]
paule
+++ kvalobs:kvoss:system:qc2:user:version:kvqc2d-1.1.2 [2022-05-31 09:29:32] (current)
@@ Line 1: / Line 1: @@
+====== kvqc2d Version: 1.1.2 ======
-.... in preparation ... FIXME
+Source: [[https://svn.met.no/viewvc/kvoss/kvqc2d/branches/kvqc2d-1.1.2/]]
-====== kvqc2d Version: 1.1.0 ======
+**Significant Changes**
-^ 8-) **To Do** 8-)  When deploying into operation the kvalobs control scripts have to be updated to call "kvqc2d" and not "kvqc2" ^
+   * Release of DipTest (including new strategy with corresponding configuration files)
+   * Algorithm Called by Name
-In consequence, these are suggested steps to follow:
+In addition, it is proposed to manage the weekly run of the Redistribution  algorithm with a cron.
-  * kvstop -l {and select the value for kvqc2}
+**Algorithms in this release:**
-  * SYSTEM: apt-get remove kvqc2
-  * SYSTEM: apt-get install kvqc2d
-  * run kvqc2d on the command line first to check all paths are in order (and also all the log files with new names are created) ... Ctrl+C to Quit
-  * update /etc/kvalobs/kv_ctl.conf ... i.e. replace all kvqc2 with kvqc2d
-  * kvstart
+^ Algorithm(+) ^Description  ^
+| SingleLinear | As in previous release kvqc2d_1.1.0: {{kvalobs:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.pdf| specification}}|
+| Redistribute | As in previous release kvqc2d_1.1.0: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:rarr_24 |
+| DipTest | New algorithm: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:diptest03 |
-Source: [[https://svn.met.no/viewvc/kvoss/kvQc2/branches/kvqc2d-1.1.0/]]
+(+) This is the name to specify in the configuration file.
-**Algorithms in this release:**
+**Install Configuration Files and Cron Jobs**
+For this release all the configuration files need to be re-installed since the use of AlgoCode="a number" is now replaced with Algorithm="the algorithm name". The changes are illustrated in the example configuration files below.
-^ AlgoCode ^Name  ^Description  ^
+The new configuration files are held in subversion in the directory https://svn.met.no/viewvc/kvoss/kvqc2d/branches/kvqc2d-1.1.2/src/Reference/
-| 10 | SingleLinear_v32 | Replaces a single missing value with a linear interpolated value. If a corresponding max or min paramids are also specified then checks that the correction lies within the available max ... min range, if the interpolated values falls outside this range then the algorithm sets the corrected value to the nearest of max or min. Run for any paramid and optional max, min specified in the configuration file. New version updated with respect to a {{:kvoss:system:qc2:test:algorithms:qc2-d2_flagg_08_2010_v32.pdf|revised specification}}|
-| 1 | Redistribution of RR_24 | See https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:rarr_24 and config file below.|
-**Sample data from the new algorithm** (scroll right for cfailed format)
+This includes a tar file of all of the required contents for easier deployment
-**NB** on dev-vm101 this data could only be generated with I_fd=0 ... normally this is set to I_fd=1 so that only QC1-controlled data is used for interpolation model values for the redistribution.
+To install new configuration files and setup the associated cronjobs:
+**New cfg files**
 <code>
-kvalobs=# select * from data where obstime between '2010-12-07' and 'today' and paramid=110 and cfailed like '%QC2-redist%' order by stationid, obstime;
+# log into the operational machine as the kvalobs user
- stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |                                  cfailed
------------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------------------------------------------------------------------------
-| 2010-12-07 06:00:00 |   -32767 |     110 | 2010-12-08 00:30:21 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_25320_25630_25640_25100_25830_24890,QC2-redist
-| 2010-12-08 06:00:00 |   -32767 |     110 | 2010-12-09 00:30:27 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_25320_25630_25640_25100_53950_25830_24890,QC2-redist
-| 2010-12-09 06:00:00 |       -1 |     110 | 2010-12-09 07:21:52 |    302 | 0      |     0 |        -1 | 0110004000007000 | 5336900000000000 | QC1-7-110,QC2N_25320_25630_25640_25100_53950_25830_24890,QC2-redist
-| 2010-12-07 06:00:00 |   -32767 |     110 | 2010-12-08 00:30:35 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_42940_45530_45600_45870_44760_42810_42520,QC2-redist
+# stop the existing kvqc2d
-| 2010-12-08 06:00:00 |   -32767 |     110 | 2010-12-09 00:30:25 |    302 | 0      |     0 |       5.3 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_42940_45530_45600_45870_44760_42810_42520_44800,QC2-redist
+$ kvstop -l
-| 2010-12-09 06:00:00 |        9 |     110 | 2010-12-09 11:40:20 |    302 | 0      |     0 |       3.7 | 0110004000007000 | 5336900000000000 | QC1-7-110,QC2N_42940_45530_45600_45870_44760_42810_42520_44800,QC2-redist
-| 2010-12-07 06:00:00 |   -32767 |     110 | 2010-12-08 00:30:48 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_84190,QC2-redist
+#setup new configuration
-| 2010-12-08 06:00:00 |   -32767 |     110 | 2010-12-09 00:30:33 |    302 | 0      |     0 |       0.2 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_84190_85180_83710,QC2-redist
-| 2010-12-09 06:00:00 |      1.5 |     110 | 2010-12-09 07:03:10 |    302 | 0      |     0 |       1.3 | 0140004000007000 | 5336900000000001 | QC1-2-72.b12,QC1-7-110,QC2N_84190_83710,QC2-redist
+$ cd /etc/kvalobs/Qc2Config
-(9 rows)
+# remove the old files (-i is suggested below so that the user is prompted to confirm removal and verify that only the desired files are removed).
+$ rm -i ./*.cfg
+$ rm -i ./*.hold
+# fetch the new files, e.g.:
+$ wget https://svn.met.no/kvoss/kvqc2d/branches/kvqc2d-1.1.2/src/Reference/cfg.tar.gz
+$ tar xvzf cfg.tar.gz
+$ rm cfg.tar.gz
+# The configuration files are now in place.
+# restart kvqc2d when ready:
+$ kvstart
 </code>
+**cron jobs**
+<code>
+# In order to set up the crontab for the weekly redistribution run
+$ cd /etc/kvalobs/Qc2Config
+# The scripts are present in the cfg.tar.gz archive and should be present if the new cfg file installation steps above have been followed.
+$ chmod +x Redistribution_14days_off_cron.sh
+$ chmod +x Redistribution_14days_on_cron.sh
+# next edit the crontab and insert the following two lines
+# i.e. as user kvalobs:
+$ crontab -e
+# insert the following lines:
+0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh
+0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
+</code>
+Notes: the new scripts in cron renames "Redistribution_14days.hold" to "Redistribution_14days.cfg" every Wednesday and the sets it back to "Redistribution_14days.hold" 24 hours later.
+The Redistribution algorithm will then run automatically to the following schedule:
+   * typically daily run at **06:35** looking at the last 4 days
+   * **01:20** Wednesday Night/Thursday Morning run to check the last 14 days
+Previously the management of the Redistribution configuration files (i.e. hold -> cfg -> hold) has been handled manually.
 **Example Configuration Files**
 ^SingleLinear_v32^
-*NB* Different/simpler checks applied to the neighbours (CF version 1.0.5)
 <code>
-# AlgCode 10 corresponds to https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/algorithms/SingleLinear.cc
 # Performs simple linear interpolation for a single point replacement.
-# If the result is outside an available max min range, the nearest of the
+Algorithm=SingleLinear
-# max or min values are used.
+RunAtHour=6
-AlgoCode=10
+RunAtMinute=50
-# Time to run the algorithm
+Last_NDays=1
-RunAtHour=16
-RunAtMinute=47
-# Check data from the last three days
-Last_NDays=40
 #[Specific Data Type and Paramters ids etc.]
-ParamId=211
+ParamId=104
-# MaxParamId=215   # Optional  !!!! The Max and Min options are no longer required (to be modified).
+# Only write back the result if not prviously controlled
-# MinParamId=213   # Optional
-# Only write back the result if not previously controlled
 W_fhqc=0
@@ Line 78: / Line 105: @@
 change_fmis=0->4
 change_fmis=2->4
+#Flag to check if the algorithm shall be applied ...
-#If there is already a 1 in place then we do not want run the algorithm again!
+U_2=0
-U_2=0
 </code>
@@ Line 87: / Line 113: @@
 <code>
 # Algorithm configuration for redistribution of accumulated precipitation
-AlgoCode=1
+Algorithm=Redistribute
 InterpCode=2
-RunAtHour=23
+RunAtHour=6
 RunAtMinute=35
-Last_NDays=14
+Last_NDays=4
 #[Time Range] The time window for when to apply the algorithm
@@ Line 122: / Line 148: @@
 #
 # Only write data it it has not been analysed by HQC
+# W_fhqc=9  # 9 set here because of the test data being used ...
 W_fhqc=0
-# Added for a special test run
-# CfailedString="any text"
 # Condition for running the algorithm
@@ Line 142: / Line 165: @@
 </code>
-**Notes**
-   * This algorithm will redistribute data flagged with fd=2 (A_fd)  and use neighbours with fd=1 for interpolation (I_fd) and set fd=7 (S_fd) for the new corrected data. But fhqc must be 0 for data to be written back.
+^DipTest^
-   * To re-run the algorithm on the same data again set A_fd=7 and keep W_fhqc=0, i.e. do not re-write any decision by HQC.
+The DipTest configuration comprises two files:
+**DipTest.cfg**
+<code>
+# DIP-TEST
+Algorithm=DipTest
+RunAtHour=5
+RunAtMinute=5
+#[Time Range]
+Last_NDays=1
-The schedule for the algorithm is as follows:
+#[Specific Data Type and Paramters ids etc.]
-   * **06:35** perform a run looking at the last 4 days - the last 4 days is selected since from the first run on murky (see above) about 30 redistributions were made of which 15 were over 3 days and 10 over 4 days.
+ParValFilename="/etc/kvalobs/Qc2Config/DipTestParameters.txt"
-   * **01:20** Wednesday Night/Thursday Morning run to check the last 14 days (Note: Could we run every night for the last two weeks to pick up any changes?)
+# Only write data it it has not been analysed by HQC
+# W_fhqc=9  # 9 set here because of the test data being used ...
+W_fhqc=0
+#Flag required for neighbour to be used for an interpolation
+U_2=0
+</code>
+That works through the parameters and corresponding delta values specified in **DipTestParameters.txt** as below:
+<code>
+   9.0
+   60.0
+   10.0
+   10.0
+   12.5
+   10.0
+   12.5
+   15.0
+   12.5
+   12.5
+   15.0
+   15.0
+   15.0
+   3.0
+   15.0
+   60.0
+   12.0
+   12.0
+   12.0
+   12.0
+   12.0
+   1000.0
+   500.0
+   500.0
+   1000.0
+   7.5
+   7.5
+   7.5
+   7.5
+   15.0
+   15.0
+</code>
+**Note:** Previously "104   100" was also an entry in the above file.
+====== Notes on first production run ======
+   * Released into operations on 2011-05-18 circa 09:38 UTC
+Algorithms ran as expected:
+^DipTest^
+First result (note __only one__ Dip was identified):
+<code>
+kvalobs=# select * from data where stationid=79600 and obstime between '2011-05-18 00:00:00' and '2011-05-19 00:00:00' and paramid=90;
+ stationid |       obstime       | original | paramid |           tbtime           | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |           cfailed
+-----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+-----------------------------
+| 2011-05-18 15:00:00 |      2.9 |      90 | 2011-05-18 14:53:12.296014 |    311 | 0      |     0 |       2.9 | 0101000000000000 | 7000000000000000 |
+| 2011-05-18 16:00:00 |      3.1 |      90 | 2011-05-18 15:53:08.861966 |    311 | 0      |     0 |       3.1 | 0101000000000000 | 7000000000000000 |
+| 2011-05-18 17:00:00 |      3.5 |      90 | 2011-05-18 16:53:21.316684 |    311 | 0      |     0 |       3.5 | 0101000000000000 | 7000000000000000 |
+| 2011-05-18 18:00:00 |       22 |      90 | 2011-05-18 17:53:13        |    311 | 0      |     0 |       3.3 | 0409000000000000 | 5033300000000002 | QC1-1-90,QC1-3a-90,QC2d-1-A
+| 2011-05-18 19:00:00 |      3.1 |      90 | 2011-05-18 18:53:08        |    311 | 0      |     0 |       3.1 | 0404000000000001 | 1000000000000022 | QC1-3a-90,QC2d-1,hqc
+| 2011-05-18 20:00:00 |      3.5 |      90 | 2011-05-18 19:53:09.543286 |    311 | 0      |     0 |       3.5 | 0101000000000000 | 7000000000000000 |
+| 2011-05-18 21:00:00 |      2.4 |      90 | 2011-05-18 20:53:16.250386 |    311 | 0      |     0 |       2.4 | 0101000000000000 | 7000000000000000 |
+| 2011-05-18 22:00:00 |        2 |      90 | 2011-05-18 21:53:10.393342 |    311 | 0      |     0 |         2 | 0101000000000000 </code>
+**Propagated downstream?**
+In this link the data appears as 'x' ...
+[[http://klapp/kdvhpub/production/Report?ct=text/html&sh=0&del=space&ddel=comma&no=0&nod=blank&nod=&sflag=1&flag=9&qa=5&co=NO&la=no&co=MX&co=US&dup=V&batch=Henter+rapport...&st=1&re=8&fd=18.05.2011&td=19.05.2011&w=1&p=FG_1&s=79600]]
+But when resetting the limit qa=6 ...
+[[http://klapp/kdvhpub/production/Report?ct=text/html&sh=0&del=space&ddel=comma&no=0&nod=blank&nod=&sflag=1&flag=9&qa=6&co=NO&la=no&co=MX&co=US&dup=V&batch=Henter+rapport...&st=1&re=8&fd=18.05.2011&td=19.05.2011&w=1&p=FG_1&s=79600]]
+the data is visible downstream.
+**...more DipTest data from later runs are included hereafter as they are spotted:**
+<code>
+| 2011-05-21 02:00:00 |     20.1 |      81 | 2011-05-21 01:51:31.275198 |    330 | 0      |     0 |      20.1 | 0411400000000000 | 7020100000000002 | QC1-1-81,QC1-4-81
+| 2011-05-21 03:00:00 |     18.5 |      81 | 2011-05-21 02:51:34.196886 |    330 | 0      |     0 |      18.5 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
+| 2011-05-21 04:00:00 |      2.7 |      81 | 2011-05-21 03:51:30        |    330 | 0      |     0 |      17.8 | 0119100000000010 | 5033300000000001 | QC1-3a-81,QC2d-1-L
+| 2011-05-21 05:00:00 |       17 |      81 | 2011-05-21 04:51:33        |    330 | 0      |     0 |        17 | 0114100000000010 | 5000000000000001 | QC1-1-81,QC1-3a-81,QC1-4-81,QC2d-1
+| 2011-05-21 06:00:00 |     17.2 |      81 | 2011-05-21 05:51:26.594347 |    330 | 0      |     0 |      17.2 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
+| 2011-05-21 07:00:00 |     19.4 |      81 | 2011-05-21 06:51:32.539021 |    330 | 0      |     0 |      19.4 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
+</code>
+^Single Linear^
+<code>
+| 2011-05-18 11:00:00 |      9.9 |     213 | 2011-05-18 10:51:44.559729 |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 |
+| 2011-05-18 12:00:00 |      9.9 |     213 | 2011-05-18 11:51:49.58822  |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 |
+| 2011-05-18 13:00:00 |     10.2 |     213 | 2011-05-18 12:51:45.981462 |    330 | 0      |     0 |      10.2 | 0111000000000000 | 7000000000000000 |
+| 2011-05-18 14:00:00 |     9999 |     213 | 2011-05-18 13:51:43        |    330 | 0      |     0 |      10.1 | 0700001100000000 | 5892900000000001 | QC1-1-213,QC1-1-213x,QC2d-2
+| 2011-05-18 15:00:00 |       10 |     213 | 2011-05-18 14:51:49.52721  |    330 | 0      |     0 |        10 | 0110000000000000 | 7000000000000000 |
+| 2011-05-18 16:00:00 |      9.9 |     213 | 2011-05-18 15:51:41.77344  |    330 | 0      |     0 |       9.9 | 0111000000000000 | 7000000000000000 |
+| 2011-05-18 17:00:00 |      9.8 |     213 | 2011-05-18 16:51:46.538881 |    330 | 0      |     0 |       9.8 | 0111000000000000 | 7000000000000000 |
+</code>
+^Redistribution^
+The algorithm ran but no redistribution took for place for the last 4 days. Results will be checked again after the next runs ... and here they are:
+<code>
+| 2011-05-11 06:00:00 |       -1 |     110 | 2011-05-11 05:54:59.95515  |    302 | 0      |     0 |        -1 | 0110000000003000 | 7220900000000000 | QC1-7-110
+| 2011-05-12 06:00:00 |      1.9 |     110 | 2011-05-12 05:49:59.465124 |    302 | 0      |     0 |       1.9 | 0110000000001000 | 7000000000000000 |
+| 2011-05-13 06:00:00 |   -32767 |     110 | 2011-05-14 00:30:23        |    302 | 0      |     0 |       3.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_50120_48500_50150_50080_47820_47890_48450_46450_51130,QC2-redist
+| 2011-05-14 06:00:00 |   -32767 |     110 | 2011-05-15 00:31:09        |    302 | 0      |     0 |      11.7 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
+| 2011-05-15 06:00:00 |   -32767 |     110 | 2011-05-16 00:31:27        |    302 | 0      |     0 |       9.3 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
+| 2011-05-16 06:00:00 |   -32767 |     110 | 2011-05-17 00:30:31        |    302 | 0      |     0 |      12.8 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
+| 2011-05-17 06:00:00 |   -32767 |     110 | 2011-05-18 00:31:26        |    302 | 0      |     0 |       5.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
+| 2011-05-18 06:00:00 |     54.8 |     110 | 2011-05-18 09:22:21        |    302 | 0      |     0 |      12.2 | 0140004000007000 | 5336900000000001 | QC1-2-72.b12,QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
+| 2011-05-19 06:00:00 |      8.2 |     110 | 2011-05-19 05:06:08.533353 |    302 | 0      |     0 |       8.2 | 0110000000001000 | 7000000000000000 |
+</code>
+^crontab control of weekly, two week redistribution^
+The crontab called the scripts successfully but the timing was wrong so the wrong script was called resulting in no algorithm running (but no other impact).
+Problem identified as the wrong time in the crontab (hours and minutes mixed up ... pilot error!)
+The erroneous entry:
+<code>
+# kvqc2d
+0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh
+0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
+</code>
+shall be changed to
+<code>
+# kvqc2d
+23 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh
+23 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
+</code>
+and installed on Friday (i.e. day=5) so that the cron runs next week. The belated 14 day run is now ready to go for this week.
+These are real time notes on 2011-05-19
+**Update** For 2011-05-25 (Wednesday) to 26 (Thursday) : the cron cycling worked OK.