kvqc2d Version: 1.1.2
Source: https://svn.met.no/viewvc/kvoss/kvqc2d/branches/kvqc2d-1.1.2/
Significant Changes
- Release of DipTest (including new strategy with corresponding configuration files)
- Algorithm Called by Name
In addition, it is proposed to manage the weekly run of the Redistribution algorithm with a cron.
Algorithms in this release:
| Algorithm(+) | Description |
|---|---|
| SingleLinear | As in previous release kvqc2d_1.1.0: specification |
| Redistribute | As in previous release kvqc2d_1.1.0: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:rarr_24 |
| DipTest | New algorithm: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:diptest03 |
(+) This is the name to specify in the configuration file.
Install Configuration Files and Cron Jobs
For this release all the configuration files need to be re-installed since the use of AlgoCode=“a number” is now replaced with Algorithm=“the algorithm name”. The changes are illustrated in the example configuration files below.
The new configuration files are held in subversion in the directory https://svn.met.no/viewvc/kvoss/kvqc2d/branches/kvqc2d-1.1.2/src/Reference/
This includes a tar file of all of the required contents for easier deployment
To install new configuration files and setup the associated cronjobs:
New cfg files
# log into the operational machine as the kvalobs user # stop the existing kvqc2d $ kvstop -l #setup new configuration $ cd /etc/kvalobs/Qc2Config # remove the old files (-i is suggested below so that the user is prompted to confirm removal and verify that only the desired files are removed). $ rm -i ./*.cfg $ rm -i ./*.hold # fetch the new files, e.g.: $ wget https://svn.met.no/kvoss/kvqc2d/branches/kvqc2d-1.1.2/src/Reference/cfg.tar.gz $ tar xvzf cfg.tar.gz $ rm cfg.tar.gz # The configuration files are now in place. # restart kvqc2d when ready: $ kvstart
cron jobs
# In order to set up the crontab for the weekly redistribution run $ cd /etc/kvalobs/Qc2Config # The scripts are present in the cfg.tar.gz archive and should be present if the new cfg file installation steps above have been followed. $ chmod +x Redistribution_14days_off_cron.sh $ chmod +x Redistribution_14days_on_cron.sh # next edit the crontab and insert the following two lines # i.e. as user kvalobs: $ crontab -e # insert the following lines: 23 0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh 23 0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
Notes: the new scripts in cron renames “Redistribution_14days.hold” to “Redistribution_14days.cfg” every Wednesday and the sets it back to “Redistribution_14days.hold” 24 hours later.
The Redistribution algorithm will then run automatically to the following schedule:
- typically daily run at 06:35 looking at the last 4 days
- 01:20 Wednesday Night/Thursday Morning run to check the last 14 days
Previously the management of the Redistribution configuration files (i.e. hold → cfg → hold) has been handled manually.
Example Configuration Files
| SingleLinear_v32 |
|---|
# Performs simple linear interpolation for a single point replacement. Algorithm=SingleLinear RunAtHour=6 RunAtMinute=50 Last_NDays=1 #[Specific Data Type and Paramters ids etc.] ParamId=104 # Only write back the result if not prviously controlled W_fhqc=0 #Flag to set if value is corrected S_ftime=1 change_fmis=3->1 change_fmis=0->4 change_fmis=2->4 #Flag to check if the algorithm shall be applied ... U_2=0
| Redistribution |
|---|
# Algorithm configuration for redistribution of accumulated precipitation Algorithm=Redistribute InterpCode=2 RunAtHour=6 RunAtMinute=35 Last_NDays=4 #[Time Range] The time window for when to apply the algorithm #Start_YYYY=2010 #Start_MM=10 #Start_DD=8 #Start_hh=6 #Start_mm=0 #Start_ss=0 #End_YYYY=2010 #End_MM=10 #End_DD=11 #End_hh=6 #End_mm=0 #End_ss=0 Start_hh=6 End_hh=6 # NB 24 hour values are at 06:00 UTC #[Time Step] Step_DD=1 #[Specific Data Type and Paramters ids etc.] ParamId=110 TypeIds=302 TypeIds=402 # # Only write data it it has not been analysed by HQC # W_fhqc=9 # 9 set here because of the test data being used ... W_fhqc=0 # Condition for running the algorithm A_fd=2 # Flags to set on successful completion S_fd=7 change_fmis=3->1 change_fmis=0->4 #Only interpolate the following data I_fd=1 InterpolationDistance=50.0
| DipTest |
|---|
The DipTest configuration comprises two files:
DipTest.cfg
# DIP-TEST Algorithm=DipTest RunAtHour=5 RunAtMinute=5 #[Time Range] Last_NDays=1 #[Specific Data Type and Paramters ids etc.] ParValFilename="/etc/kvalobs/Qc2Config/DipTestParameters.txt" # Only write data it it has not been analysed by HQC # W_fhqc=9 # 9 set here because of the test data being used ... W_fhqc=0 #Flag required for neighbour to be used for an interpolation U_2=0
That works through the parameters and corresponding delta values specified in DipTestParameters.txt as below:
1 9.0 2 60.0 81 10.0 85 10.0 86 12.5 87 10.0 88 12.5 89 15.0 90 12.5 91 12.5 92 15.0 93 15.0 94 15.0 105 3.0 113 15.0 123 60.0 172 12.0 174 12.0 175 12.0 176 12.0 177 12.0 197 1000.0 198 500.0 199 500.0 200 1000.0 211 7.5 212 7.5 213 7.5 215 7.5 221 15.0 222 15.0
Note: Previously “104 100” was also an entry in the above file.
Notes on first production run
- Released into operations on 2011-05-18 circa 09:38 UTC
Algorithms ran as expected:
| DipTest |
|---|
First result (note only one Dip was identified):
kvalobs=# select * from data where stationid=79600 and obstime between '2011-05-18 00:00:00' and '2011-05-19 00:00:00' and paramid=90;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+-----------------------------
79600 | 2011-05-18 15:00:00 | 2.9 | 90 | 2011-05-18 14:53:12.296014 | 311 | 0 | 0 | 2.9 | 0101000000000000 | 7000000000000000 |
79600 | 2011-05-18 16:00:00 | 3.1 | 90 | 2011-05-18 15:53:08.861966 | 311 | 0 | 0 | 3.1 | 0101000000000000 | 7000000000000000 |
79600 | 2011-05-18 17:00:00 | 3.5 | 90 | 2011-05-18 16:53:21.316684 | 311 | 0 | 0 | 3.5 | 0101000000000000 | 7000000000000000 |
79600 | 2011-05-18 18:00:00 | 22 | 90 | 2011-05-18 17:53:13 | 311 | 0 | 0 | 3.3 | 0409000000000000 | 5033300000000002 | QC1-1-90,QC1-3a-90,QC2d-1-A
79600 | 2011-05-18 19:00:00 | 3.1 | 90 | 2011-05-18 18:53:08 | 311 | 0 | 0 | 3.1 | 0404000000000001 | 1000000000000022 | QC1-3a-90,QC2d-1,hqc
79600 | 2011-05-18 20:00:00 | 3.5 | 90 | 2011-05-18 19:53:09.543286 | 311 | 0 | 0 | 3.5 | 0101000000000000 | 7000000000000000 |
79600 | 2011-05-18 21:00:00 | 2.4 | 90 | 2011-05-18 20:53:16.250386 | 311 | 0 | 0 | 2.4 | 0101000000000000 | 7000000000000000 |
79600 | 2011-05-18 22:00:00 | 2 | 90 | 2011-05-18 21:53:10.393342 | 311 | 0 | 0 | 2 | 0101000000000000
Propagated downstream?
In this link the data appears as 'x' …
But when resetting the limit qa=6 …
the data is visible downstream.
…more DipTest data from later runs are included hereafter as they are spotted:
15270 | 2011-05-21 02:00:00 | 20.1 | 81 | 2011-05-21 01:51:31.275198 | 330 | 0 | 0 | 20.1 | 0411400000000000 | 7020100000000002 | QC1-1-81,QC1-4-81
15270 | 2011-05-21 03:00:00 | 18.5 | 81 | 2011-05-21 02:51:34.196886 | 330 | 0 | 0 | 18.5 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
15270 | 2011-05-21 04:00:00 | 2.7 | 81 | 2011-05-21 03:51:30 | 330 | 0 | 0 | 17.8 | 0119100000000010 | 5033300000000001 | QC1-3a-81,QC2d-1-L
15270 | 2011-05-21 05:00:00 | 17 | 81 | 2011-05-21 04:51:33 | 330 | 0 | 0 | 17 | 0114100000000010 | 5000000000000001 | QC1-1-81,QC1-3a-81,QC1-4-81,QC2d-1
15270 | 2011-05-21 06:00:00 | 17.2 | 81 | 2011-05-21 05:51:26.594347 | 330 | 0 | 0 | 17.2 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
15270 | 2011-05-21 07:00:00 | 19.4 | 81 | 2011-05-21 06:51:32.539021 | 330 | 0 | 0 | 19.4 | 0211400000000010 | 7010100000000002 | QC1-1-81,QC1-4-81
| Single Linear |
|---|
39100 | 2011-05-18 11:00:00 | 9.9 | 213 | 2011-05-18 10:51:44.559729 | 330 | 0 | 0 | 9.9 | 0111000000000000 | 7000000000000000 |
39100 | 2011-05-18 12:00:00 | 9.9 | 213 | 2011-05-18 11:51:49.58822 | 330 | 0 | 0 | 9.9 | 0111000000000000 | 7000000000000000 |
39100 | 2011-05-18 13:00:00 | 10.2 | 213 | 2011-05-18 12:51:45.981462 | 330 | 0 | 0 | 10.2 | 0111000000000000 | 7000000000000000 |
39100 | 2011-05-18 14:00:00 | 9999 | 213 | 2011-05-18 13:51:43 | 330 | 0 | 0 | 10.1 | 0700001100000000 | 5892900000000001 | QC1-1-213,QC1-1-213x,QC2d-2
39100 | 2011-05-18 15:00:00 | 10 | 213 | 2011-05-18 14:51:49.52721 | 330 | 0 | 0 | 10 | 0110000000000000 | 7000000000000000 |
39100 | 2011-05-18 16:00:00 | 9.9 | 213 | 2011-05-18 15:51:41.77344 | 330 | 0 | 0 | 9.9 | 0111000000000000 | 7000000000000000 |
39100 | 2011-05-18 17:00:00 | 9.8 | 213 | 2011-05-18 16:51:46.538881 | 330 | 0 | 0 | 9.8 | 0111000000000000 | 7000000000000000 |
| Redistribution |
|---|
The algorithm ran but no redistribution took for place for the last 4 days. Results will be checked again after the next runs … and here they are:
48780 | 2011-05-11 06:00:00 | -1 | 110 | 2011-05-11 05:54:59.95515 | 302 | 0 | 0 | -1 | 0110000000003000 | 7220900000000000 | QC1-7-110
48780 | 2011-05-12 06:00:00 | 1.9 | 110 | 2011-05-12 05:49:59.465124 | 302 | 0 | 0 | 1.9 | 0110000000001000 | 7000000000000000 |
48780 | 2011-05-13 06:00:00 | -32767 | 110 | 2011-05-14 00:30:23 | 302 | 0 | 0 | 3.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_50120_48500_50150_50080_47820_47890_48450_46450_51130,QC2-redist
48780 | 2011-05-14 06:00:00 | -32767 | 110 | 2011-05-15 00:31:09 | 302 | 0 | 0 | 11.7 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
48780 | 2011-05-15 06:00:00 | -32767 | 110 | 2011-05-16 00:31:27 | 302 | 0 | 0 | 9.3 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
48780 | 2011-05-16 06:00:00 | -32767 | 110 | 2011-05-17 00:30:31 | 302 | 0 | 0 | 12.8 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
48780 | 2011-05-17 06:00:00 | -32767 | 110 | 2011-05-18 00:31:26 | 302 | 0 | 0 | 5.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
48780 | 2011-05-18 06:00:00 | 54.8 | 110 | 2011-05-18 09:22:21 | 302 | 0 | 0 | 12.2 | 0140004000007000 | 5336900000000001 | QC1-2-72.b12,QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
48780 | 2011-05-19 06:00:00 | 8.2 | 110 | 2011-05-19 05:06:08.533353 | 302 | 0 | 0 | 8.2 | 0110000000001000 | 7000000000000000 |
| crontab control of weekly, two week redistribution |
|---|
The crontab called the scripts successfully but the timing was wrong so the wrong script was called resulting in no algorithm running (but no other impact).
Problem identified as the wrong time in the crontab (hours and minutes mixed up … pilot error!)
The erroneous entry:
# kvqc2d 23 0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh 23 0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
shall be changed to
# kvqc2d 25 23 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh 25 23 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh
and installed on Friday (i.e. day=5) so that the cron runs next week. The belated 14 day run is now ready to go for this week.
These are real time notes on 2011-05-19
Update For 2011-05-25 (Wednesday) to 26 (Thursday) : the cron cycling worked OK.