kvqc2d Version: 1.1.2

Source: https://svn.met.no/viewvc/kvoss/kvqc2d/branches/kvqc2d-1.1.2/

Significant Changes

Release of DipTest (including new strategy with corresponding configuration files)

Algorithm Called by Name

In addition, it is proposed to manage the weekly run of the Redistribution algorithm with a cron.

Algorithms in this release:

Algorithm(+)	Description
SingleLinear	As in previous release kvqc2d_1.1.0: specification
Redistribute	As in previous release kvqc2d_1.1.0: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:rarr_24
DipTest	New algorithm: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:diptest03

(+) This is the name to specify in the configuration file.

Install Configuration Files and Cron Jobs

For this release all the configuration files need to be re-installed since the use of AlgoCode=“a number” is now replaced with Algorithm=“the algorithm name”. The changes are illustrated in the example configuration files below.

The new configuration files are held in subversion in the directory https://svn.met.no/viewvc/kvoss/kvqc2d/branches/kvqc2d-1.1.2/src/Reference/

This includes a tar file of all of the required contents for easier deployment

To install new configuration files and setup the associated cronjobs:

New cfg files

# log into the operational machine as the kvalobs user
# stop the existing kvqc2d
$ kvstop -l
#setup new configuration
$ cd /etc/kvalobs/Qc2Config
# remove the old files (-i is suggested below so that the user is prompted to confirm removal and verify that only the desired files are removed).
$ rm -i ./*.cfg
$ rm -i ./*.hold
# fetch the new files, e.g.:
$ wget https://svn.met.no/kvoss/kvqc2d/branches/kvqc2d-1.1.2/src/Reference/cfg.tar.gz
$ tar xvzf cfg.tar.gz
$ rm cfg.tar.gz
# The configuration files are now in place.
# restart kvqc2d when ready:
$ kvstart

cron jobs

# In order to set up the crontab for the weekly redistribution run
$ cd /etc/kvalobs/Qc2Config
# The scripts are present in the cfg.tar.gz archive and should be present if the new cfg file installation steps above have been followed.
$ chmod +x Redistribution_14days_off_cron.sh
$ chmod +x Redistribution_14days_on_cron.sh
# next edit the crontab and insert the following two lines
# i.e. as user kvalobs:
$ crontab -e
# insert the following lines:
23 0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh
23 0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh

Notes: the new scripts in cron renames “Redistribution_14days.hold” to “Redistribution_14days.cfg” every Wednesday and the sets it back to “Redistribution_14days.hold” 24 hours later.

The Redistribution algorithm will then run automatically to the following schedule:

typically daily run at 06:35 looking at the last 4 days

01:20 Wednesday Night/Thursday Morning run to check the last 14 days

Previously the management of the Redistribution configuration files (i.e. hold -> cfg -> hold) has been handled manually.

Example Configuration Files

SingleLinear_v32

# Performs simple linear interpolation for a single point replacement.
Algorithm=SingleLinear
RunAtHour=6
RunAtMinute=50
Last_NDays=1
#[Specific Data Type and Paramters ids etc.]
ParamId=104
# Only write back the result if not prviously controlled
W_fhqc=0
#Flag to set if value is corrected
S_ftime=1
change_fmis=3->1
change_fmis=0->4
change_fmis=2->4
#Flag to check if the algorithm shall be applied ...
U_2=0

Redistribution

# Algorithm configuration for redistribution of accumulated precipitation
Algorithm=Redistribute
InterpCode=2
RunAtHour=6
RunAtMinute=35
Last_NDays=4
#[Time Range] The time window for when to apply the algorithm
#Start_YYYY=2010
#Start_MM=10
#Start_DD=8
#Start_hh=6
#Start_mm=0
#Start_ss=0
#End_YYYY=2010
#End_MM=10
#End_DD=11
#End_hh=6
#End_mm=0
#End_ss=0
Start_hh=6
End_hh=6 # NB 24 hour values are at 06:00 UTC
#[Time Step]
Step_DD=1
#[Specific Data Type and Paramters ids etc.]
ParamId=110
TypeIds=302
TypeIds=402
#
# Only write data it it has not been analysed by HQC
# W_fhqc=9 # 9 set here because of the test data being used ...
W_fhqc=0
# Condition for running the algorithm
A_fd=2
# Flags to set on successful completion
S_fd=7
change_fmis=3->1
change_fmis=0->4
#Only interpolate the following data
I_fd=1
InterpolationDistance=50.0

DipTest

The DipTest configuration comprises two files:

DipTest.cfg

# DIP-TEST
Algorithm=DipTest
RunAtHour=5
RunAtMinute=5
#[Time Range]
Last_NDays=1
#[Specific Data Type and Paramters ids etc.]
ParValFilename="/etc/kvalobs/Qc2Config/DipTestParameters.txt"
# Only write data it it has not been analysed by HQC
# W_fhqc=9 # 9 set here because of the test data being used ...
W_fhqc=0
#Flag required for neighbour to be used for an interpolation
U_2=0

That works through the parameters and corresponding delta values specified in DipTestParameters.txt as below:

1 9.0
2 60.0
81 10.0
85 10.0
86 12.5
87 10.0
88 12.5
89 15.0
90 12.5
91 12.5
92 15.0
93 15.0
94 15.0
105 3.0
113 15.0
123 60.0
172 12.0
174 12.0
175 12.0
176 12.0
177 12.0
197 1000.0
198 500.0
199 500.0
200 1000.0
211 7.5
212 7.5
213 7.5
215 7.5
221 15.0
222 15.0

Note: Previously “104 100” was also an entry in the above file.

Notes on first production run

Released into operations on 2011-05-18 circa 09:38 UTC

Algorithms ran as expected:

DipTest

First result (note only one Dip was identified):

kvalobs=# select * from data where stationid=79600 and obstime between '2011-05-18 00:00:00' and '2011-05-19 00:00:00' and paramid=90;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+----------------------------+--------+--------+-------+-----------+------------------+------------------+-----------------------------
79600 | 2011-05-18 15:00:00 | 2.9 | 90 | 2011-05-18 14:53:12.296014 | 311 | 0 | 0 | 2.9 | 0101000000000000 | 7000000000000000 |
79600 | 2011-05-18 16:00:00 | 3.1 | 90 | 2011-05-18 15:53:08.861966 | 311 | 0 | 0 | 3.1 | 0101000000000000 | 7000000000000000 |
79600 | 2011-05-18 17:00:00 | 3.5 | 90 | 2011-05-18 16:53:21.316684 | 311 | 0 | 0 | 3.5 | 0101000000000000 | 7000000000000000 |
79600 | 2011-05-18 18:00:00 | 22 | 90 | 2011-05-18 17:53:13 | 311 | 0 | 0 | 3.3 | 0409000000000000 | 5033300000000002 | QC1-1-90,QC1-3a-90,QC2d-1-A
79600 | 2011-05-18 19:00:00 | 3.1 | 90 | 2011-05-18 18:53:08 | 311 | 0 | 0 | 3.1 | 0404000000000001 | 1000000000000022 | QC1-3a-90,QC2d-1,hqc
79600 | 2011-05-18 20:00:00 | 3.5 | 90 | 2011-05-18 19:53:09.543286 | 311 | 0 | 0 | 3.5 | 0101000000000000 | 7000000000000000 |
79600 | 2011-05-18 21:00:00 | 2.4 | 90 | 2011-05-18 20:53:16.250386 | 311 | 0 | 0 | 2.4 | 0101000000000000 | 7000000000000000 |
79600 | 2011-05-18 22:00:00 | 2 | 90 | 2011-05-18 21:53:10.393342 | 311 | 0 | 0 | 2 | 0101000000000000

Propagated downstream?

In this link the data appears as 'x' ...

http://klapp/kdvhpub/production/Report?ct=text/html&sh=0&del=space&ddel=comma&no=0&nod=blank&nod=&sflag=1&flag=9&qa=5&co=NO&la=no&co=MX&co=US&dup=V&batch=Henter+rapport...&st=1&re=8&fd=18.05.2011&td=19.05.2011&w=1&p=FG_1&s=79600

But when resetting the limit qa=6 ...

http://klapp/kdvhpub/production/Report?ct=text/html&sh=0&del=space&ddel=comma&no=0&nod=blank&nod=&sflag=1&flag=9&qa=6&co=NO&la=no&co=MX&co=US&dup=V&batch=Henter+rapport...&st=1&re=8&fd=18.05.2011&td=19.05.2011&w=1&p=FG_1&s=79600

the data is visible downstream.

Single Linear

39100 | 2011-05-18 11:00:00 | 9.9 | 213 | 2011-05-18 10:51:44.559729 | 330 | 0 | 0 | 9.9 | 0111000000000000 | 7000000000000000 |
39100 | 2011-05-18 12:00:00 | 9.9 | 213 | 2011-05-18 11:51:49.58822 | 330 | 0 | 0 | 9.9 | 0111000000000000 | 7000000000000000 |
39100 | 2011-05-18 13:00:00 | 10.2 | 213 | 2011-05-18 12:51:45.981462 | 330 | 0 | 0 | 10.2 | 0111000000000000 | 7000000000000000 |
39100 | 2011-05-18 14:00:00 | 9999 | 213 | 2011-05-18 13:51:43 | 330 | 0 | 0 | 10.1 | 0700001100000000 | 5892900000000001 | QC1-1-213,QC1-1-213x,QC2d-2
39100 | 2011-05-18 15:00:00 | 10 | 213 | 2011-05-18 14:51:49.52721 | 330 | 0 | 0 | 10 | 0110000000000000 | 7000000000000000 |
39100 | 2011-05-18 16:00:00 | 9.9 | 213 | 2011-05-18 15:51:41.77344 | 330 | 0 | 0 | 9.9 | 0111000000000000 | 7000000000000000 |
39100 | 2011-05-18 17:00:00 | 9.8 | 213 | 2011-05-18 16:51:46.538881 | 330 | 0 | 0 | 9.8 | 0111000000000000 | 7000000000000000 |

Redistribution

The algorithm ran but no redistribution took for place for the last 4 days. Results will be checked again after the next runs ... and here they are:

48780 | 2011-05-11 06:00:00 | -1 | 110 | 2011-05-11 05:54:59.95515 | 302 | 0 | 0 | -1 | 0110000000003000 | 7220900000000000 | QC1-7-110
48780 | 2011-05-12 06:00:00 | 1.9 | 110 | 2011-05-12 05:49:59.465124 | 302 | 0 | 0 | 1.9 | 0110000000001000 | 7000000000000000 |
48780 | 2011-05-13 06:00:00 | -32767 | 110 | 2011-05-14 00:30:23 | 302 | 0 | 0 | 3.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_50120_48500_50150_50080_47820_47890_48450_46450_51130,QC2-redist
48780 | 2011-05-14 06:00:00 | -32767 | 110 | 2011-05-15 00:31:09 | 302 | 0 | 0 | 11.7 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
48780 | 2011-05-15 06:00:00 | -32767 | 110 | 2011-05-16 00:31:27 | 302 | 0 | 0 | 9.3 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
48780 | 2011-05-16 06:00:00 | -32767 | 110 | 2011-05-17 00:30:31 | 302 | 0 | 0 | 12.8 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
48780 | 2011-05-17 06:00:00 | -32767 | 110 | 2011-05-18 00:31:26 | 302 | 0 | 0 | 5.4 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
48780 | 2011-05-18 06:00:00 | 54.8 | 110 | 2011-05-18 09:22:21 | 302 | 0 | 0 | 12.2 | 0140004000007000 | 5336900000000001 | QC1-2-72.b12,QC1-7-110,QC2N_49351_49070_48500_50150_47820_47890_48450_46450,QC2-redist
48780 | 2011-05-19 06:00:00 | 8.2 | 110 | 2011-05-19 05:06:08.533353 | 302 | 0 | 0 | 8.2 | 0110000000001000 | 7000000000000000 |

crontab control of weekly, two week redistribution

The crontab called the scripts successfully but the timing was wrong so the wrong script was called resulting in no algorithm running (but no other impact).

Problem identified as the wrong time in the crontab (hours and minutes mixed up ... pilot error!)

The erroneous entry:

# kvqc2d
23 0 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh
23 0 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh

shall be changed to

# kvqc2d
25 23 * * 3 /etc/kvalobs/Qc2Config/Redistribution_14days_on_cron.sh
25 23 * * 4 /etc/kvalobs/Qc2Config/Redistribution_14days_off_cron.sh

and installed on Friday (i.e. day=5) so that the cron runs next week. The belated 14 day run is now ready to go for this week.

These are real time notes on 2011-05-19