Installation and test on dev-vm101 notes [kvqc2_1.0.1]

The following notes describe the installation and use of kvqc2 version 1.0.1 Sources: https://svn.met.no/viewvc/kvoss/kvQc2/branches/kvqc2-1.0.1/

Algorithms in this release:

AlgoCode	Description	Status
9	SingleMinMaxAverage	kvqc2_1.0.1 For a given paramid and corresponding Max paramid and Min paramid calculates a correction for a single missing value as average of max and min. If no max or min available or specified then reverts to simple linear.
10	SingleLinear	kvqc2_1.0.1 Replaces a single missing value with a linear interpolated value. If specified checks that the correction lies within the available max ... min range, if not sets to nearest of max or min. Run for any paramid and optional max, min specified in the configuration file.

Installation

Step 1	Installation

Log into dev-vm101 as user kvalobs

If an old kvqc2 exists stop the running kvqc2 process {e.g. use kvstop ? terminate the process pid ?}

Remove active configuration files (this is to just ensure that when the new kvqc2 version is set running it does not find and use an old configuration file)

For example:

$ cd /etc/kvalobs/Qc2Config
$ mv XXX.cfg XXX.hold
$ mv ProcessUnit.cfg ProcessUnit.hold

( PS NOTE: actually now all the earliy config files have been moved to /etc/kvalobs/Qc2Config/old )

Install the latest version of kvqc2

Log in as a user with sudoer permission

NB one may need to run “apt-get update” first in order to update the package list.

$ sudo apt-get install kvqc2

An alternative is to install the new kvqc2 directly from where the debian package was built on dev-vm101:

i.e. using “sudo dpkg -i kvqc2_1.0.1-1_i386.deb” as illustrated below.

paule@dev-vm101:~/kvqc2-1.0.1$ ls
kvqc2-1.0.1 kvqc2_1.0.1-1_i386.changes kvqc2_1.0.1.orig.tar.gz
kvqc2_1.0.1-1.diff.gz kvqc2_1.0.1-1_i386.deb kvqc2-1.0.1.tar.gz
kvqc2_1.0.1-1.dsc kvqc2_1.0.1-1_i386.upload
paule@dev-vm101:~/kvqc2-1.0.1$ sudo dpkg -i kvqc2_1.0.1-1_i386.deb
(Reading database ... 52403 files and directories currently installed.)
Preparing to replace kvqc2 1.0.0-1 (using kvqc2_1.0.1-1_i386.deb) ...
Unpacking replacement kvqc2 ...
Setting up kvqc2 (1.0.1-1) ...
paule@dev-vm101:~/kvqc2-1.0.1$

Bug Fix and Confirmation of apt-get install

Initial tests illustrated that some data was written back to thedatabase at too high a precision. This is fixed in revision 1782: https://svn.met.no/viewvc/kvoss?view=rev&revision=1782 and led to a kvqc2-1.0.2 revision. This also provided the opportunity to check that the install from apt-get install works, i.e.:

paule@dev-vm101:~$ sudo apt-get install kvqc2
Reading package lists... Done
Building dependency tree... Done
The following packages will be upgraded:
kvqc2
1 upgraded, 0 newly installed, 0 to remove and 78 not upgraded.
Need to get 288kB of archives.
After unpacking 0B of additional disk space will be used.
Get:1 http://repo.met.no etch/main kvqc2 1.0.2-1 [288kB]
Fetched 288kB in 0s (12.7MB/s)
(Reading database ... 52403 files and directories currently installed.)
Preparing to replace kvqc2 1.0.1-1 (using .../kvqc2_1.0.2-1_i386.deb) ...
Unpacking replacement kvqc2 ...
Setting up kvqc2 (1.0.2-1) ...
paule@dev-vm101:~$

Run and initial tests

Step 2	Run with “kvstart”

“kvstart” should start the new version, just test to see if this works:

Login as user kvalobs and then:

kvalobs@dev-vm101:~$ kvstart
KVBIN=/usr/bin
KVPID=/var/run/kvalobs
TIMEOUT=60
Starter kvalobs dette kan ta noe tid!
Hvis det ikke skjer noe på MER enn 60 sekund
bruk CTRL-C for å avbryte!
Starter kvQabased ....running
Starter kvManagerd ....running
Starter kvDataInputd ....running
Starter kvServiced ....running
Starter kvAgregated ....running
Starter kvsynopd ....running
Starter norcom2kv ....running
Starter kvqc2 ....Ok!

Stop kvqc2 {use whatever is the operational procedure to stop an individual daemon}

Step 3	Run “kvqc2” from command line

Run from the command line and check output

kvalobs@dev-vm101:~$ kvqc2 # NB the s/w is installed under /usr/bin

Note the following features:

-- kvqc2-1.0.1: starting .... the new version is installed

Logging to file </var/log/kvalobs/Qc2.log>!

Scanning For Files {repeated} ... i.e. no configuration files are active

All of the above can be seen below!

kvalobs@dev-vm101:~$ which kvqc2
/usr/bin/kvqc2
kvalobs@dev-vm101:~$ kvqc2
INFO:
-- Reading configuration from file </etc/kvalobs/kvalobs.conf>!
INFO:
-- Configuration file loaded!
INFO:
-- Using 'database.dbconnect' from configuration file
FROM CONFIGURATION FILE: pgdriver.so
Logging to file </var/log/kvalobs/Qc2.log>!
INFO: (Qc2 ...)
-- kvqc2-1.0.1: starting ....
INFO: (Qc2 ...)
-- Using <kvtest-dev-vm101/> as path in CORBA nameserver
INFO: (Qc2 ...)
-- Using CORBA nameserver at: corbans.met.no
INFO: (Qc2 ...)
-- Loading driver for database engine </usr/lib/kvalobs/db/pgdriver.so>!
INFO: (Qc2 ...)
-- Driver <PostgreSQL> loaded!
INFO: (Qc2 ...)
-- Writing pid to file </var/run/kvalobs/kvqc2-dev-vm101.pid>!
INFO:
-- Qc2Work: starting work thread!
INFO:
-- New database connection (PostgreSQL) created!
DEBUG:
-- Created a new connection to the database!
INFO:
-- %%%%%%%%%%%%%%%%%%%%%%%%
Scanning For Files
Scanning For Files
Scanning For Files

Step 3	Check configuration files

Create an empty configuration file:

# As user kvalobs
kvalobs@dev-vm101:~$ touch /etc/kvalobs/Qc2Config/BlankFile.cfg

Wait for up to 1 minutes and then note the output from kvqc2:

Scanning For Files
Scanning For Files
Configuration File Found: /etc/kvalobs/Qc2Config/BlankFile.cfg
2010-06-22 20:10:17: 2010-06-22 20:00:00 -> 2010-06-22 20:00:00 /etc/kvalobs/Qc2Config/BlankFile.cfg

clean up by remoiving the blank file

kvalobs@dev-vm101:~$ rm /etc/kvalobs/Qc2Config/BlankFile.cfg

Stop (Ctrl C) the running process on the command line (NB may have to wait 1 minute) and then restart as the daemon process with “kvstart”.

Step 4	kvstart

Load Configuration files

Step 5	Install the default configuration files

The configuration files are available from the subversion repository in the src/Reference directory of the distribution https://svn.met.no/viewvc/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/.

To load the files, e.g. copy the links and use wget:

# As user kvalobs
$ cd /etc/kvalobs/Qc2Config
$ wget https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/DailyProcessUnitMissing.cfg --no-check-certificate
$ wget https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/MissingLinear.hold --no-check-certificate
$ wget https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/SingleMaxMinAverage.hold --no-check-certificate
$ wget https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/TEST3036.hold --no-check-certificate

Step 6	Deploy DailyProcessUnitMissing.cfg

In operations this is an example of the default configuration file. At a specified time each day it will set an algorithm to process the last N days of data.

Default setting is last 3 days.

Monitor output with

$ tail -f /var/log/kvalobs/Qc2.log

Edit the configuration file and set to run during the next minute, i.e. chnage the following entries:

...
RunAtHour=21
RunAtMinute=34
...

NB Times in UTC!!!

Here are some observations:

kvalobs@dev-vm101:~$ tail -f /var/log/kvalobs/Qc2.log
...
20100622221439: INFO
--------------- %%%%%%%%%%%%%%%%%%%%%%%%
20100622235953: INFO
--------------- Case 10: Single Linear
20100622235953: INFO
--------------- Single Linear

Then ... nothing else happened. Note strict filters are being applied to the data. Now rerun but remove the A_fqclevel=0 setting. i.e. in DailyProcessUnitMissing.cfg:

#A_fqclevel=0

Reset the run time and observe Qc2.log output.

--------------- Case 10: Single Linear
20100623000814: INFO
--------------- Single Linear
20100623000815: INFO
--------------- ProcessUnitT Writing Data 1.9 99754 2010-6-22 20:0:0
20100623000815: INFO
--------------- ProcessUnitT Writing Data 13.1 17150 2010-6-22 20:0:0
20100623000816: INFO
--------------- ProcessUnitT Writing Data 2.2 99754 2010-6-22 18:0:0
20100623000817: INFO
--------------- ProcessUnitT Writing Data 15.2 17150 2010-6-22 17:0:0
20100623000818: INFO
--------------- ProcessUnitT Writing Data 3.5 99754 2010-6-22 13:0:0
20100623000819: INFO
--------------- ProcessUnitT Writing Data 15.5 17150 2010-6-22 12:0:0
20100623000819: INFO
--------------- ProcessUnitT Writing Data 15.2 17150 2010-6-22 10:0:0
20100623000820: INFO
--------------- ProcessUnitT Writing Data 5.5 76905 2010-6-22 7:0:0
20100623000825: INFO
--------------- ProcessUnitT Writing Data 22.6 77007 2010-6-21 13:0:0
20100623000825: INFO
--------------- ProcessUnitT Writing Data 8.4 76930 2010-6-21 13:0:0
20100623000827: INFO
--------------- ProcessUnitT Writing Data 7.8 76928 2010-6-21 7:0:0
20100623000831: INFO
--------------- ProcessUnitT Writing Data 7.2 76928 2010-6-20 23:0:0

Pick a random entry from above and check the output in the kvalobs databse:

kvalobs@dev-vm101:~$ psql kvalobs
Welcome to psql 8.3.3, the PostgreSQL interactive terminal.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit
kvalobs=# select * from data where obstime='2010-06-21 13:00:00' and stationid=76930 and paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
76930 | 2010-06-21 13:00:00 | -32767 | 211 | 2010-06-21 13:30:06 | 22 | 0 | 0 | 8.4 | 0000001100000000 | 9894900000000000 | QC2d-2
(1 row)
kvalobs=# select * from data where obstime between '2010-06-21 12:00:00' and '2010-06-21 14:00:00' and stationid=76930 and paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
76930 | 2010-06-21 12:00:00 | 7.6 | 211 | 2010-06-21 12:36:27 | 11 | 0 | 0 | 7.6 | 0110000000100000 | 7000000000000000 |
76930 | 2010-06-21 13:00:00 | -32767 | 211 | 2010-06-21 13:30:06 | 22 | 0 | 0 | 8.4 | 0000001100000000 | 9894900000000000 | QC2d-2
76930 | 2010-06-21 14:00:00 | 9.1 | 211 | 2010-06-21 14:24:02 | 22 | 0 | 0 | 9.1 | 0110000000000000 | 7100000400000000 |
(3 rows)
kvalobs=#

Note: After seeing the above info I added CfailedString=“MIST” to the cfg file.

NB Lots of work to be done tuning the control flag settings etc!!!

Further testing

Step 7	Opportunity for ad hoc testing

Rerun the algorithm over a longer time interval, i.e. set Last_NDays to a value > 3

Observe output, repeat tests for different controlflag filter settings.

The algorithm checks <<!CheckFlags.condition(is->controlinfo(),params.Aflag)>> that none of the analysis flag settings are true, i.e. the settings in the config file prefixed with “A_”

Set the algorithm to run at the end of the day (at the time of writing it is set to 22:09 UTC).

During the working day the algorithm has not yet acted on any of the recent data. One can add a missing value for a specific case in the test database and then observe the results later. This is a laborious endeavour but allows very specific tests to be carried out.

Step 8	Bulk testing

On dev-vm101 the directory /home/kvalobs/TESTDATA contains test data for March 2036.

data-203603-GeneralSingleMissing-P7.dat	All of March 2036
ThreeDays.dat	Approximately the first three days

On each day for all stations the “original” hourly temperature has been set to a missing value -32767 for the times 05:00, 11:00, 17:00 and 23:00 UT and the original value is kept for reference, in curly bracket in CFAILED.

|12320|2036-03-01 11:00:00|-32767|211|2036-03-01 11:06:53|330|0|0|2.3|1111100000000010|7000000000000000| {2.3}

To use the testdata, clean up old entries in the kvalobs db with commands like:

sql> DELETE FROM data WHERE obstime between '2036-02-28' and '2036-04-01';

Reload test data:

sql> \copy data FROM '/home/kvalobs/TESTDATA/data-203603-GeneralSingleMissing-P7.dat' WITH DELIMITER AS '|'

And capture results with

sql> select * from data WHERE obstime between '2036-02-28' and '2036-04-01' and cfailed like '%QC2d-2%' \g | cat >> FullMissingLinear.dat;

EXAMPLE TEST

8i	Clean up previous entries
$psql kvalobs
sql > DELETE FROM data WHERE obstime between '2036-02-28' and '2036-04-01';
8ii	Load test data

sql > \copy data FROM '/home/kvalobs/TESTDATA/data-203603-GeneralSingleMissing-P7.dat' WITH DELIMITER AS '|'

8iii	/etc/kvalobs/Qc2Config/MissingLinear.hold is set up for this test ...
edit the file to change run time to the next minute, and then:
$ mv /etc/kvalobs/Qc2Config/MissingLinear.hold /etc/kvalobs/Qc2Config/MissingLinear.cfg
monitor output
tail -f /var/log/kvalobs/Qc2.log
NB This is a heavy test and the log file will fill and rotate, so “Ctrl C” the above command and restart a few times, otherwise it looks like the program has stalled
8iv	When finished

select * from data WHERE obstime between '2036-02-28' and '2036-04-01' and paramid=211 and cfailed like '%QC2d-2%' \g | cat >> FullMissingLinear.dat;

Also, put the algorithm back on hold since it will try and run everyday ...
$ mv /etc/kvalobs/Qc2Config/MissingLinear.cfg /etc/kvalobs/Qc2Config/MissingLinear.hold

ANALYSIS

cat FullMissingLinear.dat | grep -v "rows)" | sed '/^$/d' | sed '/--/d' | sed '/stationid/d' | sed 's/^.\{103\}//' | sed 's/|.*{/ /' | sed 's/}.*//' | sed '/-32767/d' > XY.dat

The above command just strips out the two columns of data to compare, e.g.just looking at the first lines

cat FullMissingLinear.dat | grep -v "rows)" | sed '/^$/d' | sed '/--/d' | sed '/stationid/d' | sed 's/^.\{103\}//' | sed 's/|.*{/ /' | sed 's/}.*//' | sed '/-32767/d' | head

gives:

3.2 5.4
8.1 8.7
2 2.3
6.9 6.7
5.8 6.5
1 0.9
6.6 6.9
6.8 6.63
6.9 6.86
6.5 6.68

Values to be rounded to 1 decimal place. Investigate and fix.

Then plot in R: (BE CAREFUL copying and pasting the long lines !!!)

$ R
FileName="XY.dat"
dim=100 # make dim smaller to zoom into the middle of the plot
UTD <- read.table(FileName, header=TRUE)
#jpeg(filename=paste(FileName,"jpg",sep="."))
pdf()
plot(UTD[,1],UTD[,2],ylim=c(-dim,dim),xlim=c(-dim,dim),xlab='Qc2 Algorithm/Celsius',ylab='Original/Celsius', main='200803 Hourly Temperature', sub='Missing Value Correction',col=4)
# pch=46 for dots
lines(c(-dim,dim),c(-dim,dim),col=3)
lines(c(-dim-10,dim+10),c(0,0))
lines(c(0,0),c(-dim-10,dim+10))
lines(c(-dim-10,dim+10),c(-dim-10,dim+10),col=3)
dev.off()
#To exit R:
quit()
# NB the script creates the plot "Rplots.pdf" picture below (converted to jpg for the wiki).

More examples of other parameters available at:

https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:test:algorithms:singlemissingpoint

--- 2010/06/22 23:25