Installation and test on dev-vm101 notes

The following notes describe the installation and use of kvqc2 version 1.0.1 Sources: https://svn.met.no/viewvc/kvoss/kvQc2/branches/kvqc2-1.0.1/

Installation

Step 1	Installation

Log into dev-vm101 as user kvalobs

If an old kvqc2 exists stop the running kvqc2 process {e.g. use kvstop ? terminate the process pid ?}

Remove active configuration files (this is to just ensure that when the new kvqc2 version is set running it does not find and use an old configuration file)

For example:

$ cd /etc/kvalobs/Qc2Config
$ mv XXX.cfg XXX.hold
$ mv ProcessUnit.cfg ProcessUnit.hold

( PS NOTE: actually now all the earliy config files have been moved to /etc/kvalobs/Qc2Config/old )

Install the latest version of kvqc2

Log in as a user with sudoer permission

$ sudo apt-get install kvqc2

Note: the above is the standard way to do this, but did not work. Here is the trace (this is a separate issue to work on!!!) Investigate and fix.

paule@dev-vm101:~$ apt-cache search kvqc2
kvqc2 - Non-real-time and geo-statistical quality controls.
paule@dev-vm101:~$ sudo apt-get install kvqc2
Reading package lists... Done
Building dependency tree... Done
kvqc2 is already the newest version.
You might want to run `apt-get -f install' to correct these:
The following packages have unmet dependencies:
libkvcpp-dev: Depends: libkvcpp1 (= 2.1.3-1) but 2.1.4-1 is to be installed
libkvcpp1: Depends: libkvalobs1 (= 2.1.4-1) but 2.1.4-3 is to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
paule@dev-vm101:~$

Instead installed the new kvqc2 directly from where the debian package was built on dev-vm101:

i.e. using “sudo dpkg -i kvqc2_1.0.1-1_i386.deb” as illustrated below.

THIS WAY WORKS FOR NOW

paule@dev-vm101:~/kvqc2-1.0.1$ ls
kvqc2-1.0.1 kvqc2_1.0.1-1_i386.changes kvqc2_1.0.1.orig.tar.gz
kvqc2_1.0.1-1.diff.gz kvqc2_1.0.1-1_i386.deb kvqc2-1.0.1.tar.gz
kvqc2_1.0.1-1.dsc kvqc2_1.0.1-1_i386.upload
paule@dev-vm101:~/kvqc2-1.0.1$ sudo dpkg -i kvqc2_1.0.1-1_i386.deb
(Reading database ... 52403 files and directories currently installed.)
Preparing to replace kvqc2 1.0.0-1 (using kvqc2_1.0.1-1_i386.deb) ...
Unpacking replacement kvqc2 ...
Setting up kvqc2 (1.0.1-1) ...
paule@dev-vm101:~/kvqc2-1.0.1$

Run and initial tests

Step 2	Run with “kvstart”

“kvstart” should start the new version, just test to see if this works:

Login as user kvalobs and then:

kvalobs@dev-vm101:~$ kvstart
KVBIN=/usr/bin
KVPID=/var/run/kvalobs
TIMEOUT=60
Starter kvalobs dette kan ta noe tid!
Hvis det ikke skjer noe på MER enn 60 sekund
bruk CTRL-C for å avbryte!
Starter kvQabased ....running
Starter kvManagerd ....running
Starter kvDataInputd ....running
Starter kvServiced ....running
Starter kvAgregated ....running
Starter kvsynopd ....running
Starter norcom2kv ....running
Starter kvqc2 ....Ok!

Stop kvqc2 {use whatever is the operational procedure to stop an individual daemon}

Step 3	Run “kvqc2” from command line

Run from the command line and check output

kvalobs@dev-vm101:~$ kvstart

Note the following features:

-- kvqc2-1.0.1: starting .... the new version is installed

Logging to file </var/log/kvalobs/Qc2.log>!

Scanning For Files {repeated} ... i.e. no configuration files are active

All of the above can be seen below!

kvalobs@dev-vm101:~$ which kvqc2
/usr/bin/kvqc2
kvalobs@dev-vm101:~$ kvqc2
INFO:
-- Reading configuration from file </etc/kvalobs/kvalobs.conf>!
INFO:
-- Configuration file loaded!
INFO:
-- Using 'database.dbconnect' from configuration file
FROM CONFIGURATION FILE: pgdriver.so
Logging to file </var/log/kvalobs/Qc2.log>!
INFO: (Qc2 ...)
-- kvqc2-1.0.1: starting ....
INFO: (Qc2 ...)
-- Using <kvtest-dev-vm101/> as path in CORBA nameserver
INFO: (Qc2 ...)
-- Using CORBA nameserver at: corbans.met.no
INFO: (Qc2 ...)
-- Loading driver for database engine </usr/lib/kvalobs/db/pgdriver.so>!
INFO: (Qc2 ...)
-- Driver <PostgreSQL> loaded!
INFO: (Qc2 ...)
-- Writing pid to file </var/run/kvalobs/kvqc2-dev-vm101.pid>!
INFO:
-- Qc2Work: starting work thread!
INFO:
-- New database connection (PostgreSQL) created!
DEBUG:
-- Created a new connection to the database!
INFO:
-- %%%%%%%%%%%%%%%%%%%%%%%%
Scanning For Files
Scanning For Files
Scanning For Files

Step 3	Check configuration files

Create an empty configuration file:

# As user kvalobs
kvalobs@dev-vm101:~$ touch /etc/kvalobs/Qc2Config/BlankFile.cfg

Wait for up to 1 minutes and then note the output from kvqc2:

Scanning For Files
Scanning For Files
Configuration File Found: /etc/kvalobs/Qc2Config/BlankFile.cfg
2010-06-22 20:10:17: 2010-06-22 20:00:00 -> 2010-06-22 20:00:00 /etc/kvalobs/Qc2Config/BlankFile.cfg

clean up by remoiving the blank file

kvalobs@dev-vm101:~$ rm /etc/kvalobs/Qc2Config/BlankFile.cfg

Stop (Ctrl C) the running process on the command line (NB may have to wait 1 minute) and then restart as the daemon process with “kvstart”.

Step 4	kvstart

Load Configuration files

Step 5	Install the default configuration files

The configuration files are available from the subversion repository in the src/Reference directory of the distribution https://svn.met.no/viewvc/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/.

To load the files, e.g. copy the links and use wget:

# As user kvalobs
$ cd /etc/kvalobs/Qc2Config
$ wget https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/DailyProcessUnitMissing.cfg --no-check-certificate
$ wget https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/MissingLinear.hold --no-check-certificate
$ wget https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/SingleMaxMinAverage.hold --no-check-certificate
$ wget https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/TEST3036.hold --no-check-certificate

Step 6	Deploy DailyProcessUnitMissing.cfg

In operations this is an example of the default configuration file. At a specified time each day it will set an algorithm to process the last N days of data.

Default setting is last 3 days.

Monitor output with

$ tail -f /var/log/kvalobs/Qc2.log

Edit the configuration file and set to run during the next minute, i.e. chnage the following entries:

...
RunAtHour=21
RunAtMinute=34
...

NB Times in UTC!!!

Here are some observations:

kvalobs@dev-vm101:~$ tail -f /var/log/kvalobs/Qc2.log
...
20100622221439: INFO
--------------- %%%%%%%%%%%%%%%%%%%%%%%%
20100622235953: INFO
--------------- Case 10: Single Linear
20100622235953: INFO
--------------- Single Linear

Then ... nothing else happened. Note strict filters are being applied to the data. Now rerun but remove the A_fqclevel=0 setting. i.e. in DailyProcessUnitMissing.cfg:

#A_fqclevel=0

Reset the run time and observe Qc2.log output.

--------------- Case 10: Single Linear
20100623000814: INFO
--------------- Single Linear
20100623000815: INFO
--------------- ProcessUnitT Writing Data 1.9 99754 2010-6-22 20:0:0
20100623000815: INFO
--------------- ProcessUnitT Writing Data 13.1 17150 2010-6-22 20:0:0
20100623000816: INFO
--------------- ProcessUnitT Writing Data 2.2 99754 2010-6-22 18:0:0
20100623000817: INFO
--------------- ProcessUnitT Writing Data 15.2 17150 2010-6-22 17:0:0
20100623000818: INFO
--------------- ProcessUnitT Writing Data 3.5 99754 2010-6-22 13:0:0
20100623000819: INFO
--------------- ProcessUnitT Writing Data 15.5 17150 2010-6-22 12:0:0
20100623000819: INFO
--------------- ProcessUnitT Writing Data 15.2 17150 2010-6-22 10:0:0
20100623000820: INFO
--------------- ProcessUnitT Writing Data 5.5 76905 2010-6-22 7:0:0
20100623000825: INFO
--------------- ProcessUnitT Writing Data 22.6 77007 2010-6-21 13:0:0
20100623000825: INFO
--------------- ProcessUnitT Writing Data 8.4 76930 2010-6-21 13:0:0
20100623000827: INFO
--------------- ProcessUnitT Writing Data 7.8 76928 2010-6-21 7:0:0
20100623000831: INFO
--------------- ProcessUnitT Writing Data 7.2 76928 2010-6-20 23:0:0

Pick a random entry from above and check the output in the kvalobs databse:

kvalobs@dev-vm101:~$ psql kvalobs
Welcome to psql 8.3.3, the PostgreSQL interactive terminal.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit
kvalobs=# select * from data where obstime='2010-06-21 13:00:00' and stationid=76930 and paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
76930 | 2010-06-21 13:00:00 | -32767 | 211 | 2010-06-21 13:30:06 | 22 | 0 | 0 | 8.4 | 0000001100000000 | 9894900000000000 | QC2d-2
(1 row)
kvalobs=# select * from data where obstime between '2010-06-21 12:00:00' and '2010-06-21 14:00:00' and stationid=76930 and paramid=211;
stationid | obstime | original | paramid | tbtime | typeid | sensor | level | corrected | controlinfo | useinfo | cfailed
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------
76930 | 2010-06-21 12:00:00 | 7.6 | 211 | 2010-06-21 12:36:27 | 11 | 0 | 0 | 7.6 | 0110000000100000 | 7000000000000000 |
76930 | 2010-06-21 13:00:00 | -32767 | 211 | 2010-06-21 13:30:06 | 22 | 0 | 0 | 8.4 | 0000001100000000 | 9894900000000000 | QC2d-2
76930 | 2010-06-21 14:00:00 | 9.1 | 211 | 2010-06-21 14:24:02 | 22 | 0 | 0 | 9.1 | 0110000000000000 | 7100000400000000 |
(3 rows)
kvalobs=#

Note: After seeing the above info I added CfailedString=“MIST” to the cfg file.

Further testing

Step 7	Opportunity for ad hoc testing

Rerun the algorithm over a longer time interval, i.e. set Last_NDays to a value > 3

Observe output, repeat tests for different controlflag filter settings.

The algorithm checks <<!CheckFlags.condition(is->controlinfo(),params.Aflag)>> that none of the analysis flag settings are true, i.e. the settings in the config file prefixed with “A_”

Set the algorithm to run at the end of the day (at the time of writing it is set to 22:09 UTC).

During the working day the algorithm has not yet acted on any of the recent data. One can add a missing value for a specific case in the test database and then observe the results later. This is a laborious endeavour but allows very specific tests to be carried out.

Step 8	Bulk testing

On dev-vm101 the directory /home/kvalobs/TESTDATA contains test data for March 2036.

data-203603-GeneralSingleMissing-P7.dat	All of March 2036
ThreeDays.dat	Approximately the first three days

On each day for all stations the “original” hourly temperature has been set to a missing value -32767 for the times 05:00, 11:00, 17:00 and 23:00 UT and the original value is kept for reference, in curly bracket in CFAILED.

|12320|2036-03-01 11:00:00|-32767|211|2036-03-01 11:06:53|330|0|0|2.3|1111100000000010|7000000000000000| {2.3}

To use the testdata, clean up old entries in the kvalobs db with commands like:

sql> DELETE FROM data WHERE obstime between '2036-02-28' and '2036-04-01';

Reload test data:

sql> \copy data FROM '/home/kvalobs/TESTDATA/data-203603-GeneralSingleMissing-P7.dat' WITH DELIMITER AS '|'

And capture results with

sql> select * from data WHERE obstime between '2036-02-28' and '2036-04-01' and cfailed like '%QC2d-2%' \g | cat >> FullMissingLinear.dat;

EXAMPLE TEST

8i	Clean up previous entries
$psql kvalobs
sql > DELETE FROM data WHERE obstime between '2036-02-28' and '2036-04-01';
8ii	Load test data

sql > \copy data FROM '/home/kvalobs/TESTDATA/data-203603-GeneralSingleMissing-P7.dat' WITH DELIMITER AS '|'

8iii	/etc/kvalobs/Qc2Config/MissingLinear.hold is set up for this test ...
edit the file to change run time to the next minute, and then:
$ mv /etc/kvalobs/Qc2Config/MissingLinear.hold /etc/kvalobs/Qc2Config/MissingLinear.cfg
monitor output
tail -f /var/log/kvalobs/Qc2.log
NB This is a heavy test and the log file will fill and rotate, so “Ctrl C” the above command and restart a few times, otherwise it looks like the program has stalled
8iv	When finished

select * from data WHERE obstime between '2036-02-28' and '2036-04-01' and paramid=211 and cfailed like '%QC2d-2%' \g | cat >> FullMissingLinear.dat;

Also, put the algorithm back on hold since it will try and run everyday ...
$ mv /etc/kvalobs/Qc2Config/MissingLinear.cfg /etc/kvalobs/Qc2Config/MissingLinear.hold

ANALYSIS

cat FullMissingLinear.dat | grep -v "rows)" | sed '/^$/d' | sed '/--/d' | sed '/stationid/d' | sed 's/^.\{103\}//' | sed 's/|.*{/ /' | sed 's/}.*//' | sed '/-32767/d' > XY.dat

The above command just strips out the two columns of data to compare, e.g.just looking at the first lines

cat FullMissingLinear.dat | grep -v "rows)" | sed '/^$/d' | sed '/--/d' | sed '/stationid/d' | sed 's/^.\{103\}//' | sed 's/|.*{/ /' | sed 's/}.*//' | sed '/-32767/d' | head

gives:

3.2 5.4
8.1 8.7
2 2.3
6.9 6.7
5.8 6.5
1 0.9
6.6 6.9
6.8 6.63
6.9 6.86
6.5 6.68

Values to be rounded to 1 decimal place. Investigate and fix.

Then plot in R: (BE CAREFUL copying and pasting the long lines !!!)

$ R
FileName="XY.dat"
dim=100 # make dim smaller to zoom into the middle of the plot
UTD <- read.table(FileName, header=TRUE)
#jpeg(filename=paste(FileName,"jpg",sep="."))
pdf()
plot(UTD[,1],UTD[,2],ylim=c(-dim,dim),xlim=c(-dim,dim),xlab='Qc2 Algorithm/Celsius',ylab='Original/Celsius', main='200803 Hourly Temperature', sub='Missing Value Correction',col=4)
# pch=46 for dots
lines(c(-dim,dim),c(-dim,dim),col=3)
lines(c(-dim-10,dim+10),c(0,0))
lines(c(0,0),c(-dim-10,dim+10))
lines(c(-dim-10,dim+10),c(-dim-10,dim+10),col=3)
dev.off()
#To exit R:
quit()
# NB the script creates the plot "Rplots.pdf" picture below (converted to jpg for the wiki).

More to follow ??? --- 2010/06/22 23:25