Differences

This shows you the differences between two versions of the page.

--- kvalobs:kvoss:system:qc2:test:vm101notes [2010-06-22 23:01:31]
paule
+++ kvalobs:kvoss:system:qc2:test:vm101notes [2022-05-31 09:29:32] (current)
@@ Line 1: / Line 1: @@
-====== Installation and test on dev-vm101 notes ======
+====== Installation and test on dev-vm101 notes [kvqc2_1.0.1 and other revisions ... ] ======
 The following notes describe the installation and use of **kvqc2 version 1.0.1** Sources: https://svn.met.no/viewvc/kvoss/kvQc2/branches/kvqc2-1.0.1/
+**Algorithms in this release:**
+^ AlgoCode ^Description  ^Status  ^
+| 9 | SingleMinMaxAverage | **kvqc2_1.0.1** For a given paramid and corresponding Max paramid and Min paramid calculates a correction for a single missing value as average of max and min. If no max or min available or specified then reverts to simple linear. |
+| 10 | SingleLinear |**kvqc2_1.0.1** Replaces a single missing value with a linear interpolated value. If specified checks that the correction lies within the available max ... min range, if not sets to nearest of max or min. Run for any paramid and optional max, min specified in the configuration file. |
 ===== Installation =====
@@ Line 8: / Line 14: @@
   * Log into dev-vm101 as user kvalobs
-  * If an old kvqc2 exists stop the running kvqc2 process
+  * If an old kvqc2 exists stop the running kvqc2 process by 'kvstop -l' (lower case L), then type the number corresponding to kvqc2
   * Remove active configuration files (this is to just ensure that when the new kvqc2 version is set running it does not find and use an old configuration file)
@@ Line 14: / Line 20: @@
 <code>
 $ cd /etc/kvalobs/Qc2Config
-$ mv ProcessUnit.cfg ProcessUnit.cfg
+$ mv XXX.cfg XXX.hold
 $ mv ProcessUnit.cfg ProcessUnit.hold
 </code>
@@ Line 22: / Line 28: @@
   * Install the latest version of kvqc2
   * Log in as a user with sudoer permission
+  * NB one may need to run "apt-get update" first in order to update the package list.
 <code>
@@ Line 27: / Line 34: @@
 </code>
-Note: the above is the standard way to do this, but did not work. Here is the trace (this is a separate issue to work on!!!)
+An alternative is to install the new kvqc2 directly from where the debian package was built on dev-vm101:
-<code>
-paule@dev-vm101:~$ apt-cache search kvqc2
-kvqc2 - Non-real-time and geo-statistical quality controls.
-paule@dev-vm101:~$ sudo apt-get install kvqc2
-Reading package lists... Done
-Building dependency tree... Done
-kvqc2 is already the newest version.
-You might want to run `apt-get -f install' to correct these:
-The following packages have unmet dependencies:
-  libkvcpp-dev: Depends: libkvcpp1 (= 2.1.3-1) but 2.1.4-1 is to be installed
-  libkvcpp1: Depends: libkvalobs1 (= 2.1.4-1) but 2.1.4-3 is to be installed
-E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
-paule@dev-vm101:~$
-</code>
-Instead installed the new kvqc2 directly from where the debian package was built on dev-vm101:
 i.e. using "sudo dpkg -i kvqc2_1.0.1-1_i386.deb" as illustrated below.
@@ Line 59: / Line 49: @@
 Setting up kvqc2 (1.0.1-1) ...
 paule@dev-vm101:~/kvqc2-1.0.1$
+</code>
+**Bug Fix and Confirmation of apt-get install**
+Initial tests illustrated that some data was written back to thedatabase at too high a precision. This is fixed in revision 1782: https://svn.met.no/viewvc/kvoss?view=rev&revision=1782 and led to a kvqc2-1.0.2 revision. This also provided the opportunity to check that the install from apt-get install works, i.e.:
+<code>
+paule@dev-vm101:~$ sudo apt-get install kvqc2
+Reading package lists... Done
+Building dependency tree... Done
+The following packages will be upgraded:
+  kvqc2
+upgraded, 0 newly installed, 0 to remove and 78 not upgraded.
+Need to get 288kB of archives.
+After unpacking 0B of additional disk space will be used.
+Get:1 http://repo.met.no etch/main kvqc2 1.0.2-1 [288kB]
+Fetched 288kB in 0s (12.7MB/s)
+(Reading database ... 52403 files and directories currently installed.)
+Preparing to replace kvqc2 1.0.1-1 (using .../kvqc2_1.0.2-1_i386.deb) ...
+Unpacking replacement kvqc2 ...
+Setting up kvqc2 (1.0.2-1) ...
+paule@dev-vm101:~$
 </code>
@@ Line 99: / Line 111: @@
 <code>
-kvalobs@dev-vm101:~$ kvstart
+kvalobs@dev-vm101:~$ kvqc2  # NB the s/w is installed under /usr/bin
 </code>
@@ Line 170: / Line 182: @@
 </code>
-   * Stop (Ctrl^C) the running process (NB may have to wait 1 minute) and then restart as the daemon process with "kvstart".
+   * Stop (Ctrl C) the running process on the command line (NB may have to wait 1 minute) and then restart as the daemon process with "kvstart".
 ^Step 4 ^kvstart^
@@ Line 294: / Line 306: @@
 Note: After seeing the above info I added CfailedString="MIST" to the cfg file.
+**NB** Lots of work to be done tuning the control flag settings etc!!! :-)
+**NB** Currently only work on data where "qclevel NOT EQUAL 0" **Therefore it is vital that a qc1 algorithm has checked the data ... otherwise there will be no Qc2 check** suspect that this might be the case. How often? How better to tune?
 ==== Further testing ====
@@ Line 300: / Line 316: @@
   * Rerun the algorithm over a longer time interval, i.e. set Last_NDays to a value > 3
-  * Observe output, repeat tests for different controlfalg filter settings.
+  * Observe output, repeat tests for different controlflag filter settings.
 The algorithm checks <<!CheckFlags.condition(is->controlinfo(),params.Aflag)>> that none of the analysis flag settings are true, i.e. the settings in the config file prefixed with "A_"
@@ Line 330: / Line 346: @@
 And capture results with
 <code>
-sql> select * from data WHERE obstime between '2036-02-28' and '2036-04-01' and cfailed like 'QC2d-2'  \g | cat >> FullMissingLinear.dat;
+sql> select * from data WHERE obstime between '2036-02-28' and '2036-04-01' and cfailed like '%QC2d-2%'  \g | cat >> FullMissingLinear.dat;
 </code>
-The following scripts can help process the data.
-EXAMPLE TEST
+**EXAMPLE TEST**
 |8i |Clean up previous entries |
 |   |$psql kvalobs  |
 |   |sql > DELETE FROM data WHERE obstime between '2036-02-28' and '2036-04-01';  |
+|8ii |Load test data |
+<code>sql > \copy data FROM '/home/kvalobs/TESTDATA/data-203603-GeneralSingleMissing-P7.dat' WITH DELIMITER AS '|'</code>
+|8iii |/etc/kvalobs/Qc2Config/MissingLinear.hold is set up for this test ... |
+|  |edit the file to change run time to the next minute, and then: |
+|  |       $ mv /etc/kvalobs/Qc2Config/MissingLinear.hold  /etc/kvalobs/Qc2Config/MissingLinear.cfg|
+|  |monitor output  |
+|  |    tail -f /var/log/kvalobs/Qc2.log|
+|  |NB This is a heavy test and the log file will fill and rotate, so "Ctrl C" the above command and restart a few times, otherwise it looks like the program has stalled  |
+|8iv |When finished  |
+<code> select * from data WHERE obstime between '2036-02-28' and '2036-04-01' and paramid=211 and cfailed like '%QC2d-2%'  \g | cat >> FullMissingLinear.dat;</code>
+|  |Also, put the algorithm back on hold since it will try and run everyday ...|
+|  |       $ mv /etc/kvalobs/Qc2Config/MissingLinear.cfg  /etc/kvalobs/Qc2Config/MissingLinear.hold|
+**ANALYSIS**
+<code>
+cat FullMissingLinear.dat | grep -v "rows)" | sed '/^$/d' | sed '/--/d' | sed '/stationid/d' | sed 's/^.\{103\}//' | sed 's/|.*{/ /' | sed 's/}.*//' | sed '/-32767/d' >  XY.dat
+</code>
+The above command just strips out the two columns of data to compare, e.g.just looking at the first lines
+<code>
+cat FullMissingLinear.dat | grep -v "rows)" | sed '/^$/d' | sed '/--/d' | sed '/stationid/d' | sed 's/^.\{103\}//' | sed 's/|.*{/ /' | sed 's/}.*//' | sed '/-32767/d' | head
+</code>
+gives:
+<code>
+.2  5.4
+.1  8.7
+  2.3
+.9  6.7
+.8  6.5
+  0.9
+.6  6.9
+.8  6.63
+.9  6.86
+.5  6.68
+</code>
+FIXME Values to be rounded to 1 decimal place. Investigate and fix.
+Then plot in R: (BE CAREFUL copying and pasting the long lines !!!)
+<code>
+$ R
+FileName="XY.dat"
+dim=100   # make dim smaller to zoom into the middle of the plot
+UTD <- read.table(FileName, header=TRUE)
+#jpeg(filename=paste(FileName,"jpg",sep="."))
+pdf()
+plot(UTD[,1],UTD[,2],ylim=c(-dim,dim),xlim=c(-dim,dim),xlab='Qc2 Algorithm/Celsius',ylab='Original/Celsius', main='200803 Hourly Temperature', sub='Missing Value Correction',col=4)
+# pch=46 for dots
+lines(c(-dim,dim),c(-dim,dim),col=3)
+lines(c(-dim-10,dim+10),c(0,0))
+lines(c(0,0),c(-dim-10,dim+10))
+lines(c(-dim-10,dim+10),c(-dim-10,dim+10),col=3)
+dev.off()
+#To exit R:
+quit()
+# NB the script creates the plot "Rplots.pdf" picture below (converted to jpg for the wiki).
+</code>
+{{kvalobs:kvoss:system:qc2:test:output.jpg|}}
+More examples of other parameters available at:
+^https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:test:algorithms:singlemissingpoint^
+ --- //2010/06/22 23:25//