Differences

This shows you the differences between two versions of the page.

--- kvalobs:kvoss:system:qc2:test:acceptance:alg004 [2010-01-10 22:26:05]
paule
+++ kvalobs:kvoss:system:qc2:test:acceptance:alg004 [2022-05-31 09:29:32] (current)
@@ Line 1: / Line 1: @@
 ====== ALG004 ======
-Working ... FIXME
 ===== Acceptance Test for single point missing temperature values with TAN, TAX method =====
@@ Line 7: / Line 5: @@
 ^Step ^Direction ^Expected Result ^Pass/Fail Criteria or Comment ^
 |001 | Prepare test data in the test database. | -- | -- |
-| | Download {{:kvoss:system:qc2:test:acceptance:data-202003.dat.gz|Test Data}} and gunzip. | -- | -- |
+| | Download {{kvalobs:kvoss:system:qc2:test:acceptance:alg004-kvalobsdb.dat.gz|Test Data}} and gunzip. | -- | -- |
 | | $ psql kvalobs | -- | -- |
-| | <code>DELETE FROM data WHERE obstime>'2020-02-28' and obstime<'2020-04-01';</code> | -- | -- |
+| | <code>DELETE FROM data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01';</code> | -- | -- |
-| | <code>\copy data FROM './data-202003.dat' WITH DELIMITER AS '|'</code> | -- | -- |
+| | <code>COPY data FROM '/home/kvalobs/alg004-kvalobsdb.dat' WITH DELIMITER AS '|';</code> | -- | -- |
 | | NB specify the actual path to the test data above as required. | -- | -- |
-|002 | -- | -- | -- |
+| | | -- | -- |
-|003 | -- | -- | -- |
+| | Check that the time interval contains no Qc2 processed data: | -- | -- |
-|004 | -- | -- | -- |
+| | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=0 | -- |
-|005 | -- | -- | -- |
+|002 |Download and gunzip {{kvalobs:kvoss:system:qc2:test:acceptance:alg004.cfg.gz|alg004.cfg.gz}} to the /etc/kvalobs/Qc2Config directory and edit RunAtHour and RunAtMinute fields to the time that the test shall be run (e.g. a few minutes UTC in the future). | -- | -- |
-|006 | -- | -- | -- |
+| |The configuration by default has W_fhqc=0 and Wbool=true. The algorithm will only work on data that ha not been controlled by HQC | -- | -- |
-|007 | -- | -- | -- |
-|008 | -- | -- | -- |
+The configuration file also needs to be edited to include further flag controls ( -> ) and include this:
-|009 | -- | -- | -- |
+<code>
-|010 | -- | -- | -- |
+...
+#Flag to set if value is corrected
+S_ftime=1
+change_fmis=3->1
+change_fmis=0->4
+...
+</code>
+|003 | Check log files to verify that the algorithm runs as scheduled, e.g. $ tail -f /var/log/Kvalobs/Qc2.log | -- | -- |
+|004 | Check that Qc2 processed data now resides in the database: | -- | -- |
+| | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=55 | -- |
+|005 | Delete any existing data output (rm ALG004-Step5.dat) and then EXTRACT THE DATA: |{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step5.dat.gz|DATA-S5}} | -- |
+<code>psql> select * from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'  \g | cat >> ./ALG004-Step5.dat;
+</code>
+|006 | Plot the data. Download and use the two scripts  {{kvalobs:kvoss:system:qc2:test:acceptance:unit_t_plot.bash.gz|./unit_t_plot.bash}} and {{kvalobs:kvoss:system:qc2:test:acceptance:unitt.r.gz|unitt.r.gz}}  | -- | -- |
+| |Make the bash file executable: | -- | -- |
+| |   chmod u+x unit_t_plot.bash| -- | -- |
+| |Run the script | -- | -- |
+|    |   ./unit_t_plot.bash ALG004-Step5.dat|{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step5.dat.xy.dat.jpg?100x100|}} |-- |
+|    | and inspect the produced image ALG004-Step5.dat.jpg file.  | -- |-- |
+**Alternative to step 006 for use on dev-vm101**
+These files are already installed under /home/kvalobs.
+As user kvalobs:
+<code>
+$ cd
+$ ./unit_t_plot_manual.bash ALG004-Step5.dat
+The data file ALG004-Step5.dat.XY.dat has been created ...
+Now run the following commands in R:
+--------------------------------------
+FileName="ALG004-Step5.dat.XY.dat"
+UTD <- read.table(FileName, header=TRUE)
+pdf()
+plot(UTD[,1],UTD[,2],ylim=c(-20,20),xlim=c(-20,20),xlab='Qc2 Algorithm/mm',ylab='Existing Correction/mm', main='200803 Hourly Temperature', sub='Missing Value Correction',col=4)
+lines(c(-20,20),c(-20,20),col=3)
+lines(c(-30,30),c(0,0))
+lines(c(0,0),c(-30,30))
+lines(c(-30,30),c(-30,30),col=3)
+dev.off()
+--------------------------------------
+</code>
+i.e. ... enter R:
+<code>
+$ R
+...
+> FileName="ALG004-Step5.dat.XY.dat"
+> UTD <- read.table(FileName, header=TRUE)
+> pdf()
+> plot(UTD[,1],UTD[,2],ylim=c(-20,20),xlim=c(-20,20),xlab='Qc2 Algorithm/mm',ylab='Existing Correction/mm', main='200803 Hourly Temperature', sub='Missing Value Correction',col=4)
+> lines(c(-20,20),c(-20,20),col=3)
+> lines(c(-30,30),c(0,0))
+> lines(c(0,0),c(-30,30))
+> lines(c(-30,30),c(-30,30),col=3)
+> dev.off()
+>
+> quit()
+</code>
+The above creates the file Rplots.pdf:
+<code>
+$ xpdf Rplots.pdf
+</code>
+**NB** if you have logged in using ssh -X kvalobs@dev-vm101 then the above will probably work!
+|007 | Edit the configuration file and set Wbool=false. Rerun the algorithm. The algorithm will reprocess the data flagged as controlled by HQC already. | -- | -- |
+| | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=121 | -- |
+| |Capture the results. (rm ALG004-Step7.dat any previous test results) |{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step7.dat.gz|DATA-S7}} | -- |
+<code>psql> select * from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'  \g | cat >> ./ALG004-Step7.dat;
+</code>
+|008 | Plot results as in step 006 above. | {{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step7.dat.xy.dat.jpg?100x100|}} | -- |
 **Prerequisites**
@@ Line 26: / Line 103: @@
 Set up the test data (these are prepatory notes for now for set up on pak)
-{{:kvoss:system:qc2:test:acceptance:data-202003.dat.gz|Test Data}} comprising original data from 2008-03 with all the years set to 2020.
+{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-kvalobsdb.dat.gz|Test Data}} comprising original data from 2008-03 with all the years set to 2024, the file only contains the parameters for the algorithm.
+{{kvalobs:kvoss:system:qc2:test:acceptance:alg004.cfg.gz|Configuration File}}
+Setting-up or refreshing the database. (some database management may be required to stop page errors and the like ????, vacuum cleaning etc ???)
   * $ psql kvalobs
-  * psql=> DELETE FROM data WHERE obstime>'2020-02-28' and obstime<'2020-04-01';
+  * psql=> DELETE FROM data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01';
-  * psql=> \copy data FROM data-200803/data-202003.dat WITH DELIMITER AS '|';
+  * psql=> COPY data FROM ./alg004-kvalobsdb.dat WITH DELIMITER AS '|';
+Script for plotting the test results. See the Step # above: {{kvalobs:kvoss:system:qc2:test:acceptance:unit_t_plot.bash.gz|}}
+... and not to forget the corresponding R-script called by the shell script: {{kvalobs:kvoss:system:qc2:test:acceptance:unitt.r.gz|}}