ALG004
Acceptance Test for single point missing temperature values with TAN, TAX method
| Step | Direction | Expected Result | Pass/Fail Criteria or Comment |
|---|---|---|---|
| 001 | Prepare test data in the test database. | – | – |
| Download Test Data and gunzip. | – | – | |
| $ psql kvalobs | – | – | |
DELETE FROM data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01'; | – | – | |
COPY data FROM '/home/kvalobs/alg004-kvalobsdb.dat' WITH DELIMITER AS '|'; | – | – | |
| NB specify the actual path to the test data above as required. | – | – | |
| – | – | ||
| Check that the time interval contains no Qc2 processed data: | – | – | |
| kvalobs=# select count(*) from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=0 | – | |
| 002 | Download and gunzip alg004.cfg.gz to the /etc/kvalobs/Qc2Config directory and edit RunAtHour and RunAtMinute fields to the time that the test shall be run (e.g. a few minutes UTC in the future). | – | – |
| The configuration by default has W_fhqc=0 and Wbool=true. The algorithm will only work on data that ha not been controlled by HQC | – | – |
The configuration file also needs to be edited to include further flag controls ( → ) and include this:
... #Flag to set if value is corrected S_ftime=1 change_fmis=3->1 change_fmis=0->4 ...
| 003 | Check log files to verify that the algorithm runs as scheduled, e.g. $ tail -f /var/log/Kvalobs/Qc2.log | – | – |
| 004 | Check that Qc2 processed data now resides in the database: | – | – |
| kvalobs=# select count(*) from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=55 | – | |
| 005 | Delete any existing data output (rm ALG004-Step5.dat) and then EXTRACT THE DATA: | DATA-S5 | – |
psql> select * from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%' \g | cat >> ./ALG004-Step5.dat;
| 006 | Plot the data. Download and use the two scripts ./unit_t_plot.bash and unitt.r.gz | – | – |
| Make the bash file executable: | – | – | |
| chmod u+x unit_t_plot.bash | – | – | |
| Run the script | – | – | |
| ./unit_t_plot.bash ALG004-Step5.dat |  | – | |
| and inspect the produced image ALG004-Step5.dat.jpg file. | – | – |
Alternative to step 006 for use on dev-vm101
These files are already installed under /home/kvalobs.
As user kvalobs:
$ cd $ ./unit_t_plot_manual.bash ALG004-Step5.dat The data file ALG004-Step5.dat.XY.dat has been created ... Now run the following commands in R: -------------------------------------- FileName="ALG004-Step5.dat.XY.dat" UTD <- read.table(FileName, header=TRUE) pdf() plot(UTD[,1],UTD[,2],ylim=c(-20,20),xlim=c(-20,20),xlab='Qc2 Algorithm/mm',ylab='Existing Correction/mm', main='200803 Hourly Temperature', sub='Missing Value Correction',col=4) lines(c(-20,20),c(-20,20),col=3) lines(c(-30,30),c(0,0)) lines(c(0,0),c(-30,30)) lines(c(-30,30),c(-30,30),col=3) dev.off() --------------------------------------
i.e. … enter R:
$ R ... > FileName="ALG004-Step5.dat.XY.dat" > UTD <- read.table(FileName, header=TRUE) > pdf() > plot(UTD[,1],UTD[,2],ylim=c(-20,20),xlim=c(-20,20),xlab='Qc2 Algorithm/mm',ylab='Existing Correction/mm', main='200803 Hourly Temperature', sub='Missing Value Correction',col=4) > lines(c(-20,20),c(-20,20),col=3) > lines(c(-30,30),c(0,0)) > lines(c(0,0),c(-30,30)) > lines(c(-30,30),c(-30,30),col=3) > dev.off() > > quit()
The above creates the file Rplots.pdf:
$ xpdf Rplots.pdf
NB if you have logged in using ssh -X kvalobs@dev-vm101 then the above will probably work!
| 007 | Edit the configuration file and set Wbool=false. Rerun the algorithm. The algorithm will reprocess the data flagged as controlled by HQC already. | – | – |
| kvalobs=# select count(*) from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=121 | – | |
| Capture the results. (rm ALG004-Step7.dat any previous test results) | DATA-S7 | – |
psql> select * from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%' \g | cat >> ./ALG004-Step7.dat;
| 008 | Plot results as in step 006 above. |  | – |
Prerequisites
Set up the test data (these are prepatory notes for now for set up on pak)
Test Data comprising original data from 2008-03 with all the years set to 2024, the file only contains the parameters for the algorithm.
Setting-up or refreshing the database. (some database management may be required to stop page errors and the like ????, vacuum cleaning etc ???)
- $ psql kvalobs
- psql⇒ DELETE FROM data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01';
- psql⇒ COPY data FROM ./alg004-kvalobsdb.dat WITH DELIMITER AS '|';
Script for plotting the test results. See the Step # above: unit_t_plot.bash.gz
… and not to forget the corresponding R-script called by the shell script: unitt.r.gz