====== ALG004 ======

===== Acceptance Test for single point missing temperature values with TAN, TAX method =====

^Step ^Direction ^Expected Result ^Pass/Fail Criteria or Comment ^
|001 | Prepare test data in the test database. | -- | -- |
| | Download {{kvalobs:kvoss:system:qc2:test:acceptance:alg004-kvalobsdb.dat.gz|Test Data}} and gunzip. | -- | -- |
| | $ psql kvalobs | -- | -- |
| | <code>DELETE FROM data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01';</code> | -- | -- |
| | <code>COPY data FROM '/home/kvalobs/alg004-kvalobsdb.dat' WITH DELIMITER AS '|';</code> | -- | -- |
| | NB specify the actual path to the test data above as required. | -- | -- |
| | | -- | -- |
| | Check that the time interval contains no Qc2 processed data: | -- | -- |
| | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=0 | -- |
|002 |Download and gunzip {{kvalobs:kvoss:system:qc2:test:acceptance:alg004.cfg.gz|alg004.cfg.gz}} to the /etc/kvalobs/Qc2Config directory and edit RunAtHour and RunAtMinute fields to the time that the test shall be run (e.g. a few minutes UTC in the future). | -- | -- |
| |The configuration by default has W_fhqc=0 and Wbool=true. The algorithm will only work on data that ha not been controlled by HQC | -- | -- |

The configuration file also needs to be edited to include further flag controls ( -> ) and include this:
<code>
...
#Flag to set if value is corrected
S_ftime=1
change_fmis=3->1
change_fmis=0->4
...
</code>


|003 | Check log files to verify that the algorithm runs as scheduled, e.g. $ tail -f /var/log/Kvalobs/Qc2.log | -- | -- |
|004 | Check that Qc2 processed data now resides in the database: | -- | -- |
| | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=55 | -- |
|005 | Delete any existing data output (rm ALG004-Step5.dat) and then EXTRACT THE DATA: |{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step5.dat.gz|DATA-S5}} | -- |

<code>psql> select * from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'  \g | cat >> ./ALG004-Step5.dat;
</code>

|006 | Plot the data. Download and use the two scripts  {{kvalobs:kvoss:system:qc2:test:acceptance:unit_t_plot.bash.gz|./unit_t_plot.bash}} and {{kvalobs:kvoss:system:qc2:test:acceptance:unitt.r.gz|unitt.r.gz}}  | -- | -- |
| |Make the bash file executable: | -- | -- |
| |   chmod u+x unit_t_plot.bash| -- | -- |
| |Run the script | -- | -- |
|    |   ./unit_t_plot.bash ALG004-Step5.dat|{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step5.dat.xy.dat.jpg?100x100|}} |-- |
|    | and inspect the produced image ALG004-Step5.dat.jpg file.  | -- |-- |

**Alternative to step 006 for use on dev-vm101**

These files are already installed under /home/kvalobs.

As user kvalobs:
<code>
$ cd
$ ./unit_t_plot_manual.bash ALG004-Step5.dat
 
The data file ALG004-Step5.dat.XY.dat has been created ...
 
Now run the following commands in R:
-------------------------------------- 
 
FileName="ALG004-Step5.dat.XY.dat"
UTD <- read.table(FileName, header=TRUE)
pdf()
plot(UTD[,1],UTD[,2],ylim=c(-20,20),xlim=c(-20,20),xlab='Qc2 Algorithm/mm',ylab='Existing Correction/mm', main='200803 Hourly Temperature', sub='Missing Value Correction',col=4)
lines(c(-20,20),c(-20,20),col=3)
lines(c(-30,30),c(0,0))
lines(c(0,0),c(-30,30))
lines(c(-30,30),c(-30,30),col=3)
dev.off()
 
-------------------------------------- 
</code>
i.e. ... enter R:
<code>
$ R
...
> FileName="ALG004-Step5.dat.XY.dat"
> UTD <- read.table(FileName, header=TRUE)
> pdf()
> plot(UTD[,1],UTD[,2],ylim=c(-20,20),xlim=c(-20,20),xlab='Qc2 Algorithm/mm',ylab='Existing Correction/mm', main='200803 Hourly Temperature', sub='Missing Value Correction',col=4)
> lines(c(-20,20),c(-20,20),col=3)
> lines(c(-30,30),c(0,0))
> lines(c(0,0),c(-30,30))
> lines(c(-30,30),c(-30,30),col=3)
> dev.off()
>
> quit()
</code>

The above creates the file Rplots.pdf:
<code>
$ xpdf Rplots.pdf 
</code>
**NB** if you have logged in using ssh -X kvalobs@dev-vm101 then the above will probably work!


|007 | Edit the configuration file and set Wbool=false. Rerun the algorithm. The algorithm will reprocess the data flagged as controlled by HQC already. | -- | -- |
| | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=121 | -- |
| |Capture the results. (rm ALG004-Step7.dat any previous test results) |{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step7.dat.gz|DATA-S7}} | -- |
<code>psql> select * from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'  \g | cat >> ./ALG004-Step7.dat;
</code>
|008 | Plot results as in step 006 above. | {{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step7.dat.xy.dat.jpg?100x100|}} | -- |

**Prerequisites**

Set up the test data (these are prepatory notes for now for set up on pak)

{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-kvalobsdb.dat.gz|Test Data}} comprising original data from 2008-03 with all the years set to 2024, the file only contains the parameters for the algorithm.
 
{{kvalobs:kvoss:system:qc2:test:acceptance:alg004.cfg.gz|Configuration File}}

Setting-up or refreshing the database. (some database management may be required to stop page errors and the like ????, vacuum cleaning etc ???)

  * $ psql kvalobs
  * psql=> DELETE FROM data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01';
  * psql=> COPY data FROM ./alg004-kvalobsdb.dat WITH DELIMITER AS '|';

Script for plotting the test results. See the Step # above: {{kvalobs:kvoss:system:qc2:test:acceptance:unit_t_plot.bash.gz|}}

... and not to forget the corresponding R-script called by the shell script: {{kvalobs:kvoss:system:qc2:test:acceptance:unitt.r.gz|}}