kvalobs:kvoss:system:qc2:test:acceptance:alg004

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
kvalobs:kvoss:system:qc2:test:acceptance:alg004 [2010-06-15 23:11:21]
paule 		kvalobs:kvoss:system:qc2:test:acceptance:alg004 [2022-05-31 09:29:32] (current)
Line 5: Line 5:
 ^Step ^Direction ^Expected Result ^Pass/Fail Criteria or Comment ^ ^Step ^Direction ^Expected Result ^Pass/Fail Criteria or Comment ^
 |001 | Prepare test data in the test database. | -- | -- | |001 | Prepare test data in the test database. | -- | -- |
-| | Download {{:kvoss:system:qc2:test:acceptance:alg004-kvalobsdb.dat.gz|Test Data}} and gunzip. | -- | -- |+| | Download {{kvalobs:kvoss:system:qc2:test:acceptance:alg004-kvalobsdb.dat.gz|Test Data}} and gunzip. | -- | -- |
 | | $ psql kvalobs | -- | -- | | | $ psql kvalobs | -- | -- |
 | | <code>DELETE FROM data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01';</code> | -- | -- | | | <code>DELETE FROM data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01';</code> | -- | -- |
Line 13: Line 13:
 | | Check that the time interval contains no Qc2 processed data: | -- | -- | | | Check that the time interval contains no Qc2 processed data: | -- | -- |
 | | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=0 | -- | | | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=0 | -- |
-|002 |Download and gunzip {{:kvoss:system:qc2:test:acceptance:alg004.cfg.gz|alg004.cfg.gz}} to the /etc/kvalobs/Qc2Config directory and edit RunAtHour and RunAtMinute fields to the time that the test shall be run (e.g. a few minutes UTC in the future). | -- | -- |+|002 |Download and gunzip {{kvalobs:kvoss:system:qc2:test:acceptance:alg004.cfg.gz|alg004.cfg.gz}} to the /etc/kvalobs/Qc2Config directory and edit RunAtHour and RunAtMinute fields to the time that the test shall be run (e.g. a few minutes UTC in the future). | -- | -- |
 | |The configuration by default has W_fhqc=0 and Wbool=true. The algorithm will only work on data that ha not been controlled by HQC | -- | -- | | |The configuration by default has W_fhqc=0 and Wbool=true. The algorithm will only work on data that ha not been controlled by HQC | -- | -- |
  
Line 30: Line 30:
 |004 | Check that Qc2 processed data now resides in the database: | -- | -- | |004 | Check that Qc2 processed data now resides in the database: | -- | -- |
 | | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=55 | -- | | | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=55 | -- |
-|005 | Delete any existing data output (rm ALG004-Step5.dat) and then EXTRACT THE DATA: |{{:kvoss:system:qc2:test:acceptance:alg004-step5.dat.gz|DATA-S5}} | -- |+|005 | Delete any existing data output (rm ALG004-Step5.dat) and then EXTRACT THE DATA: |{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step5.dat.gz|DATA-S5}} | -- |
  
 <code>psql> select * from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'  \g | cat >> ./ALG004-Step5.dat; <code>psql> select * from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'  \g | cat >> ./ALG004-Step5.dat;
 </code> </code>
  
-|006 | Plot the data. Download and use the two scripts  {{:kvoss:system:qc2:test:acceptance:unit_t_plot.bash.gz|./unit_t_plot.bash}} and {{:kvoss:system:qc2:test:acceptance:unitt.r.gz|unitt.r.gz}}  | -- | -- |+|006 | Plot the data. Download and use the two scripts  {{kvalobs:kvoss:system:qc2:test:acceptance:unit_t_plot.bash.gz|./unit_t_plot.bash}} and {{kvalobs:kvoss:system:qc2:test:acceptance:unitt.r.gz|unitt.r.gz}}  | -- | -- |
 | |Make the bash file executable: | -- | -- | | |Make the bash file executable: | -- | -- |
 | |   chmod u+x unit_t_plot.bash| -- | -- | | |   chmod u+x unit_t_plot.bash| -- | -- |
 | |Run the script | -- | -- | | |Run the script | -- | -- |
-|    |   ./unit_t_plot.bash ALG004-Step5.dat|{{:kvoss:system:qc2:test:acceptance:alg004-step5.dat.xy.dat.jpg?100x100|}} |-- |+|    |   ./unit_t_plot.bash ALG004-Step5.dat|{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step5.dat.xy.dat.jpg?100x100|}} |-- |
 |    | and inspect the produced image ALG004-Step5.dat.jpg file.  | -- |-- | |    | and inspect the produced image ALG004-Step5.dat.jpg file.  | -- |-- |
 +
 +**Alternative to step 006 for use on dev-vm101**
 +
 +These files are already installed under /home/kvalobs.
 +
 +As user kvalobs:
 +<code>
 +$ cd
 +$ ./unit_t_plot_manual.bash ALG004-Step5.dat
 + 
 +The data file ALG004-Step5.dat.XY.dat has been created ...
 + 
 +Now run the following commands in R:
 +-------------------------------------- 
 + 
 +FileName="ALG004-Step5.dat.XY.dat"
 +UTD <- read.table(FileName, header=TRUE)
 +pdf()
 +plot(UTD[,1],UTD[,2],ylim=c(-20,20),xlim=c(-20,20),xlab='Qc2 Algorithm/mm',ylab='Existing Correction/mm', main='200803 Hourly Temperature', sub='Missing Value Correction',col=4)
 +lines(c(-20,20),c(-20,20),col=3)
 +lines(c(-30,30),c(0,0))
 +lines(c(0,0),c(-30,30))
 +lines(c(-30,30),c(-30,30),col=3)
 +dev.off()
 + 
 +-------------------------------------- 
 +</code>
 +i.e. ... enter R:
 +<code>
 +$ R
 +...
 +> FileName="ALG004-Step5.dat.XY.dat"
 +> UTD <- read.table(FileName, header=TRUE)
 +> pdf()
 +> plot(UTD[,1],UTD[,2],ylim=c(-20,20),xlim=c(-20,20),xlab='Qc2 Algorithm/mm',ylab='Existing Correction/mm', main='200803 Hourly Temperature', sub='Missing Value Correction',col=4)
 +> lines(c(-20,20),c(-20,20),col=3)
 +> lines(c(-30,30),c(0,0))
 +> lines(c(0,0),c(-30,30))
 +> lines(c(-30,30),c(-30,30),col=3)
 +> dev.off()
 +>
 +> quit()
 +</code>
 +
 +The above creates the file Rplots.pdf:
 +<code>
 +$ xpdf Rplots.pdf 
 +</code>
 +**NB** if you have logged in using ssh -X kvalobs@dev-vm101 then the above will probably work!
 +
 +
 |007 | Edit the configuration file and set Wbool=false. Rerun the algorithm. The algorithm will reprocess the data flagged as controlled by HQC already. | -- | -- | |007 | Edit the configuration file and set Wbool=false. Rerun the algorithm. The algorithm will reprocess the data flagged as controlled by HQC already. | -- | -- |
 | | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=121 | -- | | | kvalobs=# select count(*) from data  WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'; | count=121 | -- |
-| |Capture the results. (rm ALG004-Step7.dat any previous test results) |{{:kvoss:system:qc2:test:acceptance:alg004-step7.dat.gz|DATA-S7}} | -- |+| |Capture the results. (rm ALG004-Step7.dat any previous test results) |{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step7.dat.gz|DATA-S7}} | -- |
 <code>psql> select * from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'  \g | cat >> ./ALG004-Step7.dat; <code>psql> select * from data WHERE obstime BETWEEN '2024-02-28' AND '2024-04-01' and cfailed like '%Qc2%'  \g | cat >> ./ALG004-Step7.dat;
 </code> </code>
-|008 | Plot results as in step 006 above. | {{:kvoss:system:qc2:test:acceptance:alg004-step7.dat.xy.dat.jpg?100x100|}} | -- |+|008 | Plot results as in step 006 above. | {{kvalobs:kvoss:system:qc2:test:acceptance:alg004-step7.dat.xy.dat.jpg?100x100|}} | -- |
  
 **Prerequisites** **Prerequisites**
Line 52: Line 103:
 Set up the test data (these are prepatory notes for now for set up on pak) Set up the test data (these are prepatory notes for now for set up on pak)
  
-{{:kvoss:system:qc2:test:acceptance:alg004-kvalobsdb.dat.gz|Test Data}} comprising original data from 2008-03 with all the years set to 2024, the file only contains the parameters for the algorithm.+{{kvalobs:kvoss:system:qc2:test:acceptance:alg004-kvalobsdb.dat.gz|Test Data}} comprising original data from 2008-03 with all the years set to 2024, the file only contains the parameters for the algorithm.
    
-{{:kvoss:system:qc2:test:acceptance:alg004.cfg.gz|Configuration File}}+{{kvalobs:kvoss:system:qc2:test:acceptance:alg004.cfg.gz|Configuration File}}
  
 Setting-up or refreshing the database. (some database management may be required to stop page errors and the like ????, vacuum cleaning etc ???) Setting-up or refreshing the database. (some database management may be required to stop page errors and the like ????, vacuum cleaning etc ???)
Line 62: Line 113:
   * psql=> COPY data FROM ./alg004-kvalobsdb.dat WITH DELIMITER AS '|';   * psql=> COPY data FROM ./alg004-kvalobsdb.dat WITH DELIMITER AS '|';
  
-Script for plotting the test results. See the Step # above: {{:kvoss:system:qc2:test:acceptance:unit_t_plot.bash.gz|}}+Script for plotting the test results. See the Step # above: {{kvalobs:kvoss:system:qc2:test:acceptance:unit_t_plot.bash.gz|}}
  
-... and not to forget the corresponding R-script called by the shell script: {{:kvoss:system:qc2:test:acceptance:unitt.r.gz|}}+... and not to forget the corresponding R-script called by the shell script: {{kvalobs:kvoss:system:qc2:test:acceptance:unitt.r.gz|}}
  
  
  • kvalobs/kvoss/system/qc2/test/acceptance/alg004.1276643481.txt.gz
  • Last modified: 2022-05-31 09:23:18
  • (external edit)