kvalobs:kvoss:system:qc2:test:acceptance:alg001

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
kvalobs:kvoss:system:qc2:test:acceptance:alg001 [2010-04-04 13:10:50]
paule 		kvalobs:kvoss:system:qc2:test:acceptance:alg001 [2022-05-31 09:29:32] (current)
Line 8: Line 8:
 | |$psql kvalobs | -- | -- | | |$psql kvalobs | -- | -- |
 | |psql> DELETE FROM data WHERE obstime>'2027-01-01' and obstime<'2028-01-01';| -- | -- | | |psql> DELETE FROM data WHERE obstime>'2027-01-01' and obstime<'2028-01-01';| -- | -- |
-|002 |Load the test data: | -- | -- |+|002 |Load the test data, e.g.: | -- | -- | 
 + 
 +{{kvalobs:kvoss:system:qc2:test:acceptance:2027processedtestdata.dat.gz|test data}} 
 | |psql> \COPY data FROM '/metno/kvalobs/TESTDATA/2027processedtestdata.dat' WITH DELIMITER AS ';' | -- | -- | | |psql> \COPY data FROM '/metno/kvalobs/TESTDATA/2027processedtestdata.dat' WITH DELIMITER AS ';' | -- | -- |
 | |Check the copy is successful: | -- | -- | | |Check the copy is successful: | -- | -- |
 | | select count(*) from data WHERE obstime>'2027-01-01' and obstime<'2028-01-01'; | count=124515 | -- | | | select count(*) from data WHERE obstime>'2027-01-01' and obstime<'2028-01-01'; | count=124515 | -- |
-|003 | Download the {{:kvoss:system:qc2:test:acceptance:alg001-qcd2027.cfg.gz|Configuration File}} and edit the run time as required to run the algorithm. | -- | -- | +|003 | Download the {{kvalobs:kvoss:system:qc2:test:acceptance:alg001-qcd2027.cfg.gz|Configuration File}} and edit the run time as required to run the algorithm. | -- | -- | 
-|004 | NB. If kvQc2 is not running ... start the daemon. | -- | -- | + 
-|005 |  | -- | -- |+ 
 +Also edit the config file so that it contains the following text:   
 +<code> 
 +... 
 +# Flags to set on successful completion 
 +S_fd=7 
 +change_fmis=3->1 
 +change_fmis=0->4 
 + 
 +... 
 +</code> 
 +(NB this is a recent update) 
 + 
 +|004 |Look for results prior to running the control: | -- | -- | 
 +| |select count(*) from data where obstime>'2027-01-01' and obstime<'2028-01-01' and cfailed like '%TEST%Qc2%'; | count=0 | -- | 
 +|005 | Start the kvQc2 daemon and monitor until the algorithm run is complete. | -- | -- | 
 +| |select count(*) from data where obstime>'2027-01-01' and obstime<'2028-01-01' and cfailed like '%TEST%Qc2%'; | count=172 | -- |
 <code>select * from data where obstime>'2027-01-01' and obstime<'2028-01-01' and cfailed like '%TEST%Qc2%' \g | cat >> ALG001.dat</code> <code>select * from data where obstime>'2027-01-01' and obstime<'2028-01-01' and cfailed like '%TEST%Qc2%' \g | cat >> ALG001.dat</code>
-|006 | Download the following processing and plotting scripts {{:kvoss:system:qc2:test:acceptance:alg001_plot.bash.gz|alg001_plot.bash}}, {{:kvoss:system:qc2:test:acceptance:binplot.r.gz|binplot.R}} and {{:kvoss:system:qc2:test:acceptance:plot_xy_analysis.r.gz|plot_xy_analysis.r}}| -- | -- | +|006 | Download the following processing and plotting scripts {{kvalobs:kvoss:system:qc2:test:acceptance:alg001_plot.bash.gz|alg001_plot.bash}}, {{kvalobs:kvoss:system:qc2:test:acceptance:binplot.r.gz|binplot.R}} and {{kvalobs:kvoss:system:qc2:test:acceptance:plot_xy_analysis.r.gz|plot_xy_analysis.r}}| -- | -- | 
-|007 | ./alg001_plot.bash ALG001.dat | {{:kvoss:system:qc2:test:acceptance:alg001.dat.xy.dat.jpg?100x100|}} | -- |+|007 | ./alg001_plot.bash ALG001.dat | {{kvalobs:kvoss:system:qc2:test:acceptance:alg001.dat.xy.dat.jpg?100x100|}} | -- | 
 +| | If files are downloaded from dokit: Please note dokit changes uploaded files to lower case ... an R script, example.R, becomes example.r Also need to "chmod +x" for scripts downloaded from dokit. | -- | -- | 
 +| | On dev-vm116, R is installed but "Rscript" is not. Best to copy data to a machine where R is installed fully. | -- | -- |
 |008 | -- | -- | -- | |008 | -- | -- | -- |
 |009 | -- | -- | -- | |009 | -- | -- | -- |
 |010 | -- | -- | -- | |010 | -- | -- | -- |
  
-Alternative configuration file for performing triangulation interpolation: {{:kvoss:system:qc2:test:acceptance:alg001-qcd2027_triangle.cfg.gz|}}+Alternative configuration file for performing triangulation interpolation: {{kvalobs:kvoss:system:qc2:test:acceptance:alg001-qcd2027_triangle.cfg.gz|}}
  
  
Line 73: Line 94:
 </code> </code>
  
-{{:kvoss:system:qc2:test:acceptance:2027.doc|}}+{{kvalobs:kvoss:system:qc2:test:acceptance:2027.doc|}}
  
-{{:kvoss:system:qc2:test:acceptance:2027processedtestdata.doc|}}+{{kvalobs:kvoss:system:qc2:test:acceptance:2027processedtestdata.dat.gz|}}
  
 +NB 
 +<code>
 +$ cksum 2027processedtestdata.dat.gz
 +1877035137 1486134 2027processedtestdata.dat.gz
 +</code>
  
-{{:kvoss:system:qc2:test:acceptance:ra24-results-idw.jpg|}}+{{kvalobs:kvoss:system:qc2:test:acceptance:ra24-results-idw.jpg|}}
  
-{{:kvoss:system:qc2:test:acceptance:ra24-results-triangle.jpg|}}+{{kvalobs:kvoss:system:qc2:test:acceptance:ra24-results-triangle.jpg|}}
  
 generator.pl generator.pl
Line 165: Line 191:
 </code> </code>
  
-{{:kvoss:system:qc2:test:acceptance:gencom2027.doc|List of generator.pl calls to generate the test data.}}+{{kvalobs:kvoss:system:qc2:test:acceptance:gencom2027.doc|List of generator.pl calls to generate the test data.}}
  
  
-====== EDIT ======+====== Further Investigations ======
  
-Add single use case examples for using generator.pl +It is possible to use the script generator.pl modify some real data and substitute missing values and an accummulated value, while also storing the original values. (NB copy and paste generator.pl from the listing above).
-It is possible to use the script generator.pl modify some real data and substitute missing values and an accummulated value, while also storing the original values.+
 ^Step ^Direction ^Expected Result ^Pass/Fail Criteria or Comment ^ ^Step ^Direction ^Expected Result ^Pass/Fail Criteria or Comment ^
-|001 | Clear all previous data: | -- | -- | +|I |Select candidate data. One needs a start time and a number of days interval. | -- | -- | 
-| |$psql kvalobs | -- | -- | +|II |Run "generator.pl" on the interval: | -- | -- | 
-| |psql> DELETE FROM data WHERE obstime>'2027-01-01' and obstime<'2028-01-01';| -- | -- |+| |     ./generator.pl \"YYYY-MM-DD 06:00:00\" NUMBER_MISSING_POINTS StationID TypeID ParamID| -- | -- | 
 +| |Repeat as many times as required to build test data. | -- | -- | 
 +|III |Run the algorithm and analyse the results. | -- | -- | 
 + 
 + 
 + 
 +===== For example: ===== 
 + 
 + 
 +**Before** 
 + 
 +<code> 
 +     58390 | 2032-03-02 06:00:00 |      0.3 |     110 | 2032-03-02 06:25:19 |    302 | 0      |     0 |       0.3 | 1110000000001001 | 3000000000000070 |  
 +     58390 | 2032-03-03 06:00:00 |      4.5 |     110 | 2032-03-03 06:10:32 |    302 | 0      |     0 |       4.5 | 1110000000001001 | 3000000000000070 |  
 +     58390 | 2032-03-04 06:00:00 |      2.5 |     110 | 2032-03-04 06:10:54 |    302 | 0      |     0 |       2.5 | 1110000000001000 | 7000000000000000 |  
 +     58390 | 2032-03-05 06:00:00 |      8.5 |     110 | 2032-03-05 06:25:24 |    302 | 0      |     0 |       8.5 | 1110000000001000 | 7000000000000000 |  
 +     58390 | 2032-03-06 06:00:00 |        7 |     110 | 2032-03-06 06:25:24 |    302 | 0      |     0 |         7 | 1110000000001000 | 7000000000000000 |  
 +     58390 | 2032-03-07 06:00:00 |      0.2 |     110 | 2032-03-07 06:25:21 |    302 | 0      |     0 |       0.2 | 1110000000001000 | 7000000000000000 |  
 +     58390 | 2032-03-08 06:00:00 |       -1 |     110 | 2032-03-08 06:25:17 |    302 | 0      |     0 |        -1 | 1110000000001000 | 7000000000000000 |  
 +     58390 | 2032-03-09 06:00:00 |      2.4 |     110 | 2032-03-09 06:06:31 |    302 | 0      |     0 |       2.4 | 1110000000001000 | 7000000000000000 |  
 +     58390 | 2032-03-10 06:00:00 |      2.2 |     110 | 2032-03-10 06:25:20 |    302 | 0      |     0 |       2.2 | 1110000000001001 | 3000000000000070 |  
 + 
 +</code> 
 + 
 +**Command** 
 +<code> 
 +./generator.pl "2032-03-09 06:00:00" 7 58390 302 110 
 +</code> 
 + 
 + 
 +**After** 
 +<code> 
 +     58390 | 2032-03-02 06:00:00 |      0.3 |     110 | 2032-03-02 06:25:19 |    302 | 0      |     0 |       0.3 | 1110000000001001 | 3000000000000070 |  
 +     58390 | 2032-03-03 06:00:00 |   -32767 |     110 | 2032-03-03 06:10:32 |    302 | 0      |     0 |       4.5 | 9999999999992990 | 3000000000000070 | TEST-DATA-RA_RR24 original=4.5 
 +     58390 | 2032-03-04 06:00:00 |   -32767 |     110 | 2032-03-04 06:10:54 |    302 | 0      |     0 |       2.5 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=2.5 
 +     58390 | 2032-03-05 06:00:00 |   -32767 |     110 | 2032-03-05 06:25:24 |    302 | 0      |     0 |       8.5 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=8.5 
 +     58390 | 2032-03-06 06:00:00 |   -32767 |     110 | 2032-03-06 06:25:24 |    302 | 0      |     0 |         7 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=7 
 +     58390 | 2032-03-07 06:00:00 |   -32767 |     110 | 2032-03-07 06:25:21 |    302 | 0      |     0 |       0.2 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=0.2 
 +     58390 | 2032-03-08 06:00:00 |   -32767 |     110 | 2032-03-08 06:25:17 |    302 | 0      |     0 |        -1 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=-1 
 +     58390 | 2032-03-09 06:00:00 |     25.1 |     110 | 2032-03-09 06:06:31 |    302 | 0      |     0 |       2.4 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=2.4 
 +     58390 | 2032-03-10 06:00:00 |      2.2 |     110 | 2032-03-10 06:25:20 |    302 | 0      |     0 |       2.2 | 1110000000001001 | 3000000000000070 |  
 + 
 +</code>
  • kvalobs/kvoss/system/qc2/test/acceptance/alg001.1270386650.txt.gz
  • Last modified: 2022-05-31 09:23:18
  • (external edit)