kvalobs:kvoss:system:qc2:test:acceptance:alg001

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision 		Previous revision
kvalobs:kvoss:system:qc2:test:acceptance:alg001 [2009-12-20 00:18:21]
paule created 		kvalobs:kvoss:system:qc2:test:acceptance:alg001 [2022-05-31 09:29:32] (current)
Line 1: Line 1:
 ====== ALG001 ====== ====== ALG001 ======
  
-===== Acceptance Test For Redistribution of 24 hour accumulated precipitation ===== 
  
 +===== Acceptance Test for Redistribution of 24 hour accumulated precipitation =====
  
 +^Step ^Direction ^Expected Result ^Pass/Fail Criteria or Comment ^
 +|001 | Clear all previous data: | -- | -- |
 +| |$psql kvalobs | -- | -- |
 +| |psql> DELETE FROM data WHERE obstime>'2027-01-01' and obstime<'2028-01-01';| -- | -- |
 +|002 |Load the test data, e.g.: | -- | -- |
 +
 +{{kvalobs:kvoss:system:qc2:test:acceptance:2027processedtestdata.dat.gz|test data}}
 +
 +| |psql> \COPY data FROM '/metno/kvalobs/TESTDATA/2027processedtestdata.dat' WITH DELIMITER AS ';' | -- | -- |
 +| |Check the copy is successful: | -- | -- |
 +| | select count(*) from data WHERE obstime>'2027-01-01' and obstime<'2028-01-01'; | count=124515 | -- |
 +|003 | Download the {{kvalobs:kvoss:system:qc2:test:acceptance:alg001-qcd2027.cfg.gz|Configuration File}} and edit the run time as required to run the algorithm. | -- | -- |
 +
 +
 +Also edit the config file so that it contains the following text:  
 +<code>
 +...
 +# Flags to set on successful completion
 +S_fd=7
 +change_fmis=3->1
 +change_fmis=0->4
 +
 +...
 +</code>
 +(NB this is a recent update)
 +
 +|004 |Look for results prior to running the control: | -- | -- |
 +| |select count(*) from data where obstime>'2027-01-01' and obstime<'2028-01-01' and cfailed like '%TEST%Qc2%'; | count=0 | -- |
 +|005 | Start the kvQc2 daemon and monitor until the algorithm run is complete. | -- | -- |
 +| |select count(*) from data where obstime>'2027-01-01' and obstime<'2028-01-01' and cfailed like '%TEST%Qc2%'; | count=172 | -- |
 +<code>select * from data where obstime>'2027-01-01' and obstime<'2028-01-01' and cfailed like '%TEST%Qc2%' \g | cat >> ALG001.dat</code>
 +|006 | Download the following processing and plotting scripts {{kvalobs:kvoss:system:qc2:test:acceptance:alg001_plot.bash.gz|alg001_plot.bash}}, {{kvalobs:kvoss:system:qc2:test:acceptance:binplot.r.gz|binplot.R}} and {{kvalobs:kvoss:system:qc2:test:acceptance:plot_xy_analysis.r.gz|plot_xy_analysis.r}}| -- | -- |
 +|007 | ./alg001_plot.bash ALG001.dat | {{kvalobs:kvoss:system:qc2:test:acceptance:alg001.dat.xy.dat.jpg?100x100|}} | -- |
 +| | If files are downloaded from dokit: Please note dokit changes uploaded files to lower case ... an R script, example.R, becomes example.r Also need to "chmod +x" for scripts downloaded from dokit. | -- | -- |
 +| | On dev-vm116, R is installed but "Rscript" is not. Best to copy data to a machine where R is installed fully. | -- | -- |
 +|008 | -- | -- | -- |
 +|009 | -- | -- | -- |
 +|010 | -- | -- | -- |
 +
 +Alternative configuration file for performing triangulation interpolation: {{kvalobs:kvoss:system:qc2:test:acceptance:alg001-qcd2027_triangle.cfg.gz|}}
 +
 +
 +===== Notes =====
 +
 +**Prerequisites**
 +
 +Set up the test data (these are prepatory notes for now for set up on pak}
 +
 +
 +  * $ psql kvalobs
 +  * psql=> DELETE FROM data WHERE obstime>'2027-01-01' and obstime<'2028-01-01';
 +  * psql=> COPY data FROM '/metno/kvalobs/TESTDATA/2027.dat' USING DELIMTERS ';';
 + 
 +
 +  * Run generator.pl commands (current set in /metno/kvalobs/TESTDATA/gencom2027.sh)
 +
 +<code>
 +kvalobs@pak:~/TESTDATA$ psql kvalobs
 +Welcome to psql 8.3.8, the PostgreSQL interactive terminal.
 +
 +Type:  \copyright for distribution terms
 +       \h for help with SQL commands
 +       \? for help with psql commands
 +       \g or terminate with semicolon to execute query
 +       \q to quit
 +
 +kvalobs=# DELETE FROM data WHERE obstime>'2027-01-01' and obstime<'2028-01-01';
 +DELETE 124515
 +kvalobs=# COPY data FROM '/metno/kvalobs/TESTDATA/2027.dat' USING DELIMITERS ';';
 +COPY 124515
 +kvalobs=# \q
 +
 +kvalobs@pak:~/TESTDATA$ ls
 +2027.dat              gencom2027.sh  generator.pl  RA24-RESULTS-20091230.dat  results20091230.dat
 +2047_paramid_110.dat  gencom.sh      plotXY.R      RA_RA24_ScriptFile.sql     Rplots.ps
 +kvalobs@pak:~/TESTDATA$ ./gencom2027.sh 
 +
 +Saved the results of the TEST data into a file and made ready for reinsert
 +
 + kvalobs=# select * from data where obstime>'2027-01-01' and obstime<'2028-01-01' \g | cat >> ./TEST-DATA-2027-SIMULATED.dat;
 + $ ./MakeDataForInsert.sh TEST-DATA-2027-SIMULATED.dat 
 +(NB afterwards replace "NULL" with "")
 + $ mv TEST-DATA-2027-SIMULATED.dat.temp 2027ProcessedTestData.dat
 +
 +
 +Run Qc2 RA_RR24 with triangle interpolation...
 +
 +kvalobs=# select * from data where obstime>'2027-01-01' and obstime<'2028-01-01' and cfailed like '%TEST%Qc2%' \g | cat >> ./RA24-RESULTS-20100101-triangle.dat;
 +</code>
 +
 +{{kvalobs:kvoss:system:qc2:test:acceptance:2027.doc|}}
 +
 +{{kvalobs:kvoss:system:qc2:test:acceptance:2027processedtestdata.dat.gz|}}
 +
 +NB 
 +<code>
 +$ cksum 2027processedtestdata.dat.gz
 +1877035137 1486134 2027processedtestdata.dat.gz
 +</code>
 +
 +{{kvalobs:kvoss:system:qc2:test:acceptance:ra24-results-idw.jpg|}}
 +
 +{{kvalobs:kvoss:system:qc2:test:acceptance:ra24-results-triangle.jpg|}}
 +
 +generator.pl
 +<code>
 +#!/usr/bin/perl
 +#
 +# set missing values to -11111 at first and then update them all later
 +# NB we rely on the -32676 for counting good values ...
 +#
 +# Note: to run psql command from the command line:
 +#
 +#    psql kvalobs -c 'select count(*) from station'
 +#    psql kvalobs -c "select count(*) from station"
 +#    echo "select count(*) from station" > com.sql
 +#    psql kvalobs -f com.sql
 +#
 +# Example use:
 +#  ./generator.pl "2032-03-06 06:00:00" 4 75410 -308 110
 +
 +system("touch RA_RA24_ScriptFile.sql");
 +system("/bin/rm RA_RA24_ScriptFile.sql");
 +
 +open (SCRIPT_FILE, '>>RA_RA24_ScriptFile.sql');
 +
 +$user_name= getpwuid $>;
 +
 +if($#ARGV < 4) {
 +       &shouldo;
 +       exit;
 +}
 +
 +
 +$DATE=$ARGV[0];
 +$N=$ARGV[1];
 +$STID=$ARGV[2];
 +$TID=$ARGV[3];
 +$PID=$ARGV[4];
 +
 +# substr(controlinfo,7,1)=3 || 1 || 2 --- dont forget need for this ???
 +
 +$M=$N-1;
 +
 +$UP1="UPDATE data SET original=((SELECT SUM(original) FROM data WHERE obstime BETWEEN date '$DATE' - interval '$M days' AND '$DATE' AND stationid=$STID AND typeid=$TID AND paramid=$PID AND original>-32767)+(SELECT COUNT(original) FROM data WHERE obstime BETWEEN date '$DATE' - interval '$M days' AND '$DATE' AND stationid=$STID AND typeid=$TID AND paramid=$PID AND original=-1)) WHERE obstime='$DATE' AND stationid=$STID AND typeid=$TID AND paramid=$PID AND original>-32767 AND (SELECT COUNT(original) FROM data WHERE obstime BETWEEN date '$DATE' - interval '$M days' AND '$DATE' AND stationid=$STID AND typeid=$TID AND paramid=$PID AND original>-32767)=$N;";
 +
 +$UP2="UPDATE data SET controlinfo='1140000000002000' WHERE obstime='$DATE' AND stationid=$STID AND typeid=$TID AND paramid=$PID AND original>-32767 AND (SELECT COUNT(original) FROM data WHERE obstime BETWEEN date '$DATE' - interval '$M days' AND '$DATE' AND stationid=$STID AND typeid=$TID AND paramid=$PID AND original>-32767)=$N;"; 
 +
 +print SCRIPT_FILE "\n";
 +print SCRIPT_FILE $UP1,"\n";
 +print SCRIPT_FILE "\n";
 +print SCRIPT_FILE $UP2,"\n";
 +
 +for ($i=1; $i < $N; $i++){
 +          $j=$i-1;
 +          print SCRIPT_FILE "UPDATE data SET controlinfo='1110000000002000' WHERE obstime BETWEEN date '$DATE' -interval '$i days' AND date '$DATE' -interval '$j days' AND stationid=$STID AND typeid=$TID AND paramid=$PID AND original>-32767 AND (SELECT COUNT(original) FROM data WHERE obstime BETWEEN date '$DATE' - interval '$M days' AND '$DATE' AND stationid=$STID AND typeid=$TID AND paramid=$PID AND original>-32767)=$N;\n";
 +print SCRIPT_FILE "\n";
 +          print SCRIPT_FILE "UPDATE data SET original=-11111 WHERE obstime BETWEEN date '$DATE' -interval '$i days' AND date '$DATE' -interval '$j days' AND stationid=$STID AND typeid=$TID AND paramid=$PID AND original>-32767 AND (SELECT COUNT(original) FROM data WHERE obstime BETWEEN date '$DATE' - interval '$M days' AND '$DATE' AND stationid=$STID AND typeid=$TID AND paramid=$PID AND original>-32767)=$N;\n";
 +print SCRIPT_FILE "\n";
 +}
 +
 +print SCRIPT_FILE "UPDATE data SET cfailed=cfailed||'TEST-DATA-RA_RR24 original='||corrected, original=-32767 WHERE obstime BETWEEN date '$DATE' - interval '$M days' AND '$DATE' AND stationid=$STID AND typeid=$TID AND paramid=$PID AND original=-11111;\n";
 +print SCRIPT_FILE "\n";
 +print SCRIPT_FILE "UPDATE data SET cfailed=cfailed||'TEST-DATA-RA_RR24 original='||corrected WHERE obstime='$DATE' AND stationid=$STID AND typeid=$TID AND paramid=$PID;\n";
 +
 +
 +
 +close (SCRIPT_FILE); 
 +
 +system("psql kvalobs -f RA_RA24_ScriptFile.sql");
 +
 +## ----------------------------------------------------
 +
 +sub shouldo {
 +  print
 +(" 
 +  Hi $user_name, usage is:
 +          $0 \"YYYY-MM-DD 06:00:00\" NUMBER_MISSING_POINTS StationID TypeID ParamID 
 +
 +          e.g.  ./generator.pl \"2032-03-06 06:00:00\" 4 75410 -308 110
 +
 +")
 +}
 +
 +## ----------------------------------------------------
 +</code>
 +
 +{{kvalobs:kvoss:system:qc2:test:acceptance:gencom2027.doc|List of generator.pl calls to generate the test data.}}
 +
 +
 +====== Further Investigations ======
 +
 +It is possible to use the script generator.pl modify some real data and substitute missing values and an accummulated value, while also storing the original values. (NB copy and paste generator.pl from the listing above).
 +^Step ^Direction ^Expected Result ^Pass/Fail Criteria or Comment ^
 +|I |Select candidate data. One needs a start time and a number of days interval. | -- | -- |
 +|II |Run "generator.pl" on the interval: | -- | -- |
 +| |     ./generator.pl \"YYYY-MM-DD 06:00:00\" NUMBER_MISSING_POINTS StationID TypeID ParamID| -- | -- |
 +| |Repeat as many times as required to build test data. | -- | -- |
 +|III |Run the algorithm and analyse the results. | -- | -- |
 +
 +
 +
 +===== For example: =====
 +
 +
 +**Before**
 +
 +<code>
 +     58390 | 2032-03-02 06:00:00 |      0.3 |     110 | 2032-03-02 06:25:19 |    302 | 0      |     0 |       0.3 | 1110000000001001 | 3000000000000070 | 
 +     58390 | 2032-03-03 06:00:00 |      4.5 |     110 | 2032-03-03 06:10:32 |    302 | 0      |     0 |       4.5 | 1110000000001001 | 3000000000000070 | 
 +     58390 | 2032-03-04 06:00:00 |      2.5 |     110 | 2032-03-04 06:10:54 |    302 | 0      |     0 |       2.5 | 1110000000001000 | 7000000000000000 | 
 +     58390 | 2032-03-05 06:00:00 |      8.5 |     110 | 2032-03-05 06:25:24 |    302 | 0      |     0 |       8.5 | 1110000000001000 | 7000000000000000 | 
 +     58390 | 2032-03-06 06:00:00 |        7 |     110 | 2032-03-06 06:25:24 |    302 | 0      |     0 |         7 | 1110000000001000 | 7000000000000000 | 
 +     58390 | 2032-03-07 06:00:00 |      0.2 |     110 | 2032-03-07 06:25:21 |    302 | 0      |     0 |       0.2 | 1110000000001000 | 7000000000000000 | 
 +     58390 | 2032-03-08 06:00:00 |       -1 |     110 | 2032-03-08 06:25:17 |    302 | 0      |     0 |        -1 | 1110000000001000 | 7000000000000000 | 
 +     58390 | 2032-03-09 06:00:00 |      2.4 |     110 | 2032-03-09 06:06:31 |    302 | 0      |     0 |       2.4 | 1110000000001000 | 7000000000000000 | 
 +     58390 | 2032-03-10 06:00:00 |      2.2 |     110 | 2032-03-10 06:25:20 |    302 | 0      |     0 |       2.2 | 1110000000001001 | 3000000000000070 | 
 +
 +</code>
 +
 +**Command**
 +<code>
 +./generator.pl "2032-03-09 06:00:00" 7 58390 302 110
 +</code>
 +
 +
 +**After**
 +<code>
 +     58390 | 2032-03-02 06:00:00 |      0.3 |     110 | 2032-03-02 06:25:19 |    302 | 0      |     0 |       0.3 | 1110000000001001 | 3000000000000070 | 
 +     58390 | 2032-03-03 06:00:00 |   -32767 |     110 | 2032-03-03 06:10:32 |    302 | 0      |     0 |       4.5 | 9999999999992990 | 3000000000000070 | TEST-DATA-RA_RR24 original=4.5
 +     58390 | 2032-03-04 06:00:00 |   -32767 |     110 | 2032-03-04 06:10:54 |    302 | 0      |     0 |       2.5 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=2.5
 +     58390 | 2032-03-05 06:00:00 |   -32767 |     110 | 2032-03-05 06:25:24 |    302 | 0      |     0 |       8.5 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=8.5
 +     58390 | 2032-03-06 06:00:00 |   -32767 |     110 | 2032-03-06 06:25:24 |    302 | 0      |     0 |         7 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=7
 +     58390 | 2032-03-07 06:00:00 |   -32767 |     110 | 2032-03-07 06:25:21 |    302 | 0      |     0 |       0.2 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=0.2
 +     58390 | 2032-03-08 06:00:00 |   -32767 |     110 | 2032-03-08 06:25:17 |    302 | 0      |     0 |        -1 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=-1
 +     58390 | 2032-03-09 06:00:00 |     25.1 |     110 | 2032-03-09 06:06:31 |    302 | 0      |     0 |       2.4 | 9999999999992990 | 7000000000000000 | TEST-DATA-RA_RR24 original=2.4
 +     58390 | 2032-03-10 06:00:00 |      2.2 |     110 | 2032-03-10 06:25:20 |    302 | 0      |     0 |       2.2 | 1110000000001001 | 3000000000000070 | 
 +
 +</code>
  • kvalobs/kvoss/system/qc2/test/acceptance/alg001.1261268301.txt.gz
  • Last modified: 2022-05-31 09:23:18
  • (external edit)