kvalobs:kvoss:system:qc2:test:algorithms:singlemissingpoint

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
kvalobs:kvoss:system:qc2:test:algorithms:singlemissingpoint [2010-06-21 01:11:42]
paule 		kvalobs:kvoss:system:qc2:test:algorithms:singlemissingpoint [2022-05-31 09:29:32] (current)
Line 1: Line 1:
 +
 +====== Useinfo Controls ======
 +
 +**Specification**
 +
 +Proposal:
 +
 +  * 8 > useinfo(0) > 0  (some qc has been done)
 +  * useinfo(1) = 0  (ok obs time)
 +  * useinfo(2) < 2  (original is ok or probably correct)
 +  * useinfo(3) < 3  (original unchanged and manually corrected or interpolated)
 +
 +Discussion:
 +
 +The condition on useinfo(0) and  useinfo(2) looks ok. Disregarding
 +observations with useinfo(2)=2 ("Very suspicious") sounds sensible
 +(although not self-evident). If useinfo(2) < 2, then useinfo(3) will
 +necessarily be 0, so no need to look at useinfo(3).
 +
 +Suggest not to take useinfo(1) into account either. For some
 +observation types arriving 11 minutes after termin time is enough to get
 +useinfo(1)=1, which in fact therefore is a very common value. We could,
 +however, consider looking instead at useinfo(7), and reject the
 +observation if "utført for tidlig eller sent": useinfo(7)=1,2,5,6.
 +Someone else than me has to decide whether these flag values should
 +disqualify the observation from being used in the QC2 interpolation. But
 +this can wait, since these flag values are very rare - no examples during last 2 days.
 +
 +**Sample cfg file constructed**
 +
 +<code>
 +kvalobs@pak:~/etc/kvalobs/Qc2Config$ cat 211-SingleLinear.cfg 
 +#[Specific Runs]
 +#Last 24hours .....
 +#Daily Run ....
 +# /disk1/KVALOBS/Qc2Config .... $KVALOBS
 +
 +# Performs simple linear interpolation for a single point replacement.
 +AlgoCode=10
 +#InterpCode=9   # 9
 +RunAtHour=23
 +RunAtMinute=18
 +
 +#[Time Range]
 +Start_YYYY=2036
 +Start_MM=2
 +Start_DD=28
 +Start_hh=0
 +Start_mm=0
 +Start_ss=0
 +
 +End_YYYY=2036
 +End_MM=4
 +End_DD=1
 +End_hh=0
 +End_mm=0
 +End_ss=0
 +
 +#[Time Step]
 +#Step_hh=12 
 +
 +#[Specific Data Type and Paramters ids etc.]
 +ParamId=211
 +MaxParamId=215
 +MinParamId=213
 +#InterpolationDistance=50.0
 +# Only write back the result if not prviously controlled
 +W_fhqc=0
 +
 +#Flag to set if value is corrected
 +S_ftime=1
 +change_fmis=3->1
 +change_fmis=0->4
 +#Flag to check if the algorithm shall be applied ...
 +Not_ftime=1   # ftime=1 is set if the algorithm is run. Do not use an interpolated value for another interpolation.
 +Not_fnum=6    # This is to cut out the data with large values
 +#NotU_0=7
 +U_0=1
 +U_0=2
 +U_0=3
 +U_0=4
 +U_0=5
 +U_0=6
 +U_0=7
 +NotU_2=2
 +#NotU_7=1
 +#NotU_7=2
 +#NotU_7=5
 +#NotU_7=6
 +kvalobs@pak:~/etc/kvalobs/Qc2Config$ 
 +</code>
 +
 +**Results**
 +
 +
 +{{kvalobs:kvoss:system:qc2:test:algorithms:useinfocontrol211.jpg|}}
 +
 +NB The above is on canned test data (same as below) now need to run on real setup.
 +
 +
 +====== Tests On Other Parameters ======
 +
 +All the following used "AlgoCode=10" except for paramid=212 that used "AlgoCode=9". **Ref:** Algorithm summary: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:user:algorithms
 +
 +{{kvalobs:kvoss:system:qc2:test:algorithms:112.dat.xy.dat.jpg?200x200|112}}
 +{{kvalobs:kvoss:system:qc2:test:algorithms:173.dat.xy.dat.jpg?200x200|173}}
 +{{kvalobs:kvoss:system:qc2:test:algorithms:211.dat.xy.dat.jpg?200x200|211}}
 +{{kvalobs:kvoss:system:qc2:test:algorithms:212.dat.xy.dat.jpg?200x200|212}}
 +{{kvalobs:kvoss:system:qc2:test:algorithms:213.dat.xy.dat.jpg?200x200|213}}
 +{{kvalobs:kvoss:system:qc2:test:algorithms:215.dat.xy.dat.jpg?200x200|215}}
 +{{kvalobs:kvoss:system:qc2:test:algorithms:262.dat.xy.dat.jpg?200x200|262}}
 +
 +The above results were obtained using the following test data
 +
 +^ {{kvalobs:kvoss:system:qc2:test:algorithms:multiparamtersinglemissingpointtest.dat.gz|}} ^
 +
 +Tests may be conducted thus:
 +
 +|Clean old data from data-base |DELETE FROM data WHERE obstime>'2036-02-28' and obstime<'2036-04-01'; |
 +|Load data ... |<code> \copy data FROM 'multiparamtersinglemissingpointtest.dat' WITH DELIMITER AS '|'</code>  |
 +|Run algorithms |e.g. example configuration files  |
 +| |           https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/211.cfg|
 +| |           https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/Reference/212.cfg|
 +|Extract results | <code>kvalobs=# select * from data where paramid=211 and obstime between '2036-02-28' and '2036-03-05' and cfailed like '%QC2d-2%'  \g | cat >> ./211.dat; </code> |
 +| |<code>kvalobs=# select * from data where paramid=212 and obstime between '2036-02-28' and '2036-03-05' and cfailed like '%QC2d-2%'  \g | cat >> ./212.dat;</code>     |
 +|Plot, analyse results ... | |
 +
 ====== Single Missing Point ====== ====== Single Missing Point ======
  
Line 7: Line 134:
 Test data has been taken from March 2008. On each day for all stations the "original" hourly temperature has been set to a missing value -32767 for the times 05:00, 11:00, 17:00 and 23:00 UT and the original value is kept for reference. The results of the algorithm are then compared to the original data: Test data has been taken from March 2008. On each day for all stations the "original" hourly temperature has been set to a missing value -32767 for the times 05:00, 11:00, 17:00 and 23:00 UT and the original value is kept for reference. The results of the algorithm are then compared to the original data:
  
-{{:kvoss:system:qc2:test:algorithms:3036-oldmethod.dat.xy.dat.jpg|}} +{{kvalobs:kvoss:system:qc2:test:algorithms:3036-oldmethod.dat.xy.dat.jpg|}} 
-{{:kvoss:system:qc2:test:algorithms:zoom.jpg?100x100|ZOOM}}+{{kvalobs:kvoss:system:qc2:test:algorithms:zoom.jpg?100x100|ZOOM}}
  
 Good results are obtained. Note the points at -99.9 arise due to this value being present in the original data record (and they are not a result of the algorithm). Good results are obtained. Note the points at -99.9 arise due to this value being present in the original data record (and they are not a result of the algorithm).
Line 21: Line 148:
 Running the new algorithm on the same data as above, the algorithm first tries to do a Max and Min average. If this is not possible a linear interpolation is performed across the gap. First results: Running the new algorithm on the same data as above, the algorithm first tries to do a Max and Min average. If this is not possible a linear interpolation is performed across the gap. First results:
  
-{{:kvoss:system:qc2:test:algorithms:3036-genmethod.dat.xy.dat.jpg|}} +{{kvalobs:kvoss:system:qc2:test:algorithms:3036-genmethod.dat.xy.dat.jpg|}} 
-{{:kvoss:system:qc2:test:algorithms:zoom2.jpg?100x100|ZOOM}}+{{kvalobs:kvoss:system:qc2:test:algorithms:zoom2.jpg?100x100|ZOOM}}
  
 It is clear that in the above graph there are additional points that also show good agreement with the original values. There are also many original values that are unphysical (e.g. high temperatures in March and filtering on flags has also to be applid so that these cases are not included). Examples of high temperatures in the database: It is clear that in the above graph there are additional points that also show good agreement with the original values. There are also many original values that are unphysical (e.g. high temperatures in March and filtering on flags has also to be applid so that these cases are not included). Examples of high temperatures in the database:
Line 34: Line 161:
 Ok, lets filter out results where control(4)=6 ... fnum=6. This gives the following improved result, but the algorithm still appears to act on some temperatures that are too high and should not be corrected by interpolating between the times before and after, that are also too high!?! Ok, lets filter out results where control(4)=6 ... fnum=6. This gives the following improved result, but the algorithm still appears to act on some temperatures that are too high and should not be corrected by interpolating between the times before and after, that are also too high!?!
  
-{{:kvoss:system:qc2:test:algorithms:3036-genmethod_4.dat.xy.dat.jpg|}}+{{kvalobs:kvoss:system:qc2:test:algorithms:3036-genmethod_4.dat.xy.dat.jpg|}}
  
 Configuration File Used: Configuration File Used:
Line 113: Line 240:
 Applying this algorithm and controls: Applying this algorithm and controls:
  
-{{:kvoss:system:qc2:test:algorithms:missinglinear.jpg|}} +{{kvalobs:kvoss:system:qc2:test:algorithms:missinglinear.jpg|}} 
- +{{kvalobs:kvoss:system:qc2:test:algorithms:missinglinearzoom.jpg?100x100|}}
-{{:kvoss:system:qc2:test:algorithms:missinglinearzoom.jpg?100x100|}}+
  
 <code> <code>
Line 123: Line 249:
 Applying the filter: do not include fhqclevel=0, gives: Applying the filter: do not include fhqclevel=0, gives:
  
-{{:kvoss:system:qc2:test:algorithms:fullmissinglinear.dat.xy.dat.jpg|}}+{{kvalobs:kvoss:system:qc2:test:algorithms:fullmissinglinear.dat.xy.dat.jpg|}} 
 + 
 +The corresponding configuration file to produce the above result: 
 + 
 +<code> 
 +# Performs simple linear interpolation for a single point replacement. 
 +AlgoCode=10 
 +#InterpCode=9   # 9 
 +RunAtHour=0  
 +RunAtMinute=42    
 + 
 +#[Time Range] 
 +Start_YYYY=2036 
 +Start_MM=2 
 +Start_DD=28 
 +Start_hh=0 
 +Start_mm=0 
 +Start_ss=0 
 + 
 +End_YYYY=2036 
 +End_MM=4 
 +End_DD=1 
 +End_hh=0 
 +End_mm=0 
 +End_ss=0 
 + 
 +#[Time Step] 
 +#Step_hh=12  
 + 
 +#[Specific Data Type and Paramters ids etc.] 
 +ParamId=211 
 +MaxParamId=215 
 +MinParamId=213 
 +#InterpolationDistance=50.0 
 +# Only write back the result if not prviously controlled 
 +W_fhqc=0 
 + 
 +CfailedString="MIST" 
 + 
 +#Flag to set if value is corrected 
 +S_ftime=1 
 +change_fmis=3->1 
 +change_fmis=0->4 
 +#Flag to check if the algorithm shall be applied ... i.e. the following are restrictions: 
 +A_ftime=1   # 
 +A_fnum=6    # This is to cut out the data with large values 
 +A_fpre=6 
 +A_fpre=7 
 +A_fhqc=A 
 +#V_fpre=1 
 +#V_fpre=2 
 +A_fqclevel=0 
 +</code> 
  
 TODO: Use obs_pgm to generate better station list ... (currently all stations used) TODO: Use obs_pgm to generate better station list ... (currently all stations used)
  • kvalobs/kvoss/system/qc2/test/algorithms/singlemissingpoint.1277082702.txt.gz
  • Last modified: 2022-05-31 09:23:19
  • (external edit)