kvalobs:kvoss:system:qc2:requirements:algorithms:diptest03

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
kvalobs:kvoss:system:qc2:requirements:algorithms:diptest03 [2011-04-16 17:15:44]
paule 		kvalobs:kvoss:system:qc2:requirements:algorithms:diptest03 [2022-05-31 09:29:32] (current)
Line 1: Line 1:
-====== Dip test - QC2d-1 - Revision 03 (Final Version ???)======+====== QC2d-1: Dip test ======
  
-{{:kvoss:system:qc2:requirements:algorithms:dip.png|}}+{{kvalobs:kvoss:system:qc2:requirements:algorithms:dip.png|}}
  
    * fs=2 for A(t),     * fs=2 for A(t), 
Line 7: Line 7:
    * fs=1 for A(t-2)     * fs=1 for A(t-2) 
  
-<code> +A dip is detected if <m>|A(t)-A(t-2)| < |A(t-1)-A(t-2)|</m> and <m>|A(t)-A(t-2)| < delta</m> 
- If   +where <m>delta</m> is specified in the algorithm configuration. 
-    | A(t) - A(t-2) | < | A(t-1) - A(t-2) |  +<m>A(t-1)</m> then receives a corrected value by interpolation.
- and  +
-    | A(t) - A(t-2) | < DELTA +
-</code>+
  
-where DELTA is dependent on the parameter+The neighboring points at t-2 and t may not have missing original values and have 
 +to match the flags specified in **linear_before_*flags** and **linear_after_*flags**, 
 +respectively.
  
-<code>     +If there are 2 additional usable data points before and 1 usable additional data point after t-1, 
-   set fs=9 for A(t-1)  +[[Akima splines]] are used for the interpolation and cfailed receives a message 
-   set fs=6 for A(t) +**QC2d-1-A**. To be usable, the original value may not be missing (-32767) and has 
-</code>+to match the flag specification **akima_*flags** in the configuration if it is not 
 +an immediate neighbor of t-1, and . 
  
-A(t-1) receives a corrected value by interpolation. The interpolation method can be linear or by [[Akima splines]]. +If akima interpolation is not possible, linear interpolation is used and cfailed receives 
 +a message **QC2d-1-L**. 
  
 +==== Configuration ====
  
-====== Strategy ======+Beyond the standard configuration options detailed in the [[kvoss:system:qc2:user|user guide]], 
 +the following options are available for DipTest.
  
-**Configuration File**+  * candidate_cflags (default: fs=2&fhqc=0) and candidate_uflags (default: /no constraint/) specify the flags expected for A(t-1). Dips that do not match these flags will not be detected. 
 +  * akima_cflags (default: /no constraint/) and akima_uflags (default: U2=0) specify the requirements for additional points for akima interpolation 
 +  * linear_before_cflags (default: fs=1) and linear_before_uflags (default: /no constraint/) specify flag requirements for A(t-2) 
 +  * linear_after_cflags (default: fs=2&fhqc=0) and linear_after_uflags (default: /no constraint/) specify flag requirements for A(t) 
 +  * dip_flagchange (default: fs=9) specifies the flag modification for t-1 if an interpolation is performed 
 +  * afterdip_flagchange (default: fs=4) specifies the flag modification for t if an interpolation is performed 
 +  * ParValFilename specifies either (no default, must be specified, see below for examples) 
 +    * a filename containing a list of parameter-delta pairs 
 +    * or a list of parameter-value pairs when starting with **list:**
  
-<code> +=== ParValFilename ===
-#Algorithm +
-Algorithm=DipTest             +
  
-RunAtHour=21 +If using a file to specify parameter-delta pairs, the file must be formatted like this: 
-RunAtMinute=53+  1   9.0 
 +  2   60.0 
 +  81  10.0 
 +  85  10.0 
 +  86  12.5
  
-# Time interval specification (can also be a specific start and stop time) +If using a list to specify parameter-delta pairs, the parameter must be given like this: 
-Last_NDays=3+  ParValFilename= list: 172 12.0, 174 12.0, 175 12.0, 176 12.0, 177 12.0, 211 7.5
  
-#[Specific Data Type and Paramters ids etc.] +==== Parameters ====
-ParValFilename="/etc/kvalobs/Qc2Config/TestParVal.txt"  +
  
-#Flag required for neighbour to be used for an interpolation +Relevant parameters are all those subject to the step check (QC1-3a). The implementation is only tested for hourly measurements.
-U_2=0+
  
-# Only write data it it has not been analysed by HQC +==== Previous met.no dip tests ====
-W_fhqc=0 +
-</code> +
- +
-There are some important new features in the configuration file: +
-  - The specification of the algorithm is now handled by a string, i.e. Algorith=DipTest (or "DipTest" will work) rather than a code (AlgoCode=11 etc). This is now adopted for all algorithms to be run - clearer and more user friendly. +
-  - For the first algorithms (e.g. SingleLinear) a separate configuration file exists for each paramid. This leads to many separate configuration files and is rather cumbersome. Here there is one configuration file that points to an additional file that includes the parameter ids to process and any other specific information. In this case the file includes the DELTA paramter (see next excerpt below). Later all this information can be managed in a Qc2 table in the database if this is preferred. +
- +
-**Paramter ID file** +
- +
-<code> +
-1   9.0 +
-2   60.0 +
-81   10.0 +
-85   10.0 +
-86   12.5 +
-... +
-</code> +
- +
-The specific criteria for the DipTest (i.e. fs = 1, 2, 2 for three consecutive points and the value checks defined above) are, for the first version, encoded in the algorithm. +
- +
-The algorithm steps back in time checking each successive set of three points (sliding window). If a dip is confirmed a linear interpolation is first applied using the immediate temporal neighbours. The algorithm then checks if there are enough neighbours available of sufficient quality (U_2=0, set in the config file) to perform an Akima Interpolation. If this is satisfied the Akima value is used in preference to that generated from the linear interpolation. +
- +
-For each case the results in the kvalobs database are as follows: +
- +
-**Linear:** +
- +
-<code> +
- stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |       cfailed         +
------------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------------------- +
-     18230 | 2018-09-09 02:00:00 |      4.3 |      87 | 2018-09-09 01:56:30 |    330 | 0      |     0 |       4.3 | 1101000000000000 | 7000000000000000 | +
-     18230 | 2018-09-09 03:00:00 |       18 |      87 | 2018-09-09 02:56:33 |    330 | 0      |     0 |       4.2 | 1109000000000000 | 5033300000000001 | QC1-3a-87:1,QC2d-1-L +
-     18230 | 2018-09-09 04:00:00 |        4 |      87 | 2018-09-09 03:57:17 |    330 | 0      |     0 |         4 | 1104000000000000 | 5000000000000001 | QC2d-1 +
-     18230 | 2018-09-09 05:00:00 |      4.1 |      87 | 2018-09-09 04:56:28 |    330 | 0      |     0 |       4.1 | </code> +
- +
-**Akima:** +
- +
-   * Note for testing only the linear value is also included in cfailed (see above). +
- +
-<code> +
- stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |              cfailed               +
------------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------ +
-     12320 | 2018-09-25 18:00:00 |    381.7 |     104 | 2018-09-25 17:54:15 |    330 | 0      |     0 |     381.7 | 1101000000100100 | 7000000000000000 | +
-     12320 | 2018-09-25 19:00:00 |    381.6 |     104 | 2018-09-25 18:53:50 |    330 | 0      |     0 |     381.6 | 1101000000100100 | 7000000000000000 | +
-     12320 | 2018-09-25 20:00:00 |    284.6 |     104 | 2018-09-25 19:54:35 |    330 | 0      |     0 |     381.6 | 1109000000100100 | 5033300000000001 | QC1-3a-104:1,QC2d-1-A,Linear=381.6 +
-    12320 | 2018-09-25 21:00:00 |    381.6 |     104 | 2018-09-25 20:54:44 |    330 | 0      |     0 |     381.6 | 1104000000100100 | 5000000000000001 | QC1-3a-104:1,QC2d-1 +
-     12320 | 2018-09-25 22:00:00 |    381.6 |     104 | 2018-09-25 21:53:48 |    330 | 0      |     0 |     381.6 | 1101000000100100 | 7000000000000000 | +
-     12320 | 2018-09-25 23:00:00 |    381.6 |     104 | 2018-09-25 22:53:46 |    330 | 0      |     0 |     381.6 | 1101000000100100 | 7000000000000000 | +
-(7 rows) +
-</code> +
- +
-More test results are compiled under: [[https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:test:algorithms:diptest]] +
- +
-**Flag specification chnages** +
- +
-   * An update of the Flag specification has been made and a test is illustrated here: [[https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:test:algorithms:diptest#flag_document_version_97_verification]] +
- +
- +
- +
-====== Parameters ====== +
- +
-Relevant parameters are all those subject to the step check (QC1-3a): +
-^ paramid ^ parameter ^ δ(X) ^ +
-|  **1**   | AA |  | +
-|  **2**   | BI|  | +
-|  **81**  | FF| 10 or 12,5  m/s TBC | +
-|  **85**  | FM|  | +
-|  **86**  | FX|  | +
-|  **87**  | FX_1|  | +
-|  **88**  | FX_6|  | +
-|  **89**  | FX_12|  | +
-|  **90**  | FG_1|  | +
-|  **91**  | FG_6|  | +
-|  **92**  | FG_12|  | +
-|  **93**  | FX_X|  | +
-|  **94**  | FG_X|  | +
-|  **104**  | RA|  | +
-|  **105**  | RR_01|  | +
-|  **113**  | SS_1|  | +
-|  **123**  | RT_1|  | +
-|  **172**  | PH|  | +
-|  **174**  | POM|  | +
-|  **175**  | PON|  | +
-|  **176**  | POX|  | +
-|  **177**  | PP|  | +
-|  **197**  | QE|  | +
-|  **198**  | QL|  | +
-|  **199**  | QLX|  | +
-|  **200**  | QO|  | +
-|  **201**  | QOX|  | +
-|  **211**  | TA| 7,5 °C TBC| +
-|  **212**  | TAM|  | +
-|  **213**  | TAN|  | +
-|  **215**  | TAX|  | +
-|  **221**  | TG|  | +
-|  **222**  | TGM|  | +
-|  **223**  | TGN|  | +
-|  **225**  | TGX|  | +
-|  **227**  | TJM|  | +
-|  **262**  | UU| 30% TBC| +
-|  **264**  | UN|  | +
-|  **265**  | UX|  | +
- +
-===== Dip test flags ===== +
-|Flag value|Definition|useinfo(2) consequence| +
-|**fs=4**|Kontrollert. Mistanke om feil detektert i QC1-3. Ingen tilsvarende feil i QC2d-1. Ingen korreksjon.|**Friskmeldt.**| +
-|fs=5|Kontrollert. Ingen mistanke om feil detektert i QC1-3. Mistanke om feil detektert i QC2d-1.Ingen korreksjon.|Originalverdi noe mistenkelig.| +
-|**fs=9**|Kontrollert. Observert endring høyere enn testverdi. Korrigert automatisk.|**Originalverdi sikkert feilaktig.**| +
- +
-===== Previous met.no dip tests =====+
  
 The report 24/93 KLIMA "Theoretical analysis of the dip-test in quality control of geophysical observations" by Petter Øgland describes a diptest run independent from a previous step test. The report 24/93 KLIMA "Theoretical analysis of the dip-test in quality control of geophysical observations" by Petter Øgland describes a diptest run independent from a previous step test.
Line 163: Line 65:
    δ(UU) = 30%  versus the Kvalobs step(UU) = 30%    δ(UU) = 30%  versus the Kvalobs step(UU) = 30%
    δ(FF) = 30 m/s versus the Kvalobs step(FF) = 10 or 12.5 m/s    δ(FF) = 30 m/s versus the Kvalobs step(FF) = 10 or 12.5 m/s
-   δ(TA) = 10.5 °C versus the Kvalobs step(TA) = 7,5 °C+   δ(TA) = 10.5 °C versus the Kvalobs step(TA) = 7.5 °C 
 + 
 +For wind and temperature the Kvalobs dip test will flag more often than Øgland's formula. For humidity Øgland's formula will flag for errors without corresponding flag from the Kvalobs dip test. This occurs in a "skew" dip when the step test criterion is met in one of the two cases.
  
-For wind and temperature the Kvalobs dip test will flag more often than by Øgland's formula. For humidity will Øgland's formula gives error flags without corresponding flag from the Kvalobs dip test. This occurs in a "skew" dip when the step test criterion is met in one of the two cases. 
  • kvalobs/kvoss/system/qc2/requirements/algorithms/diptest03.1302974144.txt.gz
  • Last modified: 2022-05-31 09:23:18
  • (external edit)