kvalobs:kvoss:system:qc2:requirements:algorithms:diptest03

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
kvalobs:kvoss:system:qc2:requirements:algorithms:diptest03 [2011-07-28 22:33:06]
paule 		kvalobs:kvoss:system:qc2:requirements:algorithms:diptest03 [2022-05-31 09:29:32] (current)
Line 1: Line 1:
-====== Dip test - QC2d-1 - Revision 03 (Final Version)======+====== QC2d-1: Dip test ======
  
-{{:kvoss:system:qc2:requirements:algorithms:dip.png|}}+{{kvalobs:kvoss:system:qc2:requirements:algorithms:dip.png|}}
  
    * fs=2 for A(t),     * fs=2 for A(t), 
Line 7: Line 7:
    * fs=1 for A(t-2)     * fs=1 for A(t-2) 
  
-<code> +A dip is detected if <m>|A(t)-A(t-2)| < |A(t-1)-A(t-2)|</m> and <m>|A(t)-A(t-2)| < delta</m> 
- If   +where <m>delta</m> is specified in the algorithm configuration. 
-    | A(t) - A(t-2) | < | A(t-1) - A(t-2) |  +<m>A(t-1)</m> then receives a corrected value by interpolation.
- and  +
-    | A(t) - A(t-2) | < DELTA +
-</code>+
  
-where DELTA is dependent on the parameter+The neighboring points at t-2 and t may not have missing original values and have 
 +to match the flags specified in **linear_before_*flags** and **linear_after_*flags**, 
 +respectively.
  
-The relevant DELTA is from the stepcheck QC1-3a. In station_param the values are named "max".+If there are 2 additional usable data points before and 1 usable additional data point after t-1, 
 +[[Akima splines]] are used for the interpolation and cfailed receives a message 
 +**QC2d-1-A**. To be usable, the original value may not be missing (-32767) and has 
 +to match the flag specification **akima_*flags** in the configuration if it is not 
 +an immediate neighbor of t-1, and . 
  
-<code>     +If akima interpolation is not possible, linear interpolation is used and cfailed receives 
-   set fs=9 for A(t-1)  +a message **QC2d-1-L**. 
-   set fs=6 for A(t) +
-</code>+
  
-A(t-1) receives a corrected value by interpolation. The interpolation method can be linear or by [[Akima splines]]. +==== Configuration ====
  
 +Beyond the standard configuration options detailed in the [[kvoss:system:qc2:user|user guide]],
 +the following options are available for DipTest.
  
-====== Strategy ======+  * candidate_cflags (default: fs=2&fhqc=0) and candidate_uflags (default: /no constraint/) specify the flags expected for A(t-1). Dips that do not match these flags will not be detected. 
 +  * akima_cflags (default: /no constraint/) and akima_uflags (default: U2=0) specify the requirements for additional points for akima interpolation 
 +  * linear_before_cflags (default: fs=1) and linear_before_uflags (default: /no constraint/) specify flag requirements for A(t-2) 
 +  * linear_after_cflags (default: fs=2&fhqc=0) and linear_after_uflags (default: /no constraint/) specify flag requirements for A(t) 
 +  * dip_flagchange (default: fs=9) specifies the flag modification for t-1 if an interpolation is performed 
 +  * afterdip_flagchange (default: fs=4) specifies the flag modification for t if an interpolation is performed 
 +  * ParValFilename specifies either (no default, must be specified, see below for examples) 
 +    * a filename containing a list of parameter-delta pairs 
 +    * or a list of parameter-value pairs when starting with **list:**
  
-**Configuration File**+=== ParValFilename ===
  
-<code> +If using a file to specify parameter-delta pairs, the file must be formatted like this: 
-#Algorithm +  1   9.0 
-Algorithm=DipTest             +  2   60.0 
 +  81  10.0 
 +  85  10.0 
 +  86  12.5
  
-RunAtHour=21 +If using a list to specify parameter-delta pairs, the parameter must be given like this: 
-RunAtMinute=53+  ParValFilename= list: 172 12.0, 174 12.0, 175 12.0, 176 12.0, 177 12.0, 211 7.5
  
-# Time interval specification (can also be a specific start and stop time) +==== Parameters ====
-Last_NDays=3+
  
-#[Specific Data Type and Paramters ids etc.] +Relevant parameters are all those subject to the step check (QC1-3a). The implementation is only tested for hourly measurements.
-ParValFilename="/etc/kvalobs/Qc2Config/TestParVal.txt"  +
  
-#Flag required for neighbour to be used for an interpolation +==== Previous met.no dip tests ====
-U_2=0 +
- +
-# Only write data it it has not been analysed by HQC +
-W_fhqc=0 +
-</code> +
- +
-There are some important new features in the configuration file: +
-  - The specification of the algorithm is now handled by a string, i.e. Algorithm=DipTest (or "DipTest" will work) rather than a code (AlgoCode=11 etc). This is now adopted for all algorithms to be run - clearer and more user friendly. +
-  - For the first algorithms (e.g. SingleLinear) a separate configuration file exists for each paramid. This leads to many separate configuration files and is rather cumbersome. Here there is one configuration file that points to an additional file that includes the parameter ids to process and any other specific information. In this case the file includes the DELTA paramter (see next excerpt below). Later all this information can be managed in a Qc2 table in the database if this is preferred. +
- +
-**Paramter ID file** +
- +
-<code> +
-1   9.0 +
-2   60.0 +
-81   10.0 +
-85   10.0 +
-86   12.5 +
-... +
-</code> +
- +
-The specific criteria for the DipTest (i.e. fs = 1, 2, 2 for three consecutive points and the value checks defined above) are, for the first version, encoded in the algorithm. +
- +
-The algorithm steps back in time checking each successive set of three points (sliding window). If a dip is confirmed a linear interpolation is first applied using the immediate temporal neighbours. The algorithm then checks if there are enough neighbours available of sufficient quality (U_2=0, set in the config file) to perform an Akima Interpolation. If this is satisfied the Akima value is used in preference to that generated from the linear interpolation. +
- +
-For each case the results in the kvalobs database are as follows: +
- +
-**Linear:** +
- +
-<code> +
- stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |       cfailed         +
------------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------------------- +
-     18230 | 2018-09-09 02:00:00 |      4.3 |      87 | 2018-09-09 01:56:30 |    330 | 0      |     0 |       4.3 | 1101000000000000 | 7000000000000000 | +
-     18230 | 2018-09-09 03:00:00 |       18 |      87 | 2018-09-09 02:56:33 |    330 | 0      |     0 |       4.2 | 1109000000000000 | 5033300000000001 | QC1-3a-87:1,QC2d-1-L +
-     18230 | 2018-09-09 04:00:00 |        4 |      87 | 2018-09-09 03:57:17 |    330 | 0      |     0 |         4 | 1104000000000000 | 5000000000000001 | QC2d-1 +
-     18230 | 2018-09-09 05:00:00 |      4.1 |      87 | 2018-09-09 04:56:28 |    330 | 0      |     0 |       4.1 | </code> +
- +
-**Akima:** +
- +
-   * Note for testing only the linear value is also included in cfailed (see above). +
- +
-<code> +
- stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |              cfailed               +
------------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------ +
-     12320 | 2018-09-25 18:00:00 |    381.7 |     104 | 2018-09-25 17:54:15 |    330 | 0      |     0 |     381.7 | 1101000000100100 | 7000000000000000 | +
-     12320 | 2018-09-25 19:00:00 |    381.6 |     104 | 2018-09-25 18:53:50 |    330 | 0      |     0 |     381.6 | 1101000000100100 | 7000000000000000 | +
-     12320 | 2018-09-25 20:00:00 |    284.6 |     104 | 2018-09-25 19:54:35 |    330 | 0      |     0 |     381.6 | 1109000000100100 | 5033300000000001 | QC1-3a-104:1,QC2d-1-A,Linear=381.6 +
-    12320 | 2018-09-25 21:00:00 |    381.6 |     104 | 2018-09-25 20:54:44 |    330 | 0      |     0 |     381.6 | 1104000000100100 | 5000000000000001 | QC1-3a-104:1,QC2d-1 +
-     12320 | 2018-09-25 22:00:00 |    381.6 |     104 | 2018-09-25 21:53:48 |    330 | 0      |     0 |     381.6 | 1101000000100100 | 7000000000000000 | +
-     12320 | 2018-09-25 23:00:00 |    381.6 |     104 | 2018-09-25 22:53:46 |    330 | 0      |     0 |     381.6 | 1101000000100100 | 7000000000000000 | +
-(7 rows) +
-</code> +
- +
-More test results are compiled under: [[https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:test:algorithms:diptest]] +
- +
-**Flag specification chnages** +
- +
-   * An update of the Flag specification has been made and a test is illustrated here: [[https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:test:algorithms:diptest#flag_document_version_97_verification]] +
- +
- +
- +
-====== Parameters ====== +
- +
-Relevant parameters are all those subject to the step check (QC1-3a): +
-^ paramid ^ parameter ^ δ(X) ^ +
-|  **1**   | AA |  | +
-|  **2**   | BI|  | +
-|  **81**  | FF| 10 or 12,5  m/s TBC | +
-|  **85**  | FM|  | +
-|  **86**  | FX|  | +
-|  **87**  | FX_1|  | +
-|  **88**  | FX_6|  | +
-|  **89**  | FX_12|  | +
-|  **90**  | FG_1|  | +
-|  **91**  | FG_6|  | +
-|  **92**  | FG_12|  | +
-|  **93**  | FX_X|  | +
-|  **94**  | FG_X|  | +
-|  **104**  | RA|  | +
-|  **105**  | RR_01|  | +
-|  **113**  | SS_1|  | +
-|  **123**  | RT_1|  | +
-|  **172**  | PH|  | +
-|  **174**  | POM|  | +
-|  **175**  | PON|  | +
-|  **176**  | POX|  | +
-|  **177**  | PP|  | +
-|  **197**  | QE|  | +
-|  **198**  | QL|  | +
-|  **199**  | QLX|  | +
-|  **200**  | QO|  | +
-|  **201**  | QOX|  | +
-|  **211**  | TA| 7,5 °C TBC| +
-|  **212**  | TAM|  | +
-|  **213**  | TAN|  | +
-|  **215**  | TAX|  | +
-|  **221**  | TG|  | +
-|  **222**  | TGM|  | +
-|  **223**  | TGN|  | +
-|  **225**  | TGX|  | +
-|  **227**  | TJM|  | +
-|  **262**  | UU| 30% TBC| +
-|  **264**  | UN|  | +
-|  **265**  | UX|  | +
- +
-===== Dip test flags ===== +
-|Flag value|Definition|useinfo(2) consequence| +
-|**fs=4**|Kontrollert. Mistanke om feil detektert i QC1-3. Ingen tilsvarende feil i QC2d-1. Ingen korreksjon.|**Friskmeldt.**| +
-|fs=5|Kontrollert. Ingen mistanke om feil detektert i QC1-3. Mistanke om feil detektert i QC2d-1.Ingen korreksjon.|Originalverdi noe mistenkelig.| +
-|**fs=9**|Kontrollert. Observert endring høyere enn testverdi. Korrigert automatisk.|**Originalverdi sikkert feilaktig.**| +
- +
-===== Previous met.no dip tests =====+
  
 The report 24/93 KLIMA "Theoretical analysis of the dip-test in quality control of geophysical observations" by Petter Øgland describes a diptest run independent from a previous step test. The report 24/93 KLIMA "Theoretical analysis of the dip-test in quality control of geophysical observations" by Petter Øgland describes a diptest run independent from a previous step test.
Line 165: Line 65:
    δ(UU) = 30%  versus the Kvalobs step(UU) = 30%    δ(UU) = 30%  versus the Kvalobs step(UU) = 30%
    δ(FF) = 30 m/s versus the Kvalobs step(FF) = 10 or 12.5 m/s    δ(FF) = 30 m/s versus the Kvalobs step(FF) = 10 or 12.5 m/s
-   δ(TA) = 10.5 °C versus the Kvalobs step(TA) = 7,5 °C+   δ(TA) = 10.5 °C versus the Kvalobs step(TA) = 7.5 °C 
 + 
 +For wind and temperature the Kvalobs dip test will flag more often than Øgland's formula. For humidity Øgland's formula will flag for errors without corresponding flag from the Kvalobs dip test. This occurs in a "skew" dip when the step test criterion is met in one of the two cases.
  
-For wind and temperature the Kvalobs dip test will flag more often than by Øgland's formula. For humidity will Øgland's formula gives error flags without corresponding flag from the Kvalobs dip test. This occurs in a "skew" dip when the step test criterion is met in one of the two cases. 
  • kvalobs/kvoss/system/qc2/requirements/algorithms/diptest03.1311892386.txt.gz
  • Last modified: 2022-05-31 09:23:18
  • (external edit)