• fs=2 for A(t),
• fs=2 for A(t-1)
• fs=1 for A(t-2)

 If  
    | A(t) - A(t-2) | < | A(t-1) - A(t-2) | 
 and 
    | A(t) - A(t-2) | < DELTA

where DELTA is dependent on the parameter

    
   set fs=9 for A(t-1) 
   set fs=6 for A(t)

A(t-1) receives a corrected value by interpolation. The interpolation method can be linear or by Akima algorithm.

Configuration File

#Algorithm
Algorithm=DipTest             

RunAtHour=21
RunAtMinute=53

# Time interval specification (can also be a specific start and stop time)
Last_NDays=3

#[Specific Data Type and Paramters ids etc.]
ParValFilename="/etc/kvalobs/Qc2Config/TestParVal.txt"  

#Flag required for neighbour to be used for an interpolation
U_2=0

# Only write data it it has not been analysed by HQC
W_fhqc=0

There are some important new features in the configuration file:

1. The specification of the algorithm is now handled by a string, i.e. Algorith=DipTest (or “DipTest” will work) rather than a code (AlgoCode=11 etc). This is now adopted for all algorithms to be run - clearer and more user friendly.
2. For the first algorithms (e.g. SingleLinear) a separate configuration file exists for each paramid. This leads to many separate configuration files and is rather cumbersome. Here there is one configuration file that points to an additional file that includes the parameter ids to process and any other specific information. In this case the file includes the DELTA paramter (see next excerpt below). Later all this information can be managed in a Qc2 table in the database if this is preferred.

Paramter ID file

1   9.0
2   60.0
81   10.0
85   10.0
86   12.5
...

The specific criteria for the DipTest (i.e. fs = 1, 2, 2 for three consecutive points and the value checks defined above) are, for the first version, encoded in the algorithm.

The algorithm steps back in time checking each successive set of three points (sliding window). If a dip is confirmed a linear interpolation is first applied using the immediate temporal neighbours. The algorithm then checks if there are enough neighbours available of sufficient quality (U_2=0, set in the config file) to perform an Akima Interpolation. If this is satisfied the Akima value is used in preference to that generated from the linear interpolation.

For each case the results in the kvalobs database are as follows:

Linear:

 stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |       cfailed        
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------------------
     18230 | 2018-09-09 02:00:00 |      4.3 |      87 | 2018-09-09 01:56:30 |    330 | 0      |     0 |       4.3 | 1101000000000000 | 7000000000000000 |
     18230 | 2018-09-09 03:00:00 |       18 |      87 | 2018-09-09 02:56:33 |    330 | 0      |     0 |       4.2 | 1109000000000000 | 5033300000000001 | QC1-3a-87:1,QC2d-1-L
     18230 | 2018-09-09 04:00:00 |        4 |      87 | 2018-09-09 03:57:17 |    330 | 0      |     0 |         4 | 1104000000000000 | 5000000000000001 | QC2d-1
     18230 | 2018-09-09 05:00:00 |      4.1 |      87 | 2018-09-09 04:56:28 |    330 | 0      |     0 |       4.1 | 

Akima:

• Note for testing only the linear value is also included in cfailed (see above).

 stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |              cfailed              
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+------------------------------------
     12320 | 2018-09-25 18:00:00 |    381.7 |     104 | 2018-09-25 17:54:15 |    330 | 0      |     0 |     381.7 | 1101000000100100 | 7000000000000000 |
     12320 | 2018-09-25 19:00:00 |    381.6 |     104 | 2018-09-25 18:53:50 |    330 | 0      |     0 |     381.6 | 1101000000100100 | 7000000000000000 |
     12320 | 2018-09-25 20:00:00 |    284.6 |     104 | 2018-09-25 19:54:35 |    330 | 0      |     0 |     381.6 | 1109000000100100 | 5033300000000001 | QC1-3a-104:1,QC2d-1-A,Linear=381.6
    12320 | 2018-09-25 21:00:00 |    381.6 |     104 | 2018-09-25 20:54:44 |    330 | 0      |     0 |     381.6 | 1104000000100100 | 5000000000000001 | QC1-3a-104:1,QC2d-1
     12320 | 2018-09-25 22:00:00 |    381.6 |     104 | 2018-09-25 21:53:48 |    330 | 0      |     0 |     381.6 | 1101000000100100 | 7000000000000000 |
     12320 | 2018-09-25 23:00:00 |    381.6 |     104 | 2018-09-25 22:53:46 |    330 | 0      |     0 |     381.6 | 1101000000100100 | 7000000000000000 |
(7 rows)

More test results are compiled under: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:test:algorithms:diptest

Flag specification chnages

• An update of the Flag specification has been made and a test is illustrated here: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:test:algorithms:diptest#flag_document_version_97_verification

Relevant parameters are all those subject to the step check (QC1-3a):

The report 24/93 KLIMA “Theoretical analysis of the dip-test in quality control of geophysical observations” by Petter Øgland describes a diptest run independent from a previous step test.

http://klima.dnmi.no/rapporter/Klimarapporter/1993/24_93.pdf

Comparison between Øgland's parameter δ and the current Kvalobs step criterium shows:

 δ(UU) = 30%  versus the Kvalobs step(UU) = 30%
 δ(FF) = 30 m/s versus the Kvalobs step(FF) = 10 or 12.5 m/s
 δ(TA) = 10.5 °C versus the Kvalobs step(TA) = 7,5 °C

For wind and temperature the Kvalobs dip test will flag more often than by Øgland's formula. For humidity will Øgland's formula gives error flags without corresponding flag from the Kvalobs dip test. This occurs in a “skew” dip when the step test criterion is met in one of the two cases.