Dip test - QC2d-1 - Revision 03 (Final Version ???)

fs=2 for A(t),

fs=2 for A(t-1)

fs=1 for A(t-2)

If
| A(t) - A(t-2) | < | A(t-1) - A(t-2) |
and
| A(t) - A(t-2) | < DELTA

where DELTA is dependent on the parameter

set fs=9 for A(t-1)
set fs=6 for A(t)

A(t-1) receives a corrected value by interpolation. The interpolation method can be linear or by Akima algorithm.

Strategy

Configuration File

#Algorithm
Algorithm=DipTest
RunAtHour=21
RunAtMinute=53
# Time interval specification (can also be a specific start and stop time)
Last_NDays=3
#[Specific Data Type and Paramters ids etc.]
ParValFilename="/etc/kvalobs/Qc2Config/TestParVal.txt"
#Flag required for neighbour to be used for an interpolation
U_2=0
# Only write data it it has not been analysed by HQC
W_fhqc=0

There are some important new features in the configuration file:

The specification of the algorithm is now handled by a string, i.e. Algorith=DipTest (or “DipTest” will work) rather than a code (AlgoCode=11 etc). This is now adopted for all algorithms to be run - clearer and more user friendly.

For the first algorithms (e.g. SingleLinear) a separate configuration file exists for each paramid. This leads to many separate configuration files and is rather cumbersome. Here there is one configuration file that points to an additional file that includes the parameter ids to process and any other specific information. In this case the file includes the DELTA paramter (see next excerpt below). Later all this information can be managed in a Qc2 table in the database if this is preferred.

Paramter ID file

1 9.0
2 60.0
81 10.0
85 10.0
86 12.5
...

The specific criteria for the DipTest (i.e. fs = 1, 2, 2 for three consecutive points and the value checks defined above) are, for the first version, encoded in the algorithm.

The algorithm steps back in time checking each successive set of three points (sliding window). If a dip is confirmed a linear interpolation is first applied using the immediate temporal neighbours. The algorithm then checks if there are enough neighbours available of sufficient quality (U_2=0, set in the config file) to perform an Akima Interpolation. If this is satisfied the Akima value is used in preference to that generated from the linear interpolation.

For each case the results in the kvalobs database are as follows:

Linear:

Akima:

Note for testing only the linear value is also included in cfailed (see above).

An update of the Flag specification has beeb made and an example of the change is illustrated here:

Parameters

Relevant parameters are all those subject to the step check (QC1-3a):

paramid	parameter	δ(X)
1	AA
2	BI
81	FF	10 or 12,5 m/s TBC
85	FM
86	FX
87	FX_1
88	FX_6
89	FX_12
90	FG_1
91	FG_6
92	FG_12
93	FX_X
94	FG_X
104	RA
105	RR_01
113	SS_1
123	RT_1
172	PH
174	POM
175	PON
176	POX
177	PP
197	QE
198	QL
199	QLX
200	QO
201	QOX
211	TA	7,5 °C TBC
212	TAM
213	TAN
215	TAX
221	TG
222	TGM
223	TGN
225	TGX
227	TJM
262	UU	30% TBC
264	UN
265	UX

Dip test flags

Flag value	Definition	useinfo(2) consequence
fs=4	Kontrollert. Mistanke om feil detektert i QC1-3. Ingen tilsvarende feil i QC2d-1. Ingen korreksjon.	Friskmeldt.
fs=5	Kontrollert. Ingen mistanke om feil detektert i QC1-3. Mistanke om feil detektert i QC2d-1.Ingen korreksjon.	Originalverdi noe mistenkelig.
fs=9	Kontrollert. Observert endring høyere enn testverdi. Korrigert automatisk.	Originalverdi sikkert feilaktig.

Previous met.no dip tests

The report 24/93 KLIMA “Theoretical analysis of the dip-test in quality control of geophysical observations” by Petter Øgland describes a diptest run independent from a previous step test.

http://klima.dnmi.no/rapporter/Klimarapporter/1993/24_93.pdf

Comparison between Øgland's parameter δ and the current Kvalobs step criterium shows:

δ(UU) = 30% versus the Kvalobs step(UU) = 30%
δ(FF) = 30 m/s versus the Kvalobs step(FF) = 10 or 12.5 m/s
δ(TA) = 10.5 °C versus the Kvalobs step(TA) = 7,5 °C

For wind and temperature the Kvalobs dip test will flag more often than by Øgland's formula. For humidity will Øgland's formula gives error flags without corresponding flag from the Kvalobs dip test. This occurs in a “skew” dip when the step test criterion is met in one of the two cases.