Duplicate Data Filter

Objective

to identify official value for each time where duplicates exist

use one value for time and space interpolations

all algorithms complicated by duplicates

avoid more lengthy bug searches

Reasons for duplicate data from a station for a give paramid and time:

observations delivered with more that one typeid

multiple sensors

multiple levels (NB this is strictly not a duplicate, but is a case to handle)

Current work arounds

Qc2 algorithms select data on both paramid and typeid to avoid picking up duplicate data, i.e.:

result = dbGate.select(Qc2Data,kvQueries::selectMissingData(params.missing,pid,tid,ProcessTime));

... but this reduces the available data / statistics of the Qc2 algorithm.

A routine is required to parse all data and set the official value for a given duplicate.

To Do

specification of rules to identify official value

specification of required configuration information

define a universal flag to identify the official value in all circumstances