kvalobs:kvoss:system:qc2:user:version:kvqc2d-1.1.2

This is an old revision of the document!

…. in preparation … FIXME

kvqc2d Version: 1.1.0

8-) To Do 8-) When deploying into operation the kvalobs control scripts have to be updated to call “kvqc2d” and not “kvqc2”

In consequence, these are suggested steps to follow:

  • kvstop -l {and select the value for kvqc2}
  • SYSTEM: apt-get remove kvqc2
  • SYSTEM: apt-get install kvqc2d
  • run kvqc2d on the command line first to check all paths are in order (and also all the log files with new names are created) … Ctrl+C to Quit
  • update /etc/kvalobs/kv_ctl.conf … i.e. replace all kvqc2 with kvqc2d
  • kvstart

Source: https://svn.met.no/viewvc/kvoss/kvQc2/branches/kvqc2d-1.1.0/

Algorithms in this release:

AlgoCode Name Description
10 SingleLinear_v32 Replaces a single missing value with a linear interpolated value. If a corresponding max or min paramids are also specified then checks that the correction lies within the available max … min range, if the interpolated values falls outside this range then the algorithm sets the corrected value to the nearest of max or min. Run for any paramid and optional max, min specified in the configuration file. New version updated with respect to a revised specification
1 Redistribution of RR_24 See https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:requirements:algorithms:rarr_24 and config file below.

Sample data from the new algorithm (scroll right for cfailed format)

NB on dev-vm101 this data could only be generated with I_fd=0 … normally this is set to I_fd=1 so that only QC1-controlled data is used for interpolation model values for the redistribution. 

kvalobs=# select * from data where obstime between '2010-12-07' and 'today' and paramid=110 and cfailed like '%QC2-redist%' order by stationid, obstime;
 stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |                                  cfailed                                  
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+---------------------------------------------------------------------------
     25260 | 2010-12-07 06:00:00 |   -32767 |     110 | 2010-12-08 00:30:21 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_25320_25630_25640_25100_25830_24890,QC2-redist
     25260 | 2010-12-08 06:00:00 |   -32767 |     110 | 2010-12-09 00:30:27 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_25320_25630_25640_25100_53950_25830_24890,QC2-redist
     25260 | 2010-12-09 06:00:00 |       -1 |     110 | 2010-12-09 07:21:52 |    302 | 0      |     0 |        -1 | 0110004000007000 | 5336900000000000 | QC1-7-110,QC2N_25320_25630_25640_25100_53950_25830_24890,QC2-redist

     45350 | 2010-12-07 06:00:00 |   -32767 |     110 | 2010-12-08 00:30:35 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_42940_45530_45600_45870_44760_42810_42520,QC2-redist
     45350 | 2010-12-08 06:00:00 |   -32767 |     110 | 2010-12-09 00:30:25 |    302 | 0      |     0 |       5.3 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_42940_45530_45600_45870_44760_42810_42520_44800,QC2-redist
     45350 | 2010-12-09 06:00:00 |        9 |     110 | 2010-12-09 11:40:20 |    302 | 0      |     0 |       3.7 | 0110004000007000 | 5336900000000000 | QC1-7-110,QC2N_42940_45530_45600_45870_44760_42810_42520_44800,QC2-redist

     84070 | 2010-12-07 06:00:00 |   -32767 |     110 | 2010-12-08 00:30:48 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_84190,QC2-redist
     84070 | 2010-12-08 06:00:00 |   -32767 |     110 | 2010-12-09 00:30:33 |    302 | 0      |     0 |       0.2 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_84190_85180_83710,QC2-redist
     84070 | 2010-12-09 06:00:00 |      1.5 |     110 | 2010-12-09 07:03:10 |    302 | 0      |     0 |       1.3 | 0140004000007000 | 5336900000000001 | QC1-2-72.b12,QC1-7-110,QC2N_84190_83710,QC2-redist
(9 rows)

Example Configuration Files

SingleLinear_v32

*NB* Different/simpler checks applied to the neighbours (CF version 1.0.5) 

# AlgCode 10 corresponds to https://svn.met.no/kvoss/kvQc2/branches/kvqc2-1.0.1/src/algorithms/SingleLinear.cc
# Performs simple linear interpolation for a single point replacement.
# If the result is outside an available max min range, the nearest of the
# max or min values are used.
AlgoCode=10
# Time to run the algorithm
RunAtHour=16
RunAtMinute=47
# Check data from the last three days
Last_NDays=40

#[Specific Data Type and Paramters ids etc.]
ParamId=211
# MaxParamId=215   # Optional  !!!! The Max and Min options are no longer required (to be modified).
# MinParamId=213   # Optional
# Only write back the result if not previously controlled
W_fhqc=0

#Flag to set if value is corrected
S_ftime=1
change_fmis=3->1
change_fmis=0->4
change_fmis=2->4

#If there is already a 1 in place then we do not want run the algorithm again!
U_2=0
Redistribution 
# Algorithm configuration for redistribution of accumulated precipitation
AlgoCode=1
InterpCode=2
RunAtHour=23
RunAtMinute=35
Last_NDays=14

#[Time Range] The time window for when to apply the algorithm
#Start_YYYY=2010
#Start_MM=10 
#Start_DD=8
#Start_hh=6
#Start_mm=0
#Start_ss=0

#End_YYYY=2010 
#End_MM=10
#End_DD=11
#End_hh=6
#End_mm=0
#End_ss=0


Start_hh=6
End_hh=6   # NB 24 hour values are at 06:00 UTC

#[Time Step]
Step_DD=1

#[Specific Data Type and Paramters ids etc.]
ParamId=110
TypeIds=302
TypeIds=402

#
# Only write data it it has not been analysed by HQC
W_fhqc=0

# Added for a special test run
# CfailedString="any text"


# Condition for running the algorithm
A_fd=2

# Flags to set on successful completion
S_fd=7
change_fmis=3->1
change_fmis=0->4

#Only interpolate the following data
I_fd=1

InterpolationDistance=50.0

Notes

  • This algorithm will redistribute data flagged with fd=2 (A_fd) and use neighbours with fd=1 for interpolation (I_fd) and set fd=7 (S_fd) for the new corrected data. But fhqc must be 0 for data to be written back.
  • To re-run the algorithm on the same data again set A_fd=7 and keep W_fhqc=0, i.e. do not re-write any decision by HQC.

Bugs and new requirements addressed in this release

Most of the bugs opened in response to emails.

General bugs identified in algorithm testing

https://kvoss.bugs.met.no/show_bug.cgi?id=1296

Test results are at the foot of this report: https://kvalobs.wiki.met.no/doku.php?id=kvoss:system:qc2:test:algorithms:redistribution-kvqc2d#interval_teston_development_machine

Name Changes

https://kvoss.bugs.met.no/show_bug.cgi?id=1302

  • renaming of kvqc2 to kvqc2d
  • renaming of Qc2.log to kvqc2d.log

Additional logging information

https://kvoss.bugs.met.no/show_bug.cgi?id=1303

The log file now looks like this: 

20101213010007: INFO 
--------------- Case 1: Redistribute
20101213010007: INFO 
--------------- Redistribute Accumulations
20101213010458: INFO 
--------------- Redistribution: 45350 2010-12-7 6:0:0 -32767 110 302 0 0 -1
20101213010458: INFO 
--------------- Redistribution: 45350 2010-12-8 6:0:0 -32767 110 302 0 0 5.3
20101213010458: INFO 
--------------- Redistribution: 45350 2010-12-9 6:0:0 9 110 302 0 0 3.7
20101213010459: INFO 
--------------- Redistribution: 25260 2010-12-7 6:0:0 -32767 110 302 0 0 -1
20101213010459: INFO 
--------------- Redistribution: 25260 2010-12-8 6:0:0 -32767 110 302 0 0 -1
20101213010459: INFO 
--------------- Redistribution: 25260 2010-12-9 6:0:0 -1 110 302 0 0 -1
20101213010459: INFO 
--------------- Redistribution: 84070 2010-12-7 6:0:0 -32767 110 302 0 0 -1
20101213010459: INFO 
--------------- Redistribution: 84070 2010-12-8 6:0:0 -32767 110 302 0 0 0.2
20101213010459: INFO 
--------------- Redistribution: 84070 2010-12-9 6:0:0 1.5 110 302 0 0 1.3
20101213010551: INFO 
--------------- Case 1: Completed

By default assume dry (0 -> -1)

https://kvoss.bugs.met.no/show_bug.cgi?id=1304

For example: 

 stationid |       obstime       | original | paramid |       tbtime        | typeid | sensor | level | corrected |   controlinfo    |     useinfo      |                                     cfailed                                      
-----------+---------------------+----------+---------+---------------------+--------+--------+-------+-----------+------------------+------------------+----------------------------------------------------------------------------------

     55550 | 2010-12-20 06:00:00 |   -32767 |     110 | 2010-12-21 00:30:55 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_53070_58480,QC2-redist
     55550 | 2010-12-21 06:00:00 |   -32767 |     110 | 2010-12-22 00:30:52 |    302 | 0      |     0 |       1.1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_53070_53700_58480,QC2-redist
     55550 | 2010-12-22 06:00:00 |      2.6 |     110 | 2010-12-22 15:16:50 |    302 | 0      |     0 |       1.5 | 0140004000007000 | 5336900000000001 | QC1-2-72.b12,QC1-7-110,QC2N_53130_54780_53070_53700_56960_57390_58480,QC2-redist

     84070 | 2010-12-24 06:00:00 |   -32767 |     110 | 2010-12-25 00:31:36 |    302 | 0      |     0 |       0.1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_84190_85180,QC2-redist
     84070 | 2010-12-25 06:00:00 |   -32767 |     110 | 2010-12-27 22:42:19 |    302 | 0      |     0 |       0.1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_84190,QC2-redist
     84070 | 2010-12-26 06:00:00 |   -32767 |     110 | 2010-12-28 07:43:30 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_84190,QC2-redist
     84070 | 2010-12-27 06:00:00 |   -32767 |     110 | 2010-12-28 19:51:24 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_84190,QC2-redist
     84070 | 2010-12-28 06:00:00 |   -32767 |     110 | 2010-12-29 00:31:48 |    302 | 0      |     0 |        -1 | 0000001000007000 | 5896900000000000 | QC1-7-110,QC2N_84190_85180,QC2-redist
     84070 | 2010-12-29 06:00:00 |      0.2 |     110 | 2010-12-29 12:05:13 |    302 | 0      |     0 |        -1 | 0140004000007000 | 5336900000000001 | QC1-2-72.b12,QC1-7-110,QC2N_84190,QC2-redist

Revised typeid policy

https://kvoss.bugs.met.no/show_bug.cgi?id=1305

Only typeids specificed in the configuration file with the new option TypeIds are redistributed. All typeids are used for interpolation.

CFAILED

The format of cfailed for the Redistribution algorithm has been changed (please see example data above).

The schedule for the algorithm is as follows:

  • 06:35 perform a run looking at the last 4 days - the last 4 days is selected since from the first run on murky (see above) about 30 redistributions were made of which 15 were over 3 days and 10 over 4 days.
  • 01:20 Wednesday Night/Thursday Morning run to check the last 14 days (Note: Could we run every night for the last two weeks to pick up any changes?)
This website uses cookies. By using the website, you agree with storing cookies on your computer. Also you acknowledge that you have read and understand our Privacy Policy. If you do not agree leave the website.More information about cookies
  • kvalobs/kvoss/system/qc2/user/version/kvqc2d-1.1.2.1304863036.txt.gz
  • Last modified: 2022-05-31 09:23:19
  • (external edit)