Kvalobs Activity Plan - Version 2.1.1  22.01.2010

Version 2.1.0 Comments: Added some few bugs reported after May 2009.  New activities:  18, 200, 87.  
Version 2.1.1 Comments: Change to higher priority in QC2 activities 72, 75 (1->0) and 74 (2->1)

Status Codes
2	CLOSED	The activity has been completed.
1	OPEN		The activity is awaiting attention.
0	ONGOING	The activity is partly done, but the overall activity is open.

To help identify the status, all CLOSED actions are written in Green Font Colour,
Red indicates the open part of ongoing actions and Black are open actions which are not in progress.
(i.e. Green=Closed    Red/Black=TODO) 
Blue indicates possible SMHI activities (except when used in Link column).	

Table is sorted by  “Pri / Act No / ”  (ascending).   (Activities related to the HQC application are kept in a separate document.)

Pri	Act No.	Status Code	Resp.	System	Task	Description	Link	Status & Comments	Delivery 
Priority details	
0	12	1	VB  1 w	Aggregation	Aggregation in non real time	Needs to run or rerun aggregation in non real time from script.
Input is period, elements / all, typeid / all, stations / all. 
Run response needs to be high at least in Histkvalobs. 
Default: Do not touch HQC marked values		OPEN

	

	
0	13	1	VB  >3 w
LA  
POK	Aggregation	Aggregation new elements	Some new diurnal values to be aggregated.
Calculation of P0 after correction of PR – specification needed
TAM, NN, UU,…		OPEN

Calculation of P0 requires metadata	
	
0	14	1	VB  3 w	Aggregation	Aggregation -parameterize	To parameterize the aggregation function…..kind of stations…
Aggregation of non real time data typeid=404 (Pluviometerdata)
	Bug: 
1175
	OPEN 
GK beskriv behovet

0	15	1	VB  ? d	Aggregation	Flags	No aggregation. Handling fmis=3. Recover the aggregation implies to put fmis=1.	Bug: 1155	OPEN		
0	17	0	VB 1 w
LA
JM	Aggregation	Flag on aggregated values	To compute QC-flag on aggregated values (from observation values)		ONGOING
See activity no. 26, 12
Specification proposal done ? 
Verification and implementation remains.
POK sjekk med LA -> wiki 		
0	18	1	VB  ? d	Aggregation	Aggregation Bug	The aggregator loops in special cases	Bug 1256	OPEN		
0	22	1	VB

12 – 24 w

Se 23	qaBase	QABase does not handle more than one typid for one paramID	Error in flagging as only one instance of the paramid is used. The other instances (other typid) are not used.
Extension to handle several typid per station in checks.
	Bugzilla: 967 	OPEN
Related to activity 23
Refaktorering
Contribution from SMHI?
Task on SMHI/met.no workshop		
0	23	1	VB
OB

Se 22	qaBase
QC1	level and sensor	qaBase handles only 1 sensor per station, and only standard level

DD at 25m is checked by FF at 10 m

Scripts flags only at level and sensor =  0
	
	Mail:
Pål 051005

Bug: 886
(960)	OPEN  
Related to activity 22
Refaktorering.
See activity 62
One station with level>0 has separate checks today. 
Autoobs is not able to assign level, because of dependencies to “monthly files” production for “visauto” – this should be terminated. Proposal: Should use the new KvDataInputClt.jar program ?	Pri 0 for qaBase
Pri 2 for autoobs	
0	29	1	VB  1 d	qaBase	Table data and text_data	Tabellene data og text_data bør få 
en ny kolonne 'edittime': tidspunkt for når siste gang noe er blitt 
skrevet til raden, eller trolig mer nyttig (når QC2 kommer på plass): 
Tidspunkt for sist gang noe nytt ble skrevet i original eller corrected.

Dessuten bør det lages nye indekser i data_history og text_data_history som gjør oppslag i disse tabellene raskere. 	Epost Pål 211108

Bug 1228	OPEN

Proposal: To be solved by a view

0	30	1	VB  1 h	qaBase	Flag specification	"fmis=4: kontrollene trenger å vite om original er 
korrigert eller tilfordelt". Arbeidsmengde forslås endret til 1d (vi 
trenger å sjekke i koden om ny verdi for fmis skaper uforutsette 
problemer)	Bug: 1240
	OPEN

0	31	1	VB	qaBase	Cfailed	Repeating controls several times for the same element. Ref cfailed.
		OPEN
Akt.nr er egentlig 51
Solved by 22 – 23 ?		
0	32	0	BM  5 d	Dataflow from Kvalobs	kvServiced2, persistent subscription	To change kvServiced2:
To remember what has been sent, and to know what to retry sending to subscribers.
		ONGOING
Changes in CORBA interface needed.
Implemented. To be tested.		
0	33	1	BM	Dataflow from Kvalobs	Kvservied2	To send updates when only flags is changed (no change in corr or original)		OPEN
Need to check how this works today ( BN )		
0	37	1	BM
PS

       4 w	Dataflow from Kvalobs	BUFR encoder

Synop encoder	To make BUFR encoding from Kvalobs. 

Errors in synop encoder
Synop encoder should be based on the useinfo flag	

Bug: 1131
1135, 1254 1257	OPEN
BUFR encoding to be done during 2009

To use kvsynopd – see 36
		
0	38	1	TR
BM
ØL
ÅMV	Dataflow from Kvalobs	kv2kro	Use the interface kv2klima to make a new kv2kro
To be sure that all data is transferred to KRO.		OPEN		
0	43	1	BM  1 w	Dataflow to Kvalobs 	Decoder – delete before update	A mix of original and corrected elements is the result as a correction of the entire observation is entered to Kvalobs. The original message should be deleted (from decoder) before entering the correction.
Remind also activity 42 for the opposite problem.	Bugzilla: 941	OPEN

Related to versioning.(av data history)
		
0	45	0	BM  1 w
VB	Dataflow to Kvalobs	Transactions	All data input to the database shall be transactions.  		ONGOING
Partially done   ????? Børge	
Sept 08	
0	46	0	BM
VB 1 w	Dataflow to Kvalobs
Model data	several sets of model data 	To improve the checks and interpolation
Model data of Kalman filtered air temperature (TAK). 
Or even better the adjusted Kalman filter model from WeatherDB (PROFF)		ONGOING
Discussion with Mariken
Better model + Kalman		
0	62	1	GK
BM	Decoding	Autoobs/Comobs-decoding system	Error in decoding IR	Bug: 1224
	OPEN
met.no specific problem
GK checks out this bug		
0	66	1	BM  2 d	Decoding	SYNOP	Some simple bugs	Bug: 1218
1220 1227 
1173 1237
1251 1255	OPEN		
0	67	1	BM  1 d	Decoding	Typeid 302	Detection of  “observations sent to early”
	Bug: 913	OPEN		
0	68	1	BM	Decoding	Official data	To distinguish official data and test data
Using information from Stinfosys…..
To set a new flag in useinfo from decoder?
		Ref  activity 153		
0	70	0	PE
TR    3w

(BM)
(VB)	QC2	Development of overall QC2 architecture including test version and production version	General task to provide a Qc2 system to run all Qc2 algorithms.
NB! QC2 implementation as a separate daemon.

Make test environment for production including activities 71 and 73.
Define tests before production release: 
  - implementation test 
  - system test.
Documentation for production release

To use DB-tables instead of config files??? Pro - contra
	
qaBase should not do any action after QC2
		ONGOING
linked to 71, 72, 73, 74 etc.

Patially done

SMHI ?
		
0	71	0	PE

	QC2	QC2m-2	Redistribution of accumulated days of RR_24
Spatial QC and interpolation. 
		ONGOING
Development of prototype done.
Discuss QC2 with Kvalobs team

0	72	0	PE    2w
	QC2	QC2d-1	Dip check. Hourly and minute data. 

		ONGOING
Minute data= Pluviometer data		
0	73	0	PE	QC2	QC2d-2	Interpolate hourly data.  Akima.
Up to 50% missing ???? for 24 hrs ????		ONGOING
First step to interpolate single missing hourly values by linear interpolation		
0	75	1	PE	QC2	QC2d-3	Spatial-timeseries-checks
	Bug: 895	ONGOING		
0	76	1	LA
PE	QC2	Nordklim acceptance
	To get acceptance of QC2 methods and flagging in Nordklim group		ONGOING		
0	77	1	Klimadiv.	QC2	Acceptance test
	Specification of acceptance test		ONGOING		
0	80	0	KJ
LA 
GK

3 w	QC1	New metadata for QC1	New algorithm for making Range check values as metadata for QC1

QC1-1 and QC1-4

Ship: Physical range values must be improved.
	Mail: Lars 140807
Bugs:
875 
891
914	ONGOING
Specification done  on Kvalobs.wiki
Implementation 
		
0	83	1	GK    ?  w
LA
JM
BN
ØL  	QC1	QC1 checks

Automatic correction	Evaluation of all QC1 checks
SMHI results, what have been made in Python
Change checks in QC1-2 to make automatic corrections for instance for the elements  N/CH		OPEN
Task on workshop SMHI / met.no
Proposal to be made by Zbigniew, Lars, Bjørn.		
0	84	1	TR  1 w	QC1	QC1-5
Position check	Checking the change in position.

		OPEN
		
0	86	1	ØL
GK  

       1 d	QC1	QC1-1
9999-data	Enkelte automatstasjoner leverer fra seg manglende verdier som 9999/999/99 etc. Disse blir lagt inn med denne verdien i original i data-tabellen; Forslag: Original = 9999 og corrected=-32767

Lage en sjekk som behandler dette.	Mail:
Lars 20.09 2005
Bugzilla: 932	OPEN

met.no specific problem 

Forslag se bug 932. Flagg spesifikasjon trengs
		
0	87	1	GK	QC1	QC1-1 auto-snow depth	To use sensor.info as metadata in the check.
Negative original values other than -1 and -3
				
0	91	0	BM
LA
GK
      1 w	HQC-1	The detection of highly suspicious data. Near real time corrections.	Svært mistenkelige data som er ekstra uheldig om går ut fra met.no hvis de er feil (gjelder spesielt temperatur og nedbør) bør ikke gå ut i synop, men heller forkastes av Kvalobs i første omgang, for evt å sendes senere som CCA hvis funnet OK. 

Lars og Gabriel må bli enige om prosedyren for utelukkelse. Vi må finne en metode for å varsle Salen om slike svært mistenkelige verdier, og Salen må være i stand til å korrigere i Kvalobs.

Forkastet gjennom QC 1-9. Godkjennelse må gi CCA.
Synopenkoder må reagere på HQC-rettelse innenfor 24 timer (på endret korrigert verdi).

Forutsetter hyppig sjeklk fra VA.	Mail: Gabriel 07.06 2005 

Bugzilla: 900
	ONGOING
met.no specific problem

QC1-9 implementert for å kunne forkaste verdier som flere sjekker har funnet svært mistenkelige. VA (Salen) har sagt seg villig til å overta ansvaret for avislistene fra IT-operatørene. 
Spesifisere flaggkriterier.  Lars, Gabriel, Pål
Implementering		
0	100	0	PS

        2 d	Hist-kvalobs	Deleting old data in Kvalobs	Need to have 4 months in Kvalobs, and the older data only in Histkvalobs.

Tables: data, text_data, rejectdecode 
		ONGOING
Cron jobs to copy from Kvalobs to Histkvalobs, and delete in Kvalobs.
Automation to be done.	Oct 08	
0	101	0	TR
BN	Hist-kvalobs	Historical Kvalobs database	Database for historical data and data not finally QC’ed in Kvalobs. Receives data from Kvalobs before deleted in Kvalobs.

Porting of Histkvalobs to etch
Operational function in Klimadiv.

Ny server må skaffes.		ONGOING
HIST_KVALOBS is running on poriaz.
Operational function in Klimadiv, missing instructions.
To be updated with metadata from Stinfosys.
New server is ordered.
		
0	130	0	POK
GK
Åse
Knut J
	Datatyper og stasjoner	Foreign stations	Per februar 2009 går 90 nordiske stasjoner inn i Kvalobs.

FoU har bestilt de såkalte EWGLAM-stasjonene via Kvalobs til Klimadb for verifisering av modellen.  
Dokumentet EWGLAM.synop.20090205 (med 301 stasjoner). Den nye stasjonslista inneholder de utenlandske stasjonene som pr. 
idag lagres i 'miopdb' Grunnen til at det er noe færre i den nye 
stasjonslista, er at den såkalte EWGLAM-lista som brukes for 
verifikasjonsformål i Hirlam-samarbeidet ikke har blitt oppdatert de 
siste årene, slik at endel stasjoner trolig har falt fra uten at nye er 
lagt til.

Dokumentet EWGLAM.temp.20090205 (med 52 radiosondestasjoner). Den inneholder de radiosondestasjonene som lagres i 'miopdb' pr. idag, bortsett fra de som ikke hadde noen observasjoner i 2008 (f.eks. Gardermoen og Oslo).

Vi ønsker flg. parametere for synop'er: 
NN, DD, FF, FX, RR_12, RR_24, PP, PR, TA, TD
Høyest prioritet har: PR, TA, DD, FF, RR24, N, FX

Fra radiosondestasjoner bruker vi: Z, DD, FF, TT og TD

Det er også ønske fra den nye værportalen om at alle utenlandske synop vi mottar blir videredistribuert til KlimaDVH via Kvalobs. 

Enkel grensesjekk ønskes på synop.
		ONGOING
met.no specific

For FoU verifisering.  Spesifikasjon fra Mariken / Eivind. (POK)

Legge inn stasjonsliste i Kvalobs for synop stasjonene. (kvdrift)

Sette opp grensesjekken på synop

Legge inn stasjonsliste i kv2klimafilter for overføring til KDVH (Åse)

Sondedata – første skritt: Ta inn stasjonene i KDVH (Knut J.) ved å bruke dagens kanal. Sjekk elementene.		
0	151	0	LA
GK
VB
PS	Flag	Useinfo(2)=2
Useinfo(3)=0	“Very uncertain” – “not corrected” values in DVH are rejected to show up in yr.no. 
Change the algorithm for controlinfo giving useinfo(2)=2 ?
Change which checks to use?
Change parameterization of checks giving the controlinfo?

Nedgradere algoritmen for useinfo?

Today: wsKlima for yr is a separate process, we have “adjusted” the “very uncertain level” to “uncertain” to get more data on yr. This will be set back as yr will take care of this later.

	Mail:
Gabriel 251108	ONGOING
Discussion in progress 		
0	152	0	GK
LA
PS
PE
	Flag	QC2 flags	Specification of controlinfo for QC2.
Spesification of useinfo after QC2
Conflict between QC2 and QC1?		ONGOING
Discussion in progress
SMHI		
0	153	0	GK
LA
PS	Flag	Test data flag	Specification of flag to distinguish official data and test data
Proposal:
Useinfo(6) = 0  main element  = official element
Useinfo(6) = 1  secondary element = test element
Useinfo(6) = 9 status not given
		ONGOING
Ref.  68
Discussion in progress
		
0	154	0	SMHI	Flag	Fd=4  ?	Needs to flag missing values as possible ongoing accumulation for precipitation element.
	Mail from SMHI	ONGOING
Discussed at met.no in feb-09
GK answer SMHI
Implementation S		
0	170	0	SA
BF	Diana	SMS-precipitation stations in the ordinary Diana	To be based on text files.
It should be QC1-checked data from Kvalobs (including SMS-precipitation). Easily implemented from KDVH

		ONGOING
High priority form VA (Skaslien)
Format ref. Bård Fjukstad.
Not a Kvalobs activity
		
0	200	0	BM  	System Library	Autotools	Bug when setting up Kvalobs on Ubuntu - Hardy	Bug 1249	OPEN		
0	300	1	TR  	Documentation	Documentation	Scripts. Updating metadata
		OPEN		
0	301	1	VB  	Documentation	Documentation	Find missing
smsedit		OPEN		
0	302	1	BM  	Documentation	Documentation	Dataflow to and from Kvalobs
Tables
		OPEN		
0
	400	0	POK
GK	Workshop	Met.no / SMHI	Activities – priorities
Resources – contribution on activities
Time schedule
QC2 development
Kvoss – common and national elements, modules and versions
Perl – Python		ONGOING
SMHI-repr.: 		
1	28	1	VB	qaBase	Interpolation as the last QC-function	QC1-4 makes both a QC and an interpolation by numerical model value.  This happens before QC1-9 rejects data values. The result is missing data values as no model interpolation is possible.

This will also be true for QC2 if QC2 rejects data. The QC1-4 interpolation function should be separated from the QC check.
	Mail:
Pål 20.11.08	OPEN
To do after 22 and 23		
1	40	1	BM  1 w
RR

	Dataflow to Kvalobs
	Watchdog on rejectdecode	IT-operators should have a message as observations are rejected by the decoding unit. IT-operators may call observers to resend observation. Write to file and use SMS for operators.	Mail: 
Per-Ove 15. mars 2007	OPEN 
(Check with PE)
Synop is used today. 
This gives trouble in HQC and dataflow to Klima.
		
1	47	1	BM
VB  1 w
OET	Dataflow to Kvalobs
Model data	qaBase with several sets of model data 	To at least improve checks/interpolation of snow/no-snow on ground. Model data of water amount in snow on ground (RWA).

GRIB-files   WeatherDB (WDB)		OPEN		
1	63	0	BM 3 d	Decoding	Meldinger med feil tidsstempling må endres til riktig obstime eller forkastes.
Mangler for visuelle værstasjoner	Dekoderne bør oppdage meldinger hvor tidsstemplingen åpenbart er feil (f.eks. når dagen som er oppgitt er i morgen) og enten endre til riktig obstime eller forkaste til rejectdecode.

Gabriel har laget et forslag til hvordan dekoderen bør behandle meldingenes tidsinformasjon. PIO-stasjoner og nedbørstasjoner er spesielle problemområder. Andre typeid?	Mail: Subject: SV:  
Pål 19. april 2005
Bugzilla: 882
885	ONGOING
met.no specific problem 

Put specification on wiki (GK), to be commented by BM.		
1	74	1	PE	QC2	QC2 consistence check in a “complete” observation	QC2 consistence check for all typeid per station. Makes it possible to check a complete observation from AWS + VIS (automatic + manual stations). It will also improve or substitute the QC1 consistency checks?

		OPEN		
1	81	1	BN
ØL	QC1	QC1-1
update

New checks QC1-2	Checks to be made for paramid in ( 47-49, 73-80, 95, 127, 227 and 250)
	Bug: 834
875, 936
1214

1232	OPEN
47-49 and 73 has now range check, Apr. 2006. See. Act. 80

1	131		0	BM ?
KJ	Datatypes and stations	METAR	METAR dataflow through Kvalobs.
Make dataflow as test.
Make controls for new elements

Another decoder made by Jurgen?		ONGOING
Decode SW for METAR is done.
Preparations of T_METAR table in Climate DataWarehouse is done. Ready for data from Kvalobs	Sept 08

	
1	140	0	Kvdrift
TR / ØL
VB
BM	Implementation of new version	Procedures for implementation and testing	Before implementation as a running system it will be required to run a test version on a complete system with respect to dataflow and related applications.

Needs a complete parallel system
Data input…..,Kv2klima to the test Climate Data Warehouse (KD2), KRO., SYNOP production, HQC		ONGOING
Needs to be specified by Kvdrift as a realistic procedure.

Status – utført ????  Børge , Vegard	
	
2	16	1	VB  ? w	Aggregation	Aggregation – not only in corr	Aggregation only in “corrected” when the obs is too late. Should also aggregate in original value	1178	OPEN		
2	42	0	BM 	Dataflow to Kvalobs	Updating single elements	Need to update time series of only one single element. Updating from DVH to Kvalobs. Correction of systematic errors in time series.	Bugzilla: 970
	ONGOING
Is this fixed now ????  Børge (pok)
Ref 34		
2	44	1	VB  8 w	Dataflow to Kvalobs 	Corrections before data input	SMSedit is an application to correct SMS-messages at met.no. It works for typeid 302, needed for typeid 308  (and 312).		OPEN

Works fine for typeid 302 (message format 2)		
2	61	1	BM	Decoding	Unknown observations	Unknown observations with respect to the observation program (obs_pgm) should be rejected during decoding.
Proposal: Not reject, but rather flag as kind of prequalification.
	Bugzilla: 902	OPEN
		
2	78	1		QC2	QC2…..	Specification of statistical checks
		OPEN		
2	79	1		QC2	QC2……	Specification of climatological checks
		OPEN		
2	85	1	VB
TR	QC1	QC1-5
Position check	Checking ship onshore position errors
Needs a land/sea metadata matrix
		OPEN
		
2	141	0	ØL / BN
TR
         2 w	Implementation of new version	Algorithm testing	Algorithm testing to be set up on a separate computer 
Virtual server.	Bug: 952	ONGOING
Rime ???? Nytte ???  
Pål sjekk med Øystein , Bjørn		
3	26	1	VB

	qaBase	QC of corrected and aggregated values – trigging of CCA	A general weakness that corrected and aggregated values are not QC’ed.
The flagging system is not designed for rerun of QC – like after HQC (as only useinfo is set). Important as corrections should trig a CCA for the SYNOP.

Flags on aggregated values, need to get flags from observations. 
3 elements to consider: Aggregation / qaBase / QC2
	Mail: Pål  161105

Bug: 984
1178	OPEN

Ref. activity 150 for QC1 – flag - specification

QC2 implies new activity?		
3	34	1	BM	Dataflow from Kvalobs	(Re)pushing av data over til KlimaDB	By direct SQL changes in table data we need an automatic pushing to subscribers and also activation of aggregation function	Bugzilla: 940 	OPEN
To be solved by a GUI		
3	36	1	BM  3 d	Dataflow from Kvalobs	Kvsynopd	The infrastructure in kvsynopd should be used for other purposes, ref. request from SMHI. The changes (below) will also make it easy to fix SYNOP problems.
   * Use the filter when data is selected from the cache database to produce SYNOP (not when data is inserted to the cache database.
  * The cache database should contain the QC flags (control- and useinfo).		OPEN

SMHI?

To be used in 37		
3	39	1	BM	Dataflow from Kvalobs	Subscribing history	Subscribe for KRO the tables: data_history and text_data_history	Bug: 893	OPEN		
3	82	1	VB
BN	QC1	Detailed information form qc-script to cfailed	More details needed.
Need to know in clear text what’s wrong.

Melding om hvilken del av en større algoritme/sjekk som slo ut kan nå sendes ut i den vanlige returvektoren med nøkkel "subcheck". Gjenstår å ta denne muligheten i bruk i kontrollskriptene. Viktigere ser det ut til å være å få delt opp skript hvor det er mulig å få samtidige utslag i mer enn en test. "geok-skriptet" er blitt delt opp. For det andre virkelig store skriptet: QC1-2-100 ser det ut til at behovet for mer detajert melding allikevel ikke er til stede.
	Mail: Audun 23.05 2005	OPEN

3	110	0	BM	Reserveløsning	Only one synop encoding system for Kvalobs and Autoobs	Hvis Kvalobs er nede, skal Autoobs ta over genereringen av synop. Imidlertid vil flere synoper da bli svært mangelfulle, ettersom det er mye funksjonalitet i Kvalobs sin synopenkoder som ikke finnes i Autoobs sin enkoder. Synopenkoderen i Autoobs bør derfor erstattes av synopenkoderen i Kvalobs.		ONGOING
met.no specific problem
Ikke Nødvendig ???? Børge.
Stabil drift av Kvalobs med reservemaskin.