====== Tseries Technical Documentation ======


===== Description: =====

//Tseries// is an open-source time-series viewer for meteorological purposes. //Tseries// is developed and implemented at the Norwegian Meteorological Institute met.no, as a part of the operational meteorological workstation. //Tseries// can be used standalone, or in  interaction with the [[https://diana.wiki.met.no/doku.php|Diana]] meteorological workstation.
The //Tseries// package contains two programs:

  - tseries.bin is the timeseries viewer with a Graphical user interface
  - btseries is a batch version of //Tseries//, using the same graphical engine


The Goal of this Document is to explain the basic structure of //Tseries// setup- and stylefiles in order to adapt //Tseries// to a specific environment. The intended readership for this document is it-personnel and meteorologists, in charge of the operational setup of a local //Tseries// implementation. The description of the //Tseries// GUI is integrated in the applications help menu. The btseries user interface is described in the “btseries –help” option.


Styles and control are located in a separate package - The source code is downloadable at [[https://svn.met.no/tseries/config|subversion]]
===== Command line parameters: =====

Most of the input in //Tseries// is given by the control file, but can be overridden by command line input:
|  		   | 
| **-s [ setupfile ]** | if the setup file is not directly defined or invalid, //Tseries// will try to find a valid setup file with the name tseries.ctl in the system. First looking in the current directory, then in $HOME/.tseries/ and at last in /etc/tseries. If no valid setupfile exists, //Tseries// will exit. |
| **-S [site]** | site is just a short string. The user can define elements in the setup-file that  are only used by certain groups. By this, several groups of users can share one setup-file to simplify maintenance.    |
| **-l [lang]** | The language for the GUI. Can be configured in the GUI itself. Available languages are English(default), Norwegian Nynorsk, Norwegian Bokmål, Swedish, German, French. New languages can be added by using the qt-linguist tool. |     
| **-T [title]** | Changes the window title, useful if you are running several instances of //Tseries// at the same time.     |
| **-H [wdbhost]** | Host for the [[https://wdb.wiki.met.no/doku.php|WDB]] connection |   
| **-u [wdbuser]** | User for the [[https://wdb.wiki.met.no/doku.php|WDB]] connection |   
| **-d [section1:key1=token1 section2:key2=token2] ** |  The user can override any entity in the setup file by this option. Example: Use a different Diagrams definition file, without changing the setup file, by the option -d files:Defs=./tsDiagrams.def |





===== Datastreams: =====

There are several possibilities to add a Datastream to a //Tseries// Diagram:

  - **HDF** is the most common used datasource. In HDF all positions are predefined and named, Hdf input is defined in the base control file tseries.ctl. All HDF streams mentioned in tseries.ctl are indexed at startup. Streams from different HDF files can be combined in a single Diagram.
  - **ASCII** is implemented. But in an operational environment ASCII streams are space-consuming and ineffective. An example of usage of ASCII streams is presented in the appendix of this Document.
  - **WDB** the Weather and Water Database [[https://wdb.wiki.met.no/doku.php|WDB]] is an  open-source database system based on postgres. The system is developed at MET Norway. [[https://wdb.wiki.met.no/doku.php|WDB]] is holding data as fields. //Tseries// retrieves interpolated timeseries  from the database.  As [[https://wdb.wiki.met.no/doku.php|WDB]] interpolates data on-demand, [[https://wdb.wiki.met.no/doku.php|WDB]] is more flexible, but slower than local HDF files. [[https://wdb.wiki.met.no/doku.php|WDB]] is the dataengine for for the web-based, free weather service yr.no 
  - **KDVH** is a MET Norway proprietary interface, based on URL. KDVH reads observations from the Norwegian climatic database. The Data are defined as Diagram entries in the control file (tseries.ctl) and can be implemented in any diagram. 

  - **Fimex**  [[fimex:start|Fimex]]  is a library to process several datastream types (like NetCDF).  [[fimex:start|Fimex]]  Interpolates on demand from files. This is time-consuming, therefore fimexstreams are read for all positions on request. After that the  [[fimex:start|Fimex]]  stream is used from cache

===== Diagrams: =====

A Diagram in //Tseries// consist of one or more datastreams connected to a given style. There are several files needed to generate a diagram:
To generate a new diagram, one have to proceed the following workflow:


  - Create a new folder.
  - Copy a tseries.ctl and a tsDiagrams file into the folder.
  - Edit tseries.ctl. If the new diagram implements a new style and not a new stream, then just change the definition for Defs to the tsDiagrams file in this folder.
  - Copy a stylefile into the folder (for instance meteogram.style) rename the stylefile.
  - Edit tsDiagrams. Compose the parameters and models for the new diagram. Connect the diagram to the stylefile in this folder.
  - Edit the stylefile in this folder.
  - Start tseries.bin in this folder, the local tseries.ctl will be the default. 
  - Repeat 6. and 7. until you are happy with the result.

===== The Main Control file – tseries.ctl =====

The control file is split into several sections. All parameters given in the start of the file can be used as substitutes by $(parameter) anywhere later in the file. Unix Environment variables can be substituted by ${parameter}.

Single lines can be commented out by “#”

Keys, Sites and sections are “case-independent”

==== Sites: ====
 Tseries.ctl is a multi-user setup-file. To avoid the administration of several setup-files within one company, the file can contain several different configurations. The file is processed top-down. If a variable is defined several times, the last definition is valid.

Elements within sites in //Tseries// are usually inactive, the parser will ignore them, unless the user starts //Tseries// with the flag that defines the specific site ( the -S flag). A site is defined by a colon separated list of strings in square brackets. A site section ends by another site or a set of empty brackets.  

<code>
Example:

server=meat.com

[FRUIT]
server=banana.com

[VEGETABLES:OTHERS]
server=potato.com
[]
</code>

Basically the server is defined as “meat.com”, if one wants to use the “banana.com” server, one have to start //Tseries// by:
 tseries -S FRUIT 

To use potato.com in this example, one can start //Tseries// either with “ -S VEGETABLES” or      “-S OTHERS”. This server is valid for for both flags.

==== Sections ====

Sections are defined by chevrons like ''“<Path>”''. A section is valid until the next section starts. Anything in tseries.ctl is organised in sections from the beginning of the first section. Entities written before the first section starts are pushed into a hash table and can be used freely as substitutes through the entire setup file.
^ Token ^ Description ^
| nowTime | this special token can be used to generate a formatted string from the clock like ''nowTime=%d-%m'' |


=== SECTION: <Path> ===

^ Token ^ Description ^
| **Etc**    | Path to find basic configurations |
| **Images** | Path to find images used in diagrams, like weather symbols |
| **Lang**   | Path to find language files, //Tseries// looks for  tseries_XX.qm files in this directory, where XX is the country code.  .qm files are generated by the the Qt linguist tool. |
| **Styles** | Path to find stylefiles in tsDiagrams.def |


=== SECTION: <Files> ===

^ Token ^ Description ^
| **Defs**      | The diagram definition file (tsDiagrams.def) | 
| **Configure** | Autogenerated File to store configuration (like window size, language etc) from  the GUI. |
| **WeatherSymbols** | The weather symbol definition file |
| **stdImage**       | Image to send to [[https://diana.wiki.met.no/doku.php|Diana]] to show positions on the map |
| **finImage**       | Image to display in [[https://diana.wiki.met.no/doku.php|Diana]] when the “find” button is triggered |
| **iconImage**      | The //Tseries// icon |
| **baseFilter**     | A filter file. By using the filter, //Tseries// will only display positions defined in the filter. The filter can be altered and saved locally by the user. |
| **wdbBookmarks**      | In [[https://wdb.wiki.met.no/doku.php|WDB]] mode one can store local bookmarks (positions) into this file |
| **fimexBookmarks**    | In  [[fimex:start|FIMEX]]  mode one can store local bookmarks (positions) into this file |
| **commonBookmarks** | A file with common bookmarks, valid for all users |

=== SECTION: <fimex> ===
^ Token ^ Description ^
| **FimexStreamTypes** | List of streamtypes that will be read by  [[fimex:start|Fimex]] , seperated by colon  ''netcdf : ncml'' |


=== SECTION: <WDB> ===
^ Token ^ Description ^
| **Host**  | The WDB host |
| **User**  | The WDB User |
| **busymovie** | Animated gif to display that the server is busy - caching |
| **readtime**  | If a request takes longer than readtime, the cache option appears ( in milliseconds ) |
| **maxRecord **| Cutoff for records in bookmarks |



=== SECTION: <klima>===
^ Token ^ Description ^
| **url** | base URL for the klima requequest |
| **maxDistance** | max distance to an observation in km (default=50) |
| **maxObservationLength** | how many hours to look into the past (default=300) |




=== SECTION: <Server> ===
The server section is used to define the connection to [[https://diana.wiki.met.no/doku.php|Diana]] via coserver

^ Token ^ Description ^
| **Client**  | Name of the Client ([[https://diana.wiki.met.no/doku.php|Diana]]) |
| **Name**    | Name that is sendt to the Client |
| **Command** | Command to start coserver |

=== SECTION: <Streams> ===

The streams section is parsed by an older system, the pets graphical engine, on which
//Tseries// is build.
All data input from HDF files and  [[fimex:start|Fimex]]  streams is defined here, always the same way.

A new model starts with a collection name, this is kept for backwards-compatibility,
these values are not used by //Tseries//
<code>
CollectionName=ECMeteogrammer
PreferredDiagram=Meteogram
InitialOpen=0
</code>

In a collection, any hdf/fimex file has to be registered the following way

== A HDF/ASCII Stream ==
^ Token ^ Description ^
| **DataFile**        | The HDF file |
| **DataDescription** | The data description. A hdf file can contain several streams. You have to find the exact stream name in the file (when in doubt use vshow). The data description is used to define values in the diagram definition. (tsDiagrams.def) |
| **DataType** | HDF or ASCII |
| **Contents** | What parameter are defined in the file, x means any.  The string is separated into four sections: Parameter,level,stream,runtime \\ ''x,x,ECMWF,12'' – means: All parameters from ECMWF, 12 \\ ''x,x,ECMWF,x'' - All parameters from ECMWF, all times |


== A FIMEX Stream ==
^ Token ^ Description ^
| **DataFile**        | The datafile or URL (opendap) - use of glob is possible  |
| **DataDescription** | The data description. Just a string to connect this stream to the diagram definition. (tsDiagrams.def)  |
| **DataType** | [ type : tag ] type has to be defined in fimexStreamTypes, tag is optional. if no tag is given, tag=type. Tag is used to connect the parameterdescription to this stream  |
| **Contents** | Not really used in Fimexstreams  |
| **DataConfig** | Connect a  [[fimex:start|Fimex]]  config file (xml) to this stream  |


=== SECTION: <wdbParameter> ===

This section maps wdb parameters to tseries parameters

<code>

# tseries parameter  = <wdb parameter> | <height> < height name>  : <scale> 
MSLP,0,x,x,APP=  air pressure at sea level               | 0   height above ground             :/100
RR,0,x,x,APP=    lwe precipitation rate                  | 0   height above ground
TT,0,x,x,APP=    kalman air temperature                  | 2   height above ground             :-273.15
UU,0,x,x,APP=    x wind                                  | 10  height above ground             :*1.94384
</code>


=== SECTION:<WDBVECTORFUNCTIONS> ===
connect two parameters to a vector( to rotate them into a greografic grid)
<code>
UU,0,x,x,x :VV,0,x,x,x
</code>




=== SECTION: <klimaparameter> ===

<code>
# tseries parameter = klimaparameter
TT_O,0,x,OBS,x=TA 
MSLP_O,0,x,OBS,x=PR 
QNH_O,0,x,OBS,x=PH

</code>

=== SECTION: <klimanormals> ===
<code>
# tseries parameter = klimaparameter
# for these parameters, tseries has to build a timeline
TAM,0,x,OBS,x=TAM
TANM,0,x,OBS,x=TANM
TAXM,0,x,OBS,x=TAXM
</code>

=== SECTION: <fimexParameters> ===



<code>

tag       = Given at the fimexstream definition  (DataType : Tag)
fimexname = Request parametername
Unit      = Udunits string
extradim  = Predefined dimensions are time and position, height or similar has to
            be defined her for this particular parameter

# tseries parameter   = tag | fimexname                | Unit | extradim1 start1 len1 | extradim2 ...
#
#


MSLP,x,x,x = netcdf | air_pressure_at_sea_level         | hPa      | height 0 1
TT,x,x,x   = netcdf | air_temperature_2m                | Celsius
UU,x,x,x   = netcdf | x_wind_10m                        | kt

# with a defined RRAC, tseries knows automatically how to extract RR from it
RRAC,x,x,x = netcdf | precipitation_amount_acc          | kg/m^2
# Postprocessed temperature - alias...
TTPP,x,x,x = netcdf | air_temperature_2m                | Celsius

## Wrf data

MSLP,x,x,x = wrf | air_pressure_at_sea_level      | hPa     
TT,x,x,x   = wrf | T2                             | Celsius
UU,x,x,x   = wrf | U10                            | kt  | 
VV,x,x,x   = wrf | V10                            | kt  | 

</code>


===== The Diagram definition file – tsDiagrams.def=====

The Diagram definition file ties the streams to the actual diagram styles. One model input can appear in several diagrams, as well as several different models can use the same diagram type (like a meteogram). It is also possible to use one parameter from a defined stream into a certain diagram that is used by another stream. An example is the use of wind speed from an atmospheric model into a wave-diagram.
//Tseries// builds its User interface based on this file. A stream appears first in the menus when it is defined here. If you want to remove a streams from //Tseries//, you have to remove it from this file as well. Otherwise it will appear as an empty entry in the menus.

===The [models] section===
All Models have to be defined in the models section, here one can change the display name of the model in the diagrams. The key has to be equal the stream name from tseries.ctl.

Example:
<code>
 ECMWF=Ecmwf
</code>

ECMWF equals the DataDescription from tseries.ctl, Ecmwf will be displayed in the //Tseries// GUI menus


=== The Diagrams ===
Each diagram definition connects some parameters from some models to a style. A style can be used several times in different contexts. For instance is the S-meteogram (symbol meteogram) a diagram that uses the meteogram style, but it has also the weather parameter (WW) as input.
Each diagram defintion starts with the [DIAGRAM] keyword and ends with the [ADD] keyword.

<code>
Examples:

# Standard meteogram
[DIAGRAM]
# The name of the diagram as it appears in the menu
NAME=Meteogram
# The connected style file
STYLEFILE=meteogram.style
# The streams connected to this diagram
[LEGALMODELS]
MODEL=ECMWF
MODEL=HARMONIE_2.5km

# Modelindependent = These parameters will be taken from 
# all models, if available

[MODELINDEPENDENT]
PARAM=UU,x,x,x
PARAM=VV,x,x,x
PARAM=FF,x,x,x
PARAM=DD,x,x,x
[END]

# Modelspecific = These parameters are only used if they come from the specified model.
# Modelspecific values are not necessary part of the legalmodels. 
# If they are not defined there, they will not occur in the menu

[MODELSPECIFIC]
MODEL=HARMONIE_2.5km
PARAM=TT_U,x,x,x
[END]

[MODELSPECIFIC]
MODEL=OBS
PARAM=TT,x,x,x
[END]

[ADD]
</code>

====Styles====
Styles are defined in stylefiles. These are connected to data by tsDiagrams.def.
By altering the stylesfiles you can control colors, line and object types in the diagram.
There are hundreds of different types in a stylefile.  You have to start with an existing stylefile and reduce that given stylefile until it fits your purpose. A style starts with the definition of the standard values: like default patterns, background color for the diagram etc.


<code>
#==================================================================
# 
# Global settings
#
#==================================================================


bgColor=      WHITE
topMargin=    50
bottomMargin= 50
leftMargin=   150
rightMargin=  150

# --------------------------------
# default values

type=         DUM_PRIMITIVE
parameter=    x,x,x,x
mother=       DIAGRAM
plotAll=      FALSE
enabled=      TRUE

color=        RED
color2=       YELLOW
font=         NORMAL
linePattern=  FULL
fillPattern=  SOLID
align=        LEFT
spacing=      MEDIUM
intSpacing=   SMALL
label=        TRUE
patternInColour= FALSE
lineWidth=    1.0
axisWidth=    2.0
tickWidth=    1.0
minRange=     0
interval=     1
delta=        2
minMargin=    0
maxIsSet=     FALSE
minIsSet=     FALSE
minValue=     100000
maxValue=    -100000
yaid=         0
centerVector= FALSE
numTickMajor= 10
numTickMinor= 2
labelSpace=   120
quantized=    FALSE
quantum=      1.0
gridxonly=    FALSE
axisgrid=     FALSE
gridwidth=    1
gridcolor=    GREY25
gridstyle=    DOTTED

[DEFAULT]

</code>

Then you have to define axes

<code>
first at least one x-axis
 # --------------------------------
# utc-time
[NEW]
type=     UTC
order=    8
height=   25
spacing=  SMALL
font=     NORMAL
color=    BLUE
minSkipX= 15
[ADD]

</code>

And then several y-axes. A curve in the diagram is connected to an axis. If you want to display, for instance several temperatures, you have to create a temperature axis. Then you have to connect the curves to that axis. By that it is guaranteed that the curves are scaled correctly to each other. Each axis has an unique identifier (y-axis-id/yaid). Yaid is used to connect parameters to the axis.


<code>
 # ----------------------------
# Yaxis (T2m)

[NEW]
type=        YAXIS_STATIC
yaid=        0
delta=       1.0
interval=    0.5
minMargin=   0.25
minRange=    5.0
minIsSet=    FALSE
maxIsSet=    FALSE
text=
align=       LEFT
axis=        LEFTLEFT
color=       RED
lineWidth=   2
linePattern= FULL
patternInColour=TRUE
[ADD]
</code>
 
With axis in place, you have to define a parameter and connect it to it. In this case, the curve is presented in 2 different colors, one for -100 to 0 degrees and one for 0-100 degrees. 

<code>
 [NEW]
type=            LINE
yaid=            0
parameter=       TT,0,x,x
text=            T2m [°C]
lineWidth=       2
linePattern=     FULL
patternInColour= FALSE
colorbyvalue=    TRUE
datalimits=      -100,0,100
colorlist=       BLUE,RED
patternInColour= FALSE
[ADD]
</code>

===Valid Types:===
There are several types in //Tseries// that can be used by the different elements. 

====Colors====

{{:tseries:pets_colors.png|}}



^ Fillpatterns     ^ 
| DIAGRIGHT        |         
| DIAGLEFT  	   | 
| DIAGCROSS 	   | 
| HORIZONTAL	   | 
| VERTICAL  	   | 
| SHORIZONTAL	   | 
| SVERTICAL 	   | 
| SQUARE    	   | 
| SOLID  	   | 


^ Linestyle	   ^ 
| DASHED           | 
| DASHDOTTED       | 
| DASHDASHDOTTED   | 
| DOTTED 	   |