Geo::BUFR

Geo::BUFR is a CPAN module for BUFR decoding and encoding developed at the Norwegian Meteorological Institute.

Below you will find a toolkit of BUFR handling programs used actively at MET Norway, with examples of use and complete source code. The first 5 programs are included in the CPAN Geo::BUFR distribution (will be installed in the bin subdirectory of the installation directory). Note that even if the programs are written in Perl, no knowledge of Perl is required to use them.

A line consisting solely of three dots (…) in the examples of output from running the programs, marks that a (not so interesting) part of the output has been left out.

Notes to users outside MET Norway:

Before trying to install Geo::BUFR you should check that you have Perl installed.

Installing Geo::BUFR will not install any BUFR tables. When invoking the utility programs you should use the option --tablepath to tell in which directory you keep your BUFR table files, and possibly --tableformat to tell the table format. Alternatively, you can set the environment variable BUFR_TABLES, or you can change the value of the constant DEFAULT_TABLE_PATH in the source code of the utility programs (possibly you might want to change the values of DEFAULT_TABLE_FORMAT, DEFAULT_TABLE and DEFAULT_CTABLE also).

Decodes and prints section 0-4 in BUFR messages in input file(s). If the BUFR message is contained in a GTS bulletin, will also print AHL (Abbreviated Header Line).

source code

Example: Executing bufrread.pl without arguments

Example: buffread.pl --help

Example: Processing a file containing several BUFR SYNOP bulletines, with no options set

Example: Same as above, but with options --param and --data_only set

Example: Same as previous example, but with option --filter also set. Note that the program bufrdump.pl described below provides the same option but with more functionality.

Example: Processing a BUFR TEMP message with option --codetables set (to resolve code and flag values)

Example: Processing a BUFR message containing operators 222000 and 223000 with option --bitmap set (to display bitmapped entities on same line)

Example: Processing a BUFR message containing associated fields and change scale and data width operators

Example: Same as above, but with options --all_operators and --data_only set

Example: Processing a file containing BUFR BUOY bulletins with recoverable formatting errors in the BUFR messages, with option --strict_checking set

Example: Filtering a file containing a large number of GTS bulletins to decode BUFR BUOYs only, using option --ahl

Displays information in BUFR tables.

source code

Example: Executing bufrresolve.pl without arguments

Example: bufrresolve.pl --help

Example: Executing bufrresolve.pl on a table D descriptor (309052)

Example: Same as above, but with option --partial

Example: Same as above, but with option --simple

Example: Executing bufrresolve.pl on a sequence of descriptors with option --noexpand

Example: Showing contents of code table (option --code)

Example: Resolving flag value (option --flag)

Example: Illustrating use of options --bufrtable and --tablepath (for a local table B descriptor)

Example: Illustrating use of option --tableformat ECCODES on a local table D descriptor

Example: Illustrating use of option --verbose 1 (to see which tables are loaded)

Encode a BUFR message based on data and metadata text files.

source code

Example: Executing bufrencode.pl without arguments

Example: bufrencode.pl --help

Example: Encoding a 2 subset BUFR SYNOP message

Encode BUFR message(s) based on a text file matching what you would get by running bufrread.pl on the resulting BUFR file. This might be a handy alternative to bufrencode.pl for encoding new BUFR messages: start with a similar BUFR message you know is correctly formatted, run bufrread.pl on this, edit the output as desired, and using this as input to bufr_reencode.pl.

source code

Example: Executing bufr_reencode.pl without arguments

Example: bufr_reencode.pl --help

Example: Encoding a single subset BUFR SYNOP message

bufralter.pl provides an even easier way to do some basic manipulation of BUFR messages than does bufr_reencode.pl (although much more restricted in what changes are possible). Besides, bufralter.pl has the nice feature of not touching the text between the BUFR messages. This might be handy if you for example have a file with GTS bulletins and want to do some changes in the BUFR messages without affecting or losing the GTS bulletin envelopes.

source code

Example: Executing bufralter.pl without arguments

Example: bufralter.pl --help

Example: Altering a single subset BUFR SYNOP message, by changing value of hour, minute and temperature in section 4, changing update sequence number and changing version number for master table and local table.

Extract (raw) BUFR messages and/or AHLs from file(s).

source code

Example: Executing bufrextract.pl without arguments

Example: bufrextract.pl --help

Example: bufrextract.pl --ahl

Example: bufrextract.pl --only_ahl

Examples: bufrextract.pl --ahl --only_ahl

A BUFR toolkit wouldn't be complete unless you have a way to dump the contents of BUFR files in a way which requires no prior knowledge whatsoever of the BUFR format. Therefore, the program bufrdump.pl is included even though it actually does not employ Geo::BUFR.pm, but rather is a wrapper around a Fortran program (named bufrdump) which uses the ECMWF BUFRDC library for the BUFR decoding. The reason for this choice was that for this particular program speed might be important (expecting users to dump huge BUFR files and grep'ing on the output), besides the fact that the main part of the program would be equally boring to code whether in Fortran or Perl.

The parameters are dumped in a name=value fashion according to what most users at Met Norway expects (see Parametere i Kvalobs-databasen), e.g. TA=-9.3 for temperature at 2 m equal to -9.3 Celsius. If your users are accustomed to other symbols and/or units, this should be rather easy to modify in source code of bufrdump.F.

So far dumping of BUFR SYNOP, TEMP, AMDAR and BUOY have been implemented (probably works also for PILOT, BATHY and TESAC). More observation types are likely to be included in the future.

source code

Example: Executing bufrdump.pl without arguments

Example: bufrdump.pl --help

Example: Dumping a multisubset multimessage BUFR SYNOP file with no options set

Example: Same as above, but with option --param set

Example: Same as the --param example above, but also with option --csv set

Example: Same as the --param example above, but also with option --filter set

Example: Same as the --param example above, but also with option --sort set, for sorting on station identification

Example: Same as the --param example above, but also with option --sort_on TA- set, for sorting on descending temperatures

Example: Dumping another BUFR SYNOP file with some selected parameters only

Example: Same as above, but this time using --transform to display wind speed in knots and cloud cover in %

Example: Dumping yet another BUFR SYNOP file with filter conditions (e.g. stations) and parameters selected on command line

Example: Dumping a BUFR TEMP message with no options set

Example: Dumping a BUFR AMDAR file with no options set

Example: Dumping a BUFR BUOY file (IOB bulletins) with option --obstype ocea

Example: Same as above, but this time with option --obstype surface

bufrresolve.pl does provide several ways of extracting information from BUFR tables, but you might have special needs not covered by the program. Making your own scripts based on Geo::BUFR should not bee too difficult, however. As an illustration, see dump_flag_tables.pl for a script that dumps all the flag tables in a (hard coded) table file. Running this script produced this output.

BUFR TEMP should according to WMO regulations be issued on the GTS in one IUK-bulletin containing data up to 100 hPa, and one IUS-bulletin containing the whole sounding. Some centres, however, reformat TAC TEMP part A-D part for part into 4 different BUFR TEMP bulletins, which are very cumbersome to handle in data processing. Therefore mergebufrtemp.pl was created in the hope that some data producers will find this program useful for doing the necessary merging of (BUFR) TEMP parts. Example of use:

  mergebufrtemp.pl <bufr.a> <bufr.b> --outfile <bufr.ab>

where the file bufr.a is supposed to contain a BUFR TEMP message converted from BUFR TEMP part A only, bufr.b a BUFR TEMP message converted from BUFR TEMP part B only, and the file bufr.ab is the result of merging these two parts into a BUFR message suitable to be included in an IUK-bulletin. Repeated use of mergebufrtemp.pl might be used to create a file bufr.abcd to be included in the IUS-bulletin, containing the whole ascent.

According to B/C 25.7: “Temperature, dew-point and wind data at pressure levels obtained during the radiosonde ascent shall be included in descending order with respect to pressure. Data at each pressure level shall be included only once”. This makes perfect sense, since it is hard to think of any application where you would not prefer to have this ordering and uniqueness of pressure levels. However, many data producers sin against this regulation, and even more astounding: in Note (1) to B/C 25.7 WMO explicitly allows violations of this regulation (for BUFR TEMP converted from TAC TEMP).

Therefore sortbufrtemp.pl was created to do the necessary sorting of pressure levels and possible merging of duplicate levels (i.e. pressure levels having same value for pressure, time and position coordinates).

Example: sortbufrtemp.pl --help