Abstract image of a woman sitting on books and studying on a laptop

FMI Open Data Interface and Data Models

Download as PPTX, PDF
Roope Tervo, profile picture
Roope Tervo

The Finnish Meteorological Institute (FMI) has been providing open meteorological data since 2013, offering various datasets including weather observations, forecasts, and air quality data in machine-readable formats. Users must register to access the data through an API with specified request limits and can utilize multiple services for viewing and downloading data. The document details the data models, usage protocols, and metadata structures that support the open data initiative, ensuring compliance with standards such as INSPIRE.

Engineering
1 of 43
Downloaded 30 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
Open Meteorological Data Usage and Best Practices Finnish Meteorological Institute Roope Tervo, Mikko Visa
Finnish Meteorological Institute opened its data in 2013. Basically everything FMI owns was opened. Data is provided freely in machine readable format. 6.10.2014 FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 2 FMI Open Data https://en.ilmatieteenlaitos.fi/open-data
FMI Open Data Portal FMI Open Data Portal follows INSPIRE requirements. Meta data Data Models Services The very same data portal works as Open Data and INSPIRE portal. FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 3
Example of Data Sets Data set Description Time Interval Estimated publish date Weather Observations Temperature, Wind, Humidity, Ground Temperature… 10 min Open, older data to be added Sun Radiation UV, Short and Long Term Radiation… 1 min Open Marine Observations Waves, Sea Temperature, Sea Level… 1 h Open Weather Radars Precipitation Rate, Precipitation Amount… 5 min Open, older and more data to be added 2015 Lightning Thunder Strikes in Finland 5 min Open FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 4
Example of Data Sets Data set Description Time Interval Estimated publish date Real Time Observations Real Time Observations from specific location(s) AWS 2010 – Soundings 1959 – Flashes 1998 – Sea Level 1971 – Waves 2005 – Open older data will be added Climatological Observations Daily and monthly temperature mean and extreme values from weather stations 1959 - Open Climatological Observations Monthly temperature and precipitation rate mean values interpolated to grid 1961 - Open Climatological Reference Climatological Reference. Temperature, humidity, pressure, precipitation amount and snow depth. Reference seasons: 1971-2000 1981- 2010 Open FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 5
Example of Data Sets Data set Description Time Interval Estimated publish date Weather forecast model HIRLAM RCR Point forecasts and grid data Latest model run (4 times a day) 0…54 h Open Sea forecast models Sea level point forecasts, Wave (WAM) and current (HBM) as grid data Latest model run (4 times a day) 0...54 h Open Environmental Monitoring Facilities Weather observation stations, radars… - 2015 Aviation Observations METAR/IWXXM 30 min Open Ground & mast observations Special observations from ground and masts 2014 FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 6
Example of Data Sets Data set Description Time Interval Estimated publish date Air Quality Observations Air Quality Observations 1h 2015-2016 Silam Model Dispersion Model for Air Quality, Forest Fire and Pollen Latest model run (once a day) 0…96h 2014-2015 HELMI Ice Model Ice forecast model Latest model run (4 times a day) 0...54 h 2014/12 Soundings Temperature, Humidity, Pressure, Wind from ground to 25 km height 2 times a day 2014 Quality information Quality information for observations FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 7
Registration o Registration is required to use View and Download Services o Working email address is the only mandatory information o After registration the user gets an API key which have to be added into all requests o GET parameter fmi-apikey=…& o Header fmi-apikey; … o Part of url http://wms.fmi.fi/fmi-apikey/…/wms? o One can create several API keys with one email FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 8
Usage Limits With one API key it’s allowed to o do at most 20 000 requests per day to Download Service o do at most 10 000 requests per day to View Service o do at most 600 requests per 5 minutes to both services o If all observations from one time step is calculated to as one, little over 17 000 new data sets are published daily o So, with one API key it’s allowed load everything once o View service can be used for testing but can not be used as a back end for popular clients FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 9
Catalog Service (CSW) o Based on GeoNetwork o High-level metadata o Provides UI o Maybe more useful sources: o http://en.ilmatieteenlaitos.fi/ope n-data-manual-fmi-wfs-services (based on WFS DescribeStoredQueries) o http://www.paikkatietoikkuna.fi/ web/fi/metatietohaku (based on GeoNetwork) FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 10
View Service (WMS) o Based on GeoServer o Visualize observation and forecast data and provides radar images o Only the most common layers are published o Can be used for prototyping - not for production use FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 11
Download Service (WFS 2.0) o Web Feature Service (WFS) 2.0 Simple Profile o Based on stored queries o Predefined data sets with possibility for additional parameters (i.e. time and area) o In-house production FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 12
GetCapabilities: • Describes capabilities of the service • Normally start point for client applications • http://data.fmi.fi/fmi-apikey/…/wfs?request=GetCapabilities 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 13 WFS Basics obseravtions and point forecasts
DescribeStoredQueries • List and describes available stored queries (pre-defined data sets) • Stored query name and description • Possible additional parameters for the data query • http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 14 WFS Basics obseravtions and point forecasts
GetFeature • Actual data query • In FMI service, stored query id is compulsory • &storedquery_id=.. • Other possible parameters depend on the stored query • Possible values can be fetched with DescribeStoredQueries request. • Location has to be often defined some how. http://data.fmi.fi/fmi-apikey/.../ wfs?request=GetFeature&storedquery_id=fmi::observations::weather::citi es::timevaluepair&bbox=21,62,22,64 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 15 WFS Basics obseravtions and point forecasts
Location parameters in stored queries & bbox  area • By default, in same srs with the (requested) srs. Specify as fifth element if needed (i.e. &bbox=132249,6433579,631999,6933329,epsg:3067) & fmisid  FMI id for observation station & wmo  WMO id for observation station 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 16 WFS Basics obseravtions and point forecasts
Location parameters in stored queries & geoid  http://geonames.org id for location & place  human readable name • use region to accurate your query (i.e. &place=kumpula,tampere) • can be used several times in one query (i.e. &place=kumpula,helsinki&place=heinola) & latlon  coordinates (used only in marine data sets) 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 17 WFS Basics obseravtions and point forecasts
GetPropertyValue • Responses only requested part of the GetFeature response • Use ValueReference=xpath to define part of interest • Same additional parameters for stored query with GetFeature request are still valid • For example: http://data.fmi.fi/fmi-apikey/…/ wfs?request=GetPropertyValue&storedquery_id=fmi::observatio ns::weather::multipointcoverage&place=kumpula,tampere&ValueReferenc e=wfs:FeatureCollection/wfs:member/omso:GridSeriesObservation/om:res ult 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 18 WFS Basics obseravtions and point forecasts
Data Models o Observations and point forecasts as GML o The same data is published in two different formats. o Gridded data is provided in appropriate binary format (grib, NetCDF, HDF…) o WFS members contains the metadata ‘envelope’ with a link to a actual data o Every XML document as self-explanatory as possivble FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 19
om:phenomenonTime • Tells start and end time of returned data 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 20 Data Models most important elements
om:procedure • Describes data producing procedure om:parameter • Changing parameters in the process • i.e. analysis time of the numerical model 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 21 Data Models most important elements
om:featureOfInterest • Describes location(s) or area of returned data • Several different kind of identifiers • target:SurfaceWeatherTargetArea  area (grid data) • fmisid  FMI id for observation station • wmo  WMO id for observation station • geoid  http://geonames.org id for location • name  Human readable name • region  county of the station 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 22 Data Models most important elements
om:observedProperty • Describes meteorological parameters in returned data • Defined as xlink • Describes label, basePhenomenon and units of measure of every parameter in the reponse • Tip: use the same service without parameter definitions to query all possible parameters 6.10.2014 • http://data.fmi.fi/fmi-apikey/.../meta?observableProperty=forecast • http://data.fmi.fi/fmi-apikey/.../ meta?observableProperty=observation • Note that all parameters are not available in all data sets FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 23 Data Models most important elements
om:result • Data in required format in observations and point forecasts and link to a external binary data in grid formatted data 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 24 Data Models most important elements
Data Models gmlcov:MultiPointCoverage gml:rangeSet gml:doubleOrNilReasonTupleList The data is listed for every point defined in domain set. gml:domainSet gmlcov:simpleMultiPoint The coverage is defined as a list of points in 4 dimensional grid (lat, lon, height, time). gmlcov:rangeType The parameters listed in range set are defined in separate element. FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 25
Cons - Not intuitive - No natural structure of XML  XSLT and Xpath don’t work Data Models gmlcov:MultiPointCoverage Pros + Compact + Efficient + Small file size + Works for many data types FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 26
Data Models wml2:MeasurementTimeseries wml2:MeasurementTimeseries One member contains time series of one parameter and one location FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 27
Cons - Lots of repetition - Large file size - Heavy for DOM-based parsers - Don’t work i.e. for thunder strikes Data Models wml2:MeasurementTime series Pros + Intuitive + Easy to use + XSLT & XPath works FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 28
Data Models SimpleFeature SimpleFeature One member contains one time step of one parameter and one location FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 29
Cons - Huge file size - Heavy for DOM-based parsers Data Models SimpleFeature Pros + Intuitive + Easy to use + Wide application support FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 30
Data Models file size comparison 81.7 52.9 Document Size [MB] Compressed DocumentSize [MB] 1.3 1.2 1.8 0.2 90 80 70 60 50 40 30 20 10 0 FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 31
• Aviation weather reposts are delivered as IWXXM • New data model coming into use in aviation • Consists of the same elements than other messages • om:phenomenonTime, om:procedure, om:featureOfInterest, om:result • Content of the METAR is in om:result part as • extracted into XML elements • original, “old fashion”, METAR string 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 32 Data Models aviation observations IWXXM / METARS
• Once published, content and structure of IWXXM messages will not change • Messages will contain a digital signature • Use GetPropertyValue to download only original METAR string • http://data.fmi.fi/fmi-apikey/…/ wfs?request=GetPropertyValue&storedquery_id=f mi::avi::observations::finland::latest::iwxxm&valuereference= wfs:FeatureCollection/wfs:member/avi:VerifiableMessage/av i:metadata/avi:MessageMetadata/avi:source/avi:Process/avi: input 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 33 Data Models aviation observations IWXXM / METARS
• Messages have been automatically created based on the original METAR/SPECI code messages. • Messages are NOT an authoritative aviation weather report. They SHOULD NOT be used as a weather report to be used in flight planning or other aviation related use. 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 34 Data Models aviation observations IWXXM / METARS
• WFS response contains the same meta data information than in observations and point forecasts • om:result contains gmlcov:RectifiedGridCoverage • Basically the same with multipointcoverage • domainSet defines the grid (now as regular grid) • rangeSet contains data as an external link to the binary content • rangeType defines the parameters 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 35 Data Models binary data
• Note that fileReference points often to an other service • Can be used directly, but • It is always good practice to consult WFS for available times and parameters 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 36 Data Models binary data Client WFS WMS FMI Data Server Available radar images? Available weather model outputs?
• Area, time and weather parameters may be defined in the request • For weather models, intersection of requested area and available data area is returned • For radar images, all images which intersects requested spatial and temporal space are returned 6.10.2014 Radar images Weather models Model coverage FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 37 Data Models binary data Requested area Radar images Requested area
Radar images • The reference points to a “original” gray scale GeoTiff image • Images can also be downloaded as color images • Remove &styles=raster from the data request • But then information is lost • Used SLD-files can be downloaded from https://github.com/fmidev/opendata-resources/tree/master/sld • Consult om:parameter element for single radar measurement parameters • Scanning angle, bin count and bin length 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 38 Data Models binary data
Numerical models • The reference points to (often a subset) the model output in appropriate format • GRIB2 for weather models • NetCDF for marine models • Note that whole model output can be large • Up to 17 GB • Download always only area, parameters and time range of interest 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 39 Data Models binary data
Helpful Software For binary data • PanoPly • http://www.giss.nasa.gov/tools/panoply/ • Simple • Handle only very basic data • IDV • http://www.unidata.ucar.edu/software/idv/ • Handle most of the data 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 40
Helpful Software For exploring the data • Google Earth & QGIS • Start from GetCapabilities, UI guides thereafterhttp://wms.fmi.fi/fmi-apikey/.../ geoserver/wms?service=WMS&version=1.3.0 &request=GetCapabilities 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 41
MetoLib o Open source JavaScript library produced by Finnish Meteorological Institute o Helps users to load and use the data Easy requests Cache Parse the data to as JSON o Supports multi point coverage data format o Python version is on the list FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 42
www.fmi.fi

More Related Content

PPTX
Open Weather Data as Part of Big Data
Roope Tervo
 
PPTX
Open Data and and INSPIRE
Roope Tervo
 
PPTX
Producing INSPIRE compliant datasets
Roope Tervo
 
PPTX
Meteorological and Aviation Weather Open Data implementation utilising OGC st...
Roope Tervo
 
PPTX
Inspire Compliant Weather Data
Roope Tervo
 
PPTX
FMI Open Data Interface and Usage
Roope Tervo
 
PDF
Aaltoes opendata 20130206
Roope Tervo
 
PPTX
SmartMet Server OSGeo
Roope Tervo
 
Open Weather Data as Part of Big Data
Roope Tervo
 
Open Data and and INSPIRE
Roope Tervo
 
Producing INSPIRE compliant datasets
Roope Tervo
 
Meteorological and Aviation Weather Open Data implementation utilising OGC st...
Roope Tervo
 
Inspire Compliant Weather Data
Roope Tervo
 
FMI Open Data Interface and Usage
Roope Tervo
 
Aaltoes opendata 20130206
Roope Tervo
 
SmartMet Server OSGeo
Roope Tervo
 

What's hot (18)

PPTX
20161028 strahlendorff fmi experience in openness
Mikko Strahlendorff
 
PPTX
Linked Sensor Data cube
Laurent Lefort
 
PPTX
Using the Data Cube vocabulary for Publishing Environmental Linked Data on la...
Laurent Lefort
 
PDF
Kokemuksia tiedon avaamisesta, Tarja Riihisaari
Tilastokeskus
 
PPTX
WMTS Performance Tests
Roope Tervo
 
PPT
2004-10-09 MANE-VU Status Report on CATT and FASTNET
Rudolf Husar
 
PPT
Fast Cat M V[1]
Rudolf Husar
 
PPT
TU1.L10 - Globwave and applications of global satellite wave observations
grssieee
 
PPT
2005-01-08 MANE-VU Status Report on CATT and FASTNET
Rudolf Husar
 
PPT
Fast Cat Mv3
Rudolf Husar
 
PPTX
FMI Open Data on AWS Public dataset program
Roope Tervo
 
PDF
Morales, Randulph: Spatio-temporal kriging in estimating local methane source...
Integrated Carbon Observation System (ICOS)
 
PPTX
Air quality challenges and business opportunities in China: Fusion of environ...
CLIC Innovation Ltd
 
PPT
0411 Spec Nat Assess Tmp
Rudolf Husar
 
PPT
2004-11-24 Assessment of the Speciated PM Network (Initial Draft, November 20...
Rudolf Husar
 
PPT
0411 Spec Nat Assess
Rudolf Husar
 
PPTX
Eposa english
Robert Karas
 
PDF
CLIM Program: Remote Sensing Workshop, The Earth System Grid Federation as a ...
The Statistical and Applied Mathematical Sciences Institute
 
20161028 strahlendorff fmi experience in openness
Mikko Strahlendorff
 
Linked Sensor Data cube
Laurent Lefort
 
Using the Data Cube vocabulary for Publishing Environmental Linked Data on la...
Laurent Lefort
 
Kokemuksia tiedon avaamisesta, Tarja Riihisaari
Tilastokeskus
 
WMTS Performance Tests
Roope Tervo
 
2004-10-09 MANE-VU Status Report on CATT and FASTNET
Rudolf Husar
 
Fast Cat M V[1]
Rudolf Husar
 
TU1.L10 - Globwave and applications of global satellite wave observations
grssieee
 
2005-01-08 MANE-VU Status Report on CATT and FASTNET
Rudolf Husar
 
Fast Cat Mv3
Rudolf Husar
 
FMI Open Data on AWS Public dataset program
Roope Tervo
 
Morales, Randulph: Spatio-temporal kriging in estimating local methane source...
Integrated Carbon Observation System (ICOS)
 
Air quality challenges and business opportunities in China: Fusion of environ...
CLIC Innovation Ltd
 
0411 Spec Nat Assess Tmp
Rudolf Husar
 
2004-11-24 Assessment of the Speciated PM Network (Initial Draft, November 20...
Rudolf Husar
 
0411 Spec Nat Assess
Rudolf Husar
 
Eposa english
Robert Karas
 
CLIM Program: Remote Sensing Workshop, The Earth System Grid Federation as a ...
The Statistical and Applied Mathematical Sciences Institute
 
Ad

Similar to FMI Open Data Interface and Data Models (20)

PPTX
Producing INSPIRE Compliant Data Sets
Roope Tervo
 
PPTX
Strahlendorff - EO and insitu for weather, water and climate
Mikko Strahlendorff
 
PPTX
FMI Open Data on S3
Roope Tervo
 
PDF
Available data sources & Real-time data collection
CLEEN_Ltd
 
PPT
Dynamic Integrations of Crop Data and Corresponding Meteorological Data based...
AIMS (Agricultural Information Management Standards)
 
PPT
Dynamic integrations of crop data and corresponding meteorological data based...
AIMS (Agricultural Information Management Standards)
 
PDF
Efficiently Implementing INSPIRE & Creating INSPIRE Mashups with FME
Safe Software
 
PPTX
Strahlendorff - Insitu searching challenges
Mikko Strahlendorff
 
PDF
DSD-INT 2019 Global Data Services and Analysis Frameworks-Luijendijk
Deltares
 
PPT
2005-01-28 Assessment of the Speciated PM Network (Initial Draft, November 2004)
Rudolf Husar
 
PPTX
Why we need open data? FMI Open Data on AWS
Roope Tervo
 
PPTX
FMI Information Management System
Roope Tervo
 
PPT
FR2.L10.1: MONITORING SMOS BRIGHTNESS TEMPERATURES AT GLOBAL SCALE. A PRELIMI...
grssieee
 
PPTX
SmartMet Server in INSPIRE
Roope Tervo
 
PPTX
RDMRose 2.5 Metadata and data citation
RDMRose
 
PPT
2004-10-09 MANE-VU Status Report on CATT and FASTNET
Rudolf Husar
 
PPT
2005-04-14 The Great Midwestern PM2.5 Episode of February 2005
Rudolf Husar
 
PPT
2004-09-29 Status Report on CATT and FASTNET
Rudolf Husar
 
PPTX
Fmi Open Data on S3
Roope Tervo
 
PDF
Hosting open data endpoints at IRCEL-CELINE serving air quality data from the...
Open Knowledge Belgium
 
Producing INSPIRE Compliant Data Sets
Roope Tervo
 
Strahlendorff - EO and insitu for weather, water and climate
Mikko Strahlendorff
 
FMI Open Data on S3
Roope Tervo
 
Available data sources & Real-time data collection
CLEEN_Ltd
 
Dynamic Integrations of Crop Data and Corresponding Meteorological Data based...
AIMS (Agricultural Information Management Standards)
 
Dynamic integrations of crop data and corresponding meteorological data based...
AIMS (Agricultural Information Management Standards)
 
Efficiently Implementing INSPIRE & Creating INSPIRE Mashups with FME
Safe Software
 
Strahlendorff - Insitu searching challenges
Mikko Strahlendorff
 
DSD-INT 2019 Global Data Services and Analysis Frameworks-Luijendijk
Deltares
 
2005-01-28 Assessment of the Speciated PM Network (Initial Draft, November 2004)
Rudolf Husar
 
Why we need open data? FMI Open Data on AWS
Roope Tervo
 
FMI Information Management System
Roope Tervo
 
FR2.L10.1: MONITORING SMOS BRIGHTNESS TEMPERATURES AT GLOBAL SCALE. A PRELIMI...
grssieee
 
SmartMet Server in INSPIRE
Roope Tervo
 
RDMRose 2.5 Metadata and data citation
RDMRose
 
2004-10-09 MANE-VU Status Report on CATT and FASTNET
Rudolf Husar
 
2005-04-14 The Great Midwestern PM2.5 Episode of February 2005
Rudolf Husar
 
2004-09-29 Status Report on CATT and FASTNET
Rudolf Husar
 
Fmi Open Data on S3
Roope Tervo
 
Hosting open data endpoints at IRCEL-CELINE serving air quality data from the...
Open Knowledge Belgium
 
Ad

More from Roope Tervo (8)

PDF
FMI Open Data Impact Survey 2019
Roope Tervo
 
PDF
Predicting weather inflicted train delays
Roope Tervo
 
PPTX
Forecasting Electricity Outages Caused by Convective Storms
Roope Tervo
 
PPTX
Possibilities of Open Source Code
Roope Tervo
 
PDF
AvoinData-workshop käyttöesimerkki
Roope Tervo
 
PDF
AvoinData aineistot
Roope Tervo
 
PDF
AvoinData-workshop aikasarjat
Roope Tervo
 
PDF
Avoindata workshop tekninen_yleiskuvaus
Roope Tervo
 
FMI Open Data Impact Survey 2019
Roope Tervo
 
Predicting weather inflicted train delays
Roope Tervo
 
Forecasting Electricity Outages Caused by Convective Storms
Roope Tervo
 
Possibilities of Open Source Code
Roope Tervo
 
AvoinData-workshop käyttöesimerkki
Roope Tervo
 
AvoinData aineistot
Roope Tervo
 
AvoinData-workshop aikasarjat
Roope Tervo
 
Avoindata workshop tekninen_yleiskuvaus
Roope Tervo
 

Recently uploaded (20)

PDF
Design of Material Handling Equipment Lecture Note
Barsena
 
PPTX
Micro1New.ppt.pptx the mai themes of micfrobiology
nehasingh160179
 
PDF
Unit1 - AIML Chapter 1 concept and ethics
muthusamyvijay762
 
PDF
Java Basics-Introduction and program control
lohithsrikarb719
 
PDF
UEFA_Carbon_Footprint_Calculator_Methology_2.0.pdf
tuchinad02
 
PDF
Unit I -OPERATING SYSTEMS_SRM_KATTANKULATHUR.pptx.pdf
HarshDudhatra4
 
PDF
First part_B-Image Processing - 1 of 2).pdf
barabre
 
PDF
MLpara ingenieira CIVIL, meca Y AMBIENTAL
MarceloArielTisera
 
PPT
Chapter 1 - Introduction to Manufacturing Technology_2.ppt
garciajeremiah070
 
PDF
VSL-Strand-Post-tensioning-Systems-Technical-Catalogue_2019-01.pdf
junaidhussain07049
 
PPTX
PRASUNET_20240614003_231416_0000[1].pptx
singlagaurav1311
 
PPTX
Chemical Technological Processes, Feasibility Study and Chemical Process Indu...
Raihan87
 
PDF
Beginners-Guide-to-Artificial-Intelligence.pdf
TamerAMekhimar
 
PDF
Computer System Architecture 3rd Edition-M Morris Mano.pdf
Nune SrinivasRao
 
PPTX
"Array and Linked List in Data Structures with Types, Operations, Implementat...
usham61
 
PPTX
ai_satellite_crop_management_20250815030350.pptx
kathiravanh1919
 
PDF
Computer organization and architecuture Digital Notes....pdf
Nune SrinivasRao
 
PDF
20250617 - IR - Global Guide for HR - 51 pages.pdf
JanBaumann6
 
PPTX
Principal presentation for NAAC (1).pptx
GMKosgiker
 
PPTX
CN_Unite_1 AI&DS ENGGERING SPPU PUNE UNIVERSITY
versetooni85
 
Design of Material Handling Equipment Lecture Note
Barsena
 
Micro1New.ppt.pptx the mai themes of micfrobiology
nehasingh160179
 
Unit1 - AIML Chapter 1 concept and ethics
muthusamyvijay762
 
Java Basics-Introduction and program control
lohithsrikarb719
 
UEFA_Carbon_Footprint_Calculator_Methology_2.0.pdf
tuchinad02
 
Unit I -OPERATING SYSTEMS_SRM_KATTANKULATHUR.pptx.pdf
HarshDudhatra4
 
First part_B-Image Processing - 1 of 2).pdf
barabre
 
MLpara ingenieira CIVIL, meca Y AMBIENTAL
MarceloArielTisera
 
Chapter 1 - Introduction to Manufacturing Technology_2.ppt
garciajeremiah070
 
VSL-Strand-Post-tensioning-Systems-Technical-Catalogue_2019-01.pdf
junaidhussain07049
 
PRASUNET_20240614003_231416_0000[1].pptx
singlagaurav1311
 
Chemical Technological Processes, Feasibility Study and Chemical Process Indu...
Raihan87
 
Beginners-Guide-to-Artificial-Intelligence.pdf
TamerAMekhimar
 
Computer System Architecture 3rd Edition-M Morris Mano.pdf
Nune SrinivasRao
 
"Array and Linked List in Data Structures with Types, Operations, Implementat...
usham61
 
ai_satellite_crop_management_20250815030350.pptx
kathiravanh1919
 
Computer organization and architecuture Digital Notes....pdf
Nune SrinivasRao
 
20250617 - IR - Global Guide for HR - 51 pages.pdf
JanBaumann6
 
Principal presentation for NAAC (1).pptx
GMKosgiker
 
CN_Unite_1 AI&DS ENGGERING SPPU PUNE UNIVERSITY
versetooni85
 

FMI Open Data Interface and Data Models

  • 1. Open Meteorological Data Usage and Best Practices Finnish Meteorological Institute Roope Tervo, Mikko Visa
  • 2. Finnish Meteorological Institute opened its data in 2013. Basically everything FMI owns was opened. Data is provided freely in machine readable format. 6.10.2014 FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 2 FMI Open Data https://en.ilmatieteenlaitos.fi/open-data
  • 3. FMI Open Data Portal FMI Open Data Portal follows INSPIRE requirements. Meta data Data Models Services The very same data portal works as Open Data and INSPIRE portal. FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 3
  • 4. Example of Data Sets Data set Description Time Interval Estimated publish date Weather Observations Temperature, Wind, Humidity, Ground Temperature… 10 min Open, older data to be added Sun Radiation UV, Short and Long Term Radiation… 1 min Open Marine Observations Waves, Sea Temperature, Sea Level… 1 h Open Weather Radars Precipitation Rate, Precipitation Amount… 5 min Open, older and more data to be added 2015 Lightning Thunder Strikes in Finland 5 min Open FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 4
  • 5. Example of Data Sets Data set Description Time Interval Estimated publish date Real Time Observations Real Time Observations from specific location(s) AWS 2010 – Soundings 1959 – Flashes 1998 – Sea Level 1971 – Waves 2005 – Open older data will be added Climatological Observations Daily and monthly temperature mean and extreme values from weather stations 1959 - Open Climatological Observations Monthly temperature and precipitation rate mean values interpolated to grid 1961 - Open Climatological Reference Climatological Reference. Temperature, humidity, pressure, precipitation amount and snow depth. Reference seasons: 1971-2000 1981- 2010 Open FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 5
  • 6. Example of Data Sets Data set Description Time Interval Estimated publish date Weather forecast model HIRLAM RCR Point forecasts and grid data Latest model run (4 times a day) 0…54 h Open Sea forecast models Sea level point forecasts, Wave (WAM) and current (HBM) as grid data Latest model run (4 times a day) 0...54 h Open Environmental Monitoring Facilities Weather observation stations, radars… - 2015 Aviation Observations METAR/IWXXM 30 min Open Ground & mast observations Special observations from ground and masts 2014 FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 6
  • 7. Example of Data Sets Data set Description Time Interval Estimated publish date Air Quality Observations Air Quality Observations 1h 2015-2016 Silam Model Dispersion Model for Air Quality, Forest Fire and Pollen Latest model run (once a day) 0…96h 2014-2015 HELMI Ice Model Ice forecast model Latest model run (4 times a day) 0...54 h 2014/12 Soundings Temperature, Humidity, Pressure, Wind from ground to 25 km height 2 times a day 2014 Quality information Quality information for observations FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 7
  • 8. Registration o Registration is required to use View and Download Services o Working email address is the only mandatory information o After registration the user gets an API key which have to be added into all requests o GET parameter fmi-apikey=…& o Header fmi-apikey; … o Part of url http://wms.fmi.fi/fmi-apikey/…/wms? o One can create several API keys with one email FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 8
  • 9. Usage Limits With one API key it’s allowed to o do at most 20 000 requests per day to Download Service o do at most 10 000 requests per day to View Service o do at most 600 requests per 5 minutes to both services o If all observations from one time step is calculated to as one, little over 17 000 new data sets are published daily o So, with one API key it’s allowed load everything once o View service can be used for testing but can not be used as a back end for popular clients FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 9
  • 10. Catalog Service (CSW) o Based on GeoNetwork o High-level metadata o Provides UI o Maybe more useful sources: o http://en.ilmatieteenlaitos.fi/ope n-data-manual-fmi-wfs-services (based on WFS DescribeStoredQueries) o http://www.paikkatietoikkuna.fi/ web/fi/metatietohaku (based on GeoNetwork) FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 10
  • 11. View Service (WMS) o Based on GeoServer o Visualize observation and forecast data and provides radar images o Only the most common layers are published o Can be used for prototyping - not for production use FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 11
  • 12. Download Service (WFS 2.0) o Web Feature Service (WFS) 2.0 Simple Profile o Based on stored queries o Predefined data sets with possibility for additional parameters (i.e. time and area) o In-house production FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 12
  • 13. GetCapabilities: • Describes capabilities of the service • Normally start point for client applications • http://data.fmi.fi/fmi-apikey/…/wfs?request=GetCapabilities 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 13 WFS Basics obseravtions and point forecasts
  • 14. DescribeStoredQueries • List and describes available stored queries (pre-defined data sets) • Stored query name and description • Possible additional parameters for the data query • http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 14 WFS Basics obseravtions and point forecasts
  • 15. GetFeature • Actual data query • In FMI service, stored query id is compulsory • &storedquery_id=.. • Other possible parameters depend on the stored query • Possible values can be fetched with DescribeStoredQueries request. • Location has to be often defined some how. http://data.fmi.fi/fmi-apikey/.../ wfs?request=GetFeature&storedquery_id=fmi::observations::weather::citi es::timevaluepair&bbox=21,62,22,64 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 15 WFS Basics obseravtions and point forecasts
  • 16. Location parameters in stored queries & bbox  area • By default, in same srs with the (requested) srs. Specify as fifth element if needed (i.e. &bbox=132249,6433579,631999,6933329,epsg:3067) & fmisid  FMI id for observation station & wmo  WMO id for observation station 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 16 WFS Basics obseravtions and point forecasts
  • 17. Location parameters in stored queries & geoid  http://geonames.org id for location & place  human readable name • use region to accurate your query (i.e. &place=kumpula,tampere) • can be used several times in one query (i.e. &place=kumpula,helsinki&place=heinola) & latlon  coordinates (used only in marine data sets) 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 17 WFS Basics obseravtions and point forecasts
  • 18. GetPropertyValue • Responses only requested part of the GetFeature response • Use ValueReference=xpath to define part of interest • Same additional parameters for stored query with GetFeature request are still valid • For example: http://data.fmi.fi/fmi-apikey/…/ wfs?request=GetPropertyValue&storedquery_id=fmi::observatio ns::weather::multipointcoverage&place=kumpula,tampere&ValueReferenc e=wfs:FeatureCollection/wfs:member/omso:GridSeriesObservation/om:res ult 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 18 WFS Basics obseravtions and point forecasts
  • 19. Data Models o Observations and point forecasts as GML o The same data is published in two different formats. o Gridded data is provided in appropriate binary format (grib, NetCDF, HDF…) o WFS members contains the metadata ‘envelope’ with a link to a actual data o Every XML document as self-explanatory as possivble FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 19
  • 20. om:phenomenonTime • Tells start and end time of returned data 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 20 Data Models most important elements
  • 21. om:procedure • Describes data producing procedure om:parameter • Changing parameters in the process • i.e. analysis time of the numerical model 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 21 Data Models most important elements
  • 22. om:featureOfInterest • Describes location(s) or area of returned data • Several different kind of identifiers • target:SurfaceWeatherTargetArea  area (grid data) • fmisid  FMI id for observation station • wmo  WMO id for observation station • geoid  http://geonames.org id for location • name  Human readable name • region  county of the station 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 22 Data Models most important elements
  • 23. om:observedProperty • Describes meteorological parameters in returned data • Defined as xlink • Describes label, basePhenomenon and units of measure of every parameter in the reponse • Tip: use the same service without parameter definitions to query all possible parameters 6.10.2014 • http://data.fmi.fi/fmi-apikey/.../meta?observableProperty=forecast • http://data.fmi.fi/fmi-apikey/.../ meta?observableProperty=observation • Note that all parameters are not available in all data sets FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 23 Data Models most important elements
  • 24. om:result • Data in required format in observations and point forecasts and link to a external binary data in grid formatted data 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 24 Data Models most important elements
  • 25. Data Models gmlcov:MultiPointCoverage gml:rangeSet gml:doubleOrNilReasonTupleList The data is listed for every point defined in domain set. gml:domainSet gmlcov:simpleMultiPoint The coverage is defined as a list of points in 4 dimensional grid (lat, lon, height, time). gmlcov:rangeType The parameters listed in range set are defined in separate element. FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 25
  • 26. Cons - Not intuitive - No natural structure of XML  XSLT and Xpath don’t work Data Models gmlcov:MultiPointCoverage Pros + Compact + Efficient + Small file size + Works for many data types FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 26
  • 27. Data Models wml2:MeasurementTimeseries wml2:MeasurementTimeseries One member contains time series of one parameter and one location FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 27
  • 28. Cons - Lots of repetition - Large file size - Heavy for DOM-based parsers - Don’t work i.e. for thunder strikes Data Models wml2:MeasurementTime series Pros + Intuitive + Easy to use + XSLT & XPath works FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 28
  • 29. Data Models SimpleFeature SimpleFeature One member contains one time step of one parameter and one location FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 29
  • 30. Cons - Huge file size - Heavy for DOM-based parsers Data Models SimpleFeature Pros + Intuitive + Easy to use + Wide application support FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 30
  • 31. Data Models file size comparison 81.7 52.9 Document Size [MB] Compressed DocumentSize [MB] 1.3 1.2 1.8 0.2 90 80 70 60 50 40 30 20 10 0 FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 31
  • 32. • Aviation weather reposts are delivered as IWXXM • New data model coming into use in aviation • Consists of the same elements than other messages • om:phenomenonTime, om:procedure, om:featureOfInterest, om:result • Content of the METAR is in om:result part as • extracted into XML elements • original, “old fashion”, METAR string 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 32 Data Models aviation observations IWXXM / METARS
  • 33. • Once published, content and structure of IWXXM messages will not change • Messages will contain a digital signature • Use GetPropertyValue to download only original METAR string • http://data.fmi.fi/fmi-apikey/…/ wfs?request=GetPropertyValue&storedquery_id=f mi::avi::observations::finland::latest::iwxxm&valuereference= wfs:FeatureCollection/wfs:member/avi:VerifiableMessage/av i:metadata/avi:MessageMetadata/avi:source/avi:Process/avi: input 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 33 Data Models aviation observations IWXXM / METARS
  • 34. • Messages have been automatically created based on the original METAR/SPECI code messages. • Messages are NOT an authoritative aviation weather report. They SHOULD NOT be used as a weather report to be used in flight planning or other aviation related use. 6.10.2014 FMI Open Meteorological Data Services Roope Tervo, Mikko Visa 34 Data Models aviation observations IWXXM / METARS
  • 35. • WFS response contains the same meta data information than in observations and point forecasts • om:result contains gmlcov:RectifiedGridCoverage • Basically the same with multipointcoverage • domainSet defines the grid (now as regular grid) • rangeSet contains data as an external link to the binary content • rangeType defines the parameters 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 35 Data Models binary data
  • 36. • Note that fileReference points often to an other service • Can be used directly, but • It is always good practice to consult WFS for available times and parameters 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 36 Data Models binary data Client WFS WMS FMI Data Server Available radar images? Available weather model outputs?
  • 37. • Area, time and weather parameters may be defined in the request • For weather models, intersection of requested area and available data area is returned • For radar images, all images which intersects requested spatial and temporal space are returned 6.10.2014 Radar images Weather models Model coverage FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 37 Data Models binary data Requested area Radar images Requested area
  • 38. Radar images • The reference points to a “original” gray scale GeoTiff image • Images can also be downloaded as color images • Remove &styles=raster from the data request • But then information is lost • Used SLD-files can be downloaded from https://github.com/fmidev/opendata-resources/tree/master/sld • Consult om:parameter element for single radar measurement parameters • Scanning angle, bin count and bin length 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 38 Data Models binary data
  • 39. Numerical models • The reference points to (often a subset) the model output in appropriate format • GRIB2 for weather models • NetCDF for marine models • Note that whole model output can be large • Up to 17 GB • Download always only area, parameters and time range of interest 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 39 Data Models binary data
  • 40. Helpful Software For binary data • PanoPly • http://www.giss.nasa.gov/tools/panoply/ • Simple • Handle only very basic data • IDV • http://www.unidata.ucar.edu/software/idv/ • Handle most of the data 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 40
  • 41. Helpful Software For exploring the data • Google Earth & QGIS • Start from GetCapabilities, UI guides thereafterhttp://wms.fmi.fi/fmi-apikey/.../ geoserver/wms?service=WMS&version=1.3.0 &request=GetCapabilities 6.10.2014 FMI Open Meteorological Data Service Roope Tervo, Mikko Visa 41
  • 42. MetoLib o Open source JavaScript library produced by Finnish Meteorological Institute o Helps users to load and use the data Easy requests Cache Parse the data to as JSON o Supports multi point coverage data format o Python version is on the list FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa 6.10.2014 42