Performance and Application of GIS and Big Data ETL Processes Using FME

Editor's Notes

• #6: Amazon’s Simple Storage Service (S3) Amazon Web Service’s cloud storage service Capable of storing practically unlimited amounts of data Data is stored in directories called buckets Stored input and output data DynamoDB AWS’s fully-managed No-SQL database service DynamoDB was utilized to store geohash indexes and metadata for output imagery PostGIS Spatial database extender for PostgreSQL Most powerful open source spatial database engine Used to store output composite Landsat imagery and metadata
• #7: ExpanDrive and TntDrive Provide ability to mount S3 bucket as a drive Copy, move, add, and delete files through file directory explorer Utilized to download data from S3 and to allow FME to read data directly from S3 FME Desktop All ETL workflows designed for this project were developed using FME Desktop FME Cloud Runs on AWS Instance runs a virtual FME Server machine and PostGIS database Provides scalability - Instance can be expanded up to 16 cores, with up to 4 engines per core Maximum of 64 jobs running in parallel Was used to run workspaces and to host the output PostGIS database. Two Payment structures Pay by hour Annual subscription
• #10: Landsat 8 single-band imagery stored in a public Amazon S3 bucket Imagery for 2015 alone contains over 150 TB of data. Catalogued using the Worldwide Reference System (WRS), which divides the satellite-imaged area into paths and rows. The Landsat 8 satellite records imagery in 11 spectral bands, and each image is stored separately. In S3, the data is organized by scene (WRS Path-Row combination) -grouped in sets based landsat pass Each sub-directory contains 11 GeoTIFF’s (1 for each individual Landsat 8 band) and a metadata text file
• #12: Reads and processes the single-band Landsat imagery and metadata text files Outputs composite images, using 11 common band combinations. 6 main parts
• #13: Uses the MetadataExtractor custom transformer to create and populate metadata attributes Uses regular expressions Creates attributes such as time, date, station, id, WRS path and row, and cloud cover
• #14: Developed to increase performance Eliminates the need to read every GeoTIFF within the study area Text files are sorted by scene and filtered to find the most recent file with the lowest cloud coverage per Landsat scene – 1 per scene Only the corresponding GeoTIFF files are read into the workspace In FME 2016, a single AttributeManager can be used to create and rename different attributes
• #16: Images are filtered by band. Bands are sent through FullBandCombiners Combines images based on band combo type Stretches pixel values to allow for color distinction Outputs combined images
• #17: Performs raster enhancement techniques Adding an alpha channel for transparency Add Pan Sharpening Uses the high-resolution data from Band 8 to “sharpen” the lower-resolution data RasterExpressionEvaluator to apply a weighted Brovey transformation equation
• #19: Merges all composite images with the same band combination into a single image
• #20: List of the 11 band combinations produced by the base FME Workspace Map on the right shows an example of 4 composite images produced for Landsat scene 159-073
• #22: 4 case studies were developed, which explored 3 variables. The case studies range from being entirely local to entirely in the cloud.
• #23: Case Study 1 was completed entirely on-premises. Data was read from, processed on, and written to a local machine. The study area dataset was downloaded from S3 to a local drive, using both TntDrive and ExpanDrive. The base workspace was run locally, and the output GeoTIFFs were written to a local directory.
• #24: Case Study 2 explores the use of cloud storage for the input data, while retaining the locality of the rest of the process. The source text files and GeoTIFFs were read directly from the Amazon S3 bucket, using TntDrive. The reader and writer in the CS2 workspace are identical to those used in CS1, but the pathnames were redirected to the landsat-pds bucket, mounted as the Y: drive.
• #25: For Case Study 3, all input and output data was stored in the cloud, and the actual image processing took place on-premises. There were 2 parts to Case Study 3: Part A wrote to Amazon S3 and DynamoDB, while Part B wrote to a PostGIS database hosted on FME Cloud. In both parts, the FME workspaces were run locally, but they read the Landsat data directly from the landsat-pds public S3 bucket using TntDrive.
• #26: Case Study 4 was executed entirely in the Cloud. Source data was read from FME Server resources, the workspaces were run using FME Server, and data was written a PostGIS database that was hosted on FME Cloud. The process was executed twice – with 1 engine, then with 8 engines running in parallel.
• #28: CS1 – Longest total time – due to downloading CS2 – Fastest Single-threaded process CS4 Multi-Threaded - Fastest total time CS3 Dynamo was cheapest CS4 Single thread – most expensive CS4 Multi-threaded – Cheaper than single
• #29: Successfulness determined through analyzing common set of success criteria 1 = best, 5 = worst for each criterion Case Study 1 Slowest total time Easiest setup Largest strain on local machine Case study 4 least local strain Case Study 4 8 engines – fastest overall time CS with highest score (CS3 Dynamo) is least successful CS with lowest score (CS4 PostGIS 8 Engines) is most successful