Building a Data Analytics PaaS for Smart Cities

1© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved.
Building a Data Analytics
PaaS for Smart Cities
Smiti Sharma, EMC Virtustream
Keith Manthey, EMC ETD
BRDC

Intelligent Communities
Cities and Regions that use technology not just to
save money or make things work better, but also to
create high-quality employment, increase citizen
participation and become great places to live and
work.
ICF – Intelligent Community Forum

3© Copyright 2015 EMC Corporation. All rights reserved.
“Smart Cities that use Big Data are neither about intuition nor about
looking back and analyzing what went wrong and could be better.
They spot patterns. They look forward. They predict potential crisis situations.
They find what could be better and make it better.
Smart cities don’t guess.
Theyaresure!

4© Copyright 2015 EMC Corporation. All rights reserved. 4© Copyright 2015 EMC Corporation. All rights reserved.
VISION FOR CITIES OF THE FUTURE
Become an innovative city
SAFE
Anticipate risks and
protect people and
information
EFFICIENT
Optimized use of
city resources
SEAMLESS
Integrated daily
life services
IMPACTFUL
Enriched life and
business experiences
for all

IMPLICATIONS TO THE CITY
Empower the city, citizens, visitors, and businesses
IMPROVE
Quality of
urban living
CREATE
Efficient city
and transparent
government
DEVELOP
Vital
economy
REDUCE
Environmental
impact
ADDRESS
Infrastructure,
buildings and
urban planning
IMPROVE
Tourism,
recreation,
and city image

“BIG DATA” ENABLESCITIES OF THE FUTURE
Any data-set that cannot be processed with traditional systems
Social Networks, UGCPublic records
Location DataInternet of things
Emerging Data Sources
Unstructured Data
Dark Data
Structured Data
Traditional Data Sources

BUILD SMART CITY: THE PROBLEM
Understand the city data challenge
Geo
Distributed
Data
Source
Satellite-borne
Imaging Device
Airborne
Imaging
Device
Webcam
Environmental
Monitor
Health
Monitor
Traffic
monitor
Industrial
Process
Monitor
Data Center
Centralized Storage
and Analytics Systems
City Network
DATA CHALLENGE
There are massive endpoints for these systems.
How to manage massive and heterogeneous data
becomes an enormous challenge.
Diverse data sources requires normalization
and standardization to address Data
Orchestration and integration.
DATA USAGE CHALLENGE
How could we create some innovative business
to use these data to create more value
and fully use current investment?

Architecture

Guiding Principles
Agile
Open
Portable
Extensible Modular/Ftl.
Blocks
Analytics
Driven
Software
Defined

Architecture
Ingestion Layer
Spring XD
Transformation Layer
Python
/Transformed_Files
KPIs
Métricas
exploration
Maps & Graphs
Visualization Layer
API
Open Data
Data Integration Layer
Python
/Transformed_Files
Schema and
Instance
Alignment
Data Sources
GUIs, Dashboard that access the
underlying databases and promote an
excellent User experience
Data modelling, metrics , ETL
mechanisms, definitions and
variable selectionProprietary and
Open Data
sources
APIs to expose data
Analytics
Data mining prediction

Ingestion
Ingestion Layer
Spring XD
Data Sources

Transformation
Transformation Layer
Python
/Transformed_Files
Data Cleaning Conversion

Integration
Schema
Integration
Instance
Alignment
Integration Layer
Python
/Transformed_Files
Schema and
Instance
Alignment

Integration
• Schema DB (INPUT)
• Schema Matching (Algorithms &
Heuristics)
• Suggest Attribute Mappings
(OUTPUT : SEMI-SUPERVISED)
• Instances of DB tables & Integration Rules (INPUT)
• Deduplication, Record Consolidation (Algorithms & Heuristics)
• Instance alignment using 2 phase-pass algorithm to avoid
duplicate insertion in a semi supervised data integration
tool)
• Attribute name similarity: fuzzy string comparisons
(cosine similarity)
• Levenshtein similarity: Categorical/String Data
• http://pgsimilarity.projects.pgfoundry.org/
• List of deduplicated instances (OUTPUT - SEMI-SUPERVISED)
Schema
Integration
Instance
Alignment
Integration Layer
Python
/Transformed_Files
Schema and
Instance
Alignment

Integration
Schema
Integration
Instance
Alignment
Camada de Integração
Python
/Transformed_Files
Schema and
Instance
Alignment
Deduplication
Similarity Join
Mapeamento de
atributos (Inserir)
Mapeamento de
atributos (Selecionar)
Cosine
Levenshtein

Visualization
KPIs
Métricas
exploration
Maps & Graphs
Visualization Layer

API implementation
API
Open Data

Example use Case
Transportation

• Available Data:
– City bus movement information from on board devices (lat-
long, time, date, bus line, bus ID)
• Goals:
– Predict the time of the arrival in a bus stop
• Challenges
– Lack of data in certain areas of the city
– GPS precision
Prediction of bus arrival

Architecture
GPS Ônibus Gemfire XD
Routes & Bus Stop
Data Lake
Streaming
Scheduler
Lazy-write
GPS

• Use each bus stop as a node, and the street as
edges
Transportation Network
𝑥𝑖 𝑥𝑗
𝑎𝑖𝑗 = +1
𝑎𝑖𝑗 = −1
𝑋 = 𝑥1, … , 𝑥 𝑁 , 𝑥𝑖 = 𝑙𝑎𝑡 𝑖 , 𝑙𝑜𝑛𝑔(𝑖)
𝐸 = 𝑒1, … , 𝑒 𝑀 , 𝑒 𝑘 = (𝑥𝑖, 𝑥𝑗)

• The goal is to find, for each 𝑒𝑗, an estimation of the
average speed in a instant 𝑡, 𝑣 𝑒𝑗, 𝑡 .
• Default model - estimate the velocity in each edge,
using historical data from the last month.
– Different hourly models for each day of the week
• Online Model - Use real-time date to calculate the
speed.
The model

Average speed(km/h)
Default Model

Prediction of Bus Arrival

• Based on the information of last use case,
extrapolate to verify the quality of the service
• Need to identify each bus trip, to evaluate the time
interval between two buses of the same line, at each
bus stop.
Another use case - Auditing

Screen shot - Auditing

Data Quality Issues
Route A
Route B
Bus GPS

Extending
PaaS for
Smart Cities

Real-time
Dashboard
Personal
Dashboard
Government
Transactional
Applications
Commercial
Big Data
Application
Government
Big Data
Application
Commercial
Transactional
Applications
Unified
Control
Center
Application
Layer
Security
Rules
Payment
Gateway
Trust
Authentication
Identity
Management
Locations &
Mapping
Platform as
a Service
Data
Governance
DATA ANALYTICS TOOLS Historic & Predictive/DATA APIs
Transactiona
l Data Store
Data
Transformation
Unstructured
Data
Structured
Data
City
Semantics
Audit
Open Standards Data Ingestion Interfaces and Storage
CITY IoT INFRASTRUCTURE CITY DATA SOURCES CITY ICT INFRASTRUCTURE
Government
Devices
Commercia
l Devices
Utility
Devices
Personal
Devices
IoT Data Aggregation
Governmen
t Systems
Social
Media
Commercial
Systems
Archived
Data
Fixed &
Wireless
Networks
Cloud
Services
Enablement
Layer
Data
Orchestration
Layer
Infrastructure
Layer
SECURITY
Smart City Platform requirements

Real-time
Dashboard
Personal
Dashboard
Government
Transactional
Applications
Commercial
Big Data
Application
Government
Big Data
Application
Commercial
Transactional
Applications
Unified
Control
Center
Application
Layer
Security
Rules
Payment
Gateway
Trust
Authentication
Identity
Management
Locations &
Mapping
Platform as
a Service
Data
Governance
DATA ANALYTICS TOOLS Historic & Predictive/DATA APIs
Transactiona
l Data Store
Data
Transformation
Unstructured
Data
Structured
Data
City
Semantics
Audit
Open Standards Data Ingestion Interfaces and Storage
CITY IoT INFRASTRUCTURE CITY DATA SOURCES CITY ICT INFRASTRUCTURE
Government
Devices
Commercia
l Devices
Utility
Devices
Personal
Devices
IoT Data Aggregation
Governmen
t Systems
Social
Media
Commercial
Systems
Archived
Data
Fixed &
Wireless
Networks
Cloud
Services
Enablement
Layer
Data
Orchestration
Layer
Infrastructure
Layer
SECURITY
High level Smart City Platform components
PCF Pivotal Cloud Foundry
E M C S T O R A G E
IISILON and./or
CLOUD NATIVE SOFTWARE DEFINED STORAGE
V M W A R E v R e a l i z e C l o u d S u i t e & B I G D A T A
E X T E N S I O N S
P I V O T A L B I G D A T A S U I T E
A D V A N C E D A N A L Y T I C SA P P L I C A T I O N S
A T S C A L E
D A T A
P R O C E S S I N G
GREENPLUM
DATABASE
HAWQ
SPRING XD SPARK
REDIS
RABBITMQ
GEMFIRE
H A D O O P

Pivotal GPDB Delivers
 Massively Parallel
Analytics Performance
 In-Database Analytical
Extensions
 Industry-Leading Load
Speed
 Rich SQL with Schema
Agnosticism
 Industry-Leading
Workload Mgmt.
 SAS Acceleration
Options
 Parallel Co-Processing
with Hadoop
 No-Forklift Scalability
 Multi-Level
Redundancy
 Rich, Easy-to-Use
Administration Tools
 Big Data Backup
 Comprehensive
Security
 Software-only or DCA

Simple to manage
Single file system, single volume, global namespace
Massively scalable
Scales from 16 TB to over 50 PB in a single cluster
200GB/s throughput, 3.75M IOPS
Unmatched efficiency
Over 80% storage utilization, automated tiering and SmartDedupe
Enterprise data protection
Efficient backup and disaster recovery, and N+1 thru N+4 redundancy
Robust security and compliance options
RBAC, Access Zones, WORM data security, File System Auditing
Data At Rest Encryption with SEDs, STIG hardening
CAC/PIV Smartcard authentication, FIPS OpenSSL support
Operational flexibility
Multi-protocol support including NFS, SMB, HTTP, FTP and HDFS
Object and Cloud computing including OpenStack Swift
Isilon Scale-Out NAS

Lots of Little Files
Hadoop Impact on Telemetry
AKA - Small Files Problem for Hadoop
Rio Smart Sensors - ESRI
NameNode = 512 GB for RAM
Each file eats away 1K in RAM
512GB / 1K = At Most 500M Files
assuming no other processes on the
box.
Rio has 12.5K sensors for the 2016
Olympics. Assuming each sensor
sent a file every minute, 18M files in
1 day.
EMC believes in storing Metadata
on SSD. This allows a scale out
for the NameNode to get around
the limitations of file growth on
the scale-up NameNode.

Building a Data Analytics PaaS for Smart Cities

Building a Data Analytics PaaS for Smart Cities

More Related Content

What's hot (20)

Similar to Building a Data Analytics PaaS for Smart Cities (20)

More from DataWorks Summit/Hadoop Summit (20)

Recently uploaded (20)

Building a Data Analytics PaaS for Smart Cities