Supporting Financial Services with a More Flexible Approach to Big Data

Supporting Financial Services
With a More Flexible Approach
to Big Data
October 21, 2014

WWW.WANDISCO.COMREALIZING THE POSSIBILITIES OF BIG DATA
Our Presenters
Jus$n
Sears
is
a
Product
Marke$ng
Manager
at
Hortonworks,
where
he
writes
stories

about
how
enterprise
customers
use
Apache
Hadoop
to
solve
big
data
business

challenges.
He
also
manages
product
launch
marke$ng
and
campaign
content
for

Hortonworks.
For
seventeen
years,
Jus$n
has
led
teams
in
Silicon
Valley
to
create

and
posi$on
enterprise
soCware,
risk-‐controlled
consumer
banking
products,

desktop
and
mobile
web
proper$es,
and
services
for
La$no
customers
in
the
US
and

La$n
America.
He
lives
with
his
family
in
his
na$ve
San
Francisco
Bay
Area.

BreH
Rudenstein
has
an
extensive
background
in
Applica$on
Lifecycle
Management,

High
Performance
Compu$ng
and
Open
Source
SoCware
Analysis.
He
has
held
senior

sales
engineering
and
management
posi$ons
at
Ra$onal
SoCware,
PureAtria,

IBM,
Appistry
and
Palamida.
Throughout
his
career,
he
has
enabled
organiza$ons
to

accelerate
technology
adop$on
by
understanding
their
needs
and
providing
just-‐in-‐
$me
business
solu$ons.
As
WANdisco
Director
of
Product
Management
for
Big

Data,
BreH
works
with
partners,
prospects
and
customers
to
help

them
understand
and
evolve
the
requirements
for
enterprise-‐ready
Hadoop.

© Hortonworks Inc. 2011 – 2014. All Rights Reserved
Hortonworks
We Do Hadoop

Our Mission:
Power your Modern Data Architecture
with HDP and Enterprise Apache Hadoop
Who we are
June 2011: Original 24 architects, developers, operators of Hadoop from Yahoo!
June 2014: An enterprise software company with 420+ Employees
Key Partners
Our model
Innovate and deliver Apache Hadoop as a complete enterprise data platform
completely in the open, backed by a world class support organization

Fastest growing Fortune 1000 customer base
Customer Momentum
•  300+ customers in seven quarters, growing at 75+/quarter
•  Two thirds of customers come from F1000
•  100% renewal rate
Largest Cluster in North America
32,000 Nodes
Largest Cluster in Europe
1,000 Nodes
Some notable migrations include many of the early adopters of Hadoop:
Experience at Scale
80,000 nodes under contract
Largest Known Cluster in APAC
400 Nodes
30+ customers migrated from other distributions

Hortonworks: A Leader In Hadoop
The Forrester Wave™: Big Data Hadoop Solutions, Q1 2014
“Hortonworks loves and lives
open source innovation”
Vision & Execution for Enterprise Hadoop.
Hortonworks leads with a strong strategy and roadmap for open source innovation
with Hadoop and a strong delivery of that innovation in Hortonworks Data Platform.
World Class Support and Services.
Hortonworks' Customer Support received a maximum score
and was significantly higher than both Cloudera and MapR.
Key Strategic Partnerships.
Hortonworks’ unique strategic partnerships with Microsoft, SAP, Teradata and others
are a key strength as part of its overall strategy of ecosystem partnership to
accelerate Hadoop adoption in the enterprise.

HDP IS Apache Hadoop
There is ONE Enterprise Hadoop: everything else is a vendor derivation
HDP
•  Reliable
•  Consistent
•  Current

Enabling a Modern Data Architecture
with HDP and Apache Hadoop
Hortonworks. We do Hadoop.

APPLICATIONSDATASYSTEM
Business
Analytics
Custom
Applications
Packaged
Applications
Traditional systems under pressure
•  Silos of Data
•  Costly to Scale
•  Constrained Schemas
Clickstream
Geolocation
Sentiment, Web Data
Sensor, Machine Data
Unstructured docs, emails
Server logs
SOURCES
Existing Sources
(CRM, ERP,…)
RDBMS EDW MPP
New Data Types
…and difficult to
manage new data

Traditional Hadoop, challenges & limitations
1 ° ° ° ° °
° ° ° ° ° N
HDFS
(Hadoop Distributed File System)
MapReduce
Largely Batch Processing
SOURCES
EXISTING

Systems

Clickstream
Web
&Social
Geoloca9on
Sensor
&

Machine

Server
Logs
Unstructured

Architectural Limitations
•  Single-purpose clusters, specific data sets
•  Primarily a batch system using MapReduce
Enterprise Challenges
•  Limited enterprise capabilities:  
Operations, Security & Governance
•  Created additional Silos

Interoperability Challenges
•  Diﬃcult to natively integrate existing applications

Commercial add-ons opportunistically emerged  
in the early days to address these shortcomings
Business
Analytics
Custom
Applications
Packaged
Applications
RDBMS EDW MPP

20092006
1
°
°
°
°
°

°
°
°
°
°
N

HDFS

(Hadoop
Distributed
File
System)

MapReduce

Largely
Batch
Processing

Hadoop
w/
MapReduce
YARN: Data Operating System
1
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
N
HDFS  
Hadoop2 & YARN based Architecture
Siloed clusters
Largely batch system
Diﬃcult to integrate
MR-‐279:
YARN
Hadoop 2 & YARN
Interactive Real-TimeBatch
Architected &  
led development
of YARN to enable
the Modern Data
Architecture
October 23, 2013

HDP2 and YARN enable the Modern Data Architecture
Hortonworks architected and  
led development of YARN
Common data set, multiple applications
•  Optionally land all data in a single cluster
•  Batch, interactive & real-time use cases
•  Support multi-tenant access, processing
& segmentation of data
YARN: Architectural center of Hadoop
•  Consistent security, governance & operations
•  Ecosystem applications certiﬁed  
by Hortonworks to run natively in Hadoop
SOURCES
EXISTING

Systems

Clickstream
Web

&Social

Geoloca9on
Sensor

&
Machine

Server

Logs

Unstructured

Business
Analytics
Custom
Applications
Packaged
Applications
RDBMS EDW MPP YARN: Data Operating System
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° N
HDFS
Interactive Real-TimeBatch

A Blueprint for Enterprise Hadoop
Load data
and manage
according
to policy
Deploy and
effectively
manage the
platform
Store and process all of your Corporate Data Assets
Access your data simultaneously in multiple ways
(batch, interactive, real-time) Provide layered
approach to
security through
Authentication,
Authorization,
Accounting, and
Data Protection
DATA MANAGEMENT
SECURITYDATA ACCESS
GOVERNANCE
& INTEGRATION
OPERATIONS
Enable both existing and new applications to
provide value to the organization
PRESENTATION & APPLICATION
Empower existing operations and
security tools to manage Hadoop
ENTERPRISE MGMT & SECURITY
Provide deployment choice across physical, virtual, cloud
DEPLOYMENT OPTIONS
YARN Data Operating System

Hortonworks Data Platform 2.2
HDP Delivers Enterprise Hadoop
YARN: Data Operating System
(Cluster Resource Management)
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
Tez
Tez
Java
Scala
Cascading
Tez
° °
° °
° ° ° ° °
° ° ° ° °
HDFS
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
Slider
Slider
SECURITYGOVERNANCE OPERATIONSBATCH, INTERACTIVE & REAL-TIME DATA ACCESS
In-Memory
Spark
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Data Workflow,
Lifecycle &
Governance
Falcon
Sqoop
Flume
Kafka
NFS
WebHDFS
Authentication
Authorization
Accounting
Data Protection
Storage: HDFS
Resources: YARN
Access: Hive, …
Pipeline: Falcon
Cluster: Knox
Deployment ChoiceLinux Windows On-
Premises
Cloud
YARN is the architectural
center of HDP
•  Common data set across all
applications
•  Batch, interactive & real-time
workloads
•  Multi-tenant access & processing
Provides comprehensive
enterprise capabilities
•  Governance
•  Security
•  Operations
Enables broad
ecosystem adoption
•  ISVs can plug directly into Hadoop
The widest range of deployment options
•  Linux & Windows
•  On-premises & cloud
Others
ISV
Engines

The Modern Data Architecture w/ HDP

Clickstream
Capture and analyze
website visitors’ data
trails and optimize
your website
Sensors
Discover patterns in
data streaming
automatically from
remote sensors and
machines
Server Logs
Research logs to
diagnose process
failures and prevent
security breaches
New Types of DataHadoop Value:
Sentiment
Understand how
your customers feel
about your brand
and products –
right now
Geographic
Analyze location-
based data to
manage operations
where they occur
Unstructured
Understand patterns
in files across millions
of web pages, emails,
and documents

New analytic applications for new types of data
$
•  Supplier Consolidation
•  Supply Chain and Logistics
•  Assembly Line Quality Assurance
•  Proactive Maintenance
•  Crowdsourced Quality Assurance
•  New Account Risk Screens
•  Fraud Prevention
•  Trading Risk
•  Maximize Deposit Spread
•  Insurance Underwriting
•  Accelerate Loan Processing
•  Call Detail Records (CDRs)
•  Infrastructure Investment
•  Next Product to Buy (NPTB)
•  Real-time Bandwidth
Allocation
•  New Product Development
•  360° View of the Customer
•  Analyze Brand Sentiment
•  Localized, Personalized
Promotions
•  Website Optimization
•  Optimal Store Layout
Financial
Services
Retail Telecom Manufacturing
Healthcare
Utilities,
Oil & Gas
Public
Sector
•  Genomic data for medical trials
•  Monitor patient vitals
•  Reduce re-admittance rates
•  Store medical research data
•  Recruit cohorts for
pharmaceutical trials
•  Smart meter stream analysis
•  Slow oil well decline curves
•  Optimize lease bidding
•  Compliance reporting
•  Proactive equipment repair
•  Seismic image processing
•  Analyze public sentiment
•  Protect critical networks
•  Prevent fraud and waste
•  Crowdsource reporting for
repairs to infrastructure
•  Fulfill open records requests

..to shift from reactive to proactive interactions
HDP and Hadoop allow
organizations to shift
interactions from…
Reactive
Post Transaction
Proactive
Pre Decision
…to Real-time PersonalizationFrom static branding
…to repair before breakFrom break then fix
…to Designer MedicineFrom mass treatment
…to Automated AlgorithmsFrom Educated Investing
…to 1x1 TargetingFrom mass branding
A shift in Advertising
A shift in Financial Services
A shift in Healthcare
A shift in Retail
A shift in Telco

Data Lake: An architectural shift
SCALE
SCOPE
Unlocking the Data Lake

RDBMS
MPP
EDW
Data Lake
Enabled by YARN
•  Single data repository,
shared infrastructure
•  Multiple biz apps
accessing all the data
•  Enable a shift from
reactive to proactive
interactions
•  Gain new insight across
the entire enterprise
New Analytic Apps
or IT Optimization
HDP 2.1
Governance
&Integration
Security
Operations
Data Access
Data Management
YARN

OPERATIONAL
TOOLS

DEV
&
DATA
TOOLS

INFRASTRUCTURE

HDP is deeply integrated in the data centerSOURCES
EXISTING

Systems

Clickstream
Web
&Social
Geoloca9on
Sensor
&

Machine

Server
Logs
Unstructured

DATASYSTEM
RDBMS
EDW
MPP

HANA
APPLICATIONS

BusinessObjects BI
Deep Partnerships
Hortonworks engages
in deep engineered relationships
with the leaders in the data center,
such as Microsoft, Teradata, Redhat,
HP, SAS & SAP
Broad Partnerships
Over 600 partners work with us to
certify their applications to work with
Hadoop so they can extend big data
to their users
HDP 2.1
Governance
&Integration
Security
Operations
Data Access
Data Management
YARN

HDP Use Cases in Financial Services
Hortonworks. We do Hadoop.

Monetize Anonymous & Aggregate Banking Data
Problem
Valuable banking data needed to be anonymous & unified
•  Bank possesses data that indicates larger macro-economic trends, which can be
monetized in secondary markets
•  Regulations and company policies protect customer privacy
•  Data sets are isolated in legacy silos controlled by LOBs
•  IT challenged by joining data while guaranteeing anonymity
Solution
Cross-bank data lake for aggregate data with secure access
•  Multiple data sets abstracted from source platforms
•  Single point of security & privacy for de-identification, masking, encryption,
authentication and access control
•  Mortgage bankers, consumer bankers, credit card group and treasury bankers have
access to the same cross-sell data
•  Interoperability with partners SAS, R, RedHat & Splunk
•  Economies of scale for compression & archiving data
•  Significant reduction in storage costs from prior platforms
Creating Opportunity
Data: Structured,
Clickstream, Social &
Unstructured
Banking
One of the largest US banks

Insurance Data Lake to Manage Risk
Problem
Challenges merging new & old data hamper analysis
•  Traditional and newer types of data were both growing quickly but were difficult to
combine in the EDW
•  “Schema on load” requirements of EDW platform limited ingest of some data with
significant predictive power
•  Company missed data-driven ways to serve customers
•  Process of separating legitimate from fraudulent claims created “needle-in-a-
haystack” problem
Solution
Common platform for all types of data improves up-sell and reduces fraud
•  “Schema on read” Hadoop architecture means that more data sources can be
easily ingested to enrich predictive analytics
•  Agents use big data insights to determine the best action for valued customers and
recommend those in real-time
•  Claims analysts and underwriters process streaming data to quickly flag fraud risks
and fast-track legitimate claims
Creating Opportunity
Data: Structured,
Clickstream, Server Log
Health Insurance
Large US medical insurer
>$30B in revenue
>20M members
~35K employees

Maintaining SLAs for Equity Trading Information
Problem
Meeting 12 millisecond SLAs for “ticker plant”
•  Daily ingest: 50GB server log data from 10,000 feeds
•  Four times daily, this data is pushed into DB2
•  Applications query this data 35K times per second
•  70% of queries are for data <1 year old, 30% for >1 year old
•  Current architecture can only hold 10 years of trading data
•  Growing volume puts performance at risk of missing SLAs
Solution
Meeting SLAs with confidence
•  HBase provides super-fast queries within SLA targets
•  ETL offloading to Hadoop allows longer data retention, without jeopardizing fast
response times
Improving Efficiency
Data: Server Log & ETL
Investment
Services
Highly trafficked website
providing business and
financial information
~15K employees

Hadoop is a Platform Decision
Open Leadership
Drive innovation in the open via
the Apache community-driven
open source process
Enterprise Rigor
Engineer, test and certify
Apache Hadoop with the
enterprise in mind
Ecosystem Endorsement
Focus on deep integration with
existing data center technologies
and skills
Fastest Growing Customer and Partner Base
Largest and most experienced Hadoop adopters have standardized on Hortonworks
The data center leaders have standardized on Hortonworks

Supporting Financial Services with a More Flexible Approach to Big Data

27
WANdisco Background
•  WANdisco: Wide Area Network Distributed Computing
–  Enterprise-ready, high availability software solutions that enable globally distributed
organizations to meet today’s data challenges of secure storage, scalability and availability
•  Leader in tools for software engineers – Subversion
–  Apache Software Foundation sponsor
•  Highly successful IPO, London Stock Exchange, June 2012 (LSE:WAND)
•  US patented active-active replication technology granted, November 2012
•  Global locations
–  San Ramon (CA)
–  Chengdu (China)
–  Tokyo (Japan)
–  Boston (MA)
–  Sheffield (UK)
–  Belfast (UK)

28
Customers

29
Non-Stop Hadoop
Non-Intrusive Plugin
to Hortonworks HDP
Provides Continuous Availability
In the LAN / Across the WAN
Active/Active

30
3 Problems For Sharing Data Across Clusters
LAN / WAN

31
•  Require Continuous Availability
–  SLA’s, regulatory compliance
•  Require HDFS to be Deployed Globally
–  Share data between data centers
–  Data is consistent, not eventual
•  Ease Administrative Burden
–  Reduce operational complexity
–  Simplify disaster recovery
–  Lower RTO/RPO
•  Allow Maximum Utilization of
Resources
–  Within the data center
–  Across data centers
Enterprise-Ready Hadoop
Characteristics of Mission-critical Financial Applications

32
Single Standby
•  Inefficient utilization of resource
–  Journal Nodes
–  ZooKeeper Nodes
–  Standby Node
•  Performance Bottleneck
•  Still tied to the beeper
•  Limited to LAN scope
Breaking Away from Active/Passive
What’s in a NameNode

33
Single Standby
•  Inefficient utilization of resource
–  Journal Nodes
–  ZooKeeper Nodes
–  Standby Node
•  Performance Bottleneck
•  Still tied to the beeper
•  Limited to LAN scope
Active / Active
•  All resources utilized
–  Only NameNode configuration
–  Scale as the cluster grows
–  All NameNodes active
•  Load balancing
•  Set resiliency (# of active NN)
•  Global Consistency
What’s in a NameNode

34
Standby Data Center
•  Idle Resource
–  Single Data Center Ingest
–  Disaster Recovery Only
•  One way synchronization
–  DistCp
•  Error Prone
–  Clusters can diverge over time
•  Difficult to scale > 2 Data Centers
–  Complexity of sharing data
increases
What’s in a Data Center

35
Standby Data Center
•  Idle Resource
–  Single Data Center Ingest
–  Disaster Recovery Only
•  One way synchronization
–  DistCp
•  Error Prone
–  Clusters can diverge over time
•  Difficult to scale > 2 Data Centers
–  Complexity of sharing data
increases
Active / Active
•  DR Resource Available
–  Ingest at all Data Centers
–  Run Jobs in both Data Centers
•  Replication is Multi-Directional
–  active/active
•  Absolute Consistency
–  Single HDFS spans locations
•  ‘N’ Data Center support
–  Global HDFS allows appropriate
data to be shared
What’s in a Data Center

37
•  Data is as current as possible (no
periodic synchs)
•  Doesn’t require monitoring and
consistency checking
•  Virtually zero downtime to recover
from regional data center failure
•  Regulatory compliance
Use Case: Disaster Recovery

38
•  Ingest and analyze anywhere
•  Analyze everywhere
–  Fraud detection
–  Equity trading information
–  New business
–  Etc…
•  Backup data center(s) can be used
for work
–  No idle resources
Use Case: Multi-Data Center
Ingest and multi-tenant workloads

39
•  Mixed Hardware Profiles
–  Memory, disk, CPU
–  Isolate memory-hungry
processing (Storm/Spark) from
regular jobs
•  Share data, not processing
–  Isolate lower priority (dev/
test) work
Use Case: Heterogeneous Hardware
In-memory analytics

40
The difficulty realizing the data lake…

41
…is that data spans the entire world

42
Data

Ocean

Feeder

Site

Accoun$ng

Mart

Banking

Mart

•  Data Marts
–  Restrict access to relevant
data
–  Create quick clusters
•  Feeder Sites (Data
Tributaries)
–  Ingest only
Data Reservoir
Use Cases

43
•  Basel III
–  Consistency of data
•  Data Privacy Directive
–  Data sovereignty
•  Data doesn’t leave country of
origin
Compliance

Regula$on

Guidelines

Regulatory Compliance

Technical Comparison
Hadoop Powered by WANdisco

45
Periodic Synchronization
DistCp
Parallel Data Ingest
Load Balancer, Streaming
Multi-Data Center Hadoop Today
What's wrong with the status quo

46
Periodic Synchronization
DistCp
Hacks currently in use
•  Runs as MapReduce
•  DR data center is read-only
•  Over time, Hadoop clusters
become inconsistent
•  Manual and labor-intensive
process to reconcile differences
•  Inefficient use of the network

47
Parallel Data Ingest
Load Balancer, Flume
Hacks currently in use
•  Hiccups in either of the Hadoop
clusters causes the two file
systems to diverge
•  Potential to run out of buffer when
WAN is down
•  Requires constant attention and
sys-admin hours to keep running
•  Data created on the cluster is not
replicated
•  Use of streaming technologies
(like flume) for data redirection are
only for streaming

48
Architecture of a Non-Stop Hadoop

49
Q&A
Question and Answer
Submit your questions using the “ASK A QUESTION” button

50
Thank you

Supporting Financial Services with a More Flexible Approach to Big Data

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Supporting Financial Services with a More Flexible Approach to Big Data (20)

More from Hortonworks (20)

Recently uploaded (20)

Supporting Financial Services with a More Flexible Approach to Big Data