Cloudera Operational DB (Apache HBase & Apache Phoenix)

Cloudera Operational DB
(powered by Apache HBase and
Apache Phoenix)
Beyond the Tyranny of the Schema
December 2019
Timothy Spann

© 2019 Cloudera, Inc. All rights reserved. 2
Welcome to Future of Data - Princeton
@PaasDev
https://www.meetup.com/futureofdata-princeton/
From Big Data to AI to Streaming to Containers to
Cloud to Analytics to Cloud Storage to Fast Data to
Machine Learning to Microservices to ...

Who Am I? Timothy Spann
Data in Motion Field Engineer
@PaasDev
DZone Zone Leader and Big Data MVB;
Princeton NJ Future of Data Meetup;
ex-Pivotal Field Engineer;
Author of Apache Kafka RefCard
https://github.com/tspannhw
https://www.datainmotion.dev/

This Meetup Made Possible Thanks To:
Paul Vidal from https://www.meetup.com/futureofdata-philadelphia/ for CDP HBase
Environment and Cloud Magic
Josh Elser and Josiah Goodson for OpDB Slides and HBase Guidance
Milind Pandit from https://www.meetup.com/TechnologySolutionsHub
Mehul Shah from https://www.meetup.com/TechnologySolutionsHub
Vijay Garg from https://pga.fund/
Madhavi from https://www.nuwaysolutions.com/
https://www.slideshare.net/bunkertor/tracking-crime-as-it-occurs-with-apache-phoenix-apache-hbase-and-apache-nifi

WHAT HAVE Apache HBase 2.0 & Apache Phoenix ENABLED?
• Operationalizing ML / AI to revolutionize
healthcare, public utilities, etc
• Serving real-time content at webscale
• Empowering big data analytics for operational
and oﬄine uses
• Acting as a resilient store of record
CLOUDERA OPERATIONAL DB (powered by Apache HBase & Apache Phoenix)
Operational DB is DBMS used to manage dynamic and changing data in real time and enable
applications that drive the business

6© Cloudera, Inc. All rights reserved.
APACHE HBASE FAST FACTS
Largest Database
14 Petabytes
Best Known App
Siri
Fastest
Ingestion
20M Events/s
Users
750+

HBASE ARCHITECTURE
HMaster
Orchestration layer
ZooKeeper
Region
Server
DataNode
Data plane
ColFam ColFam
Col Col Col Col
R Val Val Val Val
R Val Val Val Val
ColFam ColFam
Col Col Col Col
R Val Val Val Val
R Val Val Val Val
Region
ColFam ColFam
Col Col Col Col
R Val Val Val Val
R Val Val Val Val
• Regions are table
segments
• Read and write path
are in the data plane
• DDL operations
• Region assignment
• Recovery orchestration
• Heartbeat
• Server
state
• Services client reads &
writes
• Maximizes in-memory
operations for low-latency
operations
• Provides data resiliency

SCHEMA-LESS DATA MODEL
• Column families deﬁned at time of table creation
• Columns created as required (at time of data insertion)
• No limits to number of columns
• Tables can grow in two dimensions – columns and rows
• Compression & encoding applied at column family level
• No declaration of data types (i.e., a column can contain multiple data types)
Column Family Column Family
Column Column Column Column
RowKey Cell Cell Cell Cell
RowKey Cell Cell Cell Cell

HIGHLY AVAILABLE OUT OF THE BOX (<1 MINUTE RECOVERY)
Region Server 1
HDFS (3 copies of data)
Region Server 2 Region Server 3
What happens when a region crashes
1. Region server crashes
2. Writes and reads time-out for regions
in impacted region server
3. Regions are redistributed to other
region servers
4. WAL is replayed in other region
servers
5. Reads & writes are able to continue to
impacted regions
Typical recovery period < 1 minute (for impacted regions only)
No manual intervention

SECURITY MODEL
Authentication • Kerberos
Role Based
Access control
• Permissions & Scope enable flexible role based access control
• Scope: Global, Namespace, Table, Column Family, Cell
DB security &
encryption
• Transparent encryption of data on the wire and data on disk
(HFile for data at rest, secure WAL for data in motion within
HBase)
• Logging & auditability: configurable & fixed event

11
stmt.executeUpdate(“UPSERT INTO TABLE_NAME
VALUES(rowKey, GREETINGS) ");
stmt.execute();
Phoenix
What Phoenix adds to HBase
Pros:
Cons:
• Maximally flexible & customizable
• SQL only for data remediation
• Unfamiliar to SQL developers
• Requires non-traditional data architecture
• Programmatic ANSI SQL support
• RDBMS-like data architecture
• Auto-applies performance best practices
• Can co-exist with HBase apps
• Reduced flexibility vis-à-vis vanilla HBase
• Phoenix specific data format means you
can’t use HBase APIs directly
Put put = new Put(Bytes.toBytes(rowKey));
put.addColumn(COLUMN_FAMILY_NAME, COLUMN_NAME,
Bytes.toBytes(GREETINGS));
table.put(put);
HBase
RDBMS-like, scale-out databaseFlexible, scale-out, no-sql database

12
Key Phoenix capabilities
• ANSI SQL including joins
• Flexible Schemas / Dynamic Columns
• Secondary Indexes
• Aggregation pushdowns
• Cross-language client support
• Query logging
• Security through Ranger (supports RBAC, ABAC,
etc)
• JDBC/ODBC connectivity for operational reporting
• Plugs in to any JDBC/ODBC-compatible BI tool
to enable self-service analytics and insight
Phoenix
Applications

13
ANSI SQL 92 Support
Supported today Roadmap
Standard SQL Data Types UNION
SELECT, UPSERT, DELETE Windowing Functions
JOINs: Inner and Outer Transactions
Subqueries Cross Joins
Secondary Indexes Authorization
GROUP BY, ORDER BY, HAVING Replication Management
AVG, COUNT, MIN, MAX, SUM Column Constraints and Defaults
Primary Keys, Constraints UDFs
CASE, COALESCE
VIEWs
Flexible Schema
UNION ALL

CLOUD OPTIMIZED : HBASE backed by both HDFS and S3
Cloudera provides HBase backed by Amazon’s S3
● Cloudera Data Platform (CDP) provides an out-of-the-box solution that allows Apache HBase
deployments to use Amazon Simple Storage Service (S3) as its main persistence layer for saving
table data
● Amazon’s Simple Storage Service (S3) is an eventually consistent object store, and HBase requires a
consistent and atomic filesystem which means that it cannot directly use S3. Let's look at the topology.

CLOUD OPTIMIZED : Cloudera HBASE backed by both HDFS and S3
Cloudera with CDP has built a solution where when you launch an Operational Database (HBase)
cluster on CDP, HBase StoreFiles (the backing files for HBase tables) are stored in S3 and HBase
write-ahead-logs (WAL) are stored in an HDFS instance run alongside HBase per usual.

CLOUD OPTIMIZED : HBASE backed by both HDFS and S3
● Configuring HBase to use S3 for its StoreFiles has many benefits to our users.
● One such benefit is that users can decouple their storage and compute.
● If there are times in which no access to HBase is necessary, HBase can be cleanly
shut down and all compute resources reclaimed to eliminate any cost of compute.
● When HBase access is needed again, the HBase cluster can be recreated, pointing
to the same data in S3. Upon startup, HBase can re-initialize itself solely from the
data in S3.

WHERE IS APACHE HBASE TODAY
• Large ecosystem (Niﬁ, Spark, Hive, Impala,
SOLR, Ranger, Atlas, etc)
• Supports NoSQL, SQL, Geospatial, Graph,
TimeSeries, Key Value and other modes1
1. In conjunction with other open source projects built on top of HBase

Cloudera and Apache HBase
● The upstream community is pretty huge and very active with contributions coming
from multiple developers from Cloudera, Microsoft, Amazon, Alibaba, Apple Salesforce
and Xiaomi etc.
● Cloudera is a very active contributor to upstream HBase along with Apache Phoenix.
○ Currently > 8 PMCs and > 2 committers.
● CDP is based off latest HBase v2 and Phoenix v5.

New features in HBase 2+
● Operational simplicity
○ Assignment Manager V2 (using Procedure Framework 2)
○ Offline compaction tool (outside regionservers to save I/O thrasing)
○ Replication: namespace & serial and for bulk-loads
● Performance
○ Off-heap cache improvements (Uses DirectByteByffers to manage buckets outside
of the JVM heap to eliminate impact of gc to get better read perf)
● Space Quotas (to support multi tenancy)
● S3 support
● Spark 2 integration
● Async Client

• Provides familiar & easy interface for
developers
• Advanced multi-tenancy capabilities
• Support near 100% availability for mission
critical applications & many traditional
transactional apps
• Scale to billions of rows and millions of
columns
• Easily combine data sources that use a wide
variety of different structures and schemas
Storage for business apps that require big-data
Ingest Store Primary Use
Query &
Remediate
NO(T ONLY)SQL PHOENIX

None of this command line mess:

HUE FOR SQL & DATA BROWSING FOR REMEDIATION
Supports SQL based insert, update,
delete query for data in HBase
Supports search, insert, update,
delete, DDL for HBase
SQL interface using Impala or Hive for query processing GUI based data browser

Visualization

HBase CRUD

ENABLING DATA-DRIVEN APPS
Fast
• Real time model serving w/ <5ms latency
• Limitless concurrency (>100M updates/sec)
Easy
• Stream and bulk ingest & processing
• Process automation
• Consolidate multiple databases
• Schema ﬂexibility
• SQL & NoSQL interface
Scalable
• Multi-petabyte scale
• Unlimited Tenants
Highly-available
• Automatic recovery from server failure
• Advanced replication & synchronization
topographies
• Multiple backup methodologies
Multi-tenant capable
• Resource isolation
• Throttling & Quotas
Secure
• Role based access control
• Fine-grained authorizations (e.g., tenant, table,
column family, cell)

NiFi Integration
with HBase
• PutHBaseRecord
• PutHBaseJSON
• PutHBaseCell
• FetchHBaseRow
• GetHBase
• ScanHBase
• DeleteHBaseRow
• DeleteHBaseCells
https://www.datainmotion.dev/2019/11/exploring-apache-niﬁ-110-parameters.html
ELT/ETL Lookup Services
• HBase_1_1_2_ListLookupService
• HBase_2_RecordLookupService
• HBase_2_ClientService
• HBase_2_ClientMapCacheService
https://community.cloudera.com/t5/Community-Articles/Reading-OpenData-JSON-and-Stori
ng-into-Phoenix-Tables/ta-p/247323

https://github.com/tspannhw/HBase2

SPRING BOOT APPLICATION TO PHOENIX
https://community.cloudera.com/t5/Community-Articles/Creating-a-Spring-Boot-Java-8-Microservice-To-Read-Apache/ta-p/247379
https://github.com/tspannhw/phillycrime-springboot-phoenix

TH N Y U

Cloudera Operational DB (Apache HBase & Apache Phoenix)

More Related Content

What's hot (20)

Similar to Cloudera Operational DB (Apache HBase & Apache Phoenix) (20)

More from Timothy Spann (20)

Recently uploaded (20)

Cloudera Operational DB (Apache HBase & Apache Phoenix)