SlideShare a Scribd company logo
1 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Apache	Spark	– Apache	HBase Connector
Feature	Rich	and	Efficient	Access	to	HBase
through	Spark	SQL
Weiqing Yang	
Mingjie Tang	
October,	2017
2 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
About	Authors
à Weiqing Yang
• Contribute	to	Apache	Spark,	Apache	Hadoop,	Apache	HBase,	Apache	Ambari
• Software	Engineer	at	Hortonworks
à Mingjie Tang
• SparkSQL,	Spark	Mllib,	Spark	Streaming,	Data	Mining,	Machine	Learning
• Software	Engineer	at	Hortonworks
à …	All	Other	SHC	Contributors
3 ©	Hortonworks	 Inc.	2011	– 2016.	All	Rights	Reserved
Agenda
Motivation
Overview
Architecture	&	Implementation
Usage	&	Demo
4 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Motivation
à Limited	Spark	Support	in	HBase Upstream
– RDD	level
– But	Spark	Is	Moving	to	DataFrame/Dataset
à Existing	Connectors	in	DataFrame Level
– Complicated	Design
• Embedding	Optimization	Plan	inside	Catalyst	Engine
• Stability	Impact	with	Coprocessor
• Serialized	RDD	Lineage	to	HBase
– Heavy	Maintenance	Overhead
5 ©	Hortonworks	 Inc.	2011	– 2016.	All	Rights	Reserved
Overview
6 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Apache	Spark– Apache	HBase Connector	(SHC)
à Combine	Spark	and	HBase
– Spark	Catalyst	Engine	for	Query	Plan	and	Optimization
– HBase as	Fast	Access	KV	Store
– Implement	Standard	External	Data	Source	with	Build-in	Filter,	Maintain	Easily
à Full	Fledged	DataFrameSupport
– Spark	SQL
– Integrated	Language	Query
à High	Performance
– Partition	Pruning,	Data	Locality,	Column	Pruning,	Predicate	Pushdown
– Use	Spark	UnhandledFilters API
– Cache	Spark	HBase Connections
7 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Data	Coder	&	Data	Schema
à Support	Different	Data	Coders
– PrimitiveType:	Native	Support	Java	Primitive	Types
– Avro:	Native	Support	Avro	Encoding/Decoding
– Phoenix:	Phoenix	Encoding/Decoding
– Plug-In	Data	Coder
– Can	Run	on	the	Top	of	Existing	HBase Tables
à Support	Composite	Key
– def cat	=	s"""{
|"table":{"namespace":"default",	"name":"shcExampleTable",	"tableCoder":”Phoenix"},
|"rowkey":"key1:key2",
|"columns":{
|"col00":{"cf":"rowkey",	"col":"key1",	"type":"string”},
|"col01":{"cf":"rowkey",	"col":"key2",	"type":"int"},
…
...
8 ©	Hortonworks	 Inc.	2011	– 2016.	All	Rights	Reserved
Architecture	&	Implementation
9 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Architecture
…...
Driver
Executor Executor Executor
Region	
Server
Region	
Server
Region	
Server…...
Spark
HBase
Picture	1.	SHC	architecture
Host	1
10 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Architecture
…...
Driver
Executor Executor Executor
Region	
Server
Region	
Server
Region	
Server…...
Picture	1.	SHC	architecture
Task
Query
Partition
Filters,	Required	
Columns
RS	start/end	
point
sqlContext.sql("select	
count(col1)	from	table1	
where	key	<	'row050'")
PP P
Scans
BulkGets
11 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Implementation
…...
Driver
Executor Executor Executor
Region	
Server
Region	
Server
Region	
Server…...
Picture	1.	SHC	architecture
Task
Query
Partition
Filters,	Required	
ColumnsPartition	Pruning:	Task	Only
Performed	in	Region	Server	
Holding	Requested	Data
PP P
Scans
BulkGets Filters	->	Multiple	Scan	Ranges		
															∩
(Start	point,	end	point)
RS	start/end	
point
12 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Implementation
…...
Driver
Executor Executor Executor
Region	
Server
Region	
Server
Region	
Server…...
Picture	1.	SHC	architecture
Task
Query
Partition
Filters,	Required	
Columns
RS	start/end	
point
Data	Locality:	Move	
Computation	to	Data.	
PP P
Scans
BulkGets
RDD	Partition	has	preferred	location:
getPreferredLocations(partition) {
return RS.hostName}
13 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Implementation
…...
Driver
Executor Executor Executor
Region	
Server
Region	
Server
Region	
Server…...
Picture	1.	SHC	architecture
Task
Query
Partition
Filters,	Required	
Columns
RS	start/end	
point
Column	Pruning:	Required	
Column
Predicate	Pushdown:	HBase
built-in	Filters
PP P
Filters,	Required	
Columns
Filters,	Required	
Columns
Scans
BulkGets
Filters,	Required	
Columns
14 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Implementation
…...
Driver
Executor Executor Executor
Region	
Server
Region	
Server
Region	
Server…...
Picture	1.	SHC	architecture
Task
Query
Partition
Filters,	Required	
Columns
RS	start/end	
point
Scan	and	BulkGets:	Grouped	
by	region	server.	
PP P
Scans
BulkGets
WHERE	column	>	x	and	
column	<	y for	scan
and WHERE	column	=	
x for	get.
15 ©	Hortonworks	 Inc.	2011	– 2016.	All	Rights	Reserved
Usage	&	Demo
16 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
How	to	Use	SHC?
à Github
– https://github.com/hortonworks-spark/shc	 	
à SHC	Examples
– https://github.com/hortonworks-spark/shc/tree/master/examples
à Apache	HBase Jira
– https://issues.apache.org/jira/browse/HBASE-14789
17 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Demo
à Interactive	Jobs	through	Spark Shell
à Batch	Jobs
18 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Acknowledgement
à HBase Community	&	Spark	Community
à All	SHC Contributors,	Zhan	Zhang
19 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Reference
à Hortonworks	Public	Repo
– http://repo.hortonworks.com/content/repositories/releases/com/hortonworks/
à Apache	Spark
– http://spark.apache.org/
à Apache	HBase
– https://hbase.apache.org/
20 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Thanks
Q	&	A
Emails:	
wyang@hortonworks.com
21 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
BACKUP
22 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Kerberos	Cluster
à Kerberos	Ticket
– kinit	-kt foo.keytab foouser or		Principle/Keytab
à Long	Running	Service
– --principal,	--keytab
à Multiple	Secure	HBase Clusters
– Spark	only	Supports	Single	Secure	HBase Cluster
– Use	SHC	Credential	Manager
– Refer	LRJobAccessing2Clusters	Example	in	github
23 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Usage
Define	the	catalog	for	the	schema	mapping:
24 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Usage
à Prepare	the	data	and	populate	the	HBase table
val data	=	(0	to	255).map	{	i =>	 HBaseRecord(i,	“extra”)}
sc.parallelize(data).toDF.write.options(
Map(HBaseTableCatalog.tableCatalog ->	catalog,	HBaseTableCatalog.newTable->	“5”))
.format(“org.apache.spark.sql.execution.datasources.hbase”)
.save()
25 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Usage
à Load	the	DataFrame
def withCatalog(cat:	String):	DataFrame =	{
sqlContext
.read
.options(Map(HBaseTableCatalog.tableCatalog->cat))
.format(“org.apache.spark.sql.execution.datasources.hbase”)
.load()
}
val df =	withCatalog(catalog)
26 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Usage
à Query
Language	integrated	query:
val s	=	df.filter((($"col0ʺ	<=	“çrow050ʺ	&&	$”col0”	>	“row040”)	||
$”col0ʺ	===	“row005”	&& ($”col4ʺ	===	1	|| $”col4ʺ	===	42))
.select(“col0”,	“col1”,	“col4”)
SQL:
val s	=	df.filter((($”col0ʺ	<=	“row050ʺ	&&	$”col0”	>	“row040”)
df.registerTempTable(“table”)
sqlContext.sql(“select	count(col1)	from	table”).show
27 ©	Hortonworks	 Inc.	2011	– 2017.	All	Rights	Reserved
Usage
à Work	with	different	data	sources
//	Part	1:	write	data	into	Hive	table	and	read	data	from	it
val df1	=	sql("SELECT	*	FROM	shcHiveTable")
//	Part	2:	read data from Hbase table
val df2	=	withCatalog(cat)
//	Part	3:	join the two dataframes
val s1	=	df1.filter($"key"	<=	"40").select("key",	"col1")
val s2	=	df2.filter($"key"	<=	"20"	&&	$"key"	>=	"1").select("key",	"col2")
val result =		s1.join(s2,	Seq("key"))
result.show()

More Related Content

What's hot (20)

PPT
OSI 7 Layer Model
Pritom Chaki
 
PDF
Jumpstart Global 2020 introduction
Jimmy Chengmy
 
PPTX
Multi_Core_Processor_2015_(Download it!)
Sudip Roy
 
PDF
Generations+of+computer
sam shankar
 
PPTX
الشبكات اللاسلكية
Dhoha Mazen
 
DOCX
Cn lab manual 150702
Hardiksinh Solanki
 
PPT
Ch01 introducing operating systems
Jacob Cadeliña
 
PPTX
Network design consideration
lavanya marichamy
 
PDF
The Physical Layer
adil raja
 
PPTX
Chipset PowerPointPresentation
Amilma
 
DOCX
مكونات الحاسب
hakmhamdy
 
PPT
Hyper Threading technology
Karunakar Singh Thakur
 
PPTX
Operating System Mac OS X
mirazhosain
 
PPTX
Lec1 :- Data communication and network
Dhrumil Shah
 
PPT
NETWORK COMPONENTS
bwire sedrick
 
PPTX
4th gen microprocessor
AvantikaBadgujar
 
PDF
Data 101: Introduction to Data Visualization
David Newbury
 
PPTX
Intel i7
Justin k.
 
PPTX
Digital Technology Culture - Anti-Dote Series
Enimil Kweku Boateng
 
PPT
BCS302- Digital Design and computer organization -VTU-2022 scheme-Expectation...
ciyamala kushbu
 
OSI 7 Layer Model
Pritom Chaki
 
Jumpstart Global 2020 introduction
Jimmy Chengmy
 
Multi_Core_Processor_2015_(Download it!)
Sudip Roy
 
Generations+of+computer
sam shankar
 
الشبكات اللاسلكية
Dhoha Mazen
 
Cn lab manual 150702
Hardiksinh Solanki
 
Ch01 introducing operating systems
Jacob Cadeliña
 
Network design consideration
lavanya marichamy
 
The Physical Layer
adil raja
 
Chipset PowerPointPresentation
Amilma
 
مكونات الحاسب
hakmhamdy
 
Hyper Threading technology
Karunakar Singh Thakur
 
Operating System Mac OS X
mirazhosain
 
Lec1 :- Data communication and network
Dhrumil Shah
 
NETWORK COMPONENTS
bwire sedrick
 
4th gen microprocessor
AvantikaBadgujar
 
Data 101: Introduction to Data Visualization
David Newbury
 
Intel i7
Justin k.
 
Digital Technology Culture - Anti-Dote Series
Enimil Kweku Boateng
 
BCS302- Digital Design and computer organization -VTU-2022 scheme-Expectation...
ciyamala kushbu
 

Viewers also liked (20)

PDF
Building the Ideal Stack for Real-Time Analytics
SingleStore
 
PDF
빅데이터윈윈 컨퍼런스_데이터시각화자료
ABRC_DATA
 
PDF
Softnix Security Data Lake
Softnix Technology
 
PDF
Softnix Messaging Server
Softnix Technology
 
PPTX
Using Big Data to Transform Your Customer’s Experience - Part 1

Cloudera, Inc.
 
PDF
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice Machine
Data Con LA
 
PDF
Zoomdata
Vivek Mohan
 
PDF
The Fast Path to Building Operational Applications with Spark
SingleStore
 
PPTX
Ibm watson
Vivek Mohan
 
PDF
CWIN17 Frankfurt / Cloudera
Capgemini
 
PPTX
Webinar - Sehr empfehlenswert: wie man aus Daten durch maschinelles Lernen We...
Cloudera, Inc.
 
PPTX
Security implementation on hadoop
Wei-Chiu Chuang
 
PDF
Spark meetup - Zoomdata Streaming
Zoomdata
 
PPTX
Real-Time Analytics Visualized w/ Kafka + Streamliner + MemSQL + ZoomData, An...
confluent
 
PDF
Cloudera and Qlik: Big Data Analytics for Business
Data IQ Argentina
 
PPTX
Put Alternative Data to Use in Capital Markets

Cloudera, Inc.
 
PPTX
Partner Ecosystem Showcase for Apache Ranger and Apache Atlas
DataWorks Summit
 
PPTX
The Evolution of Data Architecture
Wei-Chiu Chuang
 
PDF
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
Spark Summit
 
PPTX
Benefits of Transferring Real-Time Data to Hadoop at Scale
Hortonworks
 
Building the Ideal Stack for Real-Time Analytics
SingleStore
 
빅데이터윈윈 컨퍼런스_데이터시각화자료
ABRC_DATA
 
Softnix Security Data Lake
Softnix Technology
 
Softnix Messaging Server
Softnix Technology
 
Using Big Data to Transform Your Customer’s Experience - Part 1

Cloudera, Inc.
 
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice Machine
Data Con LA
 
Zoomdata
Vivek Mohan
 
The Fast Path to Building Operational Applications with Spark
SingleStore
 
Ibm watson
Vivek Mohan
 
CWIN17 Frankfurt / Cloudera
Capgemini
 
Webinar - Sehr empfehlenswert: wie man aus Daten durch maschinelles Lernen We...
Cloudera, Inc.
 
Security implementation on hadoop
Wei-Chiu Chuang
 
Spark meetup - Zoomdata Streaming
Zoomdata
 
Real-Time Analytics Visualized w/ Kafka + Streamliner + MemSQL + ZoomData, An...
confluent
 
Cloudera and Qlik: Big Data Analytics for Business
Data IQ Argentina
 
Put Alternative Data to Use in Capital Markets

Cloudera, Inc.
 
Partner Ecosystem Showcase for Apache Ranger and Apache Atlas
DataWorks Summit
 
The Evolution of Data Architecture
Wei-Chiu Chuang
 
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
Spark Summit
 
Benefits of Transferring Real-Time Data to Hadoop at Scale
Hortonworks
 
Ad

Similar to Apache Spark—Apache HBase Connector: Feature Rich and Efficient Access to HBase through Spark SQL with Weiqing Yang (20)

PDF
Spark HBase Connector: Feature Rich and Efficient Access to HBase Through Spa...
Databricks
 
PDF
HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...
HBaseCon
 
PPTX
Spark + HBase
DataWorks Summit/Hadoop Summit
 
PPTX
Hspark index conf
Chester Chen
 
PDF
Hbase mhug 2015
Joseph Niemiec
 
PPTX
HiveWarehouseConnector
Eric Wohlstadter
 
PDF
HBaseCon 2013: Integration of Apache Hive and HBase
Cloudera, Inc.
 
PPTX
Spark sql
Zahra Eskandari
 
PPTX
Hive edw-dataworks summit-eu-april-2017
alanfgates
 
PPTX
An Apache Hive Based Data Warehouse
DataWorks Summit
 
PPTX
Spark meetup v2.0.5
Yan Zhou
 
PDF
Hortonworks Technical Workshop: HBase and Apache Phoenix
Hortonworks
 
PPTX
Intro to Spark with Zeppelin
Hortonworks
 
PPTX
Crash Course HS16Melb - Hands on Intro to Spark & Zeppelin
DataWorks Summit/Hadoop Summit
 
PDF
Sparkling Water 5 28-14
Sri Ambati
 
PPTX
Big data processing engines, Atlanta Meetup 4/30
Ashish Narasimham
 
PPTX
Apache Phoenix and HBase: Past, Present and Future of SQL over HBase
DataWorks Summit/Hadoop Summit
 
PPTX
Unit II Hadoop Ecosystem_Updated.pptx
BhavanaHotchandani
 
PPTX
Learning spark ch09 - Spark SQL
phanleson
 
PPTX
Apache Phoenix and HBase: Past, Present and Future of SQL over HBase
DataWorks Summit/Hadoop Summit
 
Spark HBase Connector: Feature Rich and Efficient Access to HBase Through Spa...
Databricks
 
HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...
HBaseCon
 
Hspark index conf
Chester Chen
 
Hbase mhug 2015
Joseph Niemiec
 
HiveWarehouseConnector
Eric Wohlstadter
 
HBaseCon 2013: Integration of Apache Hive and HBase
Cloudera, Inc.
 
Spark sql
Zahra Eskandari
 
Hive edw-dataworks summit-eu-april-2017
alanfgates
 
An Apache Hive Based Data Warehouse
DataWorks Summit
 
Spark meetup v2.0.5
Yan Zhou
 
Hortonworks Technical Workshop: HBase and Apache Phoenix
Hortonworks
 
Intro to Spark with Zeppelin
Hortonworks
 
Crash Course HS16Melb - Hands on Intro to Spark & Zeppelin
DataWorks Summit/Hadoop Summit
 
Sparkling Water 5 28-14
Sri Ambati
 
Big data processing engines, Atlanta Meetup 4/30
Ashish Narasimham
 
Apache Phoenix and HBase: Past, Present and Future of SQL over HBase
DataWorks Summit/Hadoop Summit
 
Unit II Hadoop Ecosystem_Updated.pptx
BhavanaHotchandani
 
Learning spark ch09 - Spark SQL
phanleson
 
Apache Phoenix and HBase: Past, Present and Future of SQL over HBase
DataWorks Summit/Hadoop Summit
 
Ad

More from Spark Summit (20)

PDF
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
Spark Summit
 
PDF
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
Spark Summit
 
PDF
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Spark Summit
 
PDF
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Spark Summit
 
PDF
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
Spark Summit
 
PDF
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
Spark Summit
 
PDF
Apache Spark and Tensorflow as a Service with Jim Dowling
Spark Summit
 
PDF
Apache Spark and Tensorflow as a Service with Jim Dowling
Spark Summit
 
PDF
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
Spark Summit
 
PDF
Next CERN Accelerator Logging Service with Jakub Wozniak
Spark Summit
 
PDF
Powering a Startup with Apache Spark with Kevin Kim
Spark Summit
 
PDF
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Spark Summit
 
PDF
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Spark Summit
 
PDF
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Spark Summit
 
PDF
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spark Summit
 
PDF
Goal Based Data Production with Sim Simeonov
Spark Summit
 
PDF
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Spark Summit
 
PDF
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Spark Summit
 
PDF
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Spark Summit
 
PDF
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
Spark Summit
 
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
Spark Summit
 
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
Spark Summit
 
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Spark Summit
 
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Spark Summit
 
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
Spark Summit
 
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
Spark Summit
 
Apache Spark and Tensorflow as a Service with Jim Dowling
Spark Summit
 
Apache Spark and Tensorflow as a Service with Jim Dowling
Spark Summit
 
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
Spark Summit
 
Next CERN Accelerator Logging Service with Jakub Wozniak
Spark Summit
 
Powering a Startup with Apache Spark with Kevin Kim
Spark Summit
 
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Spark Summit
 
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Spark Summit
 
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Spark Summit
 
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spark Summit
 
Goal Based Data Production with Sim Simeonov
Spark Summit
 
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Spark Summit
 
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Spark Summit
 
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Spark Summit
 
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
Spark Summit
 

Recently uploaded (20)

PDF
A GraphRAG approach for Energy Efficiency Q&A
Marco Brambilla
 
PDF
apidays Singapore 2025 - Building a Federated Future, Alex Szomora (GSMA)
apidays
 
PPTX
SHREYAS25 INTERN-I,II,III PPT (1).pptx pre
swapnilherage
 
PPTX
How to Add Columns and Rows in an R Data Frame
subhashenia
 
PPTX
03_Ariane BERCKMOES_Ethias.pptx_AIBarometer_release_event
FinTech Belgium
 
PPTX
美国史蒂文斯理工学院毕业证书{SIT学费发票SIT录取通知书}哪里购买
Taqyea
 
PDF
The Best NVIDIA GPUs for LLM Inference in 2025.pdf
Tamanna36
 
PDF
Unlocking Insights: Introducing i-Metrics Asia-Pacific Corporation and Strate...
Janette Toral
 
PPTX
Listify-Intelligent-Voice-to-Catalog-Agent.pptx
nareshkottees
 
PDF
Development and validation of the Japanese version of the Organizational Matt...
Yoga Tokuyoshi
 
PDF
Data Science Course Certificate by Sigma Software University
Stepan Kalika
 
PPTX
big data eco system fundamentals of data science
arivukarasi
 
PPTX
Aict presentation on dpplppp sjdhfh.pptx
vabaso5932
 
PDF
UNISE-Operation-Procedure-InDHIS2trainng
ahmedabduselam23
 
PDF
Research Methodology Overview Introduction
ayeshagul29594
 
PPTX
在线购买英国本科毕业证苏格兰皇家音乐学院水印成绩单RSAMD学费发票
Taqyea
 
PDF
1750162332_Snapshot-of-Indias-oil-Gas-data-May-2025.pdf
sandeep718278
 
PDF
Business implication of Artificial Intelligence.pdf
VishalChugh12
 
PDF
Using AI/ML for Space Biology Research
VICTOR MAESTRE RAMIREZ
 
PPTX
BinarySearchTree in datastructures in detail
kichokuttu
 
A GraphRAG approach for Energy Efficiency Q&A
Marco Brambilla
 
apidays Singapore 2025 - Building a Federated Future, Alex Szomora (GSMA)
apidays
 
SHREYAS25 INTERN-I,II,III PPT (1).pptx pre
swapnilherage
 
How to Add Columns and Rows in an R Data Frame
subhashenia
 
03_Ariane BERCKMOES_Ethias.pptx_AIBarometer_release_event
FinTech Belgium
 
美国史蒂文斯理工学院毕业证书{SIT学费发票SIT录取通知书}哪里购买
Taqyea
 
The Best NVIDIA GPUs for LLM Inference in 2025.pdf
Tamanna36
 
Unlocking Insights: Introducing i-Metrics Asia-Pacific Corporation and Strate...
Janette Toral
 
Listify-Intelligent-Voice-to-Catalog-Agent.pptx
nareshkottees
 
Development and validation of the Japanese version of the Organizational Matt...
Yoga Tokuyoshi
 
Data Science Course Certificate by Sigma Software University
Stepan Kalika
 
big data eco system fundamentals of data science
arivukarasi
 
Aict presentation on dpplppp sjdhfh.pptx
vabaso5932
 
UNISE-Operation-Procedure-InDHIS2trainng
ahmedabduselam23
 
Research Methodology Overview Introduction
ayeshagul29594
 
在线购买英国本科毕业证苏格兰皇家音乐学院水印成绩单RSAMD学费发票
Taqyea
 
1750162332_Snapshot-of-Indias-oil-Gas-data-May-2025.pdf
sandeep718278
 
Business implication of Artificial Intelligence.pdf
VishalChugh12
 
Using AI/ML for Space Biology Research
VICTOR MAESTRE RAMIREZ
 
BinarySearchTree in datastructures in detail
kichokuttu
 

Apache Spark—Apache HBase Connector: Feature Rich and Efficient Access to HBase through Spark SQL with Weiqing Yang

  • 1. 1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache Spark – Apache HBase Connector Feature Rich and Efficient Access to HBase through Spark SQL Weiqing Yang Mingjie Tang October, 2017
  • 2. 2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved About Authors à Weiqing Yang • Contribute to Apache Spark, Apache Hadoop, Apache HBase, Apache Ambari • Software Engineer at Hortonworks à Mingjie Tang • SparkSQL, Spark Mllib, Spark Streaming, Data Mining, Machine Learning • Software Engineer at Hortonworks à … All Other SHC Contributors
  • 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Motivation Overview Architecture & Implementation Usage & Demo
  • 4. 4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Motivation à Limited Spark Support in HBase Upstream – RDD level – But Spark Is Moving to DataFrame/Dataset à Existing Connectors in DataFrame Level – Complicated Design • Embedding Optimization Plan inside Catalyst Engine • Stability Impact with Coprocessor • Serialized RDD Lineage to HBase – Heavy Maintenance Overhead
  • 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Overview
  • 6. 6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache Spark– Apache HBase Connector (SHC) à Combine Spark and HBase – Spark Catalyst Engine for Query Plan and Optimization – HBase as Fast Access KV Store – Implement Standard External Data Source with Build-in Filter, Maintain Easily à Full Fledged DataFrameSupport – Spark SQL – Integrated Language Query à High Performance – Partition Pruning, Data Locality, Column Pruning, Predicate Pushdown – Use Spark UnhandledFilters API – Cache Spark HBase Connections
  • 7. 7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Data Coder & Data Schema à Support Different Data Coders – PrimitiveType: Native Support Java Primitive Types – Avro: Native Support Avro Encoding/Decoding – Phoenix: Phoenix Encoding/Decoding – Plug-In Data Coder – Can Run on the Top of Existing HBase Tables à Support Composite Key – def cat = s"""{ |"table":{"namespace":"default", "name":"shcExampleTable", "tableCoder":”Phoenix"}, |"rowkey":"key1:key2", |"columns":{ |"col00":{"cf":"rowkey", "col":"key1", "type":"string”}, |"col01":{"cf":"rowkey", "col":"key2", "type":"int"}, … ...
  • 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Architecture & Implementation
  • 9. 9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Architecture …... Driver Executor Executor Executor Region Server Region Server Region Server…... Spark HBase Picture 1. SHC architecture Host 1
  • 10. 10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Architecture …... Driver Executor Executor Executor Region Server Region Server Region Server…... Picture 1. SHC architecture Task Query Partition Filters, Required Columns RS start/end point sqlContext.sql("select count(col1) from table1 where key < 'row050'") PP P Scans BulkGets
  • 11. 11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Implementation …... Driver Executor Executor Executor Region Server Region Server Region Server…... Picture 1. SHC architecture Task Query Partition Filters, Required ColumnsPartition Pruning: Task Only Performed in Region Server Holding Requested Data PP P Scans BulkGets Filters -> Multiple Scan Ranges ∩ (Start point, end point) RS start/end point
  • 12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Implementation …... Driver Executor Executor Executor Region Server Region Server Region Server…... Picture 1. SHC architecture Task Query Partition Filters, Required Columns RS start/end point Data Locality: Move Computation to Data. PP P Scans BulkGets RDD Partition has preferred location: getPreferredLocations(partition) { return RS.hostName}
  • 13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Implementation …... Driver Executor Executor Executor Region Server Region Server Region Server…... Picture 1. SHC architecture Task Query Partition Filters, Required Columns RS start/end point Column Pruning: Required Column Predicate Pushdown: HBase built-in Filters PP P Filters, Required Columns Filters, Required Columns Scans BulkGets Filters, Required Columns
  • 14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Implementation …... Driver Executor Executor Executor Region Server Region Server Region Server…... Picture 1. SHC architecture Task Query Partition Filters, Required Columns RS start/end point Scan and BulkGets: Grouped by region server. PP P Scans BulkGets WHERE column > x and column < y for scan and WHERE column = x for get.
  • 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Usage & Demo
  • 16. 16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved How to Use SHC? à Github – https://github.com/hortonworks-spark/shc à SHC Examples – https://github.com/hortonworks-spark/shc/tree/master/examples à Apache HBase Jira – https://issues.apache.org/jira/browse/HBASE-14789
  • 17. 17 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Demo à Interactive Jobs through Spark Shell à Batch Jobs
  • 18. 18 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Acknowledgement à HBase Community & Spark Community à All SHC Contributors, Zhan Zhang
  • 19. 19 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Reference à Hortonworks Public Repo – http://repo.hortonworks.com/content/repositories/releases/com/hortonworks/ à Apache Spark – http://spark.apache.org/ à Apache HBase – https://hbase.apache.org/
  • 20. 20 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Thanks Q & A Emails: [email protected]
  • 21. 21 © Hortonworks Inc. 2011 – 2017. All Rights Reserved BACKUP
  • 22. 22 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Kerberos Cluster à Kerberos Ticket – kinit -kt foo.keytab foouser or Principle/Keytab à Long Running Service – --principal, --keytab à Multiple Secure HBase Clusters – Spark only Supports Single Secure HBase Cluster – Use SHC Credential Manager – Refer LRJobAccessing2Clusters Example in github
  • 23. 23 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Usage Define the catalog for the schema mapping:
  • 24. 24 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Usage à Prepare the data and populate the HBase table val data = (0 to 255).map { i => HBaseRecord(i, “extra”)} sc.parallelize(data).toDF.write.options( Map(HBaseTableCatalog.tableCatalog -> catalog, HBaseTableCatalog.newTable-> “5”)) .format(“org.apache.spark.sql.execution.datasources.hbase”) .save()
  • 25. 25 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Usage à Load the DataFrame def withCatalog(cat: String): DataFrame = { sqlContext .read .options(Map(HBaseTableCatalog.tableCatalog->cat)) .format(“org.apache.spark.sql.execution.datasources.hbase”) .load() } val df = withCatalog(catalog)
  • 26. 26 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Usage à Query Language integrated query: val s = df.filter((($"col0ʺ <= “çrow050ʺ && $”col0” > “row040”) || $”col0ʺ === “row005” && ($”col4ʺ === 1 || $”col4ʺ === 42)) .select(“col0”, “col1”, “col4”) SQL: val s = df.filter((($”col0ʺ <= “row050ʺ && $”col0” > “row040”) df.registerTempTable(“table”) sqlContext.sql(“select count(col1) from table”).show
  • 27. 27 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Usage à Work with different data sources // Part 1: write data into Hive table and read data from it val df1 = sql("SELECT * FROM shcHiveTable") // Part 2: read data from Hbase table val df2 = withCatalog(cat) // Part 3: join the two dataframes val s1 = df1.filter($"key" <= "40").select("key", "col1") val s2 = df2.filter($"key" <= "20" && $"key" >= "1").select("key", "col2") val result = s1.join(s2, Seq("key")) result.show()