Performing Data Science with HBase

Performing Data Science with HBase

Aaron Kimball – CTO
Kiyan Ahmadizadeh – MTS

MapReduce and log files

Log files
Batch analysis
Result data set

The way we build apps is changing

HBase plays a big role
• High performance random access
• Flexible schema & sparse storage
• Natural mechanism for time series data
… organized by user

Data science lays the groundwork
• Feature selection requires insight
• Data lies in HBase, not log files
• MapReduce is too cumbersome for
exploratory analytics

Data science lays the groundwork
• Feature selection requires insight
• Data lies in HBase, not log files
• MapReduce is too cumbersome for
exploratory analytics

• This talk: How do we explore data in HBase?

Why not Hive?

• Need to manually sync column schemas
• No complex type support for HBase + Hive
– Our use of Avro facilitates complex record types
• No support for time series events in columns

Why not Pig?

• Tuples handle sparse data poorly
• No support for time series events in columns
• UDFs and main pipeline are written in
different languages (Java, Pig Latin)
– (True of Hive too)

Our analysis needs
• Read from HBase
• Express complex concepts
• Support deep MapReduce pipelines
• Be written in concise code
• Be accessible to data scientists more
comfortable with R & python than Java

Our analysis needs

• Concise: We use Scala
• Powerful: We use Apache Crunch
• Interactive: We built a shell

wibi>

WDL: The WibiData Language

wibi> 2 + 2 wibi> :tables
res0: Int = 4 Table Description
======== ===========
page Wiki page info
user per-user stats

Outline
• Analyzing Wikipedia
• Introducing Scala
• An Overview of Crunch
• Extending Crunch to HBase + WibiData
• Demo!

Analyzing Wikipedia

• All revisions of all English pages
• Simulates real system that could be built on
top of WibiData
• Allows us to practice real analysis at scale

Per-user information
• Rows keyed by Wikipedia user id or IP address
• Statistics for several metrics on all edits made
by each user

Introducing
• Scala language allows declarative statements
• Easier to express transformations over your
data in an intuitive way
• Integrates with Java and runs on the JVM
• Supports interactive evaluation

Example: Iterating over lists
def printChar(ch: Char): Unit = {
println(ch)
}

val lst = List('a', 'b', 'c')
lst.foreach(printChar)

… with an anonymous function
val lst = List('a', 'b', 'c')
lst.foreach( ch => println(ch) )

• Anonymous function can be specified as
argument to foreach() method of a list.
• Lists, sets, etc. are immutable by default

Example: Transforming a list
val lst = List(1, 4, 7)
val doubled = lst.map(x => x * 2)

• map() applies a function to each element,
yielding a new list. (doubled is the list [2,
8, 14])

Example: Filtering
• Apply a boolean function to each element of a
list, keep the ones that return true:

val lst = List(1, 3, 5, 6, 9)
val threes =
lst.filter(x => x % 3 == 0)
// ‘threes’ is the list [3, 6, 9]

Example: Aggregation
val lst = List(1, 2, 12, 5)
lst.reduceLeft( (sum, x) => sum + x )
// Evaluates to 20.

• reduceLeft() aggregates elements left-to-right,
in this case by keeping a running sum.

Crunch: MapReduce pipelines
def runWordCount(input, output) = {
val wordCounts = read(From.textFile(input))
.flatMap(line =>
line.toLowerCase.split("""s+"""))
.filter(word => !word.isEmpty())
.count
wordCounts.write(To.textFile(output))
}

PCollections: Crunch data sets
• Represent a parallel record-oriented data set
• Items in PCollections can be lines of text,
tuples, or complex data structures
• Crunch functions (like flatMap() and
filter()) do work on partitions of
PCollections in parallel.

PCollections… of WibiRows
• WibiRow: Represents a row in a Wibi table
• Enables access to sparse columns
• … as a value: row(someColumn): Value
• … As a timeline of values to iterate/aggregate:
row.timeline(someColumn): Timeline[Value]

Introducing: Kiyan Ahmadizadeh

Demo: Visualizing Distributions
• Suppose you have a metric taken on some population of
users.
• Want to visualize what the distribution of the metric among
the population looks like.
– Could inform next analysis steps, feature selection for models,
etc.
• Histograms can give insight on the shape of a distribution.
– Choose a set of bins for the metric.
– Count the number of population members whose metric falls
into each bin.

Demo: Wikimedia Dataset
• We have a user table containing the average
delta for all edits made by a user to pages.
• Edit Delta: The number of characters added or
deleted by an edit to a page.
• Want to visualize the distribution of average
deltas among users.

Code: Accessing Data
val stats =
ColumnFamily[Stats]("edit_metric_stats")

val userTable = read(
From.wibi(WibiTableAddress.of("user"), stats))

val stats =


Will act as a handle
for accessing the
column family.

val stats =


Type annotation tells
WDL what kind of
data to read out of
the family.

val stats =


userTable is a PCollection[WibiRow] obtained by reading the
column family “edit_metric_stats” from the Wibi table
“user.”

Code: UDFs
def getBin(bins: Range, value: Double): Int = {
bins.reduceLeft ( (choice, bin) =>
if (value < bin) choice else bin )
}

def inRange(bins: Range, value: Double): Boolean =
range.start <= value && value <= range.end

Code: UDFs
def getBin(bins: Range, value: Double): Int = {
bins.reduceLeft ( (choice, bin) =>
if (value < bin) choice else bin )
}

def inRange(bins: Range, value: Double): Boolean =
range.start <= value && value <= range.end

Everyday Scala function declarations!

Code: Filtering
val filtered = userTable.filter { row =>
// Keep editors who have edit_metric_stats:delta defined
!row(stats).isEmpty && row(stats).get.contains("delta")
}

Code: Filtering
}

Boolean predicate on elements in the PCollection

Code: Filtering
}

filtered is a PCollection of rows that have the column edit_metric_stats:delta

Code: Filtering
}

Use stats variable we declared earlier to access the column family.
val stats = ColumnFamily[Stats]("edit_metric_stats")

Code: Binning
val binCounts = filtered.map { row =>
// Bucket mean deltas for histogram
getBin(bins, abs(row(stats).get("delta").getMean))
}.count()

binCounts.write(To.textFile("output_dir"))

Code: Binning
}.count()


Map each editor to the bin their mean delta falls into.

Code: Binning
}.count()


Count how many times each bin occurs in the resulting collection.

Code: Binning
}.count()


binCounts contains the number of editors that fall in each bin.

Code: Binning
}.count()


Writes the result to HDFS.

Code: Visualization
Histogram.plot(binCounts,
bins,
"Histogram of Editors by Mean Delta",
"Mean Delta",
"Number of Editors",
"delta_mean_hist.html")

Analysis Results: Full Data Set

Conclusions

+ +

= Scalable analysis of sparse data

Aaron Kimball – aaron@wibidata.com
Kiyan Ahmadizadeh – kiyan@wibidata.com

Performing Data Science with HBase

More Related Content

What's hot (20)

Similar to Performing Data Science with HBase (20)

More from WibiData (6)

Recently uploaded (20)

Performing Data Science with HBase