How Google Does Big Data - DevNexus 2014

How Google Does Big Data
Solving Big Data Problems at Google Scale
James Chittenden
Cloud Platform Engineer

Google confidential │ Do not distribute

Agenda
1

Quick Google Cloud Platform Overview

2

Google & Big Data

3

Running Hadoop on GCE & Leveraging BigQuery

4

Q&A


Enterprise

Google confidential | Do not distribute

Google has been running some
of the world’s largest distributed
systems with unique and
stringent requirements.

Images by Connie Zhou


"This is what makes Google Google:
its physical network, its thousands of
fiber miles, and those many thousands
of servers that, in aggregate, add up to
the mother of all clouds."
- Wired 'Google Throws Open Doors To Its Top Secret Data Center',

Wired, October

2012


For the past 15 years, Google
has been building out the
world’s fastest, most powerful,
highest quality cloud
infrastructure on the planet.



Cloud Platform is built on the same
infrastructure that powers Google



Google's Global OpenFlow Network

Google Innovations in Software
MapReduce

2002

2004

GFS

Cloud Platform

Dremel

2006

Big Table

2008

Colossus

2010

2012

Spanner


Investing In Our Cloud
$2.9B in additional data center investments worldwide

Cloud Platform


Compute

Storage

App Services

Compute Engine

Cloud Storage

BigQuery

App Engine

Cloud SQL

Cloud Endpoints

Cloud Datastore

Cloud Platform


Complex Big Data Landscape
Big Data Landscape

The Flood of BIG DATA

5.3

US $14.3

191

BILLION EMAILS
SENT EVERY DAY 2

BILLION SPENT IN
U.S. ON MOBILE ADS IN 2013 4

US $109.7

TRILLION ONLINE AD
IMPRESSIONS IN 2012 1

500+

TERABYTES OF DATA
UPLOADED DAILY 5

BILLION ESTIMATED
FOR 2014 3

MOBILE
AD

EMAIL
ONLINE AD
IMPRESSIONS

FACEBOOK
ONLINE
AD SPEND

How does Google process
Big Data?


Big Data processing example - application logs

User
requests

Application
servers

Note: diagram is dramatically simplified
Interop NYC - Oct 2, 2012


App persistence

BigTable / GFS

App logging

User
requests

Application
servers

Logs cluster


MapReduce

App persistence

BigTable / GFS
Dremel
MapReduce
App logging

User
requests

Application
servers

Logs cluster


MapReduce

App persistence

SQL

BigTable / GFS
Dremel
MapReduce

Analysts
& reporting

App logging

User
requests

Application
servers

Logs cluster


How does this apply
to my business?


Social Data

Internal Data

WHAT DO CUSTOMERS
SAY ABOUT MY
BUSINESS?

HOW CAN I COMBINE ALL
OF MY TRANSACTIONAL
DATA WITH OTHER
DATA?

Google Data
Sensor Data
HOW CAN I CAPTURE
MORE CUSTOMER
INTERACTIONS?

01001010110010001001000100
100110001110011v10011000100110001
01001010110010001001000100010001
01100011100010 01100011100001110
10011001000011001100101 01

HOW CAN GOOGLE
DATA HELP ME WITH
MULTI-CHANNEL
MODELING?

Sample use cases - BigQuery usage today
Display ads analytics
(global top-5 media agency)

Ads network reporting
(3rd party mobile ads)

Fleet reservations
(online travel operations)

Mobile app statistics
(online reading vendor)

Revenue optimization
(holiday/travel properties)

Analyze global campaign performance for F500 clients
1 client = 20GB/day of DoubleClick impressions logs

Deliver x-platform performance analytics dashboards
1B events/day x 100s of ads customers

Monitor customer demand vs supply shortfalls
10,000 routes x 1000s customers = millions of daily events

Usage analysis on 60M installs; 10M active users
2B API requests/day, 20GB log data/day

Correlate marketing effectiveness vs global reservations
10MBs / day from multiple data warehouses


Today we see the same trends happening in industry
Opportunities
Single place to capture data
Combine data from different sources

Challenges
The data is large
Up to 100-200 TB
High rate of growth
Up to 100-200 GB / day

Detect patterns and correlations
Easily share data insights with org
Distributed decision making

Non-relational
Unstructured or semi-structured
Multiple sources
SaaS, Mobile devices, legacy


Processing Data: MapReduce
●

2004: Google releases the MapReduce* framework paper

●

2011: Big Data "analysis" using Hadoop and other Google inspired technology

Visualization

1
Google App Engine
1

Data Collection

2

ETL

2

Raw Data Storage

4

6

Auto-scale
web services

Data
Ingestion

4
Google Compute Engine

3

Web Apps

Scalable VMs

BI tools

Aggregation

Collection
Transformation

Data processing
Cleansing

3
Google Cloud Storage
5

Analytics Storage

6

TBs of Data

Visualization

Raw Data Storage
Big Query Staging

Cloud Storage
Hadoop Connector

5
Google Big Query
Raw Data Storage

Interactive
Dashboards + apps

Ad-hoc queries
REST API

Google
Spreadsheets

1

Visualization

1

2

Scalable VMs

6

Data Collection
Collection
Transformation

ETL

Data processing
Cleansing

BI tools
3

Raw Data Storage
2

4

4

Aggregation

5

Analytics Storage

6

Visualization

Interactive
Dashboards + apps

Google
Spreadsheets

1

Visualization
Google Cloud Storage

1

2

TBs of Data

6

Data Collection
Raw Data Storage
Big Query Staging

ETL

BI tools
3

Huge Storage Capacity

Raw Data Storage
3

4

Aggregation

5

Analytics Storage

6

Visualization

Consistent, fast
speeds
Multiple storage
types

Interactive
Dashboards + apps

Google
Spreadsheets

1

Visualization
Google BigQuery

1

2

Ad-hoc queries

6

Data Collection
AdHoc Queries
REST API

ETL

Visualization
Partners

BI tools
3

Raw Data Storage
5

4

Aggregation

5

Analytics Storage

6

Visualization

Interactive
Dashboards + apps

Google
Spreadsheets


+


Demo
Hadoop on Google Compute Engine


BigData at Google
Running Hadoop on Google Compute Engine

1. Set up Hadoop
./compute_cluster_for_hadoop.py setup google-platform-demo
hadoop-demo-jameschi


BigData at Google

2. Start Cluster
./compute_cluster_for_hadoop.py start google-platform-demo
hadoop-demo-jameschi 500


BigData at Google

3. Start MapReduce
./compute_cluster_for_hadoop.py mapreduce google-platformdemo hadoop-demo-jameschi
--input gs://hadoop-demo-jameschi-input
--output gs://hadoop-demo-jameschi-output
--mapper sample/shortest-to-longest-mapper.pl
--reducer sample/shortest-to-longest-reducer.pl
--mapper-count 5
--reducer-count 1


BigData at Google

4. Shutdown Cluster
./compute_cluster_for_hadoop.py shutdown google-platformdemo


Google Compute Engine Costs
Total Costs for Running Hadoop:

5 node Hadoop cluster x 20 minutes = $0.21
5,000 node Hadoop cluster x 20 min = $8.67


How Google Approaches Analytics
●

MapReduce based analysis can be slow for ad-hoc queries

●

Managing data centers and tuning software takes time & money

●

Analytics tools should be services

How Google Approaches Analytics
Dremel
Ad-hoc query system for terabyte datasets
●
●
●

Query execution tree
Column Oriented records
...and a lot of nodes

What is BigQuery?
Google BigQuery: An Overview

BigQuery: a fully-managed data analytics service in the cloud.
Unlimited storage. Interactive analysis on multi-terabyte datasets.
Adwords
DCLK
Analytics

Other BI
Tools

Google
Spreadsheets

Scalable Storage
SQL

API

Corporate data
3rd party data
Store all your data
in the cloud

App Engine
App

Co-Workers

Analyze interactively
Mash it up

Securely Share/
distribute the results

Cloud-based data analytics pipeline
Application
level code

Log data

BI tools

Logstore

Backends +
MapReduce
BigQuery
Structured data

Datastore
SQL

API

Interactive
Dashboards + apps

Hadoop

Unstructured data

Google Spreadsheets
Cloud Storage

Store

Extract &
Transform

Analyze
interactively

Serve

Custom logic & 3rd party
libraries


Origins of BigQuery

+


Origins of BigQuery

How can a business
analyst find Top 20
Apps in matter of
seconds?

Origins of BigQuery

How can a business analyst find Top 20 Apps in
matter of seconds?

SELECT
top(appId, 20) AS app,
count(*) AS count
FROM installlog.2012;
ORDER BY
count DESC

Result in ~20 seconds!

Origins of BigQuery

How do you find slow
running servers from
billions of log entries in seconds?


Origins of BigQuery

How do you find slow running servers from
billions of log entries - in seconds?
SELECT
count(*) AS count, source_machine AS
machine
FROM product.product_log.live
WHERE
elapsed_time > 4000
GROUP BY
source_machine
ORDER BY
count DESC

Result in ~20 seconds!

Origins of BigQuery

Google BigQuery


BigQuery -- Sweetspots

Use cases/applications where
Interactive analysis on large data sets is a requirement
Data mashups at scale and ease is important
Elasticity of environment is important
Minimal administration is required
Empowering Analysts or Data scientist with self service capabilities


Demo
Ad-hoc Analysis with BigQuery


Why is BigQuery so Special?

+


Why BigQuery is Special?

0

ZERO Administration for Performance and Scale

No complex data architecture required.
●
●

Simple denormalized data structure
Data can be loaded in csv or JSON format

Supports Open Standards
●
●
●

SQL like language
Standard BI/ETL tools supported
REST API Support for integrating analytics programmatically


Nested Field Support
●
●

Analyze details for headers (items of the order, clicks of the session in
single SQL statement)
Ingest Web data (Typically in JSON format) directly into BigQuery

Query Result Caching at Scale
●
●

Caching of query results at Scale (24 hours best effort)
Option to re-generate results / update cache

Real-time Data Ingest and Reporting
●
●

Data ingest at 100 rows/second (with peak 1000 rows/second) into
BigQuery
Support for real-time to near real-time query/reporting


And many more...
SQL like Query Language and REST based APIS
High performance queries on strings (use of REGEXP and pattern matching)
Data window functions and many aggregation capabilities
Table Wildcards
Table decorators for time based queries..

Integration with Google Stack
Google Analytics Premium (hit level data) - BigQuery
Google App Engine Datastore to BigQuery - Trusted Tester

What BigQuery is Not?

+


Why BigQuery is Not?

Relational Database Management System (RDBMS)
●
●

Data is stored in BigQuery in Columnar fashion
BigQuery tables are immutable (no updates or deletes)*

Not a replacement for existing Data warehouse en-mass
●
●

Specifically designed for large data sets
Queries requiring massive processing

On-premise solution or Appliance
●
●

It is available only in Google Cloud in a fully hosted fashion
Data needs to be in Google Cloud (BigQuery) for analysis

* alternate ways to support these capabilities

Columnar Storage and Tree Architecture of Dremel
Why Dremel can be so drastically fast as the examples show?
The answer can be found in two core technologies which gives Dremel this
unprecedented performance:
● Columnar Storage. Data is stored in a columnar storage fashion which
makes possible to achieve very high compression ratio and scan throughput.
● Tree Architecture is used for dispatching queries and aggregating results
across thousands of machines in a few seconds.


Dremel: Key to Run Business at “Google Speed”
Google has been using Dremel in production since 2006 and
has been continuously evolving it for the last 6 years.
Examples of applications include:
●
●
●
●
●
●
●
●
●
●
●

Analysis of crawled web documents
Tracking install data for applications in the Android Market
Crash reporting for Google products
OCR results from Google Books
Spam analysis
Debugging of map tiles on Google Maps
Tablet migrations in managed Bigtable instances
Results of tests run on Google’s distributed build system
Disk I/O statistics for hundreds of thousands of disks
Resource monitoring for jobs run in Google’s data centers
Symbols and dependencies in Google’s codebase

BigQuery Sample Use Cases


Sensor Data Analytics


Sensor Data Analytics
Data Sensing Lab
Sensors

Generate

Endpoints/App Engine

Ingest/Process

Datastore

Store

Compute Engine/
BigQuery

Compute

Visualize
Analyze

Mobile and Social Gaming User Analysis

Notice trend change

Slice user data, identify
segments

Compare segments
vs general population

BigQuery versus MapReduce
● BigQuery is designed as an interactive data analysis tool for
large datasets
● MapReduce is designed as a programming framework to
batch process large datasets


BigQuery: Key Differentiators
Agile : BigData Analytic Hosted Environment
● Fully managed cloud service
● Interactive querying on Terabytes of data
● Always on
Cost Effective: Low TCO / TCD
● No CapEx as no hardware/software to own
● No administration or management overhead
● Pay for only what you use.
Non disruptive, fits right into IT landscape
● Expanding ISV ecosystem (BI/ETL)
● Use of standards (SQL like, REST API etc...)
● Integrated with other Google Services (GCE, GAE, GCE etc...)

Secure and Reliable
● Uptime SLA (99.9% uptime)
● OAUTH 2.0 Support, User/Group ACL
● Uses Google's core network/data center backbone
Innovative
● Innovation momentum: 2 release since launch
● New capabilities/concepts: e.g. Queryable Nested fields
● e.g. Use of RegExp without performance loss


Recent Manufacturer POC

Local
Processing

BIME
Dashboards

Fusion
Tables

AppEngine

Cloud Storage

GCE

Cloud Storage

Big Query

CSV

●

●

Achieved a 200,000 rows per
second ingestion rate, and it took
about 10 minutes from when the
file is first dropped into GCS to
when you can query the data in
BigQuery.
Demonstrated ingesting 120
million rows in 10 minutes into
BigQuery.

●

●

Leveraged Google Fusion Tables
to easily collaborate on datasets
with business and partners.
Also discovered manufacturer
had machines that consumed
250+ billion gallons of fuel as of
July 30th.

●

●

Spun up BIME Dashboard to
interactively connect to BigQuery
dataset.
With dashboards connected to BQ
(BIME, Tableau, QlikView, etc),
we can drill into billions of rows
near real-time data with ease.

BigQuery: Ever Expanding Ecosystem of Partners


cloud.google.com


How Google Does Big Data - DevNexus 2014

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to How Google Does Big Data - DevNexus 2014 (20)

Recently uploaded (20)

How Google Does Big Data - DevNexus 2014