How Klout is changing the landscape of social media with Hadoop and BI
Download as PPTX, PDF15 likes8,873 views
Klout leverages big data and Hadoop to process and analyze vast amounts of social media data, handling over 1 billion social signals daily from 120 terabytes of storage. The platform's architecture supports various aspects of business intelligence, including data warehousing and real-time analytics for user engagement and optimization. Key features include the use of Hive for query execution and a flexible schema for event tracking and analytics.
![Event Tracker Architecture event_log
tstamp string
{ project string
"project":"plusK", string
event
session_id bigint
"event":"spend",
insights3:9003/track/{"project":”plu
ks_uid bigint
sK","event":”spend”,"session_id":"0",
Warehouse
ip string
"ip":"50.68.47.158",
"ks_uid":123456,”type":”add_topic"}
json_keys array<string>
"kloutId":“123456",
json_values
“cookie_id":”123456",
array<string>
"ref":"http://klout.com/",
json_text string
"type":"add_topic",
Tracker API Log Process Cube
dt string Klout UI
"time":"1338366015"
Scala, Flume Analysis Scala,
} hr string
node.JS Services AJAX UX
SELECT { [Measures].[Counter], [Measures].[PreviousPeriodCounter]}
ON COLUMNS,
will be saved in HDFS at:
NON EMPTY CROSSJOIN ( /logs/events_tracking/2012-05-30/0100
exists([Date].[Date].[Date].allmembers,
[Date].[Date].&[2012-05-19T00:00:00]:[Date].[Date].&[2012-06-
02T00:00:00]),
[Events].[Event].[Event].allmembers ) DIMENSION PROPERTIES
MEMBER_CAPTION
ON ROWS
FROM [ProductInsight]
WHERE ({[Projects].[Project].[plusK]})
Instrument Collect Persist Query Report
9](https://image.slidesharecdn.com/hadoopsummit2012klouthadoopandbi-120704001241-phpapp02/85/How-Klout-is-changing-the-landscape-of-social-media-with-Hadoop-and-BI-9-320.jpg)
How Klout is changing the landscape of social media with Hadoop and BI
- 1. How Klout is changing the landscape of social media with Hadoop and BI Dave Mariani VP Engineering, Klout Denny Lee Principal Program Manager Microsoft
- 2. Discover and be recognized for how you influence the world
- 3. Klout’s Big Data makes all this possible 15 Social Networks Processed Every Day 120 Terabytes of Data Storage 200,000 Indexed Users Added Every Day 140,000,000 Users Indexed Every Day 1,000,000,000 Social Signals Processed Every Day 30,000,000,000 API Calls Delivered Every Month 54,000,000,000 Rows of Data In Klout Data Warehouse 3
- 4. KLOUT DATA ARCHITECTURE THE BEST TOOL FOR THE JOB Registrations DB Klout.com (MySql) (Node.js) Mobile Profile DB (ObjectiveC) Klout API (Scala) (HBase) Signal Collectors Data Partner API (Java/Scala) Enhancement Engine (Mashery) Data Warehouse (PIG/Hive) Search Index (Hive) (Elastic Search) Streams (MongoDB) Monitoring (Nagios) Serving Stores Dashboards (Tableau) Perks Analyics (Scala) Analytics Cubes Event Tracker (SSAS) (Scala)
- 5. What is Business Intelligence? • Data Warehousing, OLAP, Dashboards, Reporting • Ability to slice and dice data in an ad-hoc manner • Getting the right data to the right people, at the right time • i.e. Now 5
- 6. Why Hadoop + BI? Hadoop BI Requirement & Query Hive Engines Capture & store all data Yes No Support queries against detail data Yes No Support interactive queries & No Yes applications Support BI & visualization tools No Yes 6
- 7. An Example: Klout Event Tracker 1 Perform A|B Testing of User Flows 2 Optimize Registration Funnels 3 Monitor consumer engagement & retention (DAUs & MAUs) 4 Flexibly track and report on user generated events 7
- 8. A Flexible, Hierarchical Schema Project: Event: Property Type: Property Value: Collection Captured Attribute Attribute of Events User Action Key Value HomePage, Source, Google Search Actions, Gender, Male Mobile iOS Location SF +K (Add a topic) event
- 9. Event Tracker Architecture event_log tstamp string { project string "project":"plusK", string event session_id bigint "event":"spend", insights3:9003/track/{"project":”plu ks_uid bigint sK","event":”spend”,"session_id":"0", Warehouse ip string "ip":"50.68.47.158", "ks_uid":123456,”type":”add_topic"} json_keys array<string> "kloutId":“123456", json_values “cookie_id":”123456", array<string> "ref":"http://klout.com/", json_text string "type":"add_topic", Tracker API Log Process Cube dt string Klout UI "time":"1338366015" Scala, Flume Analysis Scala, } hr string node.JS Services AJAX UX SELECT { [Measures].[Counter], [Measures].[PreviousPeriodCounter]} ON COLUMNS, will be saved in HDFS at: NON EMPTY CROSSJOIN ( /logs/events_tracking/2012-05-30/0100 exists([Date].[Date].[Date].allmembers, [Date].[Date].&[2012-05-19T00:00:00]:[Date].[Date].&[2012-06- 02T00:00:00]), [Events].[Event].[Event].allmembers ) DIMENSION PROPERTIES MEMBER_CAPTION ON ROWS FROM [ProductInsight] WHERE ({[Projects].[Project].[plusK]}) Instrument Collect Persist Query Report 9
- 10. Hadoop & BI Together: Query Cube using a Custom App 10
- 11. A peek into product insight > A|B test : unsorted vs. Sorted 11
- 12. A Peek into Product Insights > Projects: Mobile iOS 12
- 13. 13
- 14. Hadoop & BI Together: Query Cube Using Viz App 14
- 15. 15
- 16. 16
- 17. Hadoop & BI Together: Query Hive using CLI 17
- 18. HiveQL Example SELECT get_json_object(json_text,'$.sid') as sid, get_json_object(json_text,'$.inc') as inc, get_json_object(json_text,'$.status') as status, event FROM bi.event_log WHERE project='mobile-ios' AND dt=20120612 AND get_json_object(json_text,'$.v')<>'1.5' AND (event = 'api_error' OR event = 'api_timeout') ORDER BY sid;
- 19. 19
- 20. Hadoop & BI Together: Query Hive using Excel 20
- 21. 21
- 22. Why Hadoop + BI? Hadoop BI Requirement & Query Hive Engines Capture & store all data Yes No Support queries against detail data Yes No Support interactive queries & No Yes applications Support BI & visualization tools No Yes 22
- 23. Any Questions? 23
Editor's Notes
- #19: Copy this from notepad for demo:CREATE TABLE mobile_ios_details_20120612 asSELECT get_json_object(json_text,'$.sid') as sid, get_json_object(json_text,'$.inc') as inc, get_json_object(json_text,'$.status') as status, eventFROM bi.event_logWHERE project='mobile-ios' AND dt=20120612 AND get_json_object(json_text,'$.v')<>'1.5' AND (event = 'api_error' OR event = 'api_timeout') ORDER BY sid;
- #23: 1.Don’t throw data away, leverage Hadoop (track users and events for a/b testing)2. BI tools aggregate data, but we need to reach back to the detail to answer deeper questions (http codes)3. Hadoop != interactive queries (combined proprietary data with detail)4.Use open source, but don’t reinvent the wheel (BI tools are mature, valuable & complementary)Leverage the best tool for the function or job