Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid
Download as PPTX, PDF•17 likes•6,332 views
The document discusses Pulsar, a real-time analytics framework integrated with Kafka, Kylin, and Druid to process user behavioral data at scale. It highlights its capabilities in sessionization, bot detection, and multi-dimensional analysis, while detailing its architecture and application in various data pipelines. Additionally, it emphasizes personalized content delivery based on user behavior and offers links for more resources and documentation.
![Aggregate Computation Example
// create 10-second time window context
create context MCContext start @now and pattern
(timer:interval(10)];
// create aggreated stream within specified time window
context MCContext insert into AggStream
select count(*) as M1, guid, evt_type from RawStream
group by guid, evt_type output snapshot when terminated;
// publish aggregated stream
select * from AggStream;](https://image.slidesharecdn.com/pulsar-strata-share-161017040357/85/Pulsar-Real-time-Analytics-at-Scale-with-Kafka-Kylin-and-Druid-11-320.jpg)
![TopN Computation Example
// create 60-second time window context
create context MCContext start @now and pattern
(timer:interval(60)];
// create topN stream via sorting
context MCContext insert into TopNStream
select count(*) as M1, guid, evt_type from RawStream
group by guid, evt_type order by M1 limit 10;
// publish topN stream
select * from TopNStream;](https://image.slidesharecdn.com/pulsar-strata-share-161017040357/85/Pulsar-Real-time-Analytics-at-Scale-with-Kafka-Kylin-and-Druid-13-320.jpg)
Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid
- 1. Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid September 2016 Tony Ng Director of Engineering
- 3. *Q2 2016 data New Items 80% Fixed Price Items 86% Evolving from our Auction Roots.. Ships for Free 65%
- 4. EBAY AT A GLANCE $8.6B Revenue in 2015 $82B GMV in 2015 1B Live Listings * 164M Global Active Buyers * 190 Countries eBay apps are available in # *Q2 2016 data #Q4 2015 data 326M App downloads*
- 5. CHECKOUT WATCH LIST SHARE LOCAL PREFERENCES BID 10’s TB USER BEHAVIORAL DATA / DAY 10’s B DATABASE QUERIES / DAY 100’s PB DATA CLICK SEARCH
- 6. ENTERPRISE DATA PLATFORM Data Stores Data Streams & Processing Machine Learning Enterprise Data Ecosystem Data Ingestion Personalization / Optimization Insights / Reporting Kylin
- 7. Open-source real-time analytics and stream processing framework
- 8. Pulsar Stream • Focus on user behavioral data processing • Complex event processing – Streaming SQL with extensible annotations – Java • SQL for common stream operations (Filtering, mutation, aggregation) with time windows • Declarative topology construction • Each stage can adopt its own release and deployment cycles • Dynamic partitioning and flow control
- 9. Multi Stage Distributed Pipeline
- 10. Event Filtering and Routing Example // create filtered stream insert into FilteredStream select guid, evt_type, C1, C2, C3 from RawStream where evt_type = ‘bid’; // publish and route filtered stream @PublishOn(topics=“Topic1”) @Output(“OutboundChannel”) @ClusterAffinityTag(column = guid) select * from SubStream;
- 11. Aggregate Computation Example // create 10-second time window context create context MCContext start @now and pattern (timer:interval(10)]; // create aggreated stream within specified time window context MCContext insert into AggStream select count(*) as M1, guid, evt_type from RawStream group by guid, evt_type output snapshot when terminated; // publish aggregated stream select * from AggStream;
- 12. Stream Aggregation Time Window Sliding Window Tumbling Window
- 13. TopN Computation Example // create 60-second time window context create context MCContext start @now and pattern (timer:interval(60)]; // create topN stream via sorting context MCContext insert into TopNStream select count(*) as M1, guid, evt_type from RawStream group by guid, evt_type order by M1 limit 10; // publish topN stream select * from TopNStream;
- 15. Pulsar Behavioral Data Pipeline Sessionizer Metrics Calculator Event Distributor Real Time Consumers Metrics Store Collector Real-time Pipeline BOT Detection Enriched Sessionized Events Producing Applications Real Time Dashboard and Services Kafka DruidHaddop / Kylin Batch Loader
- 16. Sessionization: Group together events of a single user visit e1 e2 e3 e4 e5 e7 e8 User 1: >30 min of inactivity Session A (User 1): e1, e2, e4 Session B (User 2): e3, e5, e6 Session C (User 1): e7, e8 e6 . . .
- 17. Sessionization Challenges • Session state management – High read/write throughput – State recovery when node crash/fail • Session Expiration – Full table scan is not acceptable
- 18. Sessionization Solution • Long live state management (At least 30 minutes) – Local Off-Heap Cache • Instantaneous Session Expiration (<= 1sec delay) – Double-Linked Off-Heap Map (Local Access) – Order by Expiration time (O(1)) • Pluggable Sessionization logic – SQL with customized annotation – Counter – State
- 19. Sessionizer Architecture Collector Sessionizer Sessionizer Local Off-Heap Session Cache I M C Timer Remote Store Client Remote Session Store Recovery O M C BotDetection Distributor Persist Sync
- 20. Bot Detection Overview • Detect non-human activities in near realtime • May treat bot traffic differently during analysis • High level bot rules – Self-declared bots by user agent – Behavior within a session or time window • Tag events with bot flag
- 21. Bot Detection SessionizerCollector BotDetection Distributor Behavior Events And Metrics BotSignature Bot Detection Service BotSignature Bot tagged Stream
- 22. PULSAR INTEGRATION WITH OTHER SYSTEMS Divider sub-headline goes here
- 23. Pulsar Integration with Kafka •Kafka – Persistent messaging queue – High availability, scalability and throughput •Pulsar leveraging Kafka – Supports pull and hybrid messaging model – Loading of data from real-time pipeline into Hadoop and other metric stores – Use schema to validate event payload 2
- 24. Messaging Models Producer Producer Queue Kafka Producer Queue Kafka Replayer Push Model Pull Model Pause/Resume Hybrid Model (At most once delivery semantics) (At least once delivery semantics) Consumer Consumer Consumer Netty
- 25. Pulsar Integration with Kylin •Apache Kylin – Distributed analytics engine – SQL interface and multi-dimensional analysis (OLAP) on Hadoop – Interactive Query on Billions of Rows •Pulsar leveraging Kylin – Build multi-dimensional OLAP cube over long time period – Aggregate/drill-down on dimensions such as browser, OS, device, geo location – Capture metrics such as session length, page views, event counts 2
- 26. Pulsar Integration with Druid •Druid – Real-time ROLAP engine for aggregation, drill-down and slice-n-dice •Pulsar leveraging Druid – Real-time analytics dashboard – Near real-time metrics like number of visitors in the last 5 minutes, refreshing every 10 seconds – Aggregate/drill-down on dimensions such as browser, OS, device, geo location 2
- 27. BEHAVIORAL DATA DRIVEN APPLICATIONS Divider sub-headline goes here
- 29. TRENDING: ALGORITHMS NARROW THE FOCUS Algorithms and machine learning identify significant trends Humans provide the context and and interesting story
- 30. search view watch bid purchase w time NearlineOffline (historical) s vv…events… b Online (in-session) vpv s Activity Timeline for Personalization Customer Profile • Price • Category • Sale Type • Item Condition • Deals Customer Intent • Price • Category • Sale Type • Item Condition • Deals
- 31. EXAMPLE PERSONALIZED CONTENT Personalized digest for a consumer interested in jewelry and accessories Personalized digest for a consumer interested in auto and electronics
- 32. Behavior Data: A/B Testing 3 Behavioral Data: A/B Testing
- 33. More Information •GitHub: http://github.com/pulsarIO –repos: pipeline, framework, docker files •Website: http://gopulsar.io –Technical whitepaper –Getting started –Documentation •Google group: http://groups.google.com/d/forum/pulsar 3
Editor's Notes
- #3: spectrum of our inventory (old and new) … if it exists, probably on sale @ ebay