Using GraphX/Pregel on Browsing History to Discover Purchase Intent by Lisa Zhang
10 likes2,634 views
This document discusses using graph-based machine learning on browsing history data to discover customer purchase intent for advertisers. It presents challenges with existing solutions like SVD that identify general online buyers but not advertiser-specific patterns. The document proposes representing sites as a graph and using GraphX's Pregel API to propagate positive customer labels along site connections, assigning higher scores to similar sites. Evaluation shows this approach identifies advertiser-relevant sites while addressing issues like model sparsity and frequency. It also provides lessons learned on optimizing Spark jobs.
![Simplified Code
type MT = Double; type ED = Double; type VD = Double
val lambda = …; val maxIterations = …
val initialMsg = 0.0
def updateVertex(id: VertexId, w: VD, delta_w: MT): VD =
w + lambda * delta_w
def sendMessage(edge: EdgeTriplet[VD, ED]): Iterator[(VertexId, MT)] = {
Iterator((edge.srcId, edge.attr * edge.dstAttr),
(edge.dstId, edge.attr * edge.srcAttr))
}
def mergeMsgs(w1: MT, w2: MT): MT = x + y
val graph: Graph[VD, ED] = …
graph.pregel(initialMessage, maxIterations, EdgeDirection.out)(
updateVertex, sendMessage, mergeMessage)](https://image.slidesharecdn.com/3snlisazhang-160224015952/85/Using-GraphX-Pregel-on-Browsing-History-to-Discover-Purchase-Intent-by-Lisa-Zhang-20-320.jpg)
Using GraphX/Pregel on Browsing History to Discover Purchase Intent by Lisa Zhang
- 1. Using GraphX/Pregel on Browsing History to Discover Purchase Intent Zhang, Lisa Rubicon Project Buyer Cloud
- 2. Problem • Identify possible new customers for our advertisers using intent data, one of which is browsing history travel-site-101.com, spark-summit.org
- 3. Challenges Sites are numerous and ever-changing Need to build one model per advertiser Positive training cases are sparse Models run frequently: every few hours
- 4. Offline Evaluation Metrics • AUC: area under ROC curve • Precision at top 5% of score: model used to identify top users only • Baseline: Previous solution prior to Spark
- 5. Linear Dimensionality Reduction SVDINPUT GBT OUTPUT per advertiser Dimension Reduction Classification
- 6. Evaluation
- 7. SVD: Top Sites Home Improvement Advertiser deal-site-101.com chat-site-001.com ecommerce-site-001.com chat-site-002.com invitation-site-001.com classified-site-001.com Telecom Advertiser developer-forum-001.com chat-site-001.com invitation-site-001.com deal-site-101.com college-site-001.com chat-site-002.com
- 8. The Issue with SVDs • Dominated by the same signal across all advertisers • Identify online buyers, but not those specific to each advertiser • Not appropriate for our use case
- 9. SVD per Advertiser? SVDINPUT GBT OUTPUT per advertiser Dimension Reduction Classification
- 10. Non-linear Approaches? Too Complex: Cannot run frequently, we become slow to learn about new sites Too Simple: Possibly same problem as SVD Speed Complexity
- 11. Can We Simplify? Intuition: Given a known positive training case, target other users that have similar site history as the current user. One natural way is to treat sites as a graph.
- 12. Sites as Graphs • Easy to interpret • Easy to visualize • Graph algos well studied
- 13. Spark GraphX • Spark’s API for parallel graph computations • Comes with some common graph algorithms • API for developing new graph algorithms: e.g. via pregel
- 14. Pregel API • Pass messages from vertices to other, typically adjacent, vertices: “Think like a vertex” • Define an algorithm by stating: how to send messages how to merge multiple messages how to update a vertex with message repeat
- 15. Propagation Based Approach • Pass positive (converter) information across edges • Give credit to “similar” sites
- 16. Example Scenario travel-site-101.com book-my-travel-103.com canoe-travel-102.com 1 converter / 40,000 visitors 0 converter / 48,000 visitors 0 converter / 41,000 visitors
- 17. Sending Messages ω = 1/40,000 Δω = ω * edge_weight Δω = ω * edge_weight canoe-travel-102.com book-my-travel-103.com travel-site-101.com
- 18. Receiving Messages Δω1 hawaii-999.com … Δω2 Δωn ωnew = ωold + λ • Σ Δωi canoe-travel-102.com travel-site-101.com
- 19. Weights After One Iteration book-my-travel-103.com canoe-travel-102.com 2.5 x 10^(-5) 1.2 x 10^(-5) 0.8 x 10^(-5) travel-site-101.com
- 20. Simplified Code type MT = Double; type ED = Double; type VD = Double val lambda = …; val maxIterations = … val initialMsg = 0.0 def updateVertex(id: VertexId, w: VD, delta_w: MT): VD = w + lambda * delta_w def sendMessage(edge: EdgeTriplet[VD, ED]): Iterator[(VertexId, MT)] = { Iterator((edge.srcId, edge.attr * edge.dstAttr), (edge.dstId, edge.attr * edge.srcAttr)) } def mergeMsgs(w1: MT, w2: MT): MT = x + y val graph: Graph[VD, ED] = … graph.pregel(initialMessage, maxIterations, EdgeDirection.out)( updateVertex, sendMessage, mergeMessage)
- 21. Model Output & Application • Model output is a mapping of sites to final scores • To apply the model, aggregate scores of sites visited by user SITE SCORE travel-site-101.com 0.5 canoe-travel-102.com 0.4 sport-team-101.com 0.1 … …
- 22. Other Factors • Edge Weights: Cosine Similarity, Jaccard Index, Conditional Probability • Edge/Vertex Removal: Remove sites and edges on the long-tail • Hyper parameter Tuning: lambda, numIterations and others through testing (there is no convergence)
- 23. Evaluation
- 24. Propagation: Top Sites Home Improvement Advrt. label-maker-101.com laptop-bags-101.com renovations-101.com fitness-equipment-101.com renovations-102.com buy-realestate-101.com Telecom Advertiser canada-movies-101.ca canadian-news-101.ca canadian-jobs-101.ca canadian-teacher-rating-101.ca watch-tv-online.com phone-system-review-101.com Canadian Telecom Renovations
- 25. Challenges (from earlier) Sites are numerous and ever-changing Need to build one model per advertiser Positive training cases are sparse Models run frequently: every few hours
- 26. Resolutions Graph built just in time for training Need to build one model per advertiser Positive training cases are sparse Models run frequently: every few hours
- 27. Resolutions Graph built just in time for training Graph built once; propagation runs per advertiser Positive training cases are sparse Models run frequently: every few hours
- 28. Resolutions Graph built just in time for training Graph built once; propagation runs per advertiser Propagation resolves sparsity: intuitive and interpretable Models run frequently: every few hours
- 29. Resolutions Graph built just in time for training Graph built once; propagation runs per advertiser Propagation resolves sparsity: intuitive and interpretable Evaluating users fast; does not require GraphX
- 30. General Spark Learnings • Many small jobs > one large job: We split big jobs into multiple smaller, concurrent, jobs and increased throughput (more jobs could run concurrently). • Serialization: Don’t save SparkContext as a member variable, define Python classes in a separate file, check if your object serializes/deserializes well! • Use rdd.reduceByKey() and others over rdd.groupByKey(). • Be careful with rdd.coalesce() vs rdd.repartition(), rdd.partitionBy() can be your friend in the right circumstances.