Clojure at BackType
- 1. Clojure at BackType How we learned to stop worrying and love the parentheses Nathan Marz BackType @nathanmarz
- 2. BackType Data Services (APIs) Social Media Analytics Dashboard
- 3. APIs • Conversational graph for url • Comment search • #Tweets / URL • Influence scores • Top sites • Trending links stream • etc.
- 5. URL Profiles
- 8. Twitter Account Analytics
- 10. Topic Analysis
- 12. BackType’s Challenges Complex analytics
- 13. BackType’s Challenges Complex analytics on lots of data (> 30TB)
- 14. BackType’s Challenges Complex analytics on lots of data (> 30TB) in realtime
- 15. Clojure at BackType • Cascalog • ElephantDB • Storm
- 16. Let’s build an app
- 17. Let’s build an app
- 18. Cascalog Cascalog Variables and logic Abstraction Cascading Tuples, data workflows Key/value pairs, MapReduce aggregation
- 19. Cascalog basics The “age” dataset
- 20. Cascalog basics
- 21. Cascalog basics Define and execute a query
- 22. Cascalog basics Where to emit results Define and execute a query
- 23. Cascalog basics Where to emit results Output variables Define and execute a query
- 24. Cascalog basics Where to “Predicates”: constrain emit results the output variables Output variables Define and execute a query
- 25. Predicates
- 27. Predicates Input fields Output fields
- 28. Predicates Fields can be constants or variables
- 29. Predicates Fields can be constants or variables Variables are prefixed with ? or !
- 30. Predicates
- 31. Predicates • Functions • Filters • Aggregators • Generators: finite sources of tuples
- 32. Example #1 Generator Filter
- 33. Example #2 Generator Function
- 34. Example #3 Generator Aggregator Filter
- 35. Join example
- 36. Join example Triggers a join
- 37. Join example
- 38. Join example Joins are an implementation detail
- 39. Cascalog demo!
- 41. Composability expands to Using a predicate macro
- 42. Contrast to Pig Pig’s AVG is 300 lines of code
- 43. Let’s build an app
- 44. Graph Schema Reshare: true Gender: female Property Tweet: 456 Property Reaction Reactor Reactor Tweet: 123 Alice Bob Property Property Content: RT @bob Content: Data is fun! Data is fun!
- 45. ElephantDB Shard 0 Shard 1 Shard 2 Distributed Key/Value pairs Shard 3 Filesystem Pre-shard Shard 4 and index in Shard 5 MapReduce Generation of domain of data
- 46. ElephantDB DFS ElephantDB Server Shard 0 Shard 1 Shard 2 ElephantDB Server Shard 3 Shard 4 Shard 5 ElephantDB Server Serving domain of data
- 48. Stream processing • Automatically distributes computation • Horizontally scalable • Fault-tolerant • Guarantees processing of messages
- 49. Stream processing DB Queue DB DB Storm cluster
- 50. Raw data What is a query? View
- 51. Tweets What is a query? # Tweets for a URL
- 52. Tweets What is a query? Influence Score for a person
- 53. Raw data Computing a query Fully precompute view DB Query
- 54. Raw data Computing a query Do a live compute from scratch Query
- 55. Computing a query DB Raw data Precompute subviews Compute query from DB Query intermediate dbs DB
- 56. Distributed RPC Application Queue “I want to know X, and return the results to me at Y”
- 57. Distributed RPC DBs Queue App queries Storm cluster
- 59. Questions?
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n