Building Event-Based Systems for the Real-Time Web
- 1. Building Event Based Systems for the Real- Time Web Paul Dix http://pauldix.net @pauldix
- 5. The Talk
- 6. Event Based: data update => do something
- 7. Batch: need data => calculate
- 8. Real-Time: data update => notification
- 9. Scheduled: occasionally => do stuff
- 10. Event Based == Real Time
- 11. Monolithic American Backroom (http://www.flickr.com/photos/41922098@N03/4247207167/)
- 14. Scale with Complexity PhOtOnQuAnTiQuE (http://www.flickr.com/photos/67968452@N00/1876685709/)
- 15. Scale with Team Size alexkess (http://www.flickr.com/photos/34838158@N00/3370167184/)
- 17. SQL Databases
- 18. events: insert, update, delete execution: before, after, instead of Event Based: triggers
- 19. Batch: count, sum, min, max
- 23. Self Contained
- 24. Monolithic
- 26. Doesn’t Scale Darwin Bell (http://www.flickr.com/photos/53611153@N00/465459020/)
- 28. Event Based: messaging, publish subscribe, amqp
- 30. Real-Time: messaging, publish/ subscribe
- 31. Scheduled: delayed replication, data backups
- 33. Event Based: web hooks, pubsubhubub, rsscloud
- 34. Batch: polling
- 35. Real-Time: web hooks, web sockets, pubsubhubbub, rsscloud
- 36. Scheduled: check every 1 hour
- 37. Distributed (internal) Systems
- 38. Service Calls
- 39. Feed Reader (on fetch)
- 40. Feed Reader (on fetch) • Update users reading list
- 41. Feed Reader (on fetch) • Update users reading list • Language Identification
- 42. Feed Reader (on fetch) • Update users reading list • Language Identification • Named Entity Extraction
- 43. Feed Reader (on fetch) • Update users reading list • Language Identification • Named Entity Extraction • Cluster with other Articles
- 44. Feed Reader (on fetch) • Update users reading list • Language Identification • Named Entity Extraction • Cluster with other Articles • Identify Trending Entities
- 45. Tightly Coupled Steve aka Crispin Swan (http://www.flickr.com/photos/26811962@N05/3704553536/)
- 47. Publish/Subscribe (pubsub)
- 48. RabbitMQ
- 49. AMQP
- 50. Exchanges
- 51. Queues
- 52. Bindings
- 53. Routing Key
- 54. Exchange Type
- 55. Topic Exchange
- 57. # - 0 or more wildcard
- 58. * - 1 wildcard
- 64. Include Data in the Message
- 65. Data is the API
- 66. Another Example: error logging
- 69. binding: #
- 70. Loosely Coupled ° ρЯίтΛм ° (http://www.flickr.com/photos/39046851@N08/3829500850/)
- 72. Event-Based
- 73. Real-Time
- 74. Web Hooks
- 77. git push triggers hooks
- 78. POST to Post-Recieve URLs
- 79. { "before": "5aef35982fb2d34e9d9d4502f6ede1072793222d", "repository": { "url": "http://github.com/defunkt/github", "name": "github", "description": "You're lookin' at it.", "watchers": 5, "forks": 2, "private": 1, "owner": { "email": "[email protected]", "name": "defunkt" } }, "commits": [ { "id": "41a212ee83ca127e3c8cf465891ab7216a705f59", "url": "http://github.com/defunkt/github/commit/41a212ee83ca127e3 "author": { "email": "[email protected]", "name": "Chris Wanstrath" }, "message": "okay i give in", "timestamp": "2008-02-15T14:57:17-08:00",
- 80. Data is the API
- 82. Bug Tracking
- 83. Campfire
- 85. PubSubHubbub
- 86. RSS or Atom Feed <link rel=”hub” href=”http://pubsubhubbub.appspot.com/”>
- 88. Hub verifies with Subscriber
- 90. Hub gets from Publisher
- 91. Hub sends to Subscribers
- 92. Data is the API
- 93. OAuth + Web Hooks
- 94. Concerns
- 95. don’t make web-hook callbacks while a user is waiting Responsive
- 96. what if the subscriber doesn’t respond? retry logic notifications Retries
- 97. use OAuth. SSL for sensitive data Security
- 98. Possibilities
- 99. Real-Time
- 100. Event Based
- 101. Communication
- 102. Are these really real- time?
- 103. Data Consistency
- 104. in brewer’s CAP theorem he talked about the relationship bet ween three requirements when building distributed systems. consistency, availability, and partition tolerance. Eric Brewer’s CAP theorem
- 105. consistency means that an operation either works completely or fails. this is also referred to as atomic. Bug tracker example. consistency
- 106. availability is pretty self explanatory. a service is available to ser ve requests. Twitter example availability
- 107. when you replicate data across multiple systems, you create the possibility of forming a partition. this happens when one or more systems lose connectivity to other systems. partition tolerance is defined formally as “no set of failures less than total net work failure is allowed to cause the system to respond incorrectly” partition tolerance
- 108. pick two
- 109. Can have all three “is a special form of weak consistency. if no new updates are made to an object, eventually all accesses will return the last updated value.” Werner Vogels’ eventual consistency
- 112. Design Around Eventual Consistency
- 113. In Closing...
- 114. Event-Based == Real Time
- 115. Pubsub
- 116. Loosely Coupled
- 117. Expose Data
- 119. Extend and Improve
- 120. many heads are better... shoothead (http://www.flickr.com/photos/66621443@N00/519240547/)
- 121. than one. John Spooner (http://www.flickr.com/photos/29809546@N00/315157691/)
- 122. Questions? Paul Dix http://pauldix.net @pauldix