DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
1 like•437 views
The document discusses reasons for reimplementing messaging queue functionality instead of using an existing solution like RabbitMQ or Kafka. It notes that reimplementing provides full understanding and control, allowing for quick fixes and easy addition of features without needing a daemon. However, it also acknowledges that reimplementing takes a long time to reach stability. The document then describes the specific requirements for the reimplemented messaging queue and provides a brief overview of its design and performance.
![BYOMQ: Why We [re]Built
IronMQ
Reed Allman
@rdallman10](https://image.slidesharecdn.com/byomq-160414230807/85/DataEngConf-SF16-BYOMQ-Why-We-re-Built-IronMQ-1-320.jpg)
![Making [another!] Case for NIH
Reed Allman
@rdallman10](https://image.slidesharecdn.com/byomq-160414230807/85/DataEngConf-SF16-BYOMQ-Why-We-re-Built-IronMQ-2-320.jpg)
![Beating A Dead Horse [again!]
Reed Allman
@rdallman10](https://image.slidesharecdn.com/byomq-160414230807/85/DataEngConf-SF16-BYOMQ-Why-We-re-Built-IronMQ-3-320.jpg)
![DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://image.slidesharecdn.com/byomq-160414230807/85/DataEngConf-SF16-BYOMQ-Why-We-re-Built-IronMQ-10-320.jpg)
![DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://image.slidesharecdn.com/byomq-160414230807/85/DataEngConf-SF16-BYOMQ-Why-We-re-Built-IronMQ-19-320.jpg)
![DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://image.slidesharecdn.com/byomq-160414230807/85/DataEngConf-SF16-BYOMQ-Why-We-re-Built-IronMQ-24-320.jpg)
![DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://image.slidesharecdn.com/byomq-160414230807/85/DataEngConf-SF16-BYOMQ-Why-We-re-Built-IronMQ-25-320.jpg)
![DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://image.slidesharecdn.com/byomq-160414230807/85/DataEngConf-SF16-BYOMQ-Why-We-re-Built-IronMQ-27-320.jpg)
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
- 1. BYOMQ: Why We [re]Built IronMQ Reed Allman @rdallman10
- 2. Making [another!] Case for NIH Reed Allman @rdallman10
- 3. Beating A Dead Horse [again!] Reed Allman @rdallman10
- 4. There
- 5. Are
- 6. Too
- 7. Many
- 8. Options
- 9. Me Showing .gifs for 30 Minutes Reed Allman @rdallman10
- 11. NIH: Not Invented Here Syndrome A quick refresher: ...the tendency towards reinventing the wheel (reimplementing something that is already available) based on the belief that in-house developments are inherently better suited, more secure, more controlled, quicker to develop, and incur lower overall cost (including maintenance cost) than using existing implementations.
- 13. Throw this at ${USE_CASE}
- 14. Out of the Box ● Massive scale on commodity hardware in hours of easy set up ● Battle tested, many bugs worked out, lots of docs, tools, guides, etc. ● Operate JVM, hard not to code on JVM ● Any and all problems associated with any of the boxes or how they fit together ● And these are open source! Many boxes are not and suffer vendor lock-in ● At mercy of those who understand it / time for fixing bugs, making improvements
- 15. Could you do better?
- 16. Different people come from different backgrounds and, based on that, find different tools useful. Ultimately you need to use the tools that let you get your job done most effectively, however that is defined in your particular case. - Russ Cox Go
- 17. If it's a core business function -- do it yourself, no matter what. - Joel Spolsky Excel
- 18. I felt that my team, which was supposed to be made up of distributed systems engineers, was really acting more as distributed system plumbers. - Jay Kreps Kafka
- 20. Pain meow or later?
- 21. To the Queues!
- 22. Our Requirements: ● Easy to run (single binary ideal) in any environment & not hosted ● Persisted, consistent and performant ● Highly available -- fast recovery, no data loss, minimal downtime ● Horizontally scalable ● Timing based ● FIFO ● >= 1 time delivery ● Multi-Tenant capable+ ● Preferably built in Go, HTTP interface
- 23. ...Scalable, Low-Latency, Persistent MQ for Job Processing
- 26. What, briefly ● Single Go binary per server ● RocksDB embedded to store all metadata and messages ● Built the distributed database - viewstamped replication, multi-master, gossip membership - scales across queues across nodes, auto-balanced ● Linear scaling factor of .98 ● Recovery time < 300ms ● Simple HTTP interface, stupidly simple clients ● Enqueue / Dequeue avg latency < 10ms for < 4KB messages
- 28. boom
- 29. Results
- 30. Full understanding and control makes for quick fixes, easy to add features
- 31. Don’t have to build a daemon to add features, can bake everything in
- 32. Takes a long time to get to stable
- 33. Make all your own tools, plumb any data you want out
- 34. Build exactly what you need, optimized for what you are doing
- 35. Use the language(s) you know and love
- 36. My attempt to say profound shit: Be a builder, not a plumber.
- 37. Questions?