RabbitMQ Model and Some Example Applications
Download as PPTX, PDF23 likes12,879 views
RabbitMQ is used in several example applications and real case studies. Three example codes demonstrate: 1) the RabbitMQ tutorial using request/reply queues, 2) RPC using correlated messages, and 3) Celery's distributed task queue using polling. Case studies show RabbitMQ for: 1) message passing between Zenoss daemons, 2) Nova controller RPC, 3) Instagram's task queue with Celery, 4) MercadoLibre's service bus architecture, and 5) Indeed.com's distributed crawling. RabbitMQ provides reliability, scalability, and asynchronous messaging between distributed applications and microservices.
![Heartbeat
zenhub
zeneventd
zenhubwork
er
zenhubworker.py---(1)-->[RabbitMQ]
zenhubworker.py---(1)-->[RabbitMQ]
zenhubworker.py---(1)-->[RabbitMQ]
[RabbitMQ]---(3)-->java
[RabbitMQ]---(3)-->java
[RabbitMQ]---(3)-->java
zenoss
event server
HB
HB
HB
heart
beat
HB
message bus
publish exchange: zenoss.heartbeats key: zenoss.heartbeat.localhost
content header: org.zenoss.protobufs.zep.DaemonHeartbeat
content with len: 33
deliver
content header: org.zenoss.protobufs.zep.DaemonHeartbeat
content with len: 33](https://image.slidesharecdn.com/messagequeuemodelandexampleapplicationspublicversion-131124233625-phpapp01/85/RabbitMQ-Model-and-Some-Example-Applications-18-320.jpg)
![Ping Failure Messages
zenping---(1)-->[RabbitMQ]
zenping---(1)-->[RabbitMQ]
zenping---(1)-->[RabbitMQ]
[RabbitMQ]---(1)-->zeneventd.py
[RabbitMQ]---(1)-->zeneventd.py
[RabbitMQ]---(1)-->zeneventd.py
zeneventd.py---(1)-->[RabbitMQ]
zeneventd.py---(1)-->[RabbitMQ]
zeneventd.py---(1)-->[RabbitMQ]
[RabbitMQ]---(1)-->java
[RabbitMQ]---(1)-->java
[RabbitMQ]---(1)-->java
java---(6)-->[RabbitMQ]
java---(6)-->[RabbitMQ]
java---(6)-->[RabbitMQ]
publish exchange: zenoss.zenevents.raw key: zenoss.zenevent.status.ping
content header: org.zenoss.protobufs.zep.Event
content with len: 235
deliver
content header: org.zenoss.protobufs.zep.Event
content with len: 235
publish exchange: zenoss.zenevents.zep key: zenoss.zenevent.status.ping
content header: org.zenoss.protobufs.zep.ZepRawEvent
content with len: 432
deliver
content header: org.zenoss.protobufs.zep.ZepRawEvent
content with len: 432
publish exchange: zenoss.zenevents key: zenoss.zenevent.status.ping
content header: org.zenoss.protobufs.zep.EventSummary
content with len: 433](https://image.slidesharecdn.com/messagequeuemodelandexampleapplicationspublicversion-131124233625-phpapp01/85/RabbitMQ-Model-and-Some-Example-Applications-20-320.jpg)
RabbitMQ Model and Some Example Applications
- 1. RabbitMQ Model and Some Example Applications [email protected] CCMA ITRI
- 2. Outline • • • • RabbitMQ model and standard Simplified message queue model 3 example code 5 real case studies
- 3. RabbitMQ Model s1 C1 tmp1: private (Distribution) s2 direct exchange C2 q1: ! private key: q1 reply-to: tmp1 cor-id: 1 s3 key1 q3: fan-out key1 q4: to: fan-out key:key1 topic exchange C3 STK.US.* STK.IND.* STK.US.* AND type == pdf to: topic key: STK.US.C filter exchange q5: !durable q6: s4 (Fan-out)
- 4. Notations in RabbitMQ Model s4 application thread virtual host TCP connection exchange channel binding message queue consuming
- 5. RabbitMQ Model • Virtual Host • Exchange – direct – fan-out – topic • Binding – topic – cascading – message select • queue – flags: private, durable • Connection – channel: every thread work with one channel • Message – content header • Properties: Reply-To, Cor-Id, Message-Id, Key • queue server may add properties , wont remove/ modify – content body (wont modify) • binary/ file/ stream • Application – client/ server
- 6. Other Items in AMQP 0.8, not included in Model – AMQP transaction – deliver ACK to sender after message received – AMQP distributed transaction – deliver ACK to a 3rd party software after message received – Access – a realm based access control model we think these are irrelevant to ADDM transaction detection
- 7. Simplified Message Queue Model C1 s1 send rpc:q1.1 rpc q1.* s2 reply-to: tmp1 cor-id: (2) q2.* tmp1 C2 rpcresult send update:q34 s3 update q34 cor-id: 2 (Distribution) s4 s5 rpc q5 (Fan out) s6 q5 queue server
- 8. Model Notation C1 queue server application thread $name exchange ($destination: key) message $key $properties binding queue consuming
- 9. 3 Example Code 1. RabbitMQ tutorial code (synchronous) 2. RabbitMQ RPC (synchronous) 3. Celery framework (distributed task queue / asynchronous)
- 10. Classical Scenario in Example Code – one request queue per service; one reply queue per client – properties used on message • correlate-id • message-id • reply-to – steps 1. 2. 3. 4. client thread send REQUEST and wait on reply queue server thread receive REQUEST, process it and send REPLY to reply queue client thread get reply from reply queue if asynchronous mode, client may polls processing status by another status update REQUEST message, and server reply another status update REPLY message
- 11. Example Code 1: RabbitMQ tutorial C1 (q1.1) s1 tmp1 reply-to: (tmp1) (q1) C1 q1 tmp1 1 (tmp1) q1 3 reply-to: (tmp1) 2 C1 tmp1 q1 s1 s1 C1 (tmp1) s1 tmp1 q1 4 • coordinate REQUEST-REPLY into transaction by reply queue id
- 12. Example Code 2: RabbitMQ RPC (rpcq1) C1 s1 tmp1 reply-to: (tmp1) cor-id: (1) (rpcq1) C1 rpcq1 tmp1 q1 1 tmp1 rpcq1 reply-to: (tmp1) cor-id: (1) 2 (tmp1) C1 cor-id: (1) s1 C1 (tmp1) cor-id: (1) s1 rpcq1 tmp1 3 • s1 Coordinate REQUEST-REPLY into transaction by correlate-id 4
- 13. Example Code 3: Celery (Distributed Task Queue) (celery:celery) (celery:celery) C1 s1 add (1,1) celery result: 1 celery: celery cor-id: (1) C1 s1 celery result: 1 step 1 ( add (1,1) cor-id: (1) step 2 (celery-result:1) step 3: polling status of result queue many times: celery: celery C1 (celery-result:1) C1 poll step a empty step b )*
- 14. Example Code 3: Celery (Distributed Task Queue) (celery-result:1) C1 (celery-result:1) worker celeryresult: 1 celery: celery step 4 cor-id: (1) C1 worker cor-id: (1) celeryresult: 1 celery: celery step 5 • Coordinate REQUEST-REPLY into transaction by (correlate-id)
- 15. Celery: one client application with two worker w1 service C1 w1 2+2 1+1 cor-id: (2) cor-id: (1) worker thread pool 1. client make two calls via the same connection w1 C1 2 cor-id: (1) w2 4 cor-id: (2) 2. worker send REPLY via its own connection 3. client receive REPLYs via the same connection
- 16. 5 Real Cases Study • Applications use RabbitMQ 1. Zenoss4‘s message bus between daemons (non transaction app) 2. Nova controller’s RPC call between hosts (transaction app) • Enterprise use RabbitMQ 3. Instagram : Celery + RabbitMQ 4. MercadoLibre: RabbitMQ 5. Indeed.com MailboxApp
- 17. Case 1: Zenoss 4 • A message bus to pass event and heartbeat between daemons – RabbitMQ as message queue server – Protobufs to serialization • Non-transaction application zenhub zenoss event server heart beat event zenhubworker zeventd raw event RabbitMQ
- 18. Heartbeat zenhub zeneventd zenhubwork er zenhubworker.py---(1)-->[RabbitMQ] zenhubworker.py---(1)-->[RabbitMQ] zenhubworker.py---(1)-->[RabbitMQ] [RabbitMQ]---(3)-->java [RabbitMQ]---(3)-->java [RabbitMQ]---(3)-->java zenoss event server HB HB HB heart beat HB message bus publish exchange: zenoss.heartbeats key: zenoss.heartbeat.localhost content header: org.zenoss.protobufs.zep.DaemonHeartbeat content with len: 33 deliver content header: org.zenoss.protobufs.zep.DaemonHeartbeat content with len: 33
- 20. Ping Failure Messages zenping---(1)-->[RabbitMQ] zenping---(1)-->[RabbitMQ] zenping---(1)-->[RabbitMQ] [RabbitMQ]---(1)-->zeneventd.py [RabbitMQ]---(1)-->zeneventd.py [RabbitMQ]---(1)-->zeneventd.py zeneventd.py---(1)-->[RabbitMQ] zeneventd.py---(1)-->[RabbitMQ] zeneventd.py---(1)-->[RabbitMQ] [RabbitMQ]---(1)-->java [RabbitMQ]---(1)-->java [RabbitMQ]---(1)-->java java---(6)-->[RabbitMQ] java---(6)-->[RabbitMQ] java---(6)-->[RabbitMQ] publish exchange: zenoss.zenevents.raw key: zenoss.zenevent.status.ping content header: org.zenoss.protobufs.zep.Event content with len: 235 deliver content header: org.zenoss.protobufs.zep.Event content with len: 235 publish exchange: zenoss.zenevents.zep key: zenoss.zenevent.status.ping content header: org.zenoss.protobufs.zep.ZepRawEvent content with len: 432 deliver content header: org.zenoss.protobufs.zep.ZepRawEvent content with len: 432 publish exchange: zenoss.zenevents key: zenoss.zenevent.status.ping content header: org.zenoss.protobufs.zep.EventSummary content with len: 433
- 21. Case 2: Nova Controller • A message queue server for RPC call between hosts • Each RPC call has its own unique message-id generated by UUID library
- 22. Case 3: Instagram • Web server uses nginx • Messaging system use RabbitMQ – Scalability: self-made message broker, dispatch message in RR – Reliability: RabbitMQ cluster • Worker system use Celery – Celery is distributed task queue – worker concurrency model • slow task: pre-fork pool • fast task: gevent • Overall throughput – 25k application threads publishing task – 4K tasks per second – Also monitor queue length http://lanyrd.com/2013/pycon/scdyrp/
- 23. Nginx Server with Celery nginx w1 client1 1 cor-id: (c1’s request) w2 client client client 10k 10k 10k 2 cor-id: (c2’s request) queue server Nginx • single thread • event driven • asynchronous (non-blocking) • Task complete and return REPLY messages
- 24. Case 4: MercadoLibre • MercadoLibre is 8th largest ecommerce company in Latin American • original architecture of their ecommerce system: requests requests requests Redis nodeJS nginx proxy to route requests 20 million request per minutes, 4GB per seconds Grails Groovy 7000+ vm insance (by openstack) MongoDb
- 25. Case 4: MercadoLibre • Changes from LB + web app to enterprise service bus architecture • Benefit – – – – error handling routing scalability event driven for higher performance web server app1 req app2 item feed req enterprise service bus app3 persist
- 26. MercadoLibre’s News Update System • The items-app is not transaction based
- 27. MercadoLibre’s Selectively Receive Message • The items-app is not transaction based
- 28. Case 5: indeed.COM crawler engine Rabbit MQ crawler engine Rabbit MQ crawler engine Rabbit MQ Asian DC Job write service Job write service DB Job write service Rabbit Rabbit MQ MQ HA DB DB pacific US DC ocean • crawler engine app is not transaction based
- 29. Reference • • • • • • • RabbitMQ specification 0.8 RabbitMQ Tutorial, example 1/6 and its packet dump Celery: Distributed Task Queue and its packet dump Messaging at Scale at Instagram How is nginx different from apache Zenoss source code nova controller source code