Kafka Summit SF 2017 - Running Streaming Apps on Docker
1 like•1,364 views
This document discusses running Kafka Streams applications on Docker. It outlines some lessons learned around tuning Java Virtual Machines to work well within Docker container resource constraints, dealing with networking complexity when using containers, and ensuring there is a good use case for Docker before adopting it. The key takeaways are that Kafka Streams applications need to be tuned as regular Java applications; networking adds complexity; and Docker should only be used if it meets real infrastructure or deployment needs rather than just for the sake of using the technology.
Kafka Summit SF 2017 - Running Streaming Apps on Docker
- 1. 1 R U N N I N G S T R E A M I N G A P P S O N D O C K E R O N E C O U L D . . . B U T S H O U L D O N E ?
- 2. 2
- 5. 5 HOW (NOT?) TO SET UP A STREAMING PLATFORM
- 6. 6 K A F K A S T R E A M S “ H e y , C o n f l u e n t c r e a t e d t h i s r e a l l y c o o l n e w s t r e a m i n g d a t a f r a m ewo r k w e s h o u l d t r y o u t . ” HOW (NOT?) TO SET UP A STREAMING PLATFORM
- 7. 7 “But what if someone else’s app eats up all of my resources?”
- 9. 9
- 10. 10 K A F K A S T R E A M S D O C K E R “ H e y , C o n f l u e n t c r e a t e d t h i s r e a l l y c o o l n e w s t r e a m i n g d a t a f r a m ewo r k w e s h o u l d t r y o u t . ” “ Yo u k n o w w h a t w e s h o u l d d o ? R U N O U R S T R E A M I N G A P P S O N D O C K E R ! ” HOW (NOT?) TO SET UP A STREAMING PLATFORM
- 11. 11 Docker images made it easy for us to deploy anywhere very quickly. N O M O R E C H E F ! ( W E L L , M O S T L Y . )
- 12. 12 Isolated containers freed us from systemwide requirements.
- 13. 13 Build-time arguments made our deploy setup faster and more flexible.
- 14. 14 Docker also helped with a Kafka Connect issue: rebalancing.
- 15. 15 Adding a new connector to a KC cluster causes a rebalance of all workers.
- 16. 16
- 17. 17
- 18. 18 K A F K A S T R E A M S D O C K E R C O N F U S I O N “ H e y , C o n f l u e n t c r e a t e d t h i s r e a l l y c o o l n e w s t r e a m i n g d a t a f r a m ewo r k w e s h o u l d t r y o u t . ” “ Yo u k n o w w h a t w e s h o u l d d o ? R U N O U R S T R E A M I N G A P P S O N D O C K E R ! ” “ H o w d o e s K a f k a S t r e a m s w o r k ? W h a t ’s h a p p e n i n g i n s i d e m y a p p ? H ow d o yo u eve n D o c ke r ? ” HOW (NOT?) TO SET UP A STREAMING PLATFORM
- 19. 19 GAMEDAY Docker and the JVM
- 20. 20 GAMEDAY (n): All your coworkers get together to try and make your brand new system fall over.
- 21. 21 It’s great for testing the resilience of your system...
- 22. 22 It’s great for testing the resilience of your system... But it also becomes a test of its viability.
- 23. 23 High latency / Slow recovery = NOT VIABLE
- 24. 24
- 25. 25 KA FKA STRE AMS
- 26. 26 KA FKA STRE AMS DO CKER SWAR M
- 27. 27 8 replicas Multithreaded > 1 billion records / day
- 28. 28
- 29. 29 STEP 1: Kill all but the Swarm manager.
- 30. 30 STEP 1: Kill all but the Swarm manager. STEP 2: Wait for Swarm to restart replicas.
- 31. 31 1 replica + Lots of messages + Lots of processing
- 32. 32 “It is better to fail in originality than to succeed in imitation.” H e r m a n M e l v i l l e
- 33. 33 After we restarted all the replicas, lag continued to stagnate.
- 34. 34 Record lag now numbered in the hundreds of millions.
- 35. 35 “It is better to fail in originality than to succeed in imitation.” H e r m a n M e l v i l l e
- 36. 36 “It is better to fail in originality than to succeed in imitation.” H e r m a n M e l v i l l e
- 37. 37 “It is better to fail in originality than to succeed in imitation.” H e r m a n M e l v i l l e
- 38. 38 In the end, a simple `docker stats` revealed the problem:
- 40. 40 Why do we want to run JVMs inside Docker containers?
- 41. 41 So we can pack lots of them onto one server ... Image credit: datadoghq.com
- 42. 42 So we can pack lots of them onto one server ... and take advantage of Docker’s resource constraints. Image credit: datadoghq.com
- 43. 43 The JVM doesn’t care that you’ve set resource constraints.
- 44. 44 It’s not cgroup aware. P R O J E C T D E S C R I P T I O N Image credit: mairin.wordpress.com
- 45. 45 JVM ERGONOMICS Th e p r o ce s s by w h i c h t h e J V M t u n e s i t s e l f d e p e n d i n g o n i t s e nv i r o n m e n t . G A R B A G E C O L L E C T I O N I N I T I A L H E A P S I Z E M A X I M U M H E A P S I Z E R U N T I M E C O M P I L E R
- 46. 46 It can be easy to run a single JVM on a server-class machine with minimal tuning.
- 47. 47 But how does the JVM behave when you combine ergonomics with cgroups?
- 48. 48 We’ve established that my apps were suffering due to memory constraints.
- 49. 49 But they weren’t being reaped by the OOM killer… Image credit: turnoff.us
- 50. 50 But they weren’t being reaped by the OOM killer… They were just running incredibly slowly. Image credit: turnoff.us
- 51. 51 This is generally not something you want to see.
- 52. 52 Limiting memory like this allows you 2x as much swap...
- 53. 53 ... while this constrains your swap limit to 300M. Limiting memory like this allows you 2x as much swap...
- 54. 54 Tune your JVMs or you might break your system by deploying to a different server.
- 55. 55 WHAT HAVE WE LEARNED SO FAR? L E S S O N O N E Your Kafka Streams app is just a regular Java app. Tune settings carefully and intentionally so you can take full advantage of containerization.
- 57. 57 My manager came to me with an urgent proposal.
- 58. 58 T R E N D I N G S E A R C H E S O N E T S Y
- 59. 59 RECREATE THIS... USING STREAMING DATA. T R E N D I N G S E A R C H E S O N E T S Y
- 61. 61 A company is an association or collection of individuals, whether natural persons, legal persons,. Y O U R T I T L E H E R E A company is an association or collection of individuals, whether natural persons, legal persons,. Y O U R T I T L E H E R E A company is an association or collection of individuals, whether natural persons, legal persons,. Y O U R T I T L E H E R E VALUE NAME 1 Y O U C A N W R I T E H E R E E n t r e p r e n e u r i a l a c t i v i t i e s d i f f e r s u b s t a n t i a l l y d e p e n d i n g o n t h e t y p e o f o r g a n i z a t i o n
- 62. 62 The aggregation worked ... but the interactive queries piece didn’t.
- 63. 63 Sometimes I would see data…
- 64. 64 ... and sometimes I would see this. Sometimes I would see data… *Etsy’s 404 image.
- 65. 65 It didn’t seem to be following any sort of pattern.
- 67. 67 At this point I was starting to tear my hair out...
- 68. 68 …and then I decided to RTFM.
- 69. 69 This was the code for the RPC layer. P R O J E C T D E S C R I P T I O N
- 70. 70 This was the Docker ser vice definition. P R O J E C T D E S C R I P T I O N
- 71. 71 Can you guess what we did wrong?
- 72. 72
- 73. 73
- 74. 74
- 76. 76 Overlay network: More overhead, higher latency
- 77. 77 Host network: Lower latency, less reusability
- 78. 78
- 80. 80 WHAT HAVE WE LEARNED SO FAR? L E S S O N O N E Your Kafka Streams app is just a regular Java app. Tune settings carefully and intentionally so you can take full advantage of containerization. L E S S O N T W O Networking in containers is not always straightforward. It adds an extra layer of complexity to your application deployment.
- 81. 81 K A F K A S T R E A M S D O C K E R C O N F U S I O N “ H e y , C o n f l u e n t c r e a t e d t h i s r e a l l y c o o l n e w s t r e a m i n g d a t a f r a m ewo r k w e s h o u l d t r y o u t . ” “ Yo u k n o w w h a t w e s h o u l d d o ? R U N O U R S T R E A M I N G A P P S O N D O C K E R ! ” “ H o w d o e s K a f k a S t r e a m s w o r k ? W h a t ’s h a p p e n i n g i n s i d e m y a p p ? H ow d o yo u eve n D o c ke r ? ” HOW (NOT?) TO SET UP A STREAMING PLATFORM
- 82. 82 K A F K A S T R E A M S D O C K E R C O N F U S I O N “ H e y , C o n f l u e n t c r e a t e d t h i s r e a l l y c o o l n e w s t r e a m i n g d a t a f r a m ewo r k w e s h o u l d t r y o u t . ” “ Yo u k n o w w h a t w e s h o u l d d o ? R U N O U R S T R E A M I N G A P P S O N D O C K E R ! ” “ H o w d o e s K a f k a S t r e a m s w o r k ? W h a t ’s h a p p e n i n g i n s i d e m y a p p ? H ow d o yo u eve n D o c ke r ? ” HOW (NOT?) TO SET UP A STREAMING PLATFORM L E A R N A L L T H E T H I N G S !
- 83. 83 WHAT HAVE WE LEARNED SO FAR? L E S S O N O N E L E S S O N T H R E E Learning two things at the same time is hard! Docker will change how you think about deployment and failure recovery. Your Kafka Streams app is just a regular Java app. Tune settings carefully and intentionally so you can take full advantage of containerization. L E S S O N T W O Networking in containers is not always straightforward. It adds an extra layer of complexity to your application deployment.
- 84. 84 In the end, running Kafka Streams on Docker turned out just fine.
- 85. 85 Working out the best way to use Docker took by far the most time.
- 86. 86 WHAT HAVE WE LEARNED SO FAR? L E S S O N O N E L E S S O N T H R E E L E S S O N F O U R Learning two things at the same time is hard! Docker will change how you think about deployment and failure recovery. Your Kafka Streams app is just a regular Java app. Tune settings carefully and intentionally so you can take full advantage of containerization. Make sure you’re using Docker for the right reasons! Otherwise, it’s not worth the time. L E S S O N T W O Networking in containers is not always straightforward. It adds an extra layer of complexity to your application deployment.
- 87. 87 If you’re thinking about running your Streams apps in containers...
- 89. 89 Do you have a good use case for Docker?
- 90. 90 M U LT I P L E D E P L O Y S E T U P S ( F L E X I B L E I M A G E S )
- 91. 91 M U LT I P L E D E P L O Y S E T U P S ( F L E X I B L E I M A G E S ) R U N N I N G AT L A R G E S C A L E ( R E S O U R C E C O N S T R A I N T S )
- 92. 92 Do you have a good use case for Docker? Do you have an existing Docker setup?
- 93. 93 M U LT I P L E D E P L O Y S E T U P S ( F L E X I B L E I M A G E S ) R U N N I N G AT L A R G E S C A L E ( R E S O U R C E C O N S T R A I N T S ) I N T E G R AT I N G W I T H E X I S T I N G D O C K E R S E T U P F O R M I C R O S E R V I C E S
- 94. 94 Do you have a good use case for Docker? Do you have an existing Docker setup? Do you have Docker expertise on your team?
- 95. 95 Do you have a good use case for Docker? Do you have an existing Docker setup? Do you have Docker expertise on your team? Or were you already running Kafka Streams in production?
- 96. 96 Even if the answer to all these questions is no...
- 97. 97 Don’t let me stop you.
- 98. 98 “It is better to fail in originality than to succeed in imitation.” H e r m a n M e l v i l l e
- 99. 99 FURTHER QUESTIONS? [email protected] E M A I L @NikkiThean T W I T T E R http://bit.ly/2weFcJz L I N K E D I N