Cassandra and Docker Lessons Learned
59 likes40,539 views
The document discusses the integration of Docker with Cassandra, emphasizing their compatibility and the advantages of using Docker for deployment and development. It covers Docker's functionality, how it helps manage dependencies, and addresses challenges like networking, filesystem, process capabilities, and signal handling. The author also shares tips for utilizing Docker in development environments for Cassandra to optimize performance and stability.
Cassandra and Docker Lessons Learned
- 1. Cassandra and Docker Two buzzwords or a match made in heaven? instaclustr.com @Instaclustr
- 2. Who am I and what do I do? • Ben Bromhead • Co-founder and CTO of Instaclustr -> www.instaclustr.com <sales> • Instaclustr provides Cassandra-as-a-Service in the cloud. • Currently in AWS, Azure and Google Cloud in private beta with more to come. • We currently manage 50+ nodes for various customers, who do various things with it. </sales>
- 3. Objectives • A quick intro on docker. • Why docker matters and how it works. • Working with Cassandra and docker. • Running C* in a constrained env w/ docker
- 4. The Challenge Static website Web frontend DB Queue Background workers API endpoint nginx 1.5 + modsecurity + openssl + bootstrap 2 Java + Cassandra + Spark Ruby + Rails + sass + Unicorn Redis + redis-sentinel Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client DevelopmentVM QA server Public Cloud Disaster recovery Contributor’s laptop Production Servers Mul$plicity*of*Stacks* Mul$plicity*of* hardware* environments* Production Cluster Customer Data Center Do*services*and*apps* interact* appropriately?* Can*I*migrate* smoothly*and* quickly?*
- 5. Static website Web frontend Background workers DB Analytics Queue Developmen t VM QA Server Single Prod Server Onsite Cluster Public Cloud Contributor’ s laptop Customer Servers ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Dependency madness
- 7. Also a dependency mess ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
- 8. Solution: Intermodal Shipping containers Mul$plicity*of*Goods* Mul$plicity*of* methods*for* transpor$ng/storing* Do*I*worry*about* how*goods*interact* (e.g.*coffee*beans* next*to*spices)* Can*I*transport* quickly*and*smoothly* (e.g.*from*boat*to* train*to*truck)* …in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another! A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery.!
- 9. Docker, shipping containers for code Static website Web frontendUser DB Queue Analytics DB Development VM QA server Public Cloud Contributor’s laptop Mul$plicity*of*Stacks* Mul$plicity*of* hardware* environments* Production Cluster Customer Data Center Do*services*and*apps* interact* appropriately?* Can*I*migrate* smoothly*and*quickly* …that can be manipulated using standard operations and run consistently on virtually any hardware platform ! An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…!
- 10. Solves the deployment dependency matrix Static website Web frontend Background workers User DB Analytics DB Queue Developmen t VM QA Server Single Prod Server Onsite Cluster Public Cloud Contributor’ s laptop Customer Servers
- 11. Why docker matters • Finally Developers have a solution to build once and deploy anywhere • Finally Ops/Admin has a solution to configure anywhere • Finally DevOps is easy • Dev == Test == Staging == Production • Move with speed
- 12. Docker, how it works. • Runs anywhere (Linux kernel 2.6.32+) • Uses lightweight VMs: • Own process space (namespace) • Process isolation and resource control (cgroups) • Own network adapter • Own filesystem (chroot) • Linux Analog to Solaris Zones, *BSD jails
- 13. Docker, how it works. • Difference between a container and a VM Virtual Machine Container
- 14. Docker, how it works. • What about the packaging component? • Uses Union filesystem to create a git like workflow around your deployed code: ! ! Docker! Container! Image! Registry! Push% ! ! ! ! Bins/! Libs! ! ! ! ! App! A! App!Δ!! ! ! ! ! Bins/! Docker'Engine' Docker'Engine' Update' Host'is'now'running'A’’' ' App'Δ'' ' ' ' ' Bins/' ' ' ' ' Bins/' Libs' ' ' ' ' App' A' ' ' ' ' Bins/' ' ' ' ' Bins/' Libs' ' ' ' ' App' A’’' Host'running'A'wants'to'upgrade'to'A’’.' Requests'update.'Gets'only'diffs' '
- 15. Docker is it production ready?
- 16. Cassandra and Docker • So how do we get on board the hype train? Without killing performance or stability? • Build Cassandra in a docker container, run it, then test. • Run in dev to get comfortable with it. • Talk to others who use it in production • https://github.com/docker/docker/issues - You will spend a lot of time here
- 17. Docker + Networking • 1st attempt, throughput dropped in half! • Writes sucked, streaming sucked, what was going on? • Quick check with iperf showed a 50% hit in throughput • Docker uses Linux Ethernet Bridges for basic software defined routing. This will hose your network throughput. • Use the host network stack instead (—net=host), only saw a ~10% hit on network performance
- 18. Docker + Networking • Docker uses Linux Ethernet Bridges for basic software defined routing. This will hose your network throughput. • Use the host network stack instead (—net=host), only saw a ~10% hit on network performance • Also solves NAT issues in an AWS like networking environment.
- 19. Docker + Filesystem • Don’t want to throw it out when you upgrade/stop container. • Use volume mount folders to the underlying host!
- 20. Docker + Filesystem • The filesystems (AUFS, BTRFS etc) that bring great benefits to Dockers workflow around building and snapshoting containers are not very good for databases. • UnionFS (AUFS) is terrible for writing lots of big files. • BTRFS is a pain to use from an ops point of view. • Hooray volume mounts use the underlying filesystem.
- 21. Docker + Process Capabilities • Mlockall permission denied? A process needs CAP_IPC_LOCK & RLIMIT_MEMLOCK in order to perform this operation. By default docker doesn't assign this to a running container… • Can’t use native memory. Cassandra becomes slooow. • Can use --privileged and be done with it. Kind of lazy though • Use --cap-add instead
- 22. Docker + SIGTERM propagation • When stopping the process docker will send a SIGTERM. • PID 1 does not have default signal handlers! • Bad if you use a bash script to launch Cassandra Images shameless copied from https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
- 23. Docker + SIGTERM propagation • Java to the rescue! • Make sure you run the cassandra bash script with -f (foreground) • exec causes the JVM to replace the bash process… making the world a happier place
- 24. Docker + SIGTERM propagation • Tools like OpsCenter Server will have trouble with this. • Can be fixed using a wacky combination of trap and wait stanzas in your OpsCenter Server script (see http://veithen.github.io/ 2014/11/16/sigterm-propagation.html) • But now you have a bash script that duplicates init/systemd/ supervisord • The debate rages on…
- 25. Docker + CoreOS • Docker + fav OS + CM?, CoreOS + etcd, Swarm + Machine, Deis etc • We chose CoreOS (Appeared to be sane, etcd is cool, systemd if you are into that kind of thing)
- 26. Docker + CoreOS • Disable automatic updates + restarts (seriously do this) • Fix logging, otherwise you will log to 3 locations (/var/log/ cassandra, journalctl and dockers json based log • JVM will exit with error 143 (128 + 15 for SIGTERM). Need to ignore that in your systemd service definition.
- 27. Docker + Dev Env • Docker relies on Linux kernel capabilites… so no native docker in OS X • We use OSX for dev, so we run vagrant and the CoreOS vagrant file https://github.com/coreos/coreos-vagrant • Look at https://github.com/tobert/cassandra-docker for something more off the shelf
- 28. Docker + C* + Dev Env • How do I run lots of C* instances on a VM or my dev laptop without it falling over? • Make it run as slowly, but as stable as possible! • This is actually a great learning exercise as you discover a lot about how Cassandra works under the hood.
- 29. Docker + C* + Dev Env • Set Memory to be super low, edit your cassandra-env.sh: MAX_HEAP_SIZE="128M" HEAP_NEWSIZE="24M"
- 30. Docker + C* + Dev Env • Tune compaction to have free reign and to smash the disk concurrent_compactors: 1 in_memory_compaction_limit_in_mb: 2 compaction_throughput_mb_per_sec: 0
- 31. Docker + C* + Dev Env • Let’s use HSHA thrift server as it reduces the memory per thread used. rpc_server_type: hsha
- 32. Docker + C* + Dev Env • The HSHA server also lets us limit the number of threads serving in flight requests, but still have a large number of clients connected. concurrent_reads: 4 concurrent_writes: 4 rpc_min_threads: 2 rpc_max_threads: 2 • You can play with these to get the right numbers based on how your clients connect, but keep them low.
- 33. Docker + C* + Dev Env • This is Dev! Caches have no power here! key_cache_size_in_mb: 0 reduce_cache_sizes_at: 0 reduce_cache_capacity_to: 0
- 34. Docker + C* + Dev Env • How well does this work?!?! • Will survive running the insane workload in the c* 2.1 new stresstest tool. • We run this on AWS t1.micro instances • Sign up at https://www.instaclustr.com and give our new Developer nodes a spin!
- 35. Go forth and conquer! Questions?