Running 400-node Cassandra + Spark Clusters in Azure (Anubhav Kale, Microsoft) | C* Summit 2016
Download as PPTX, PDF4 likes3,799 views
Anubhav Kale discusses the management of over 400 Cassandra clusters in Azure for Microsoft Office 365, highlighting achievements and user statistics. The document details challenges faced, such as SSD reliability, data streaming slowdowns, and schema update issues, alongside automated solutions implemented to mitigate these problems. It emphasizes the importance of continuous learning and automation in optimizing operations and maintaining performance in a high-scale environment.
Running 400-node Cassandra + Spark Clusters in Azure (Anubhav Kale, Microsoft) | C* Summit 2016
- 1. Anubhav Kale Running 400+ node Cassandra clusters in Azure
- 2. Running 400+ node Cassandra clusters in Azure Anubhav Kale Senior Software Engineer – Microsoft Office 365 [email protected]
- 3. 1 The big picture 2 Our stack 3 Challenges 4 Solutions 5 Path forward 3© DataStax, All Rights Reserved.
- 4. Office 365 – Productivity services at scale ! 1.6 billion – Sessions / month 59% - Commercial seat growth in FY16 Q2 20.6 million - Consumer Subscribers >30 Million – iOS and Android devices run Outlook
- 5. Why ? Are users happy with O365? Are users fully utilizing the services they are paying us for? How do we proactively find issues ? Do we understand our users experience over their lifetime? Linear Scale Fast Ingestion Advanced Analytics
- 6. How, where, what ? Cassandra 2.1.13 running on Azure Linux VMs Apache Kafka as the intermediate queue Multiple Clusters to serve different teams / scale profiles Common management stack for all clusters Home grown internal and external monitoring, recovery Tooling for On Call Activities, Backups et. al. Datastax Ops Center does the heavy lifting
- 7. Architecture Spark Streaming Spark Batch Processing Kafka Cassandra Store O365 servers Apps/Clients Commerce systems Support systems Serving Admin Portal Support Tools Ad Hoc Querying Zeppelin Splunk
- 8. The Cassandra side 10 Clusters - DSE 4.8.5 30 - 400+ nodes (300+ TB) RF: 5 Virtual nodes G1 GC Gossiping-Snitch
- 9. Azure Networking Public IP Addresses Allow geo-redundant replication over Internet Not secure Virtual Networks No bandwidth limit within a VNET, Allow replication via 1. High-Performance Gateway – Max 200Mbs. 2. Express Route – Max 10Gbs 3. VNET Peering (Public Preview) – No Limit We use VNETs due to security requirements and dedicated bandwidth guarantees
- 10. Azure Deployment ARM Templates with post-deployment scripts
- 11. Challenges
- 12. SSDs can vanish SSDs give best latency and IOPs, but they are not persistent. Why is this a problem? • SSDs are ephemeral. When Azure moves VMs, VMs will lose SSDs ! • "A node with address %s already exists, cancelling join. " Use cassandra.replace_address if you want to replace this node." How to fix it? . Restart using ‘–Dcassandra.replace_address=<ip address of node>’ in JVM opts . Don’t forget to remove once the node joins the ring . We built automation to detect and fix this, running continuously on all nodes
- 13. Are you really Rack-aware ? We use Azure Fault Domains for rack-awareness. Why is this a problem? • When Azure moves nodes they can change FD. • This invalidates the rack configured in cassandra-rackdc.properties How to fix it? . Cassandra won’t let you change DC / Racks of existing node. Must remove and add the node. . Automation to detect, and change rackdc.properties as needed
- 14. Streaming is slow “nodetool rebuild” or equivalent is often necessary while replacing / adding new nodes. Why is this a problem? • If the node crashes while streaming, it starts from the beginning • The source node does not transfer SS Table files in parallel How to fix it? . Set “nodetool streamthroughput” and “nodetool compactionthroughput” to 0 . sudo sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10 . Wait for JIRAs 4663 , 9766 to get fixed in your version of DSE !
- 15. We use 30GB heaps, so heap dump files tend to be large on our nodes. Why is this a problem? • A standard Azure OS disk is 30 GB. • The default heap dump dir is /var/lib/cassandra. • We often ran out of OS disk space when Cassandra crashed How to fix it? If you are running Datastax Enterprise: Edit /etc/init.d/dse to add this line at the top: DUMPFILE_DIR=/mnt/dumps Or, just don’t let Cassandra crash, whichever is easiest. Heap dumps are big !!
- 16. Memory ! Running Spark and Cassandra on same node poses unique challenges Why is this a problem? • Bad models / spark code can easily fill up 30 GB heap • Linux Kernel will kill DSE when VM running low on memory • system.log won’t show this. sudo dmesg –T is the way to go ! How to fix it? . Set -1000 as OOM Score in /proc/<pid>/omm_score_adj file . Need automation since the pid will change on DSE process restarts
- 17. SS Table Compactions If not tuned correctly, this will melt down clusters ! Why is this a problem? • IO heavy activity • Incorrect compaction strategy parameters is a time bomb ! • Makes tmp files requiring double the space until it finishes. How to fix it? . Use DTCS only if you can explain what Target.GetBuckets does ! . Use the more stable and easy to reason about thing : STCS . Pay close attention to PendingTasks, PendingCompactions and “nodetool compactionstats”
- 18. Schema Updates If nodes are down while schema changes are made, badness happens !! Why is this a problem? • Schema version changes don’t get updated in saved_caches if a node is down. This leads to “couldn’t find cfid=foo exceptions” in logs • Very easy to repro in debugger • Column add / removes are okay, renaming tables not so much ! How to fix it? . Known problem in community, 3.0 should have a new storage engine . Don’t rename tables. Create new tables, and migrate data. Yes, it sucks !!
- 19. SS Table Corruption SS Tables can get corrupt! Plan for fault tolerance Why is this a problem? • Azure node restarts can cause this. • Shows up as apache.cassandra.io.sstable.CorruptSSTableException: • Node won’t start if disk_failure_policy is set to 0 How to fix it? . “nodetool scrub” doesn’t usually fix such SS Tables. . sudo find /mnt/cassandra/data -type f -path ./system -prune -o -path ./system_auth -prune -o -cmin -<Number of Minutes> -print | sudo xargs rm. . Automation to automatically detect, delete bad tables and restart DSE.
- 20. Mutation Drops Even with reasonable tuning, our nodes were showing mutation drops. LocalMutationRunnable thread wasn’t getting scheduled to run within the configured read and write timeouts. Contributed below as part of diving deep in code. JIRA Description 10866 Expose dropped mutations metrics per table 10605 MUTATION and COUNTER MUTATION using same thread pool 10580 Expose metrics for dropped messages latency
- 21. Backup / Restore With RF = 5 and TBs of data, we need efficient data movement Explored using a Data Center with RF =1 as “Backup DC”. Failed quickly because “restore” was slow ! Built rsync based solution to snapshot and backup periodically to 1 TB HDDs attached to every node. Also lets us restore in staged fashion while taking live traffic https://github.com/anubhavkale/CassandraTools
- 22. Takeaways
- 23. Don’t underestimate learning curve Teach, coach, grow, help, assist your fellow team members ! Quiz : https://blogs.msdn.microsoft.com/anubhavk/2016/08/22/the-cassandra-challenge/ Debug multi-node Cassandra locally You will be surprised often at how things work How: https://blogs.msdn.microsoft.com/anubhavk/2016/08/25/debugging-multi-node-cassandra-cluster-on- windows/ JIRA and Mailing Lists Cassandra Devs are fantastic at explaining things deeply You will find great workarounds there : e.g. CASSANDRA-10371 !
- 24. Looking forward
- 25. Azure Premium Storage Network attacked SSD storage with local SSD cache DS 14 VMs = 550 GB local cache ! Great IOPS and Latency if you RAID disks: Read here and here We added DS VMs to our existing clusters and did not see any performance degradation. Working through more formal tests.
- 26. Questions ?
- 27. Leverage Ops Center Metrics Read and Write Latencies SS Table Counts Tombstone Counts OS : CPU, Memory, Disk Compactions Blocked Tasks Aggressively invest in automation Use Chef or equivalent Local Monitoring and recovery Learn concepts deeply Datastax Enterprise support !
- 28. More … Enable encryption at rest VNET Peering Helps with connecting legacy services deployed on old Azure classic stack with new ARM stack Use Azure HDInsight as Spark compute cluster DSE Spark version is usually behind the industry latest Running Spark + Cassandra on same node makes debugging performance issues tricky Cassandra 3.x Mostly for bug fixes, and general improvements to repairs
- 29. Cleanup !! By design, Cassandra doesn’t delete data from disk if another node starts owning it. Why is this a problem? • When adding nodes to a ring, disk space on old nodes won’t be reclaimed automatically ! • Disk pressure will bring nodes down How to fix it? . “nodetool cleanup” . Safe to run in parallel on all nodes. . Be sure to increase concurrent_compactors if needed.
Editor's Notes
- #7: Compliance requirements a major design decision