Cassandra Day SV 2014: Basic Operations with Apache Cassandra
14 likes3,184 views
Operations and tuning for Cassandra involve: 1) Ensuring a good data model first before trying to optimize operations, as a bad model cannot be fixed by operations alone. 2) Sizing for latency and operations depends on factors like CPU, memory, disk type and replication factor, with SSDs offering much faster performance than mechanical disks. 3) Various tuning techniques are described like disabling access time, warming the buffer cache, using SSDs, adjusting read ahead and schedulers on SSDs, choosing appropriate compaction strategies, and Cassandra heap size settings.
Cassandra Day SV 2014: Basic Operations with Apache Cassandra
- 2. Data Model First Operations Can’t Fix a Bad Data Model
- 4. Sizing for Latency CPU Buffer Cache Disk JVM
- 5. Sizing for Operations (C * Nodes * Cores) / Replication Factor = Ops/Second C is a hardware constant! • 3000 AWS Non-SSD! • 4000 Commodity SAS Hardware! • 12000 Commodity SSD Hardware
- 6. Sizing for Latency • 30-130 microseconds in memory! • 100 microseconds to 12 milliseconds for SSD! • 9 milliseconds to 60 milliseconds for mechanical disks
- 7. Shared Storage DO NOT USE SHARED STORAGE!!!!!
- 8. Disable Access Time /dev/xvdb /data1 auto defaults,noatime,nodiratime 0 2
- 9. Warm The Buffer Cache ➜ ~ find /var/lib/cassandra -name '*.db' -exec cat {} > /dev/null ;
- 10. Disable NUMA Zone Reclaim ➜ ~ echo 0 > /proc/sys/vm/zone_reclaim_mode
- 11. Use SSD 25% more expensive! 10x Faster
- 12. Tuning SSD: Read Ahead ➜ ~ sudo blockdev --report RO RA SSZ BSZ StartSec Size Device rw 256 512 4096 0 6442450944 /dev/xvda rw 256 512 4096 1920 6144000000 /dev/xvda1 rw 8 512 4096 0 343568023552 /dev/xvdb rw 256 512 4096 0 343568023552 /dev/xvdc ➜ ~ sudo blockdev --setra 8 /dev/xvdb
- 13. Tuning SSD: Scheduler ➜ ~ echo 0 > /sys/block/xvdb/queue/rotational ➜ ~ echo noop > /sys/block/xvdb/queue/scheduler
- 14. Compaction Mechanical Disk = Size Tiered! SSD = Leveled Compaction
- 16. Cassandra Heap Settings 8G MAX_HEAP ! 2G NEW_HEAP
- 17. Solr/Cassandra Heap Settings 14G MAX_HEAP ! 2G NEW_HEAP
- 18. Hadoop/Cassandra Heap Settings 10G MAX_HEAP ! 2G NEW_HEAP
- 19. Kernel Modern Kernel = 30% performance boost
- 20. Monitoring
- 21. nodetool cfhistograms Offset SSTables Write Latency Read Latency Row Size Column Count 1 3579 0 0 0 0 2 0 0 0 0 0 . . . 35 0 0 0 0 0 42 0 0 27 0 0 50 0 0 187 0 0 60 0 10 460 0 0 72 0 200 689 0 0 86 0 663 552 0 0 103 0 796 367 0 0 124 0 297 736 0 0 149 0 265 243 0 0 179 0 460 263 0 0 . . . 25109160 0 0 0 0 0
- 22. nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 0 15 0 0 RequestResponseStage 0 0 0 0 0 MutationStage 0 0 3674 0 0 ReadRepairStage 0 0 0 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage 0 0 0 0 0 MemoryMeter 0 0 1 0 0 MemtablePostFlusher 0 0 267 0 0 FlushWriter 0 0 9 0 1 MiscStage 0 0 0 0 0 commitlog_archiver 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 0 0 0 0 0 ! Message type Dropped RANGE_SLICE 0 READ_REPAIR 0 BINARY 0 READ 0 MUTATION 0 _TRACE 0 REQUEST_RESPONSE 0
- 23. Warning Signs • Pending compactions > 15! • Parnew collections > 1 second! • CMS collections > 5 minutes! • Disk utilization above 40% or 70%! • CFHistograms write throughput > 150 μs! • CFHistograms read throughput > 50 ms! • Really large rows in CFHistograms! • Anything blocked in TPStats
- 24. Tools