Pythian: My First 100 days with a Cassandra Cluster
4 likes2,217 views
The document is a presentation from Cassandra Summit 2015 by Gustavo René Antúnez and Carlos Rolo, detailing their experiences and insights regarding Cassandra, a NoSQL database. It covers the architecture, installation, data management, and operational aspects of Cassandra, emphasizing its distributed nature and the challenges of managing data across nodes. Additionally, it compares Cassandra with traditional relational databases, outlines key features, and provides best practices for database administrators.
Pythian: My First 100 days with a Cassandra Cluster
- 1. My First 100 days with a Cassandra Cluster Presented by : Gustavo René Antúnez DBA Team Lead Carlos Rolo Cassandra MVP September, 2015
- 2. 2 Welcome to Cassandra Summit 2015
- 3. • 18 Years of Data infrastructure management consulting • 200+ Top brands • 6000+ databases under management • Over 400 DBA’s, in 35 countries • Top 5% of DBA work force, 9 Oracle ACE’s, 2 Microsoft MVP’s, 1 Cassandra MVP • Oracle, Microsoft, MySQL, Datastax partners, Netezza, Hadoop and MongoDB plus UNIX Sysadmin and Oracle apps About Pythian
- 4. Where does René come from – Oracle DBA • Started with Version 9.2 in 2004 – Speaker at Oracle Open World, Developers Day and Collaborate – APress Q1 2016: “Prac%cal Data Refresh” – Movie Fanatic & Music Lover – Bringing the best from México (Mexihtli) to the rest of the world and in the process photographing it :) – rene-‐ace.com – @rene_ace 4
- 5. Where does Carlos come 5 • Cassandra Consultant • First contact was 0.8 • Cassandra MVP & DataStax Certified Architect • Lisbon Cassandra Meetup • Passion for distributed systems • Loves a good challenge • Waterpolo is my sport • @cjrolo
- 6. How did you get to be a DBA 6
- 7. 6th Happiest Job of 2015! 7 http://www.forbes.com/sites/susanadams/2014/03/20/the-happiest-and-unhappiest-jobs-in-2014/ Work-life balance Relationship with boss and co-workers Daily tasks Job resources Field will grow by 15% between 2012 and 2022 DBA can be the key driver of success
- 8. Happiest Job of 2034? Oxford University: THE FUTURE OF EMPLOYMENT: HOW SUSCEPTIBLE ARE JOBS TO COMPUTERISATION? • 47 percent of American jobs are at high risk of being taken by computers within the next two decades. – 1st Wave • Computers will start replacing people in especially vulnerable fields like transportation/logistics, production labor, and administrative support. – 2nd Wave • Dependent upon the development of good artificial intelligence. This could next put jobs in management, science and engineering, and the arts at risk. 8
- 9. What is Cassandra ? • NoSQL database, developed in JavaOne • Fully distributed DB • Meaning that there is no master DB, unlike Oracle or MySQL. • Linearly scalable • Based on 2 core technologies, Google’s Big Table and Amazon’s Dynamo • 2 versions of Cassandra • Community Edition.-‐ This is distributed under the Apache™ License • Enterprise Edition .-‐ This is distributed by Datastax 9 ≠
- 10. CAP Theorem • In a distributed system you can only have two out of the following three guarantees across a write/read pair: • Consistency.-‐ A read is guaranteed to return the most recent write for a given client. • Availability.-‐A non-‐failing node will return a reasonable response within a reasonable amount of time (no error or timeout). • Partition Tolerance.-‐The system will continue to function when network partitions occur. 10 N1 N2 X X N1 N2 N1 N2 What is Cassandra ?
- 11. What is Cassandra ? • Cassandra is a BASE (Basically Available, Soft state, Eventually consistent) type system 11 • Not an ACID (Atomicity, Consistency, Isolation, Durability) type system
- 12. It Can be as easy as … • Start your machine and install the following: • ntp (Packages are normally ntp, ntpdata and ntp-‐ doc) • wget (Unless you have your packages copied over via other means) • vim (Or your favorite text editor) • Yum Package Management • Root or sudo access to the install machine • Latest version of Oracle Java SE Runtime Environment (JRE) 8 (recommended) or OpenJDK 7. • Python 2.6+ (needed if installing OpsCenter) 12
- 13. It Can be as easy as … 13 • Install Cassandra. ~$ sudo yum install dsc21-2.1.5-1 cassandra2.1.5-1 • Install optional utilities. ~$ sudo yum install cassandra21-tools-2.1.5-1 • Start Cassandra service ~$ sudo service cassandra stop ~$ sudo rm -rf /var/lib/cassandra/data/system/* • In the cassandra-‐rackdc.properties file # indicate the rack and dc for this node dc=Pythian rack=RAC1 ~$ sudo service cassandra start
- 14. Where is everything in Cassandra? 14 Directories Description /var/lib/cassandra Data directories /var/log/ cassandra Log directory /var/run/ cassandra Runtime files /usr/share/ cassandra Environment settings /usr/share/ cassandra/ lib JAR files /usr/bin Optional utilities, such as sstablelevelreset, sstablerepairedset, and sstablesplit /usr/bin Binary files /usr/sbin /etc/cassandra Configuration files /etc/init.d Service startup script /etc/security/ limits.d Cassandra user limits /etc/default /usr/share/ doc/ cassandra/examples Sample cassandra.yaml files for stress testing
- 15. I come from this world… 12c Version Architecture… 15
- 16. I come from this world… Oracle… 16 101010 Online Redo Log10100 Data Files Control Files Segment Database Tablespace Extent Oracle data block Schema Data file OS block Logical Datafile Physical Datafile
- 17. I come from this world… 17 RAC -‐ For Node Point of Failure RAC Cluster Node3Node2 ASM Disks Node1 Public Network Storage Network ASM Network CSS Network ASM ASM ASM DBB DBBDBB Global Data Services – Service Failover / Load Balancing
- 18. I come from this world… 18 Dataguard -‐ For Failover Primary Standby Far Sync Instance SYNC ASYNC Zero data loss failover
- 19. Cassandra Architecture Cassandra Cluster 19 N1 Node N2 Node Rack 1 Datacenter México N3 Node N4 Node Rack 2 Datacenter Portugal
- 20. One Ring to Rule them All 20 • The total amount of data managed by the cluster is represented as a ring • Each node is assigned a part of the database to hold based on each table’s primary key. • To guarantee both availability and durability multiple nodes will be assigned to the same data. • There is no master node all nodes can perform all operations 1 4 3 2 A-F,T-Z,M-S G-L,A-F,T-Z M-S,G-L,A-F T-Z,M-S,G-L
- 21. Gossip 21 • Peer-‐to-‐peer communication protocol in which nodes periodically exchange state information • Runs every second and exchanges state messages with up to three other nodes in the cluster • Failure detection • It determines locally from gossip state and history if another node in the system is down or has come back up.
- 22. Consistent Hashing 22 • A hash consists of one or more arithmetic operations on a piece of data • Common way of load balancing across several nodes • Hash function must have a upper and lower bound so objects can be mapped in a circle • Common Hash algorithms – Simple checksums – Message Digest (MD5) – Secure Hash Algorithm (SHA-‐1/2) – MurmurHash
- 23. Partitioners 23 • Determines how data is distributed across the nodes in the cluster • Function for deriving a token representing a row from its partition key Cassandra Offers: – Murmur3Partition – RandomPartitioner – ByteOrderedPartitioner
- 24. Virtual Nodes 24 • Solution for avoiding calculating node tokens and thinking about the cluster size before hand • Each node has multiple virtual nodes • Each node virtual node own a much smaller subset of data
- 25. Coordinators 25 • Acts as a proxy between the client application and the nodes that own the data being requestedAny client request can be sent to any node.
- 26. Snitch 26 • Is responsible for keeping all of the nodes up to date on what node has what data, what nodes are currently down, what nodes are bootstrapping, etc. • It Interprets the topology The most popular are: – Gossiping property file snitch – EC2 Snitch – EC2 Multi-‐region snitch – Dynamic Snitch
- 27. 27
- 28. Logical database container 28 Data is Stored in Keyspaces
- 29. A CASSANDRA TABLE OR COLUMN FAMILY 29 Coordinator Snitch Commitlog Writer Mem table writer Mem Table Flush (Sstable writer) Reader Mem tables Bloom Filters Cassandra Node CommitLog 10100 SSTables
- 30. A CASSANDRA TABLE OR COLUMN FAMILY 30 • Consists of one or more SStables and 0 or more MEMtables • SStable stands for Sorted String Table. • E.G. all of the Columns in the SStable are sorted in order by key. • Each SStable consists of the data table, bloom filter, index and some other minor files. • SStables are immutable. Once written they are never altered only read and eventually deleted videogames-events-data-jb-1.db videogames-events-filters-jb-1.db videogames-events-index-jb-1.db videogames-events-data-jb-2.db videogames-events-filters-jb-2.db videogames-events-index-jb-2.db videogames-events-data-jb-3.db videogames-events-filters-jb-3.db videogames-events-index-jb-3.db videogames-events-data-jb-4.db videogames-events-filters-jb-4.db videogames-events-index-jb-4.db SStables on disk /var/lib/cassandra
- 31. REPLICATION FACTOR (RF) AND CONSISTENCY 31 • Replication Factor is the number of copies of columns stored in the ring • Replication factor should not exceed the number of nodes in the cluster – RF=1 is one copy this means that the data for each column is stored only once in the ring. – RF=3 (default) means every column stored in the database is stored three times. – Quorum .-‐ The read and write must be acked/returned from a quorum of nodes.
- 32. REPLICATION FACTOR (RF) AND CONSISTENCY 32 • Consistency – When write or read is performed the application can choose to wait for n copies of the data to be written or read this is referred to as consistency of n. – There is a special consistency value called quorum which means a response from RF/2+1 nodes is required.
- 33. HOW TO MAKE SURE WE DON’T LOOSE DATA 33 • Three anti-‐entropy mechanisms in Cassandra 1) Hinted handoff 2) Read repair 3) Repair A.K.A. Anti-‐Entropy
- 34. WRITE PATH 34
- 35. COMPACTIONS 35 • SStables are immutable. • Deletes and updates are just new writes • SStables are merged together by partitioned key.Old obsolete data is discarded. • Lots of SStables become a few. • Compaction can require a lot of disk space. DO NOT LET your disks get more than 50% full.
- 36. CQL - Cassandra Query Language 36 CQL is not SQL • Default and primary interface into the Cassandra Database (since 2.0) • Cassandra does not support joins or subqueries • Only way to create users and user based permissions • Very similar: cqlsh> CREATE KEYSPACE sandbox WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', DC1 : 1}; cqlsh> USE sandbox; cqlsh:sandbox>CREATE TABLE data (id uuid, data text, PRIMARY KEY (id)); cqlsh:sandbox> INSERT INTO data (id, data) values (c37d661d-7e61-49ea-96a5-68c34e83db3a, 'testing'); cqlsh:sandbox> SELECT * FROM data;
- 37. 37
- 38. 38 Feature/Function DSE/Cassandra Oracle RDBMS Core architecture “Masterless”; peer-to-peer with all nodes being the same Traditional standalone High availability Continuous availability with built in redundancy and hardware rack awareness in both single and multiple data centers Oracle Dataguard (for failover) and Oracle RAC (Node SPOF) GoldenGate Data model Google Bigtable Relational/tabular Data consistency model Tunable consistency (CAP theorem consistency per operation Traditional ACID Storage model Targeted directories with separation Tablespaces Logical database container Keyspace Database Backup/recovery Online, point-in-time restore Online, point-in-time restore Enterprise management/ monitoring DataStax OpsCenter Oracle Enterprise Manager
- 39. LESSONS LEARNED 39 • Understand the Data Model Differences • Hardware Setup does Matter • Grep the logs for errors and warnings • Make sure each node is created properly • Know your tools • nodetool utility • Cassandra bulk loader (sstableloader) • jconsole/JavaVisualVM • Cassandra-‐Stress • OpsCenter
- 40. 40
- 41. FIT-ACER • F – Focus (SLOW DOWN! Are you ready?) • I – Identify server/DB name, time, authorization • T – Type the command (do not hit enter yet) • A – Assess the command (SPEND TIME HERE!) • C – Check the server / database name again • E – Execute the command • R – Review and document the results 41
- 42. 42 rene-ace.com
- 43. 43 To contact us [email protected] 1-877-PYTHIAN To follow us http://www.pythian.com/blog http://www.facebook.com/pages/The-Pythian-Group/163902527671 @pythian http://www.linkedin.com/company/pythian Thank you – Q&A