Designing Fault-Tolerant Applications with DataStax Enterprise and Apache Cassandra

© DataStax, All Rights Reserved.Confidential
Designing Fault-Tolerant
Applications with
DataStax Enterprise and
Apache Cassandra
1 © DataStax, All Rights Reserved. Confidential

© DataStax, All Rights Reserved.ConfidentialConfidential© DataStax, All Rights Reserved.
Designing Fault Tolerant
Applications
1. Why does it matter?
2. What can be done about it?
3. Let’s see it.

Why does it matter?

Outages Happen
June 17, 2019
CEO apologizes for register outages: 'A tough weekend'
Saturday’s glitch was due to an internal technical issue that caused
registers to stop working in stores nationwide, while
Sunday’s malfunction was due to problems at its vendor’s data
center.
System was down for several hours as employees worked to sort out
the situation on the biggest shopping days of the week … confusion
and long delays at stores across the U.S.
Source: Fox Business

Outages Happen
More than 60 flights to and from Heathrow and Gatwick were
canceled and more than 100 were delayed, according to the
departure boards at the two airports. The problems started when people
tried to check in for the first flights of the day and lasted for about
12 hours.
Airline would not confirm how many people have been affected but
said it had experienced a “systems issue” affecting check-in
and flight departures at Heathrow, Gatwick and London City
airports.
Source: Reuters
August 7, 2019
Airline resuming services after latest IT meltdown

Outages Happen
August 11, 2019
Banking services across Mexico go down due to data center outage
An electronic transaction services firm responsible for processing card
payments, said an electrical fault at its data centre in Santa Fe,
Mexico, was responsible for the outage that affected customers.
Customers were unable to make purchases or withdraw cash
using their credit and debit cards for several hours on Saturday.
Several Mexican media outlets reported chaos in supermarkets as
hapless shoppers were forced to abandon shopping trolleys full of
food.
Source: Techerati

Outages are Expensive
“Analysts also predicted the retailer most likely lost hundreds of millions of dollars in sales
due to the glitches.” - Fox Business
“Said a power outage that led to the cancellation of hundreds of flights last month probably
cost it about 80 million pounds ($102 million) in lost revenue and the expense of
accommodating, re-booking and compensating thousands of passengers” - Bloomberg
“Uptime Institute’s 2018 Data Center Survey polled nearly 1,500 respondents and key
findings revealed that nearly a third of all reported outages cost more than $250K; 41
respondents reported a single outage cost over $1M; and one specific incident cost over
$50M”

What is the root cause of these outages?

Network

Network Config
change

Network Config
change
Maintenance

Network Config
change
Maintenance
Humans!

Why does it matter?
Designing fault tolerant applications matters because outages
happen and they are expensive
… and the incidents show up in the news

What can be done about it?

Infrastructure Terminology
Cloud
Instances
Availability Zones (AZ)
Regions

On-Premises
Servers
- Maps to Cloud concept of Instances
Racks
- Maps to Cloud concept of AZ
Physical Data Centers
- Maps to Cloud concept of Regions
Physical Data Center
Infrastructure Terminology
Virginia California
Physical Data Center

Cassandra Concepts
Logical Groupings
Nodes
Racks
- controls placement of data replicas
within a data center / region
Data Centers
- controls placement of data replicas
across data centers / regions
Clusters
- encompases one to many data centers
Configured in cassandra-rackdc.properties
dc=dc1
rack=rack1example

Cassandra Concepts
Masterless

Cassandra Concepts
Data Distribution
Replication Factor
- Defines number of replicas for a single row
- Defines the data centers in which the data
should live
- Using “tokens”, Cassandra will evenly
distribute the data replicas across the logical
racks for each data center
CREATE KEYSPACE shopping WITH REPLICATION = {'class':'NetworkTopologyStrategy','dc1': 3,'dc2': 3};
allocate_tokens_for_local_replication_factor

Cassandra Concepts
Data Distribution
Replication Factor
should live
To protect against single
node outage, have
multiple replicas

Cassandra Concepts
Data Distribution
Replication Factor
should live
To protect against
availability zone outage,
distribute data across
multiple racks or AZs
node outage, have
multiple replicas

Cassandra Concepts
Data Distribution
Replication Factor
should live
To protect against region
outage, distribute data
across multiple physical
data centers or regions
To protect against
availability zone outage,
distribute data across
multiple racks or AZs
node outage, have
multiple replicas

Cassandra Concepts
Consistency Levels
- The number of replicas that need to acknowledge the read or write operation success to
the coordinator of the query.
- Tunable consistency model, trade-off between availability and data consistency

Cassandra Concepts
Consistency Levels
Local Data Center
- Best practice to pin driver instances to a single data center using the DCAwareRoundRobinPolicy
- Affects the coordinator selection and group of nodes that will need to respond for LOCAL consistency
levels ( LOCAL_ONE, LOCAL_QUORUM )
- The driver creates connection pools to each node in the local data center
DCAwareRoundRobinPolicy.builder().withLocalDc(“us-east-1”)

Cassandra Concepts
Consistency Levels
LOCAL_ONE
- A single replica in the local data center received the write and
only requires a single local node to be available and to confirm the
write for a request to succeed.
- The coordinator of the query will send the write request to replicas
in all data centers, the difference relies on whether the coordinator
should wait for remote replicas to acknowledge the write for the
operation to succeed.
- For RF=3 … LOCAL_ONE is 1 replica

Cassandra Concepts
Consistency Levels
LOCAL_QUORUM
- Majority of replicas in the local data center received the write and
only requires those local nodes to be available and to confirm the
write for a request to succeed.
- The coordinator of the query will send the write request to replicas
in all data centers, the difference relies on whether the coordinator
should wait for remote replicas to acknowledge the write for the
operation to succeed.
- For RF=3 … LOCAL_QUORUM is 2 replicas

Cassandra Concepts
Consistency Levels
EACH_QUORUM
- Majority of replicas in each data center received the write and
requires a quorum of nodes in each data center to be available
and to confirm the write for a request to succeed.
- For RF=3 in 2 data centers … EACH_QUORUM is 4 total replicas

Application Architecture

Availability Zone Outage

Region Outage

What can be done about it?
1. Understand the failure domains
2. Prepare and design to protect against scopes of failure
3. Use DataStax and Cassandra for a fault tolerant database

Lets see it.

Demo
https://github.com/datastax/dc-failover-demo
● 6 EC2 instances for DataStax Distribution of Apache Cassandra nodes segregated in two
data-centers:
○ Region us-east-1: 3 EC2 m5.2xlarge instances across 3 Availability Zones (AZ).
○ Region us-west-2: 3 EC2 m5.2xlarge instances across 3 AZs.
● 6 EC2 m5.large instances to be used for application services, one in each AZ.
● 2 Elastic Load Balancers (ELB), one per region, with health checks enabled.
● 1 AWS Global Accelerator as ELBs anycast frontend, with health checks enabled.
● 2 EC2 t2.small instances to be used as clients, one in each region. ( Locust )

Demo

Demo
CREATE KEYSPACE IF NOT EXISTS shopping WITH REPLICATION =
{'class':'NetworkTopologyStrategy','us-east-1': 3,'us-west-2': 3};
CREATE TABLE IF NOT EXISTS shopping.carts (
username text,
item_id int,
date_added timestamp,
item_name text,
PRIMARY KEY (username, item_id, date_added))
SELECT * FROM shopping.carts WHERE username = ?
INSERT INTO shopping.carts (username, item_id, date_added, item_name) VALUES (?,
?, toTimestamp(now()), ?)
Uses DataStax Java Driver for Cassandra, version 4.x
Schema
Reads
Writes

Demo
https://locust.io - “An Open Source Load Testing Tool”

Questions?

Thank you!

Designing Fault-Tolerant Applications with DataStax Enterprise and Apache Cassandra

More Related Content

What's hot (20)

Similar to Designing Fault-Tolerant Applications with DataStax Enterprise and Apache Cassandra (20)

More from DataStax (13)

Recently uploaded (20)

Designing Fault-Tolerant Applications with DataStax Enterprise and Apache Cassandra

Editor's Notes