Microservices Chaos Testing at Jet

Chaos Engineering at Jet.com
Rachel Reese | @rachelreese | rachelree.se
Jet Technology | @JetTechnology | tech.jet.com

InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/jet-microservices-testing

Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

Why do you need chaos testing?

The world is naturally chaotic

But do we need more testing?
Unit Sanity Random Continuous
UsabilityA/BLocalizationAcceptance
Regression Performance Integration Security

You’ve already tested all your
components in multiple ways.

Microservices Chaos Testing at Jet

It’s super important to test the interactions in your
environment

Taking on Amazon!
Launched July 22
• Both Apple & Android named our
app as one of their tops for 2015
• Over 20k orders per day
• Over 10.5 million SKUs
• #4 marketplace worldwide
• 700 microservices
We’re hiring!
http://jet.com/about-us/working-at-jet

Azure Web sites Cloud
services VMs Service bus
queues
Services
bus topics
Blob storage
Table
storage Queues Hadoop DNS Active
directory
SQL Azure R
F# Paket FSharp.Data Chessie Unquote SQLProvider Python
Deedle
FAK
E
FSharp.Async React Node Angular SAS
Storm Elastic
Search
Xamarin Microservices Consul Kafka PDW
Splunk Redis SQL Puppet Jenkins
Apache
Hive
Apache
Tez

Microservices
• An application of the single responsibility principle at the service level.
• Has an input, produces an output.
Easy scalability
Independent releasability
More even distribution of complexity
Benefits
“A class should have one, and only one, reason to change.”

It’s just wreaking havoc with your code
for fun, right?

Chaos Engineering is…
Controlled experiments on a distributed system
that help you build confidence in the system’s
ability to tolerate the inevitable failures.

Principles of Chaos Engineering
1. Define “normal”
2. Assume ”normal” will continue in both a control group
and an experimental group.
3. Introduce chaos: servers that crash, hard drives that
malfunction, network connections that are severed, etc.
4. Look for a difference in behavior between the control
group and the experimental group.

Going farther
Build a Hypothesis around Normal Behavior
Vary Real-world Events
Run Experiments in Production
Automate Experiments to Run Continuously
From http://principlesofchaos.org/

Benefits of chaos engineering
You're awake Design for failure
Healthy systems Self service

Current examples of chaos engineering

Maybe you meant Netflix’s Chaos Monkey?

We’re not testing in prod (yet).

SQL restarts & geo-replication
Start
- Checks the source db for write access
- Renames db on destination server (to create a new one)
- Creates a geo-replication in the destination region
Stop
- Shuts down cloud services writing to source db
- Sets source db as read-only
- Ends continuous copy
- Allows writes to secondary db

What FP means to us
Prefer immutability
Avoid state changes,
side effects, and
mutable data
Use data in  data out
transformations
Think about mapping
inputs to outputs.
Look at problems
recursively
Consider successively
smaller chunks of the
same problem
Treat functions as
unit of work
Higher-order functions

The F# solution offers us an order of magnitude
increase in productivity and allows one developer to
perform the work [of] a team of dedicated
developers…
Yan Cui
Lead Server Engineer, Gamesys
“
“ “

Concise and powerful code
public abstract class Transport{ }
public abstract class Car : Transport {
public string Make { get; private set; }
public string Model { get; private set; }
public Car (string make, string model) {
this.Make = make;
this.Model = model;
}
}
public abstract class Bus : Transport {
public int Route { get; private set; }
public Bus (int route) {
this.Route = route;
}
}
public class Bicycle: Transport {
public Bicycle() {
}
}
type Transport =
| Car of Make:string * Model:string
| Bus of Route:int
| Bicycle
C# F#
Trivial to pattern match on!

Concise and powerful code
public abstract class Transport{ }
public abstract class Car : Transport {
public string Make { get; private set; }
public string Model { get; private set; }
public Car (string make, string model) {
this.Make = make;
this.Model = model;
}
}
public abstract class Bus : Transport {
public int Route { get; private set; }
public Bus (int route) {
this.Route = route;
}
}
public class Bicycle: Transport {
public Bicycle() {
}
}
type Transport =
| Car of Make:string * Model:string
| Bus of Route:int
| Bicycle
| Train of Line:int
let getThereVia (transport:Transport) =
match transport with
| Car (make,model) -> ...
| Bus route -> ...
| Bicycle -> ...
Warning FS0025: Incomplete pattern
matches on this expression. For example,
the value ’Train' may indicate a case not
covered by the pattern(s)
C# F#

TickSpec – an F# project
Thanks to Scott Wlaschin for his post, Cycles and modularity in the wild

SpecFlow– a comparable C# project
Thanks to Scott Wlaschin for his post, Cycles and modularity in the wild

type Input =
| Product of Product
type Output =
| ProductPriceNile of Product * decimal
| ProductPriceCheckFailed of PriceCheckFailed
let handle (input:Input) =
async {
return Some(ProductPriceNile({Sku="343434"; ProductId = 17; ProductDescription = "My
amazing product"; CostPer=1.96M}, 3.96M))
}
let interpret id output =
match output with
| Some (Output.ProductPriceNile (e, price)) -> async {()} // write to event store
| Some (Output.ProductPriceCheckFailed e) -> async {()} // log failure
| None -> async.Return ()
let consume = EventStoreQueue.consume (decodeT Input.Product) handle interpret
What do our services look like?
Define inputs
& outputs
Define how input
transforms to output
Define what to do
with output
Read events,
handle, & interpret

Our code!
let selectRandomInstance compute hostedService = async {
try
let! details = getHostedServiceDetails compute hostedService.ServiceName
let deployment = getProductionDeployment details
let instance = deployment.RoleInstances
|> Seq.toArray
|> randomPick
return details.ServiceName, deployment.Name, instance
with e ->
log.error "Failed selecting random instancen%A" e
reraise e
}

Our code!
let restartRandomInstance compute hostedService = async {
try
let! serviceName, deploymentId, roleInstance =
selectRandomInstance compute hostedService
match roleInstance.PowerState with
| RoleInstancePowerState.Stopped ->
log.info "Service=%s Instance=%s is stopped...ignoring...”
serviceName roleInstance.InstanceName
| _ ->
do! restartInstance compute serviceName deploymentId roleInstance.InstanceName
with e ->
log.error "%s" e.Message
}

Our code!
compute
|> getHostedServices
|> Seq.filter ignoreList
|> knuthShuffle
|> Seq.distinctBy (fun a -> a.ServiceName)
|> Seq.map (fun hostedService -> async {
try
return! restartRandomInstance compute hostedService
with
e -> log.warn "failed: service=%s . %A" hostedService.ServiceName e
return ()
})
|> Async.ParallelIgnore 1
|> Async.RunSynchronously

Additional chaos finds
- Redis
- Checkpointing

If availability matters, you should be
testing for it.

Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/jet-
microservices-testing

Microservices Chaos Testing at Jet

More Related Content

Similar to Microservices Chaos Testing at Jet (20)

More from C4Media (20)

Recently uploaded (20)

Microservices Chaos Testing at Jet