Successful Scalability Principles - Part 1

Successful Scalability
Principles - Part 1
Ronald Bradford
http://ronaldbradford.com
2012.08
EffectiveMySQL.com - Performance, Scalability & Business Continuity

OUTLINE
Necessary Principles
System Architecture
Data Availability
Best Practices
Being proactive


1


ACTION

Integrated
monitoring and
instrumentation


EXAMPLE
Question:

✘

How do you know when your server is
down or not accessible?
Answer:
The users will let us know.


HOW


HOW
Monitoring/Alerting
Graphical
Historical
Necessary
Generally missing/incomplete
Useless for real-time analysis

HOW

E
V
I
T
C
A
E
R
Monitoring/Alerting

Graphical
Historical
Necessary
Generally missing/incomplete
Useless for real-time analysis


HOW

http://rb42.com/monitoring-needs

HOW
Dashboard
The state of NOW
Near real-time statistics
Sampling at 1s/3s/5s
e.g. 0.1% of throughput

HOW
Dashboard

E
V
I
T

C
A
O
R
P
The state of NOW

Near real-time statistics
Sampling at 1s/3s/5s

e.g. 0.1% of throughput


HOW
Instrumentation
Important to business viability
e.g. orders per minute (KPI)
page load time

Seamless implementation
i.e. no code changes to view real-time
extensible


HOW
Instrumentation

E
V
I
T

C
A
O
R
P

Important to business viability
e.g. orders per minute (KPI)
page load time

Seamless implementation
i.e. no code changes to view real-time
extensible


Why

Monitoring +
Instrumentation =
Preemptive Analysis


How NOT TO

✘


HOW TO

✔
< 1 second


WHY
Preemptive Analysis
Fail early
Faster user response
Less physical resource impact


TIP


TIP
Have a status website
allow for comments (e.g. blog)


TIP

Have a public dashboard website


TIP

Have a public dashboard website
Host them somewhere else!

2


ACTION

Seamless
automated
server
deployment

EXAMPLE
Actual Client
100+ production database servers

✘

Full-time DBAs
No automated software installation
No single documented installation process
No version control
No auditing/veriﬁcation


HOW
Basics
Version Control
Automated Build & Release
Integrated Monitoring


HOW
Intermediate
Health Checks
Runtime conﬁg ﬁle management
Benchmarking


HOW
Advanced
Server request queue
Automated discovery
Proactive scaling (up & down)


GOAL
Consistent and timely provisioning
Unattended installation with correct
and reproducible conﬁguration
Automated discovery

✔


3


ACTION

Disaster
is
inevitable

EXAMPLE
Question:

✘

Have you ever performed a database recovery?

Answer:
No, why?


EXAMPLE
Consultant:

✘

Do you know that your daily backups only recover
the data up to that time (e.g. 1 am).You know you
have lost all your sales(aka $2 million sales today)
and data changes since then.

Customer:
No, I didn’t know that.


HOW
Have a DR plan


HOW
Have a DR plan
Documented


HOW
Have a DR plan
Documented
Tested


HOW
Have a DR plan
Documented
Tested
Timed


HOW
Have a DR plan
Documented
Tested
Timed
Veriﬁed

HOW
Test under production conditions


HOW
System Load


HOW
System Load
Database Size


HOW
System Load
Database Size
End to End


GOAL
Know and practice for disaster
Provide conﬁdence to
management

✔

Be as conﬁdent about your
recovery as your scalability


SIDEBAR!

What does your website
look like when it’s down?


SIDEBAR

✘

Google Picasa


SIDEBAR

✘


SIDEBAR

✘

Mozilla


SIDEBAR

✔


4


ACTION

Application
Programming
Interface

EXAMPLE

✘

3 access paths to same data
with different business rules
Public API
Application web site
Batch processes


WHY an API?
One code path for business
functionality
Implied business documentation
Enforced data exchange standard
Testability

GOAL
Technology independence
Business speciﬁcation
dependence (API)

✔

Stress testable


REQUIREMENTS
Input Speciﬁcation
Output Speciﬁcation
Standard for data communication
Throughput needs
Latency requirements


SIDEBAR!

Testing is not about
what works, it’s about
breaking your software


5


ACTION

Support different
levels of data
availability


HOW
Data can be:


HOW
Data can be:
Read & Write


HOW
Data can be:
Read & Write
Read


HOW
Data can be:
Read & Write
Read
Cached


HOW
Data can be:
Read & Write
Read
Cached
No Access

EXAMPLE
Development Team:
We need a maintenance
window for software
upgrades and new
releases.

EXAMPLE
Management Team:
No Downtime


EXAMPLE
Development Team:
But we need this to ﬁx
problems and improve
performance.


EXAMPLE

What is your
deﬁnition of no
downtime?


WHY
Support downtime
Software upgrades

Support disasters
Reduce high load
Support partial functionality

GOAL
Meet the business needs of
availability
There are ways to give greater
perception of availability

✔


6


ACTION

Support different
scalability
principles


HOW
Read Scalability
Write Scalability
Caching


HOW
Depends on your R/W ratio
Depends on your business needs
Depends on rate of data change
(caching)


ISSUES
Consistency
Data Interoperability


GOAL

✔

Support large scale growth with
appropriate software architecture
Minimize human interaction

Ongoing review and improvement


SIDEBAR!

Availability options &
scalability principles
= scalable datastore

+


WHAT's NEXT

I only had
one hour!


WHAT's NEXT
Part 2


WHAT's NEXT
Part 2
Reduce deployment time
The right business metrics
Team dynamics
Managing risk
Continual improvement
Data interoperability
...


WHAT's NEXT
Part 2
Reduce deployment time
The right business metrics
Team dynamics
Managing risk
Continual improvement
Data interoperability
...

Asynchronous
Write once data
Consistency


C


CONCLUsION
Monitoring. Before, during and after NOW.
You may not be able to predict the future but
you can preempt the future.
Operate below 90%. That 10% is your
insurance policy. Invest in insurance.


CONCLUSION
Does your business exist without your data?
How long can your site be unavailable before
customers go elsewhere?


CONCLUSION
Support 3 levels of real time data access
Read/Write, Read and no access
Support 3 different aspects of scalability
Read, Write and Caching


REFERENCE
Presentation
The most common MySQL scalability
mistakes, and how to avoid them.


REFERENCE
1.

My website is slow?

2.

I want to add new H/W. How do I change my application to support this?

3.

The database is slow. My webpage takes ﬁve seconds to load.

4.

Why is my database executing so many qps?

5.

My server has crashed with a hard drive failure

6.

My replication slave can't keep up?

7.

I can't access my website?

8.

8. Why is my database so large?

9.

My website seems to freeze or responds randomly?

Ronald Bradford
http://effectiveMySQL.com

Successful Scalability Principles - Part 1

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Successful Scalability Principles - Part 1 (20)

More from Ronald Bradford (19)

Recently uploaded (20)

Successful Scalability Principles - Part 1