Christo kutrovsky oracle rac solving common scalability problems

Oracle RAC – Solving Common
Scalability Problems
Presented by: Christo Kutrovsky

© 2015 Pythian - Presentation for Oracle OpenWorld
Who Am I
2
• Oracle ACE
• 15 years in Oracle field
• Joined Pythian 2003
• Part of Pythian Consulting Group
• Special projects
• Performance tuning
• Critical services
• Presenter at: IOUG, RMOUG, UKOUG, OpenWorld
“oracle pinup”

© 2015 Pythian - Presentation for Oracle OpenWorld3 © 2011 Pythian - Confidential
Why Companies Trust Pythian
• Recognized Leader:
• Global industry-leader in remote database administration services and consulting for
Oracle, Oracle Applications, MySQL and SQL Server
• Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion
and Western Union to help manage their complex IT deployments
• Expertise:
• One of the world’s largest concentrations of dedicated, full-time DBA expertise.
Employ 7 Oracle ACEs/ACE Directors.
• Hold 7 Specializations under Oracle Platinum Partner program, including Oracle
Exadata, Oracle GoldenGate & Oracle RAC.
• Global Reach & Scalability:
• 24/7/365 global remote support for DBA and consulting, systems administration,
special projects or emergency response

Questions for Audience
5
• How many have RAC today?
• How many have RAC for HA?
• How many have RAC for scalability?
• How many plan to move to RAC?
• How many develop on RAC?

Scale UP vs Scale OUT
7
• Can have many meanings
• Scale UP
• Get a bigger machine
• Scale OUT
• Get more machines

Database Performance
8
• Has 2 primary components
• CPU – compute capacity
• IO – data transfer capacity
• Disk
• Network
• Cache (RAM)
• Implementation specifics
• Database scalability – latches, mutexes,
threading etc.

Database Performance
9
Disk
Server
CPU
RAM RAM
APP

Database Performance – Scale UP
10
Disk
Server
CPU
RAM RAM
APP
CPU
RAM RAM

Database – Scale OUT
11
Disk
Server
CPU
RAM RAM
APP
Server
CPU
RAM RAM
Server
CPU
RAM RAM

Scale UP – challenges 2011
12
• Scale UP
• Limited “max size”
• Appears unified, but
is not, NUMA
• Slower CPUs
• 12.8-GB/sec per QPI
• Up to 88 Gb/sec of network
(NEM)
• 8 PCIe slots
Sources: http://www.oracle.com/technetwork/articles/systems-hardware-architecture/sun-server-x2-8-architecture-1689083.pdf

CPU Comparison 2011
13
X3-2
• 8 cores per chip
• 2 chips max
• Server with 16 cores:
702 (SPECint®_rate2006)
• 35% capacity of x3-8
• 43.8 per core
• 1.8 x faster
X3-8
• 8 chips max
• server with 80 cores:
• 2.8 X Capacity
• 24.8 per core
• 56% speed of x3-2
Sources:
http://www.spec.org/cpu2006/results/res2012q4/cpu2006-20121104-24909.html
http://www.spec.org/cpu2006/results/res2012q3/cpu2006-20120710-23574.html

CPU Comparison 2015
14
X5-2
• 2 chips max
• 30% Capacity of X4-8
• 38.8 per core
• 45 MB CPU cache
• 1U
X4-8
• 8 chips max
• 3.2 X capacity of x5-2
• 38.4 per core
• 40 MB CPU cache
• 4U
Sources:
http://spec.org/cpu2006/results/res2014q4/cpu2006-20141007-32157.html
http://spec.org/cpu2006/results/res2014q4/cpu2006-20141120-33211.html

Scale OUT
16
• Requires Synchronisation Software – Oracle RAC
• Requires “interconnect” network
• Low latency critical
• Requires additional CPU resources for cross-
node synchronisation
• Some operations (ex. locking) significantly
increased latency (network vs. memory)

Adding CPU hardware
17
• Both Scale UP and Scale OUT increase total CPU
capacity
• and IO network bandwidth
• Does not improve IO latency
• Does not improve IO requests capacity
• Does not improve application response time
• Does not make your batch job run faster

Adding CPU hardware
18
• Adds more workers, but you must keep them
busy – at the same time
• Execute more tasks in parallel
• Parallelise existing tasks

Adding CPU hardware
19
• Unless more RAM is added
• Indirectly affects IO requests capacity
• Indirectly affects IO latency

20
Oracle RDBMS
Scale OUT with RAC

Why Scale OUT Oracle
21
• Natural High Availability
• Infinite scalability – realistically
• Smaller “block” of failure

Challenges with Oracle RAC
22
• Concurrent modifications latency increased
drastically
• Requires block transfer (CPU + network)
• Requires partial disk flush (CPU + latency)
• Concurrent reads do not share RAM cache
• Each node has local copy
• Oracle RAC Cache Fusion Reads slow
• timings similar to SSD read speeds – 10,000x
slower than memory reads

23
Practical Issues
Concurrent inserts

Data Block
24
• Oracle stores data in blocks,
in rows
• Each segment (table) has
independent blocks*
• Lowest granularity in RAC
data exchange is block*
• Modifying any data
requires the block to be
‘acquired’ in local SGA
(cache)
Oracle Table Block
8 KiB (8192)
1, Christo Kutrovsky, Canada, 99$
2, Alex Gorbachev, Canada, 98$
3, Marc Fielding, Canada, 101$

Inserting Data
25
• Inserting a row is modifying
the block
Oracle Table Block
8 KiB (8192)
4, Vanessa Simons, Canada, 199$

Block needs to move
26
• Block will bounce from node to node depending on
where the insert runs
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
APP
5, John Smith, Canada, 777$ 6, Scott Tiger, Canada, 333$

Inserting Data
27
• Well that’s not efficient, can’t each node insert
into it’s own block?

Inserting Data
28
• Yes we can! But…
• ASSM
• Instances can “own” L1 space metadata,
which maps to N blocks
• Larger segments
• To have multiple L1 space metadata,
larger segments are required

ASSM – bitmapped space mgmt
29
ASSM
L3
L2
L2
L1
1
L1
2
L1
3
L1
4
L1
5
L1
6
L1
7
L1
8
BLOCKS
1 2 3 4
5 6 7 8
BLOCKS
9 10 11 12
13 14 15 16

ASSM
30
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
APP
4, Vanessa Simons, Canada, 199$ 5, John Smith, Canada, 777$ 6, Scott Tiger, Canada, 333$
L1
1
BLOCKS
L1
2
BLOCKS
L1
3
BLOCKS

Primary Key
31
• Primary Key – an index
• Each row has a very
specific place
• Must insert into last block,
always
• Sequentially generated
• … or is it?
Oracle Table Block
8 KiB (8192)
5, John Smith, Canada, 777$
6, Scott Tiger, Canada, 333$
7, ……………………..

Sequences
32
• Sequences can be
• Cached or NOcached
• Ordered or NOordered
• Cached NOordered with large cache are your best
friend
• A sequence’s cache “sticks” with a RAC node
• cache 10,000 – 10,000 sequential values
assigned to node 1

NO ORDER
33
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
APP
?, Vanessa Simons, Canada, 199$ ?, John Smith, Canada, 777$ ?, Scott Tiger, Canada, 333$
10,001
10,002
20,001
20,002
30,001
30,002

Sequences
34
• Cached noordered 10,000 cache – almost perfect
• Oracle numbers “compress” zeroes
• 10,000 and 20,000 are next to each other
• Sequence dictionary updates moves blocks
• Order not maintained
• But what if it’s required?
• Is it required?
• When do you consider the row “inserted”?
• What if delayed due to lock/concurrency?

Index on date inserted
35
• Concurrently inserted records
have same date or very close
• Specific ordering in blocks
• No flexibility on values
• Same for index on Country or
any other “same value”
column
Oracle Table Block
8 KiB (8192)
5, John Smith, Canada, 777$
6, Scott Tiger, Canada, 333$
7, ……………………..

Dates
36
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
APP
?, Vanessa Simons, Canada, 199$ ?, John Smith, Canada, 777$ ?, Scott Tiger, Canada, 333$
11:09:01 11:09:01 11:09:01

Same Value Index
37
• Requires Partitioning
• Changes everything
• Partition key selection is non-trivial
• Local Indexes
• Lots of combinations
• Primary key local or global
• Depends on partition key and type
• Lots of considerations
• Data locality

Partition 4
BLOCKS
1 2 3 4
5 6 7 8
IDX
Date
IDX
PK
Partitioning
38
Partition 1
BLOCKS
1 2 3 4
5 6 7 8
IDX
Date
IDX
PK
Partition 2
BLOCKS
1 2 3 4
5 6 7 8
IDX
Date
IDX
PK
Partition 3
BLOCKS
1 2 3 4
5 6 7 8
IDX
Date
IDX
PK

Same Value Index
39
• Multiple partitions
• Create multiple blocks that have the “current”
date
• “current” block will still move around the
cluster, but decreases chances of “collisions”
• Still far from ideal

Same Value Index - Other
40
• Alternative solutions
• Prefix same value index with customer ID
• Prefix same value index with INSTANCE_ID
• “fake” local index
• Relies on skip scan

APP
APPApps
42
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
APP
11:09:01 11:09:01 11:09:01

Apps
43
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
APP
Server
APP
Server
APP

Application partitioning
44
What if I told you it doesn’t have to be like that?

Application partitioning
45
• By Functionality
• Not all functions of the application require
concurrent modifications
• Isolating by functionality is the easiest and
fastest way to ‘fix’ RAC unfriendly activities
• Such functionality has inherently bad
scalability properties, as number of cores
increase, this will become bigger problem
• Cache fusion will mirror concurrently read data
• Multiple nodes do not ‘add’ to caching benefits

Apps
46
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
APP
Server
APP
Server
APP
Module(s)
With
Concurrent
Modifications

Application Partitioning
47
What if a single module exceeds the performance
capacity of a single node?

More Performance
48
• Tune it
• What are you doing?
• Can you not do it?
• Can you do it more efficiently?
• Are you using the latest hardware?
• Should probably start with this
• Still need more?

© 2015 Pythian - Presentation for Oracle OpenWorld49
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
APP
Server
APP
Server
APP
Server
APP
Server
APP

Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
APP
Server
APP
Server
APP
Server
APP
Server
APP
Server
APP

Data Partitioning
51
• By Data
• Each application server processes a subset of
the data
• By Customer ID
• By Region – data balancing challenges

Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
APP
Server
APP
Server
APP
Server
APP
Server
APP
Server
APP
Load Balancer

Data Partitioning
53
• Partition the Database to match
• Consider growth – have significantly more
buckets than servers
• Local indexes everywhere
• Partition key and Primary Key generation can be
challenging
• If partitioned by primary key, key generation
from one node, must not send data to multiple
partitions
• Secondary unique indexes still challenging

Data Partitioning – Primary Keys
56
• Unique Keys
• Primary Keys usually part of partition key
• Two level partitioning (range for historical,
hash for concurrency) – global partitioned
indexes

Sharded Data – why use RAC?
57
• Most apps will “just work”
• Few components require extra attention
• Not all data is sharded
• Reference / Lookup Data is shared
• Low concurrency data does not need to be
sharded
• Redundancy – can ‘fail back’ to any node
• Caching (performance) implications

Sharded Data – why use RAC?
58
• Some data cannot be sharded one way
• Many to Many
• Buyer/Seller with portals for both customers
and stores
• Manual sharding mistakes – loose data
• Non shardable queries? Code manually?
• Sharded queries and sorting
• Schema Changes/Indexing

Why use RAC?
59
• Run reports across all data

Reports Across all Data
60
• No replication
• Cross shard consistency – even after data restore
• No data movement
• Reuse cache from each node
• Parallel queries can leverage all nodes
• MPP on OLTP data

Final Thoughts
63
• RAC Works
• With a little help
• RAC Scales
• Your CPU
• Sometimes your RAM
• Not your disk
• RAC can be
• share everything
• share nothing

© 2009/2010 Pythian - Presentation for ABC Company64
The End
Thank You
Questions?
I blog at
http://www.pythian.com/news/author/kutrovsky/

Christo kutrovsky oracle rac solving common scalability problems

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Christo kutrovsky oracle rac solving common scalability problems (20)

Recently uploaded (20)

Christo kutrovsky oracle rac solving common scalability problems

Editor's Notes