Oracle: Binding versus caging

Binding (with
processor_group_name) versus
Caging: Facts, Observations and
Customer cases
Bertrand Drouvot

About Me
 Oracle DBA since 1999
 Working for the Accenture Enkitec Group
 OCP 9i,10g,11g
 Rac certified Expert
 Exadata certified implementation specialist
 Blogger since 2012
 Oracle ACE
 @bertranddrouvot
 BasketBall fan

Introduction (1/2)
 Nowadays it is very common to consolidate multiple databases on
the same server.
 One could want to limit the CPU usage of databases and/or ensure
guarantee CPU for some databases. I would like to compare two
methods to achieve this.
 Instance Caging (available since 11.2).
 processor_group_name (available since 12.1).

Introduction (2/2)
 Facts: Describing the features and a quick overview on how to
implement them.
 Observations: What I observed on my lab server. You should
observe the same on your environment for most of them.
 OEM and AWR views for Instance Caging and
processor_group_name with CPU pressure.
 Customer cases: I cover all the cases I faced so far.

Facts: Instance Caging
 Instance caging: Purpose is to Limit or “cage” the amount of CPU
that a database instance can use at any time. It can be enabled
online (no need to restart the database) in 2 steps:
 SQL> alter system set cpu_count = 2;
 SQL> alter system set resource_manager_plan =
‘default_plan’;

Facts: Binding (1/3)
 processor_group_name: Bind a database instance to specific
CPUs / NUMA nodes.
 On Linux x86-64, the named subset of CPUs is created through a
Linux feature called control groups (cgroups).
 On Oracle Solaris 11 SRU 4, the named subset of CPUs is created
through a feature called resource pools.
 It is enabled in 4 steps on my RedHat 6.5 machine.
1. Specify the CPUs or NUMA nodes by creating a “processor group”:

Facts: Binding (3/3)
2. Start the cgconfig service.
 service cgconfig start
3. Set the Oracle parameter “processor_group_name” to the name
of this processor group:
 SQL> alter system set
processor_group_name='oracle' scope=spfile;
4. Restart the Database Instance.

Facts: Availability
 The Instance caging can be enabled online.
 For cgroups/processor_group_name the database needs to
be stopped and started.

Observations
 For brevity, “Caging” is used for “Instance Caging” and “Binding” for
“cgroups/processor_group_name” usages.
 Also “CPU” is used when CPU threads are referred to, which is the
actual granularity for caging and binding.
 The observations have been made on a 12.1.0.2 database using
large pages (use_large_pages=only). I will not cover the “non
large pages” case as not using large pages is not an option for me.

Observations: binding, NUMA memory, CPUs
locality & LIO performance (1/3)
 The SGA will be allocated according to
the cpuset.mems parameter.
 The processes will be bind to the CPUs according to
the cpuset.cpus parameter.

 Local: cpuset.cpus=”1-9″ and cpuset.mems=”0″
 Remote: cpuset.cpus=”1-9″ and cpuset.mems=”7″
 Each test will be based on the same SLOB configuration and
workload: 8 readers and fix run time of 3 minutes.

 Remote is about 2.15 times slower than with the local NUMA node
access.
 Then pay attention to configure the group correctly
with cpuset.cpus and cpuset.mems to ensure local
NUMA node access.

Observations: Without binding SGA is
interleaved

Observations: With binding SGA is not
interleaved

Observations: During LIO pressure, Instance
Caging less performant on my lab server (1/2)
 Test 1: Curiosity, launch the SLOB runs (9 readers) without cpu
binding and without Instance caging (without LIO pressure).
 Test 2: Instance caging (6) in place and LIO pressure (9 SLOB
readers).
 Test 3: cpu binding in place (6 cpus same numa node) and LIO
pressure (9 SLOB readers).
 Test 4: cpu binding (9 cpus same numa node) and Instance caging
(6) in place with LIO pressure (9 SLOB readers) due to the caging.
 Test 5: Curiosity, launch the SLOB (9 readers) runs with cpu binding
(9 cpus) and without Instance caging (without LIO pressure).

Observations: During LIO pressure, Instance
Caging less performant on my lab server (2/2)
 The ranking is the following (Logical IO per seconds in descending
order):
1. cpu binding (without LIO pressure): 8 600 000 logical reads per
second.
2. without anything (without LIO pressure): 6 300 000 logical reads
per second.
3. cpu binding only and LIO pressure: 5 800 000 logical reads per
second.
4. cpu binding and Instance caging (LIO pressure due to the
caging): 5 100 000 logical reads per second.
5. Instance caging only and LIO pressure: 3 500 000 logical reads per
second.

Observations: binding already in place,
change the number of cpus a database is
allowed to use on the fly
 Initially: cpuset.cpus="1-6”
 Alert.log: Instance started in processor group oracle (NUMA Nodes:
0 CPUs: 1-6)
 taskset -c -p `ps -ef | grep -i smon | grep BDT12CG | awk '{print $2}'`
 pid 1019968's current affinity list: 1-6
 /bin/echo 11-20 > /cgroup/cpuset/oracle/cpuset.cpus
 Alert.log: Detected change in CPU count to 10.
 taskset -c -p `ps -ef | grep -i smon | grep BDT12CG | awk '{print $2}'`
 pid 1019968's current affinity list: 11-20

Observations: Summary
 Pay attention to configure the group correctly
with cpuset.cpus and cpuset.mems to ensure local NUMA node
access.
 With the binding the SGA is not interleaved.
 With Caging the SGA is interleaved (unless on top of the Binding).
 Instance caging has been less performant (compare to the cpu
binding) during LIO pressure.
 With cpu binding already in place (using
processor_group_name) we are able to change the number of
cpus a database is allowed to use on the fly.
 All above mentioned observations are related to performance.

CPU pressure with Caging: OEM view
cpu_count parameter set to 6 and is running 9 SLOB users in parallel
In average about 6 sessions are running on the CPU and about 3 are waiting
in the “Scheduler” wait class

CPU pressure with Caging: AWR view
Approximately 60% of the DB time is spend on CPU
40% is spend waiting because of the caging (Scheduler wait class)

CPU pressure with Binding: OEM view
cgroup of 6 CPUs. The Instance is running 9 SLOB users in parallel
In average 6 sessions are running on CPU and 3 are waiting for the CPU (runqueue)

CPU pressure with Binding: AWR view
New “Top Events” section as of 12c

CPU pressure with Binding: Summary

Case number 1: One database uses too
much CPU and affects other instances’
performance
 Then we want to limit its CPU usage.
 Caging: We are able to cage the database Instance without any
restart.
 Binding: We are able to bind the database Instance, but it needs a
restart.
 Example of such a case by implementing Caging:

Case number 1: One database uses too
much CPU and affects other instances’
performance: What to choose?
 The caging is the best choice in this case (as it is the easiest and we
don’t need to restart the database).

For the following cases we need to define 2
categories of database
 The paying ones: Customer paid for guaranteed CPU resources
available.
 The non-paying ones: Customer did not pay for guaranteed CPU
resources available.

Guarantee CPU resource for some databases
at the database level: Caging (1/2)
 All the databases need to be caged (if not, there is no guaranteed
resources availability as one non-caged could take all the CPU).
 As a consequence, you can’t mix paying and non-paying customer
without caging the non-paying ones.
 You have to know how to cage each database individually to be sure
that sum(cpu_count) <= Total number of threads (If not, CPU
resources can’t be guaranteed).
 Be sure that sum(cpu_count) of non-paid <= “CPU Total – number of
paid CPU” (If not, CPU resources can’t be guaranteed).
 There are a maximum number of databases that you can put on a
machine: As cpu_count >=2 (see facts) then the maximum number
of databases = Total number of threads /2

at the database level: Caging (2/2)

at the database level: Binding (1/2)
 Each database is linked to a set of CPU (There is no overlap, means
that a CPU can be part of only one cgroup).
 One cgroup per paying database.
 One cgroup for all the non-paying databases.
 That way we can easily mix paying and non-paying customer on the
same machine.

at the database level: Binding (2/2)

at the database level: Binding + Caging Mix
(1/2)
 create a cgroup for all the paying databases and then put the
caging on each paying database.
 Put the non-paying into another cgroup (no overlap with the
paying one) without caging.

at the database level: What to choose?
 No “best choice”.
 It all depends of the number of database we are talking about (For a
large number of databases I would use the mix approach).
 don’t forget that with the caging option your can’t put more than
number of threads/2 databases.

Guarantee CPU for exactly one group of
databases at the group level: Caging option 1
(1/2)
 Cage all the non-paying databases.
 Don’t cage the paying databases.
 That way the paying databases have guaranteed at least the
resources they paid for.

(2/2)

(1/2)
 Cage all the databases (paying and non-paying).
 That way the paying databases have guaranteed exactly the

(2/2)

databases at the group level: Binding option 1
(1/2)
 Put all the non-paying databases into a “CPU Total – number of
paid CPU ” cgroup.
 Allow all the CPUs to be used by the paying databases (don’t
create a cgroup for the paying databases).
 That way the paying group has guaranteed at least the resources it
paid for.

(2/2)

(1/2)
 Put all the paying databases into a cgroup.
 Put all the non-paying databases into another cgroup (no
overlap with the paying cgroup).
 That way the paying databases have guaranteed exactly the

(2/2)

databases at the group level: What to choose?
 I like the option 1) in both cases as it allows customer to get more
than what they paid for (with the guarantee of what they paid for).
 Then the choice between caging and binding really depends of the
number of databases we are talking about (binding is preferable and
easily manageable for large number of databases).

Guarantee CPU for more than one group of
databases at the group level: Caging (1/2)
 All the databases need to be caged (if not, there is no guaranteed
resources availability as one non-caged could take all the CPU).
 As a consequence, you can’t mix paying and non-paying customer
without caging the non-paying ones.
 There are a maximum number of databases that you can put on a
machine: As cpu_count >=2 (see facts) then the maximum number
of databases = Total number of threads /2

databases at the group level: Caging (2/2)

databases at the group level: Binding (1/2)
 Create a cgroup for each paying database group.
 Create a cgroup for the non-paying ones (no overlap between all the
groups).
 Overlapping is not possible in that case (means you can’t guarantee
resources with overlapping).

databases at the group level: Binding (2/2)

databases at the group level: What to choose?
 It really depends of the number of databases and groups we are
talking about (binding is preferable and easily manageable for large
number of databases).

Remarks
 If you see (or faced) more cases, then feel free to share and to
comment on the blog post.
 I did not pay attention on potential impacts on LIO performance
linked to any possible choice (caging vs binding with or without
NUMA).
 I just mentioned them into the observations, but did not include them
into the use cases (I just discussed the feasibility of the
implementation).
 I really like the options 1 into the case with exactly one group. This
option has been proposed by Karl Arao during a tweeter
conversation.

Conclusion
 The choice between caging and binding depends of:
 Your needs.
 The number of databases you are consolidating on the server.
 Your performance expectations.

Oracle: Binding versus caging

More Related Content

What's hot (20)

Similar to Oracle: Binding versus caging (20)

Recently uploaded (20)

Oracle: Binding versus caging

Editor's Notes