Oracle Extended Clusters for Oracle RAC
15 likes13,926 views
The document discusses the features and benefits of Oracle Real Application Clusters (RAC) on extended distance clusters, highlighting its use for data mirroring and active-active data center operations. It emphasizes the importance of site-aware configurations and network considerations to ensure application performance and data synchronization over distances of up to 100 kilometers. Additionally, new capabilities introduced in Oracle RAC 12c release 2 enhance the management and resilience of extended clusters, including advancements in storage management and quorum setups.
![Copyright
©
2016,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Oracle
Extended
Cluster
-‐
BeSer
Through
IntegraVon
[grid@nodeA ~]$ $GIHOME/bin/
crsctl get cluster type
CRS-6539: The cluster type is 'flex’.
[grid@nodeA ~]$ $GIHOME/bin/crsctl query cluster site -all
Site 'sitea' identified by GUID '3dc2b783ef717f0affe5cc9e237e2623' in state
'ENABLED' contains nodes ’nodeA' and disks
'INFRA_0004,INFRA_0005,INFRA_0006,MGMT_0000'.
Site 'siteb' identified by GUID 'c34a271d4d845f55ff84c039f2e74c9c' in state
'ENABLED' contains nodes ’nodeB' and disks
'INFRA_0000,INFRA_0001,INFRA_0003,MGMT_0001'.
Site 'siteq' identified by GUID '22c889c0a831df7bbf337a5882a4c087' in state
'ENABLED' contains disks 'INFRA_0002' and no nodes .
30
[grid@nodeA ~]$ $GIHOME/bin/
crsctl get cluster extended
CRS-6577: The cluster is extended.](https://image.slidesharecdn.com/extendedoracleclustersforrac-170411200055/85/Oracle-Extended-Clusters-for-Oracle-RAC-30-320.jpg)
![Copyright
©
2016,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Oracle
Extended
Cluster
• Do
not
focus
on
maximum
network
latencies,
– As
the
applicaVon
response
Vme
will
be
negaVvely
affected
long
before
maximums
are
hit.
• Latency
increases
over
distance.
– Whether
an
increase
in
distance
negaVvely
affects
response
Vme
is
hard
to
predict.
– Latency
impact
on
applicaVon
response
Vme
depends
on
many
factors,
especially:
• Does
the
applicaVon
use
nodes
across
sites?
– Certain
latency
impact
on
on
applicaVon
response
Vme
might
get
compensated
by
storage
improvements.
• Using
two
storage
arrays
in
the
cluster
versus
one
increases
IOPs
for
reads
and
potenVally
throughput.
• The
goal
to
meet
network
requirements
is
to
hide
the
distance!
– Ideally,
users
are
not
aware
of
the
extended
setup.
– DisVnguish
users
that
are
new
to
the
applicaVon
and
– those
that
used
the
same
applicaVon
on
a
non-‐extended
setup
before.
• The
laSer
are
more
percepVve
to
changes
in
response
Vme.
37
Maximum
network
latencies
do
not
ma]er](https://image.slidesharecdn.com/extendedoracleclustersforrac-170411200055/85/Oracle-Extended-Clusters-for-Oracle-RAC-37-320.jpg)
Oracle Extended Clusters for Oracle RAC
- 2. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | E x t e n d e d Oracle RAC -‐ The New Feature Markus Michalewicz Senior Director of Product Management, Oracle RAC Development November 15th, 2016 [email protected] @OracleRACpm hSp://www.linkedin.com/in/markusmichalewicz hSp://www.slideshare.net/MarkusMichalewicz
- 3. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Oracle RAC on Oracle Extended Clusters -‐ The New Feature Markus Michalewicz Senior Director of Product Management, Oracle RAC Development November 15th, 2016 [email protected] @OracleRACpm hSp://www.linkedin.com/in/markusmichalewicz hSp://www.slideshare.net/MarkusMichalewicz
- 4. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direcVon. It is intended for informaVon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcVonality, and should not be relied upon in making purchasing decisions. The development, release, and Vming of any features or funcVonality described for Oracle’s products remains at the sole discreVon of Oracle. 4
- 5. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Program Agenda Extended Oracle RAC -‐ A Brief Review Oracle RAC on Extended Distance Clusters Basics What’s New In Oracle RAC 12c Release 2 Oracle Extended Clusters & Network Requirements 1 2 3 4 5
- 6. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Program Agenda Extended Oracle RAC -‐ A Brief Review Oracle RAC on Extended Distance Clusters Basics What’s New In Oracle RAC 12c Release 2 Oracle Extended Clusters & Network Requirements 1 2 3 4 6
- 7. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | The first version of the “Oracle Real ApplicaVon Clusters on Extended Distance Clusters” paper was authored in November 2006 and based on Oracle RAC 9i 7 In August 2010, the paper was updated to cover Oracle RAC 11g Release 2. The October 2013 version hSp://www.oracle.com/technetwork/database/opVons/ clustering/overview/extendedracversion11-‐435972.pdf covers Oracle RAC One Node and hence was renamed to “Oracle RAC and Oracle RAC One Node on Extended Distance (Stretched) Clusters”. Oracle RAC on Extended Distance Clusters is Not New
- 8. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Site A Site B 8 What Defines an Extended Distance Cluster? Most customers associate “Extended RAC” with deployment models 2 or 3. From an Oracle perspec7ve, an extended cluster configura7on is used as soon as data is mirrored between independent storage arrays (using Oracle ASM).
- 9. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Why Oracle RAC on Extended Distance Clusters? • Customers use “Extended RAC” mostly for the following reasons: 1. Mirror data between fire-‐cells in the same building. 2. Establish an acVve/acVve data center (DC) usage. • Greatest distance currently spanned: 52KM • Supported distance to be spanned: 100KM • Most customers use Extended RAC for distances < 5KM 3. Use for a 2+1 data centers (DCs) implementaVon. • "2+1" = 2 DCs with close proximity; 1 further apart for disaster recovery 9
- 10. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | When Does Oracle Recommend Using “Extended RAC”? • Oracle has no preference for “Extended RAC” compared to other Oracle soluVons. • Oracle’s Maximum Availability Architecture (MAA) recommendaVon foresees using a local Oracle RAC setup and synchronizing the sites via Oracle (AcVve) Data Guard (or Oracle Golden Gate). • “Extended RAC” alone is not a disaster recovery solu9on. – Oracle recommends using an Oracle Data Guard setup for full DR protecVon. • Oracle will recommend “Extended RAC” – to meet customer requirements regarding fire cell-‐based or data center-‐based protecVon – considering the individual infrastructure that is currently in place. 10
- 11. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Program Agenda Extended Oracle RAC -‐ A Brief Review Oracle RAC on Extended Distance Clusters Basics What’s New In Oracle RAC 12c Release 2 Oracle Extended Clusters & Network Requirements 1 2 3 4 11
- 12. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Oracle RAC on Extended Distance Clusters Basics – Part 1 1. Main consideraVon when seqng up an Oracle RAC on Extended Distance Clusters is the configuraVon of the storage and data synchronizaVon. 2. Network consideraVons are similarly important, especially with respect to applicaVon performance, and should be considered early on. 3. StarVng with Oracle RAC 12c Release 2, Oracle RAC on Extended Distance Clusters has advanced to a first class feature: • Oracle RAC on Oracle Extended Clusters – Oracle Extended Clusters use Oracle ASM for data management. 12
- 13. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 13 Disk Group and Failure Group rela7onship an7cipa7ng an Extended RAC setup Oracle ASM Storage Management for Oracle RAC Oracle ASM Disk Group Failure Group 2Failure Group 1 ASM Instance Database Instance nodeCnodeBnodeA nodeDASM ASM ASM ASM DBBDBBDBB DBB Normal Redundancy Disk Group
- 14. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 14 A more detailed architectural picture (local setup) Oracle ASM Storage Management for Oracle RAC Oracle ASM Disk Group Failure Group 2Failure Group 1 ASM Instance Database Instance nodeCnodeBnodeA nodeDASM ASM ASM ASM DBBDBBDBB DBB
- 15. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Site B Site A 15 A more detailed architectural picture -‐ over distance Oracle ASM Storage Management for Oracle RAC ASM Instance Database Instance nodeBnodeA ASM ASM DBBDBB nodeC nodeDASM ASM DBB DBB Oracle ASM Disk Group Failure Group 2Failure Group 1
- 16. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Oracle RAC on Extended Distance Clusters Basics – Part 2 • An Oracle RAC on Extended Distance Clusters must meet all requirements that must be met for a standard Oracle RAC setup. – This especially applies to network and storage requirements. – hSp://ww.slideshare.net/MarkusMichalewicz/how-‐to-‐use-‐oracle-‐rac-‐in-‐a-‐cloud-‐a-‐support-‐quesVon • Similar applies to the “Node Management in Oracle Clusterware”. – The same rules as for a standard Oracle RAC setup must be met. – The “Simple Majority Rule” for VoVng Disks is vital for Extended clusters. Details: • hSp://www.slideshare.net/MarkusMichalewicz/oracle-‐clusterware-‐node-‐management-‐and-‐voVng-‐disks • The main difference is that failure handling over distance needs to consider more scenarios compared to a standard Oracle RAC setup. 16
- 17. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 17 The “Simple Majority Rule” (in Extended Distance Clusters) hSp://www.slideshare.net/MarkusMichalewicz/oracle-‐clusterware-‐node-‐management-‐and-‐voVng-‐disks
- 18. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 18 Why an Odd Number of VoVng Disks in Three LocaVons? hSp://www.slideshare.net/MarkusMichalewicz/oracle-‐clusterware-‐node-‐management-‐and-‐voVng-‐disks
- 19. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 19 Vo7ng Disk configura7on Oracle RAC on Extended Distance Clusters Site C or Quorum VD3 Site A ASM Instance Database Instance nodeBnodeA ASM ASM DBBDBB Site B nodeC nodeDASM ASM DBB DBB Oracle ASM Disk Group Failure Group 1 VD1 Failure Group 2 VD2
- 20. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Program Agenda Extended Oracle RAC -‐ A Brief Review Oracle RAC on Extended Distance Clusters Basics What’s New In Oracle RAC 12c Release 2 Oracle Extended Clusters & Network Requirements 1 2 3 4 20
- 21. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | What’s New In Oracle RAC 12c Release 2? • Oracle Extended Cluster has become a first class feature. • The cluster understands that it is operated as an extended cluster. – The cluster is site-‐aware and manages components accordingly. • Oracle Universal Installer supports an Extended Cluster setup. – If a cluster is declared as an Oracle Extended Cluster, – site-‐awareness will be applied during install where necessary and – certain configuraVon steps will be enforced and implemented automaVcally. 21
- 22. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 22 Oracle Extended Cluster InstallaVon – Site-‐awareness is Elementary and Fully Integrated
- 23. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 23 Oracle Extended Cluster InstallaVon – NEW: Extended Redundancy Define failure group names and assign to sites.
- 24. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Flex Disk Group and Extended Redundancy • Flex Disk Group is a new Disk Group type in Oracle ASM 12c Rel. 2 that • enables various new feature for database-‐oriented storage management. • Together with the Flex Disk Group come two new redundancy seqngs: “Flex” and “Extended” – “Extended” is only offered for Extended Clusters – Disk Groups with “Extended Redundancy” are site-‐aware – Extended Disk Groups support mulVple VoVng Disk FG • MulVple VoVng Disk Failure Groups (FG) per site prevent local VoVng Disk failures from affecVng the cluster as a whole. 24 Flex Disk Group DB1 File 1 File 2 File 3 DB2 File 1 File 2 File 3 File 4 DB3 File 1 File 2 File 3 File Group Database-‐oriented storage management
- 25. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 25 Configura7on example Oracle Extended Cluster siteQ siteA ASM Instance Database Instance nodeA ASM siteB nodeB ASM INFRA INFRA FGA1 FGA3FGA2 FGB3 FGQ FGB1 FGB2 VD7 VD1 VD2 VD3 VD4 VD5 VD6 VD7
- 26. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 26 For be2er availability & reliability Why mulVple VoVng Disks with a Quorum in different sites? siteQ siteA ASM Instance Database Instance nodeA ASM siteB nodeB ASM INFRA INFRA FGA1 FGA3FGA2 FGB3 FGQ FGB1 FGB2 VD1 VD2 VD3 VD4 VD5 VD6 VD7 ✔ “Simple Majority Rule” is applied for each site” “Simple Majority Rule” is applied across sites” ✔ ✔ AddiVonal local VoVng Disks (3) per site increase reliability.
- 27. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | • 2 data sites & 1 Quorum site • Can tolerate the loss of 1 data or quorum site • Loss of addiVonal VoVng Disk FGs can be tolerated • 3 data sites • Each site will contain one copy of data • 3 data sites have no advantage over above config. • 3 data sites & 2 Quorum sites • Can tolerate the loss of • 2 data sites OR • 1 data site & 1 quorum site OR • 2 quorum sites • Loss of addiVonal VoVng Disk FGs can be tolerated • 2 data sites & 3 Quorum sites • Can tolerate the loss of one each OR 2 quorum sites 27 More Reliability and Flexibility – Supported ConfiguraVons DataSiteA DataSiteB QuorumSiteQ DataSiteA DataSiteB DataSiteC DataSiteA DataSiteB QuorumSite1 Quorum Site 2 Quorum Site 3 DataSiteA DataSiteB DataSiteC Quorum Site 1 Quorum Site 2 “Simple Majority Rule” is applied across sites”
- 28. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | • A storage split describes a failure affecVng the storage-‐related communicaVon between sites, while the cluster related communicaVon (network connecVon) between sites remains unaffected. • The result is that the cluster remains operaVonal, but the storage cannot be synchronized across data sites anymore. • “_asm_storagemaysplit” addresses storage splits since 11.2.0.2 • Oracle ASM 12c Rel. 2 automates this setup, improves availability & simplifies respec7ve configura7ons. 28 “Storagemaysplit” – or: What If It’s Not a Clear Cut? hSp://www.slideshare.net/MarkusMichalewicz/oracle-‐clusterware-‐node-‐management-‐and-‐voVng-‐disks
- 29. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 29 “Storagemaysplit” Integra7on Oracle Extended Cluster -‐ BeSer Through IntegraVon siteQ siteA ASM Instance Database Instance nodeA ASM siteB nodeB ASM INFRA FGQ ✔ DBBDBB If a storage split occurs, ASM will ensure to un-‐mount all disk groups in one site only. “Storage split”, as interconnect remains intact.
- 30. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Oracle Extended Cluster -‐ BeSer Through IntegraVon [grid@nodeA ~]$ $GIHOME/bin/ crsctl get cluster type CRS-6539: The cluster type is 'flex’. [grid@nodeA ~]$ $GIHOME/bin/crsctl query cluster site -all Site 'sitea' identified by GUID '3dc2b783ef717f0affe5cc9e237e2623' in state 'ENABLED' contains nodes ’nodeA' and disks 'INFRA_0004,INFRA_0005,INFRA_0006,MGMT_0000'. Site 'siteb' identified by GUID 'c34a271d4d845f55ff84c039f2e74c9c' in state 'ENABLED' contains nodes ’nodeB' and disks 'INFRA_0000,INFRA_0001,INFRA_0003,MGMT_0001'. Site 'siteq' identified by GUID '22c889c0a831df7bbf337a5882a4c087' in state 'ENABLED' contains disks 'INFRA_0002' and no nodes . 30 [grid@nodeA ~]$ $GIHOME/bin/ crsctl get cluster extended CRS-6577: The cluster is extended.
- 31. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Oracle Extended Cluster – Fully Integrated Part 1 • Oracle Extended Clusters are Flex Cluster based – As with Oracle Grid Infrastructure 12c Rel. 2 all new clusters will be Flex Clusters • Flex ASM Instances cardinality is also site-‐aware – Seqng ASM instance cardinality will be considered per data site • Assuming an sufficient number of nodes per data site to meet cardinality seqng – The total number of ASM instances is equal to (“data sites”*cardinality) • Assuming an sufficient number of nodes per data site to meet cardinality seqng • Preferred reads are site-‐aware – Addresses Bertrand Drouvot’s find, which is documented in: • bdrouvot.wordpress.com/2013/07/02/flex-‐asm-‐12c-‐12-‐1-‐and-‐extended-‐rac-‐be-‐careful-‐to-‐unpreferred-‐read/ 31
- 32. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | • Oracle Extended Cluster lays the foundaVon for “Extended RAC” on Exadata Database Machines. • Site-‐awareness & an Exadata-‐specific configuraVon as part of the Flex Diskgroup configuraVon allow spanning distances up to 100 meters. • 100m Infiniband cable length restricVon • Support for “Extended Exadata RAC” is currently planned for a future release of Oracle Database 12c Rel. 2 32 Oracle Extended Cluster – Fully Integrated Part 2 100 m / 200 m max. fireCellA fireCellB Installing Oracle Extended Clusters on Exadata Database Machine hSp://www.oracle.com/ technetwork/database/ availability/maa-‐ extclusters-‐ installguide-‐3748227.pdf
- 33. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Oracle Extended Cluster – Summary • StarVng with Oracle RAC 12c Release 2 – Oracle Extended Clusters is a first class feature – that can be selected as an installaVon type during the Grid Infrastructure installaVon. • Oracle Extended Clusters are “extended distance clusters” that use Oracle ASM for storage management and may or may not host RAC databases. – Extended distance clusters not using Oracle ASM for storage remain supported. – In either case, “Extended RAC” describes a setup, in which an Oracle RAC database is operated based on an extended distance cluster (preferably an Oracle Extended Cluster). • Oracle Extended Clusters are Flex Cluster-‐based – Reason: starVng with Oracle Grid Infrastructure 12c Rel. 2, all clusters will be Flex Clusters. 33
- 34. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Program Agenda Extended Oracle RAC -‐ A Brief Review Oracle RAC on Extended Distance Clusters Basics What’s New In Oracle RAC 12c Release 2 Oracle Extended Clusters & Network Requirements 1 2 3 4 34
- 35. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Oracle Extended Cluster • While site-‐awareness is the foundaVon for supporVng different subnets per site, the network requirements for extended clusters remain unchanged: – The private network needs the same subnet(s) to span sites. • Any virtualizaVon technology spanning sites typically requires the same setup. • This can be achieved using certain network configuraVons or using overlay technologies. – A public network needs to be available in all sites. • One can establish addiVonal public networks per site, as long as at least 1 common one is available. – Note: Services do not fail over between networks automaVcally. 35 Network requirements remain unchanged
- 36. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 36 Maximum network latencies Oracle Extended Cluster NFS / iSCSI connected quorum siteQ siteA ASM Instance Database Instance nodeA ASM siteB nodeB ASM DBBDBB 27 seconds 200 seconds / 27 seconds 200 seconds / 27 seconds
- 37. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Oracle Extended Cluster • Do not focus on maximum network latencies, – As the applicaVon response Vme will be negaVvely affected long before maximums are hit. • Latency increases over distance. – Whether an increase in distance negaVvely affects response Vme is hard to predict. – Latency impact on applicaVon response Vme depends on many factors, especially: • Does the applicaVon use nodes across sites? – Certain latency impact on on applicaVon response Vme might get compensated by storage improvements. • Using two storage arrays in the cluster versus one increases IOPs for reads and potenVally throughput. • The goal to meet network requirements is to hide the distance! – Ideally, users are not aware of the extended setup. – DisVnguish users that are new to the applicaVon and – those that used the same applicaVon on a non-‐extended setup before. • The laSer are more percepVve to changes in response Vme. 37 Maximum network latencies do not ma]er