ZDLRA in Action

BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
ZDLRA – in Action
Daniele Massimi – Principal Consultant
BE-IMS

Oracle Engineered Systems - Family
ZDLRA - in Action2 21.09.2016

Oracle Engineered Systems - Family

Who am I
Daniele Massimi, Trivadis AG (CH, Bern)
Principal Consultant daniele.massimi@trivadis.com
Seit 2012 bei Trivadis IMS (Infrastructure Service Management)
Oracle DBA seit >16 Jahre (Exadata ~5 Jahre)
Infrastruktur-Architekturen, Operations & Administration, Upgrades and Migration,
High Availability, Backup & Recovery, Enterprise Manager, Engineered Systems
(Exadata, ODA)
Oracle Certifications
Oracle Certified Professional
Oracle Certified RAC Expert
Oracle Exadata Implementations Specialist

Agenda
1. ZDLRA Functionality
2. Installation and Configuration from bottom up
3. Have a look inside
4. From protected Databases to backups
5. Recovery Capabilities
6. Recovery Appliance Views and Utilities
7. Conclusion

ZDLRA Functionality

Common problems with Oracle backups
RPO depends on archive backup frequency (many minutes or hours)
Backup window and nightly batches compete for resources
High network bandwidth usage for full backups
Low-speed backups and restores
Backup and restore success ratio penalized by shared infrastructure
Critical databases require Data Guard for transaction safety
– Additional space, storage, license and computation required
Expiration of backup set on ML: need to crosscheck and delete expired
Lack of backup validation
…

Recovery Appliance Architecture

Eliminate data loss – Real Time redo transport
Real Time redo transport
– Data Guard like but asynchronous
– Changes out of Logbuffer transfered
– Validate and writes to a stage area
Redolog switch on database
– RA converts the staged redo data to a
compressed archived redolog backup
– Writes this archlog backup to catalog
– Ready for future restores
Explicit Archivelog backup no more necessary
 Reduces RPO to (near) zero

Minimal Backup Overhead
10 21.09.2016
Traditional Backup
– Read/Write for Disk/Tape Backup
– Deduplication
– Compression
– Validation
– Deletion
– Merging
All on database host
 Degrade performance
Recovery Appliance
– Offloading operations
– Delta Push
– Delta Store

Delta Push
Delta Push is a highly optimized form of source-
side deduplication
– Through RMAN block change tracking
– No reading of unchanged data
– No commited undo
Two operations on the protected Database
– Incremental "forever" backup
– Real time redo transport
One-time full backup as prerequirement
Afterwards Incremental forever backup
Validate incoming Backups against corrupted
physical blocks
Compress the Backups using special block-level
Algorithm
Backuped Database
Changed Data
 Less data, less I/O, less network traffic

Delta Store – the “brains” of the Recovery Appliance
Backup
validates the incoming blocks
compresses, indexes and stores
Creates a Virtual Full Database
Backups
10 times less storage usage
High number of virtual full backups
Higher recover window
Delta Store
Day N
Incr
Day 1 Virtual Full
Day N Virtual Full
Day 1
Incr
Day 0
Full

Delta Store – fast restore/recover
Restore
Recreates a physical full backup out
of a virtual one
No need of transfer and apply
incremental backups
Roll forward with restored redologs
Uses Exadata performance and
scalability
Delta Store
Day 0
Full
Day 1
Incr
Day N
Incr
Day ‘N’ Full Backup

Delta Pools
Chunks inside the Delta Store
Backups will be indexed and stored inside the Delta Pools
Set of datafiles blocks from which RA constucts Virtual Full Backups
Each backuped datafile have his own delta Pool
Automated delta pool space management
– Protection Policy will be inherited to database retention policy
– Delete automatically expired or obseleted Backups
– Periodically reorganising for performance improving
– Delta pool optimization when old blocks are deleted and new incremental Backup
arriving

Massive Scale out – Oracle Database Backup Service
10/100/1000 DBs across datacenters
RMAN-integrated subscription-based
backup storage in the public cloud
RMAN compression and encryption
quickly and easily setup with a
secure, low-cost backup
Flexibility in backup/archive concepts
The Recovery Appliance flexibly
supports both tape and cloud archival
option

Installation and Configuration
from bottom up

Installation Steps
Create the ZDLRA configuration using
the latest OEDA
Start the Re-Imaging and Setup from the first
Computing Node (like an Exadata Setup)
[root@zdldbadm01 linux-x64]# ./install.sh -cf ./WorkDir/zdl-zdl.xml –r 1-16

Installation Steps
Recovery Appliance specific Installation Steps
[root@zdldbadm01 install]# ./ra_install.pl --help
ra_install.pl - Recovery Appliance Installer
You must be logged in as root to run this command.
All steps should be run successfully for install.
Usage: ./ra_install.pl --help | --step=STEP_NUMBER {--install|--uninstall} [--
stdout]
Options:
--help: displays this help message
--step=number: Which step to run
--stdout: Print all messages to stdout rather then the log file
--install: Installs the step [Default]
--uninstall: Uninstalls the step
Step Numbers:
Step 1 - Validation and Configuration Prep
Step 2 - OS Setup
Step 3 - Oracle User Setup
Step 4 - DBFS Setup
Step 5 - Tape Backup configuration
Step 6 - ZDLRA DB Backup Setup
Step 7 - Enable ZDLRA Services

Configuration Steps
Deployment of the Agents on the compute Nodes
Recovery Appliance Discovery
Installation of the Backup Module on any Clients
[oracle@odax30 zdlra]$ java -jar ra_install.jar -dbUser ra_orcl02 -dbPass
manager -host zdlingest-scan -port 1521 -serviceName zdlra -walletDir
/u01/app/oracle/admin/orcl02/xdb_wallet -libDir $ORACLE_HOME/lib
Recovery Appliance Install Tool, build 2015-11-03
Recovery Appliance credentials are valid.
Recovery Appliance wallet created in directory
/u01/app/oracle/admin/orcl02/xdb_wallet.
Recovery Appliance initialization file
/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/raorcl02.ora created.
Downloading Recovery Appliance Software Library from file ra_linux64.zip.
Downloaded 27189305 bytes in 5 seconds. Transfer rate was 5437861 bytes/second.
Download complete.
[oracle@odax30 zdlra]$

Have a look inside
(just particularities)

On the computing Nodes
Metadata Database
– RAC Database (also used for Recovery Catalog (VPC))
– Non Container Database
– DBFS Mount-Points for Backup and Replication purpose
– DBFS added as Cluster Resources
– HTTP Service inside the database will be started for communication between the
clients and the ZDLRA (dbms_ra.startup_recovery_appliance)
oracle@zdldbadm01:~/ [+ASM1] df -h | grep dbfs
dbfs-@repdbfs.local:/
957M 120K 956M 1% /dbfs_repdbfs
dbfs-@obdbfs.local:/ 300G 23G 278G 8% /dbfs_obdbfs
SQL> exec dbms_ra.startup_recovery_appliance;
PL/SQL procedure successfully completed.

ASM Configuration
– Two Diskgroups  DELTA (for Backups), CATALOG (Metadatas)
– New special sub-Directory CONTAINER for storing Container Files which contains
the Delta Pools and Delta Stores
– Each Container File as a size of 2 TB
ASMCMD [+delta/zdlra] > ls -l
Type Redund Striped Time Sys Name
Y ARCHIVELOG/
Y CONTAINER/
Y DATAFILE/
Y TEMPFILE/
ASMCMD [+delta/zdlra/CONTAINER] > ls -s
Block_Size Blocks Bytes Space Name
512 4294959104 2199019061248 4398059094016 con.365.920823681
512 4294959104 2199019061248 4398059094016 con.366.920823687
512 4294959104 2199019061248 4398059094016 con.367.920823699
...

Backup of Metadata Database
– Done using Export of RASYS Schema every 2 Hours
– Backup Location on DBFS FS: /dbfs_obdbfs/OSB/ra_export
– Scheduling via "anacron"
[root@zdldbadm01 ~]# cat /etc/anacrontab
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
#RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22
#period in days delay in minutes job-identifier command
1 65 cron.daily nice run-parts /etc/cron.daily
7 70 cron.weekly nice run-parts /etc/cron.weekly
30 75 cron.monthly nice run-parts /etc/cron.monthly
[root@zdldbadm01 cron.hourly]# ls -l /etc/cron.hourly/
total 4
-rwx------ 1 root root 409 Nov 10 2015 0anacron
lrwxrwxrwx 1 root root 46 Aug 25 16:20 ra_export.pl -> /opt/oracle.RecoveryAppliance/bin/ra_export.pl

On the cell Nodes
Fortunatelly nothing special 
Same appearance as in Exadata Storage Cell
During the Installation the "Write Back" Option on the Flashdisks will be enabled
Little difference on the Grid Disks compared with the Exadata, only 10 Grid Disks
instead of 12 for the CATALOG ASM Disk Group

User Model ZDLRA
Component Account Type User Name Description
EM Cloud Control Cloud Control Super User SYSMAN EM Admin User
EM Cloud Control Cloud Control Admin User specified User to manage RA and protected databases
Recovery Appliance RA Metadata Super User SYS DB Admin User, needed for create RA User
Recovery Appliance RA Admin RASYS RA Schema Owner
Recovery Appliance RA User Account User specified
User on RA for managing Metadata on VPC
of protected databases
Protected Database
Protected Database
Backup Admin
User Account with
SYSBACKUP privileges
User for doing backup, restores and recovery
of protected databases
User Accounts:

From protected Databases
to backups

Protected Databases
Once the Environment meets all the requirements we can add the databases to the
ZDLRA
The requirement are:
• Network connectivity
• Backup Module is installed for every Oracle Home
• Definition of Protection Policies are created (Retention Time)
 Block Change Tracking not mandatory, but recommended !!!
Two Methods are possible:
• Enterprise Manager
• Manually  DBMS_RA PL/SQL Package

Prerequisite for Adding Protected Database
(EM and manually)
Create an VPD User on the Metadata Database
Grant the Create Session privilege
SQL> create user ra_orcl03 identified by <pwd>;
SQL> grant create session to ra_orcl03;

Adding Protected Database with EM
Add Protected Database on Recovery Appliance Page
– Choose the Protection Policy

Adding Protected Database with EM
Configuring Backup Settings
– Register database
– Add RMAN configuration (configure...)
– Enabling Redo Transport
(add parameter to spfile and restart DB)

Scheduling Protected Database with EM
Protected Database is ready for backing up
– We need a Schedule

Configure the repeating interval for
the Incremental-Forever Backups
Configure the Archive Log Backups and
deletion rule

Also useful as Script Generator  here the Script is not modifiable !!!

Output of an Backup Schedule on EM13c

Adding Protected Database manually
Add Database for the protected database
Grant access to Virtual Private Catalog
Configure the RMAN Channel for ZDLRA
begin
dbms_ra.add_db (
db_unique_name => 'orcl03',
protection_policy_name => 'bronze',
reserved_space => '250G');
end;
begin
dbms_ra.grant_db_access (
db_unique_name => 'orcl03',
username => 'ra_orcl03');
end;
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS
"SBT_LIBRARY=/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libra.so,
ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/ra_wallet
credential_alias=zdlingest-scan:1521/zdlra:dedicated')";

Adding Protected Database manually
Enable Real-Time Redo (Delta Push)
Restart the Database
Configure the RMAN Channel for ZDLRA
ALTER SYSTEM SET REDO_TRANSPORT_USER=ra_orcl03 SCOPE=BOTH;
[oracle@odax30 ~]$ srvctl stop database -db orcl03
[oracle@odax30 ~]$ srvctl start database -db orcl03
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS
"SBT_LIBRARY=/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libra.so,
ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/ra_wallet
credential_alias=zdlingest-scan:1521/zdlra:dedicated')";

Execute Backup manually
Backups are like in traditional way
Only "forever" incremental level 1 backups should be schedule
Level 0 Backup are of course always possible
It works also with old fashion syntax  soft link from libra.so to libobk.so are
needed !!!
Scheduling is also free selectable, EM is recommended !
[oracle@odax30 ~]$ rman target / catalog ra_orcl02/manager@zdlingest-scan:1521/zdlra
...
connected to target database: ORCL02 (DBID=3592015831)
connected to recovery catalog database
RMAN> backup incremental level 1 database;
run {
allocate channel c1 type sbt_tape;
backup incremental level 1 database;
}
...
ORA-27211: Failed to load Media Management Library

Recovery Capabilities

General statement about Recoveries of protected
Databases
Recoveries of protected databases can be done with EM or manually
With EM several Recovery-Scenarios are possible
All Recovery scenarios with EM are full automated
(e.g. Creation of auxiliary Databases, etc.)
EM generates RMAN Scripts for each Scenario
with the possibility of adaption
Possibility of selection between full and point-in-time
recoveries
Duplicate Database (for standby) need some preparation
(e.g. FS-Structure (admin tree), Password-File, etc.)

Protected Database Recovery with EM
From database Page under Availability  Backup & Recovery  Perform Recovery

Choose the needed Recovery Scenario
... You can choose an other restore
location if needed...

A schedule will be created
... And you can adapt the script if needed and submit the job...

The Job can be monitored by clicking the "View Job" button...

Protected Database Recovery manually
Restore can be done also manually
– of course db must be mounted before...
– RMAN Client should be configured otherwise parameter are needed (e.g. ENV=...)
Or a point-in-time Recovery
run {
restore database;
recover database;
alter database open;
}
run {
set until time '08.09.2016 08:45:00';
restore database;
recover database;
alter database open resetlogs;
}

Recovery Appliance Views and
Utilities

Some RA Views
It is also possible to read the Recovery Appliance Information over Views from Recover
Appliace Metadata Database
View Name Description
RA_DATABASE Information about Protected Databases
RA_DB_ACCESS Describes User Accounts that have access to which protected database
RA_PROTECTION_POLICY Protection Policies defined on the Recovery Appliance
RA_DATABASE_STORAGE_USAGE Storage usage for each protected database
RA_RESTORE_RANGE Describe Restore Range for each protected database
RA_INCIDENT_LOG Describe Recovery Appliance Incidents
RA_REPLICATION_SERVER Replication Server Information (if used)

rastat.pl Utility
rastat Utility help to evaluate the performance of Recovery Appliance
– NETBACKUP/NETRESTORE
• Measure the network performance
– ASMREAD/ASMWRITE
• Measure the ASM Disk I/O Performance
– CONTAINERREAD/CONTAINERWRITE/CONTAINERALLOC
• Measure the Container Disk I/O Performance
oracle@zdldbadm01:/opt/oracle.RecoveryAppliance/client/ [zdlra1] perl
/opt/oracle.RecoveryAppliance/client/rastat.pl --test=ASMREAD --filesize=2048M --
rasys=rasys/manager@zdlingest-scan:1521/zdlra --diskgroup=+CATALOG
RECOVERY APPLIANCE READ IO TEST FROM DISK
Disk Group: +CATALOG
2048 MB, 1.90s read IO time, .10s CPU time, 1076.40 MB/s, 5.26% CPU usage
PL/SQL procedure successfully completed.

dbms_ra Package
dbms_ra provides administration function from inside the RA Database
Ca. 50 Procedure for different types of administration
– Start/Stop of Recovery Appliance
– Add/Delete Protected Databases
– Grants Access to specific User to enable to backup and restore
– Add/Update Protection Policies
– Manage Tape Configuration (if available)
– Add Repliaction Server (if available)
But, the utilization of Enterprise Manager is recommended !!!

DEMO

Demo
Recovery Appliance Page
Configure Protected Database
Run a Backup
Real Time Redo Transport
Restore a Protected Database

Conclusion

Conclusion
ZDLRA worked in our Tests as
expected
Very good implementation of well
known Backup and Recovery
Technology on Engineered System
RPO is (near) zero with Real Time
Redo Transport
Even RTO ist much better then
existing systems
Very simple Management with EM
 Risk reduced, cost reduced,
qualitiy improved
 ZDLRA will find it’s
customer !

Questions and Answers...
Daniele Massimi – Principal Consultant
Tel. +41 58 459 50 92
daniele.massimi@trivadis.com
21.09.2016 ZDLRA - in Action53

ZDLRA in Action

More Related Content

What's hot (20)

Similar to ZDLRA in Action (20)

Recently uploaded (20)

ZDLRA in Action