Oracle exalytics deployment for high availability

An Oracle White Paper
November 2014
Oracle Exalytics Deployment for High
Availability

Oracle Maximum Availability Architecture Exalytics Deployment for High Availability
Introduction ....................................................................................... 1
Deploying Exalytics for High Availability ............................................ 2
Web Tier........................................................................................ 5
Application Tier.............................................................................. 5
Data Tier ....................................................................................... 5
Exalytics Prerequisites for Deployment.............................................. 5
Exalytics and Exadata InfiniBand Configuration................................. 6
Exalytics Deployment on the First Exalytics Node ............................. 9
Post Installation Steps ................................................................. 10
Oracle WebLogic Server.............................................................. 12
Oracle Business Intelligence ....................................................... 13
Basic Exalytics Start-Stop Scripts first Node................................ 13
Exalytics Deployment on the Second Exalytics Node ...................... 15
Oracle TimesTen Installation ....................................................... 15
Oracle WebLogic Server Installation............................................ 15
Oracle Business Intelligence Installation...................................... 16
Basic Exalytics Start-Stop Scripts Second Node ......................... 17
Building Aggregates ........................................................................ 17
Create Aggregates in the Local TimesTen Instance .................... 18
Create Aggregates in the Remote TimesTen Instance ................ 19
Refresh Aggregates in the Local TimesTen instance................... 21
Refresh Aggregates in the Remote TimesTen instance............... 22
Additional Considerations................................................................ 23

1
Introduction
Oracle Exalytics In-Memory Machine provides the fastest performance for business
intelligence applications. These applications often provide critical information to business
decision-makers and that requires the application to be available. To ensure the applications
are available when they’re needed, a high availability configuration is required to eliminate the
single points of failure that can result in an application outage.
This white paper contains technical details about how to configure a two node cluster of Oracle
Exalytics systems running Oracle Business Intelligence Enterprise Edition with Oracle Exadata
to enable high availability. The Maximum Availability Architecture (MAA) configuration requires
horizontal scale out to be configured for scalability and performance (load balancing).

2
Deploying Exalytics for High Availability
Oracle Exalytics runs the Oracle Business Intelligence (BI) Foundation along with Oracle
TimesTen In-Memory Database. Oracle Exalytics supports both active-active and active-
passive clustering to provide scalability and high availability. This MAA white paper will focus
on the Oracle Exalytics active-active cluster configuration, where each Exalytics machine in
the cluster runs its own Oracle BI Foundation and Oracle TimesTen database. Each BI Server
has a separate Data Source Name (DSN) for communication with each TimesTen instance.
Shared storage such as Oracle ZFS Appliance is configured for BI server repository, BI
presentation catalog, and Global Cache.
Note: BI local cache resides on the local node on RAM disk.
Each Exalytics machine has InfiniBand connectivity to the Oracle Exadata Database Machine
running the application database to take advantage of flash technology, cell offload
processing, high availability, and other optimizations provided by Exadata.
Oracle HTTP servers must be installed on a separate pair of physical or virtual hosts between
the load balancer and Exalytics machines.

3
Figure 1: The full set of components in a High Availability cluster

4
Figure 2: High Availability configuration elements for OBI EE and TimesTen
The two environments are effectively identical other than their host name, so the configuration of the sys.odbc.ini
(TimesTen) and odbc.ini (OBI EE) files are identical. These both reference a TimesTen data store that is local
to them and one that is remote to them. The difference is in the sys.ttconnect.ini file where the network
addresses of the physical hosts are defined.
The BI Server will select the Logical Table Source (LTS) with the largest grain (the most aggregated values)
that can satisfy any given query. This results in queries to the fact table with the smallest number of rows. If it
finds two LTSes at the same grain, then it will prioritize the first one it finds in the list. Figure 2 uses numbers to
show the relative order that the LTSes should appear in the RPD to reduce “line collisions”. In the RPD the
aggregate LTSes would appear in the reverse order
Shared File Server
Global Cache
Exalytics2.mycompany.com
Web Logic EE
Presentation Server
BI Server
odbc.ini
RPD
Logical Table Source
3) Disk Oriented DB
2) Remote TT IMDB
1) Local TT IMDB
TimesTen
sys.odbc.ini
sys.ttconnect.ini
IMDB
Perm Temp
Exalytics1.mycompany.com
Web Logic EE
Presentation Server
BI Server
odbc.ini
RPD
Logical Table Source
3) Disk Oriented DB
2) Remote TT IMDB
1) Local TT IMDB
TimesTen
sys.odbc.ini
sys.ttconnect.ini
IMDB
Temp Perm
Shared webcat
RPD
Dashboards
Reports
BI Config files
Shared DB Server
RAC

5
Web Tier
In the web tier, two nodes, WEBHOST1 and WEBHOST2, run Oracle HTTP Server configured
with mod_wl_ohs.
Through mod_wl_ohs, which allows requests to be proxied from Oracle HTTP Server to Oracle
WebLogic Server, Oracle HTTP Server forwards requests to Oracle WebLogic Server running
in the application tier.
The web tier should include an external load balancer to handle external requests. External
requests are sent to the virtual host names configured on the load balancer. The load balancer
then forwards the requests to the Oracle HTTP Server.
Application Tier
Nodes in the application tier are located in the DMZ secure zone.
In this tier, Oracle WebLogic Server is configured with Managed Servers for running Oracle BI
Foundation and the Oracle TimesTen In-Memory Database. Multiple Exalytics servers are
clustered together using InfiniBand for high availability. For more information, see Oracle®
Exalytics In-Memory Machine Owner's Guide, Chapter 7 “Connecting Oracle Exalytics In-
Memory Machine to Oracle Exadata Database Machine” and Chapter 7.2.3 “Scenario 2:
Clustered Exalytics In-Memory Machine with Oracle Exadata Database Machine.”
Note: Due to lack of InfiniBand support in the Firewalls today Exalytics placement in the
Application Tier makes only sense if using Ethernet connectivity between Exalytics and
Exadata. Until InfiniBand Support is available in Firewalls we recommend Exalytics to be
located in the Data Tier.
Data Tier
Nodes in the data tier are located in the most secured network zone (the intranet).
In the data tier, the Oracle Exalytics machines are connected to Oracle Exadata Database
Machine over InfiniBand.
Exalytics Prerequisites for Deployment
1. Ensure that the prerequisites in Oracle® Fusion Middleware Installation and
Administration Guide for Oracle Exalytics In-Memory Machine for Installing Software
on Exalytics are met.
2. Connect Exalytics HCA ports to each Exadata Database Machine InfiniBand leaf
switch. One of ports 5B, 6A, 6B, 7A, 7B, or 12A on the leaf switches of the Exadata
Database Machine should be used even if additional ports are open. On partial rack
configurations, some ports are reserved for potential future expansion.

6
3. Configure DNS client settings on the Exalytics machine. It is best to use the same
DNS servers as used by the Exadata Database Machine nodes if possible.
4. Enable NTP client settings on the Exalytics machine. It is best to use the same NTP
servers as used by the Exadata Database Machine nodes if possible.
5. Configure HugePages if TimesTen requires more than 256 GB. Activate the number of
HugePages in /etc/sysctl.conf file and by setting the parameter vm.nr_hugepages =
410200.
6. The largest recommended memory for TimesTen is a number that leaves enough
memory for the other installed components and the Operating System to function
adequately; the larger the number of concurrent users the higher this memory value
needs to be. As a very rough guide no more than 80% - 85% of the memory should be
devoted to active TimesTen databases. If you are running EPM on the same Exalytics
machine then that number should be reduced to provide adequate memory resources
to the EPM system.
Exalytics and Exadata InfiniBand Configuration
1. Enable automatic RDS kernel module loading on Exalytics.
This can be done in rolling fashion
a. Edit /etc/infiniband/openib.conf to set RDS_LOAD=yes and SDP_LOAD=yes.
b. Edit the /etc/modprobe.conf file and add this at the end as single line:
options ib_sdp sdp_zcopy_thresh=0 recv_poll=0 sdp_apm_enable=0
Note:
In Exalytics physical environment, disable APM at Exalytics node level.
In Exalytics virtual environment, disable APM at guest VM (DomU) level.
The setting sdp_apm_enable=0 is if Exadata Image is 11.2.3.3.x or 12.1.1.1.x
and if this parameter is defined in the Linux kernel module ib_sdp
modinfo ib_sdp|grep sdp_apm_enable
c. Reboot after change.
2. Create InfiniBand listener on each Exadata compute nodes.
This can be done in rolling fashion
a. Edit /etc/infiniband/openib.conf to set RDS_LOAD=yes and SDP_LOAD=yes.
b. Edit /etc/ofed/libsdp.conf and change both lines.
Change
use both server * :
use both client * :

7
to
use tcp server * *:*
use tcp client * *:*
“use tcp" means that TCP is the default protocol, SDP is available on demand.
c. Edit the /etc/modprobe.conf file and add this at the end as a single line:
options ib_sdp sdp_zcopy_thresh=0 recv_poll=0 sdp_apm_enable=0
The setting sdp_apm_enable=0 is if Exadata Image is 11.2.3.3.x or 12.1.1.1.x
and if this parameter is defined in the Linux kernel module ib_sdp
modinfo ib_sdp|grep sdp_apm_enable
d. Reboot the compute node.
e. Edit /etc/hosts on each Exalytics machine and Exadata Database Machine
compute node to add the virtual IP addresses for use on the InfiniBand
network, for example, Exadata Database Machine quarter rack (2 compute
nodes). Choose a valid IP address on the IB private network for each Exadata
compute node and ensure those IPs are not in use currently.
192.168.10.21 dm01db01-ibvip.mycompany.com dm01db01-ibvip
192.168.10.22 dm01db02-ibvip.mycompany.com dm01db02-ibvip
f. As root user on one of the Exadata Database Machine compute nodes,
execute the following steps to create a network resource for the InfiniBand
network, and add start virtual IP addresses for each compute node on it. Make
appropriate substitutions for the network and subnet mask in use on your
private network (it may not be 192.168.10.0/255.255.255.0). Ask your
administrator or use ipcalc, based on network mask to calculate the network.
srvctl add network -k 2 -S 192.168.10.0/255.255.255.0/bondib0
srvctl add vip -n dm01db01 -A dm01db01-ibvip/255.255.255.0/bondib0 -k 2
srvctl add vip -n dm01db02 -A dm01db02-ibvip/255.255.255.0/bondib0 -k 2
Starting with X4 hardware using Active/Active IB, use instead:
srvctl add network -k 2 -S 192.168.10.0/255.255.255.0/ib0|ib1
srvctl add vip -n dm01db01 -A dm01db01-ibvip/255.255.255.0/ib0|ib1 -k 2
srvctl add vip -n dm01db02 -A dm01db02-ibvip/255.255.255.0/ib0|ib1 -k 2

8
Bug fix 17551223 in GI HOME is required to bring up new cluster ware
resources. It is included in 11.2.0.3 BP21, 11.2.0.4, 12.1.0.1 and
12.1.0.2.
srvctl start vip -i dbm01db01-ibvip
srvctl start vip -i dbm01db02-ibvip
g. On one of the Exadata Database Machine compute nodes, as oracle user (or
owner of the Grid Infrastructure ORACLE_HOME), execute following
command
srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522/SDP:1522
h. On each Exadata Database Machine compute node, as oracle user (or owner
of the Grid Infrastructure ORACLE_HOME), create or append to tnsnames.ora in
GRID_HOME/network/admin the following new entries
Note: *IBREMOTE and *IBLOCAL entries will be slightly different on each
node to reference the proper remote and local VIP names-you cannot copy the
entries from one node to another without some modifications:
DBM =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dm01-
scan.mycompany.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = dbm)
))
DBM_IB =
(DESCRIPTION =
(LOAD_BALANCE=on)
(ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-
ibvip.mycompany.com)(PORT = 1522))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = dbm)
))
LISTENER_IBREMOTE =
(DESCRIPTION =
(ADDRESS_LIST =
))
LISTENER_IBLOCAL =

9
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = SDP)(HOST = dm01db01-
))
LISTENER_IPLOCAL =
(DESCRIPTION =
(ADDRESS_LIST =
vip.mycompany.com)(PORT = 1521))
))
LISTENER_IPREMOTE =
(DESCRIPTION =
(ADDRESS_LIST =
scan.mycompany.com)(PORT = 1521))
))
i. On one Exadata Database Machine compute node, connect to the database
instance as sysdba.
SQL> alter system set listener_networks='((NAME=network2)
(LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LIST
ENER_IBREMOTE))',
'((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE
_LISTENER=LISTENER_IPREMOTE))' scope=both;
j. On one Exadata Database Machine compute node, as oracle user (or owner of
the Grid Infrastructure ORACLE_HOME) restart the listener named
LISTENER_IB.
srvctl stop listener -l LISTENER_IB
srvctl start listener -l LISTENER_IB
Exalytics Deployment on the First Exalytics Node
Follow Section “Deploying Oracle Exalytics for High Availability” in Oracle® Fusion Middleware
Installation and Administration Guide for Oracle Exalytics In-Memory Machine.
In the input.properties file specify the virtual IP addresses you used for the InfiniBand network.
For example:

10
DATABASE_CONNECTION_STRING= dbm01db01-ibvip.mycompany.com:1522:dbm1^dbm01db02-
ibvip.mycompany.com:1522:dbm2@dbm
Respectively in RCU
Connect String: dbm01db01-ibvip.mycompany.com:1522:dbm1^dbm01db02-
ibvip.mycompany.com:1522:dbm2@dbm
Post Installation Steps
Oracle TimesTen
1. Run the daemon configuration scripts as root to start TimesTen on boot.
/u01/app/oracle/product/TimesTen/tt1122/bin/setuproot -install
2. Configure large pages. You can configure large pages for the TimesTen database by
editing the following file and adding the following lines to
/u01/app/oracle/product/TimesTen/tt1122/info/ttendaemon.options
-linuxLargePageAlignment 2
3. Define the logical Local and Remote server name for Oracle TimesTen in
/u01/app/oracle/product/TimesTen/tt1122/info/sys.ttconnect.ini
[tt_exalytics_local]
Description=TimesTen Server
Network_Address=exalytics1.mycompany.com
TCP_PORT=53397
[tt_exalytics_remote]
Description=TimesTen Server
TCP_PORT=53397
4. Define the Local and Remote TimesTen In-Memory Database using a DSN that is
contained in the /u01/app/oracle/product/TimesTen/tt1122/info/sys.odbc.ini file.
[ODBC Data Sources]
TT_AGGR_STORE_LOCAL=TimesTen 11.2.2 Driver
TT_AGGR_STORE_REMOTE=TimesTen 11.2.2 Client Driver
# The DSN below defines a TimesTen In-Memory Database with 40GB devoted to
# tables and indexes and 40GB for handling queries and their results,

11
# the overall memory consumption will be approximately 81GB (PermSize + TempSize
+ LogBufMB + 64).
[TT_AGGR_STORE_LOCAL]
Driver=/u01/app/oracle/product/fmw/../TimesTen/tt1122/lib/libtten.so
DataStore=/u01/app/oracle/product/fmw/../aggregate_store/tt_aggr_store
LogDir=/u01/app/oracle/product/fmw/../aggregate_store/logs
DatabaseCharacterSet=AL32UTF8
ConnectionCharacterSet=AL32UTF8
LogFileSize=1024
LogBufMB=1024
LogBufParallelism=16
Preallocate=0
PermSize=40000
TempSize=40000
MemoryLock=4
CkptFrequency=30
CkptLogVolume=0
CkptRate=0
PrivateCommands=1
RecoveryThreads=40
# The DSN bellow references it remote equivalent in case the local TimesTen instance
# fails. This DSN will be used by the BI Server in cases where it detects that the
# local TimesTen database is no longer available.
[TT_AGGR_STORE_REMOTE]
TTC_SERVER = tt_exalytics_remote
TTC_SERVER_DSN = TT_AGGR_STORE_LOCAL
5. Instantiate Oracle TimesTen Database. The TimesTen In-Memory Database is
instantiated when the first connection is made to the database.
cd /u01/app/oracle/product/TimesTen/tt1122/bin
./ttisql
connect dsn=TT_AGGR_STORE_LOCAL;
create user exalytics identified by password;
grant create session to exalytics;
grant create table to exalytics;
grant select on SYS.OBJ$ to exalytics;
Use the following command to test the connection to the local TimesTen In-Memory
Database. There are no tables yet configured to return.

12
CONNECT “DSN=TT_AGGR_STORE_LOCAL;uid=exalytics”;
tables;
EXIT;
The Connectivity Test to the remote TimesTen In-Memory Database must be
postponed until it has been installed and configured.
Oracle WebLogic Server
1. Create boot.properties for Admin Server to start without prompting you for the
administrator username and password in
/u01/app/oracle/product/fmw/user_projects/domains/bifoundation_domain/servers/AdminSe
rver/security.
password=<WLS Admin User Password>
username=<WLS Admin User>
2. Enable SDP Support for JDBC.
a. Ensure that you have specified the virtual IP addresses used for InfiniBand
network during BI Domain creation. The console automatically generates the
complete JDBC URL as shown in the following example:
For mds-owsm multi data source, we have 2 generic data sources:
 mds-owsm-rac0
URL:
jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOC
OL=TCP)(HOST= dm01db01-
ibvip.mycompany.com)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME=
dbm)(INSTANCE_NAME=dbm1)))
 mds-owsm-rac1
URL:
OL=TCP)(HOST= dm01db02-
b. In the JDBC URL, replace TCP protocol with SDP protocol. For example:
 mds-owsm-rac0
URL:
OL=SDP)(HOST= dm01db01-
 mds-owsm-rac1
URL:
OL=SDP)(HOST= dm01db02-

13
c. Manually add the system property –Djava.net.preferIpv4Stack=true to the
startWebLogic.sh script, which is located in the bin directory of DOMAIN_HOME
as follows:
i. Locate the following line in the startWebLogic.sh script:
. ${DOMAIN_HOME}/bin/setDomainEnv.sh $*
ii. Add the following property immediately after the above entry:
JAVA_OPTIONS=”${JAVA_OPTIONS} –Djava.net.preferIPv4Stack=true –
Doracle.net.SDP=true”
iii. Save the file and close it.
iv. Restart the BI Domain.
Oracle Business Intelligence
Configure ODBC Connection from Oracle Business Intelligence to Oracle TimesTen in
/u01/app/oracle/product/fmw/instances/instance1/bifoundation/OracleBIApplication/coreapplication/s
etup/odbc.ini to contain references to the TimesTen database.
[ODBC Data Sources]
AnalyticsWeb = Oracle BI Server
Cluster = Oracle BI Server
SSL_Sample = Oracle BI Server
TT_AGGR_STORE_LOCAL = TimesTen 11.2.2 Driver
TT_AGGR_STORE_REMOTE = TimesTen 11.2.2 Driver
[TT_AGGR_STORE_LOCAL]
Driver = /u01/app/oracle/product/fmw/../TimesTen/tt1122/lib/libttclient.so
TTC_SERVER_DSN = TT_AGGR_STORE_LOCAL
TTC_SERVER = tt_exalytics_local
TTC_TIMEOUT = 0
[TT_AGGR_STORE_REMOTE]
Driver = /u01/app/oracle/product/fmw/../TimesTen/tt1122/lib/libttclient.so
TTC_SERVER_DSN = TT_AGGR_STORE_REMOTE
TTC_SERVER = tt_exalytics_remote
TTC_TIMEOUT = 0
Basic Exalytics Start-Stop Scripts first Node

14
These scripts are only examples and can be enhanced for better security and prerequisite
checking as required. The goal of these examples is to provide the necessary commands and
ordering for starting and stopping the processes on the Exalytics system.
startExalytics.sh
nohup
/u01/app/oracle/product/fmw/user_projects/domains/bifoundation_domain/startWebLogic.sh &
sleep 60
nohup /u01/app/oracle/product/fmw/wlserver_10.3/server/bin/startNodeManager.sh &
export ORACLE_INSTANCE=/u01/app/oracle/product/fmw/instances/instance1
/u01/app/oracle/product/fmw/instances/instance1/bin/opmnctl startall
/u01/app/oracle/product/fmw/instances/instance1/bin/opmnctl status
nohup
/u01/app/oracle/product/fmw/user_projects/domains/bifoundation_domain/bin/startManagedWeb
Logic.sh bi_server1 t3://exalytics1:7001 &
/u01/app/oracle/product/TimesTen/tt1122/bin/ttDaemonAdmin –start
stopExalytics.sh
/u01/app/oracle/product/fmw/user_projects/domains/bifoundation_domain/bin/stopManagedWeb
Logic.sh bi_server1 t3://exalytics1:7001 biadmin password
/u01/app/oracle/product/fmw/instances/instance1/bin/opmnctl stopall
/u01/app/oracle/product/TimesTen/tt1122/bin/ttAdmin –ramUnload
TT_AGGR_STORE_LOCAL (See note below)
/u01/app/oracle/product/TimesTen/tt1122/bin/ttDaemonAdmin –stop
/u01/app/oracle/product/fmw/user_projects/domains/bifoundation_domain/bin/stopWebLogic.sh
biadmin password
ps -ef|grep NodeManager|grep -v grep|perl -lane 'print $F[1]'|xargs kill -9
Note: It is extremely important that any active TimesTen databases are properly and cleanly
shut down before stopping the main daemon. Because the recommended setup on Exalytics is
for all TimesTen databases to use ramPolicy manual, it is necessary to explicitly shut down the
local database using ttAdmin -ramUnload TT_AGGR_STORE_LOCAL, and wait for it to complete
prior to stopping the main daemon. If there are any application connections open against the
database then ramUnload cannot complete.

15
Exalytics Deployment on the Second Exalytics Node
Follow Section “Configuring for High Availability” Step 2 "Preparing the Second Exalytics
Machine” in Oracle® Fusion Middleware Installation and Administration Guide for Oracle
Exalytics In-Memory Machine to deploy Exalytics.
Oracle TimesTen Installation
1. Install TimesTen using the Installer script
/home/oracle/EXALYTICS_INSTALLERS/tt/setup.sh.
2. Run the Post Installation Steps for TimesTen described for the first Exalytics Node.
The only difference will be
/u01/app/oracle/product/TimesTen/tt1122/info/sys.ttconnect.ini
[tt_exalytics_local]
Description=Local TimesTen Server
TCP_PORT=53397
[tt_exalytics_remote]
Description=Remote TimesTen Server
TCP_PORT=53397
Oracle WebLogic Server Installation
1. Use the WLS Installer script to install Oracle WebLogic Server and create the
Middleware Home.
/home/oracle/EXALYTICS_INSTALLERS/wls/wls1036_linux64.bin
In the Choose Middleware Home Directory screen, for Middleware Home Directory,
enter the same directory as the first Exalytics Machine. For example:
/u01/app/oracle/product/fmw
In the Choose Install Type screen, select Custom and deselect Coherence.
Coherence does not need to be installed.
In the Installation Complete screen, clear the Run Quickstart option and click Done.
2. Update /u01/app/oracle/product/fmw/instances/instance2/config/OPMN/opmn/opmn.xml
to point to the driver location for ODBC for Oracle TimesTen by editing the following
"variable" elements. The edited text is shown in bold.
<variable id="LD_LIBRARY_PATH"
value="$ORACLE_HOME/common/ODBC/Merant/5.3/lib$:$ORACLE_HOME/bifoundatio
n/server/bin$:$ORACLE_HOME/bifoundation/web/bin$:$ORACLE_HOME/clients/epm/Es

16
sbase/EssbaseRTC/bin$:$ORACLE_HOME/bifoundation/odbc/lib$:$ORACLE_INSTANCE$
:$ORACLE_HOME/lib:/u01/app/oracle/product/TimesTen/tt1122/lib"
append="true"/>
<variable id="TIMESTEN_DLL"
value="/u01/app/oracle/product/TimesTen/tt1122/lib/libttclient.so"/>
3. Restart OPMN.
Oracle Business Intelligence Installation
1. Use the BI runInstaller script to install the BI Software.
(/home/oracle/EXALYTICS_INSTALLERS/bi/bishiphome/Disk1/runInstaller).
In the Select Installation Type screen, select Software Only Install.
2. Set up Shared File Locations for Oracle BI.
As part of configuring Exalytics cluster, you must configure the following persistent
stores to be placed on shared storage to ensure that they are available from both
Exalytics machines.
 RPD Publishing Directory – Repository for Oracle BI Server. Specify a shared RPD
publishing directory in Fusion Middleware Control to propagate online repository
changes in a cluster.
 Oracle BI Presentation Catalog – Each Oracle BI Presentation Services instance
loads the catalog from the catalog location that is specified in Fusion Middleware
Control. Copy any existing catalogs to the shared storage before updating the
location of the Oracle BI Presentation Catalog.
 Global Cache – The global cache resides on shared storage and stores purging
events, seeding events, and results sets that are associated with seeding events.
 BI Publisher Configuration Folder
 BI Publisher Scheduler Temp Directory
3. Scale Out the Existing BI Domain using
/u01/app/oracle/product/fmw/Oracle_BI1/bin/config.sh.
4. Scale Out System Components using Fusion Middleware Control as described in
Oracle Fusion Middleware System Administrator's Guide for Oracle Business
Intelligence Enterprise Edition:
 The Oracle BI Server
 Oracle BI Presentation Services
 JavaHost
5. Configure Secondary Instances of Singleton System Components using Fusion
Middleware Control

17
 Oracle BI Cluster Controller
 Oracle BI Scheduler
6. Configure ODBC connection from Oracle Business Intelligence to Oracle TimesTen in
/u01/app/oracle/product/fmw/instances/instance2/bifoundation/OracleBIApplication/coreapp
lication/setup/odbc.ini as described for the first Exalytics Node
Basic Exalytics Start-Stop Scripts Second Node
These scripts are examples and can be enhanced for better security and prerequisite checking
as required. The goal of these examples is to provide the necessary commands and ordering
for starting and stopping the processes on the Exalytics system.
startExalytics.sh
nohup /u01/app/oracle/product/fmw/wlserver_10.3/server/bin/startNodeManager.sh &
/u01/app/oracle/product/fmw/instances/instance2/bin/opmnctl startall
/u01/app/oracle/product/fmw/instances/instance2/bin/opmnctl status
nohup
/u01/app/oracle/product/fmw/user_projects/domains/bifoundation_domain/bin/startManagedWeb
Logic.sh bi_server2 t3://exalytics1:7001 &
/u01/app/oracle/product/TimesTen/tt1122/bin/ttDaemonAdmin –start
stopExalytics.sh
/u01/app/oracle/product/fmw/user_projects/domains/bifoundation_domain/bin/stopManagedWeb
Logic.sh bi_server2 t3://exalytics1:7001 biadmin password
/u01/app/oracle/product/fmw/instances/instance1/bin/opmnctl stopall
/u01/app/oracle/product/TimesTen/tt1122/bin/ttAdmin –ramUnload
TT_AGGR_STORE_LOCAL
/u01/app/oracle/product/TimesTen/tt1122/bin/ttDaemonAdmin –stop
ps -ef|grep NodeManager|grep -v grep|perl -lane 'print $F[1]'|xargs kill -9
Building Aggregates
Aggregates are built by running an Aggregate Persistence script, this creates the aggregate
star, it populates it with data, and it updates the RPD with new LTS entries. It is a very
convenient mechanism for creating new aggregates to accelerate query performance. In

18
addition to manual execution of the Aggregate Persistence scripts, Exalytics offers the
Summary Advisor, a feature that uses query logs to recommend what aggregates to create,
and to then build the appropriate Aggregate Persistence scripts. These scripts then need to be
executed.
This section looks at how to execute Aggregate Persistence scripts; it does not explain how to
run Summary Advisor nor does it cover design consideration for building these scripts.
For more Information look at Fusion Middleware Metadata Repository Builder's Guide for
Oracle Business Intelligence Enterprise Edition, Chapter Creating and Persisting Aggregates
for Oracle BI Server Queries
We will use an example to illustrate a recommended approach.
Let us assume that we have the following RPD configuration:
Figure 3: Logical and Physical elements of the RPD prior to running aggregate create script
Create Aggregates in the Local TimesTen Instance
Execute the following script.
set variable LOGLEVEL=7 :
create aggregates "ag_F_Sales_Yr_Ln"
for "BMV"."F - Sales"(

19
"Units sum","Revenue sum",
"Units avg","Revenue avg"
)
at levels(
"BMV"."Time"."Year" using_surrogate_key,
"BMV"."Stock"."Line" using_surrogate_key
)
using connection pool
"TT_AGGR_STORE"."TT_AGGR_CP"
in "TT_AGGR_STORE".."DBO";
The BI server will read the “F - Sales” star from the most efficient source, in this case BMV
Data Base (ORCL) and create aggregates at the level of Year and Line, deploying them
under the DBO user in TimesTen as defined by the connection pool
TT_AGGR_STORE.TT_AGGR_CP. For this step we will ensure that this points to our local
TimesTen instance.
Figure 4: Logical and Physical elements of the RPD after aggregate create script
Create Aggregates in the Remote TimesTen Instance
Copy and modify the script in the previous section so that it can create aggregates in the
remote TimesTen instance. The modifications are highlighted below.

20
)
at levels(
)
"TT_AGGR_STORE_REMOTE"."TT_AGGR_CP"
in "TT_AGGR_STORE_REMOTE".."DBO”;
The BI server will read the “F - Sales” star from the most efficient source; here it will use the
TT_AGGR_STORE database and create replicas of the aggregates at the level of Year and
Line, deploying them under the DBO user in the remote TimesTen as defined by the
connection pool TT_AGGR_STORE_REMOTE.TT_AGGR_CP.
Figure 4: Logical and Physical elements of the RPD after running the 2nd aggregate create script

21
As both LTSes generated by the scripts are at the same level of aggregation, the BI Server will
select the first one in the list, and so the local TimesTen instance will be prioritized. If the local
TimesTen instance is not available, then the BI Server will go to the remote TimesTen
instance.
The RPD is now ready and should be made available to both BI Servers via a rolling restart.
Refresh Aggregates in the Local TimesTen instance
Copy and modify the initial “local” script so that it can drop and recreate the aggregates from
the disk-oriented data warehouse (a less efficient data source with fresh data) and not from the
remote in-memory TimesTen instance (a more efficient data source with state data). The
changes are highlighted in yellow.

22
delete aggregates "TT_AGGR_STORE".."DBO"."ag_F_Sales_Yr_Ln";
set variable LOGLEVEL=7,
set variable INACTIVE_SCHEMAS= '"TT_AGGR_STORE_REMOTE".."DBO"' :
for " BMV"."F - Sales"(
)
at levels(
)
"TT_AGGR_STORE"."TT_AGGR_CP"
in "TT_AGGR_STORE".."DBO";
The BI server will now ignore TT_AGGR_STORE_REMOTE and will issue the query to build
aggregates using BMV Data (ORCL) the disk-oriented database schema with the latest data.
Refresh Aggregates in the Remote TimesTen instance
Copy and modify the initial “remote” script in order for it to drop and recreate the aggregates
from the remote TimesTen instance. The changes are highlighted in yellow.
delete aggregates "TT_AGGR_STORE_REMOTE".."DBO"."ag_F_Sales_Yr_Ln";
)
at levels(
)
"TT_AGGR_STORE_REMOTE"."TT_AGGR_CP"
in "TT_AGGR_STORE_REMOTE".."DBO”;
The BI server will read the “F - Sales” star from the most efficient source; here it will use the
TT_AGGR_STORE database and create replicas of the aggregates at the level of Year and
Line, deploying them under the DBO user in the remote TimesTen as defined by the
connection pool TT_AGGR_STORE_REMOTE.TT_AGGR_CP.

23
Additional Considerations
 Tune the BI Server Cache on an Exalytics machine to reside on RAM disk so it can also
benefit from in-memory capabilities. First create the actual RAM disk. Then mount the BI
Server Cache directory to RAM disk so that it is running in memory. The tmpfs mount will give
you the power of fast reading and writing of files, from and to, the primary memory.
 Configure Usage Tracking and Summary Advisor Settings.
Usage Tracking is required for Summary Advisor Statistics. For more details refer to
"Managing Usage Tracking" in Oracle Fusion Middleware System Administrator's Guide for
Oracle Business Intelligence Enterprise Edition
 Configure SDP-enabled JDBC Drivers for GridLink Data Sources for JDBC connectivity.
For more details refer to Fusion Middleware Configuring and Managing JDBC Data Sources
for Oracle WebLogic Server and see Enable SDP Support for JDBC in this document
 Exalytics 10GbE Configuration
Currently InfiniBand on OVM on Exalytics is not supported. For details check
When Will Oracle Exalytics Support InfiniBand with OVM (Doc ID 1918880.1)
For Best Practices using 10Gb Interfaces on Exalytics for Exadata connectivity consider
Oracle VM 3: 10GbE Network Performance Tuning (Doc ID 1519875.1)
resp.
Best Practices for Oracle Solaris Network Performance with Oracle VM Server for SPARC
(Doc ID 1908136.1)
 Migration Steps from 10GbE to InfiniBand Configuration
o Create IB Listener and add Database listener_networks Parameter (see
Exalytics and Exadata InfiniBand Configuration)
o Change WLS data sources to use SDP and InfiniBand IPs (see Enable SDP
Support for JDBC above).
Note: Some of the DataSources still don’t support SDP Protocol. Use TCP
instead of SDP in such cases
 Exalytics and 12c IMDB
For supported local configurations and recommended 12c Database Parameters check
Installing Oracle Database 12c In-Memory (IMDB) for Oracle Exalytics (Doc ID 1925793.1)
 Exalytics Certification Matrix
Exalytics Certification Matrix depends on Exalytics Patchset Version and can be found at
http://www.oracle.com/technetwork/middleware/bi/oracle-exalytics-ps4-2147225.xls

Oracle Exalytics High Availability Configuration
November 2014
Authors: Ventzislav Stankov, Susan Kornberg,
Socs Cappas
Oracle Corporation
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.
Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
oracle.com
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the
contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or
fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are
formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any
means, electronic or mechanical, for any purpose, without our prior written permission.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license
and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open
Company, Ltd. 1010

Oracle exalytics deployment for high availability

More Related Content

What's hot (20)

Similar to Oracle exalytics deployment for high availability (20)

More from Paulo Fagundes (10)

Recently uploaded (20)

Oracle exalytics deployment for high availability