les_05.ppt of the Oracle course train_5 file

Editor's Notes

• #3: Adding Oracle Clusterware Homes There are mainly three methods that you can use to add and delete nodes in an Oracle Clusterware environment: Silent cloning procedures: Cloning enables you to copy images of an Oracle Clusterware installation to other nodes of new clusters or nodes to extend existing clusters that have identical hardware by using the clone.pl script. This is only for adding nodes. Enterprise Manager (EM) Grid Control: EM provides a GUI interface and automated wizards to the cloning procedures. addNode.sh/rootdeletenode.sh: Interactive or silent procedures use shell scripts that invoke a subset of OUI functionality. In this lesson, you examine the use of the addNode.sh and rootdeletenode.sh scripts. Special attention must be given to the procedure because some steps are performed on the existing nodes, whereas other steps are performed on the nodes that are being added or removed. Note: In this lesson, assume that host01 and host02 are the existing nodes, with host03 being the node that will be added or removed.
• #4: Prerequisite Steps for Running addNode.sh The addNode.sh script is used to extend an existing, successful Oracle Clusterware installation to more nodes. Before the addNode.sh script can be run, several prerequisite steps must be performed. For step 1, the new node must be physically connected to the existing cluster network infrastructure to include the public, private, storage, and other connections that may exist. Remember that all nodes must use the same adapter names for the public and private interfaces. For step 2, install a cloned image of the operating system that matches the operating system on the other nodes in the cluster, including the required service patches, drivers, and modifications to configuration files. If a cloned image is not possible, the individual modifications that were performed as prerequisite tasks for installing Oracle Clusterware will have to be performed on the new node for step 3. The provisioning of storage prerequisite task is not listed because this step has already been performed by the existing nodes. You need to ensure that Secure Shell (SSH) is configured to operate without prompts for both the fully qualified names and nonqualified host names. This involves updates to the authorized_keys and known_hosts files of the existing nodes in addition to the new nodes being added.
• #5: Prerequisite Steps for Running addNode.sh (continued) You have already seen a variety of methods by which Cluster Verify Utility (cluvfy) can be invoked. A complete reference to cluvfy is provided in the lesson titled “Troubleshooting Oracle Clusterware.” For step 4, you invoke cluvfy from an existing node (host01) to perform a post-hardware and operating system installation check against the new node (host03). Next, you perform a detailed properties comparison with cluvfy by using an existing node as a reference node, and comparing it against the new node to be added. If errors are discovered while performing these checks, they must be corrected before continuing with the addNode.sh script. One aspect of running addNode.sh in which many human errors are made is keeping track of the proper node. Always be aware of the following two questions: Which node is the command being run on: existing or new? When node(s) are being supplied as parameters, is the parameter the existing node, the new node being added, or all the nodes? In the slide examples, host01 is an existing node and host03 is the new node that is being added.
• #6: Running OUI with addNode.sh OUI is used to distribute the Oracle Clusterware software from an existing node to the new nodes being added to the existing cluster. This process is similar to the technique by which the software was distributed during the original installation. This process is started by running the <CRS Home>/oui/bin/addNode.sh script. Although this is a shell script, it invokes OUI and must be run from a session that is capable of displaying graphical windows. The addNode.sh shell script is not usually located with the PATH environment variable. Running the OUI utility with addNode.sh is different from running the utility with the runInstaller command that was used during installation. Because the installation has already been performed successfully with values supplied to all the installation questions, addNode.sh prompts for information only about the new node that is being added and skips prompts that are based on information that is already known. When the addNode.sh script is run, the Welcome window is displayed. Click the Next button to continue with the process.
• #7: Specifying the New Nodes The “Specify Cluster Nodes to Add to Installation” window displays the nodes of the existing cluster. This information cannot be changed. In the Specify New Nodes section, you can enter information about the new nodes. Multiple nodes can be added at a time. Enter the public node name, private node name, and virtual host name for each new node. Enter the short name or unqualified name of the nodes instead of the fully qualified names as displayed in the Existing Nodes section. Click the Next button to continue this process.
• #8: Reviewing the Summary Window The “Cluster Node Addition Summary” window is then displayed for your review. Check the spelling of the new host names that have been entered. Notice that the space required for the new node is 709 MB. This is larger than the 650 MB requirement suggested in the installation documentation. The Oracle Clusterware software home will grow in size over time due to log files, tracing files, patches, automatic backups of Oracle Cluster Registry (OCR), and backup copies of some files. The 709 MB requirement for space is based on the current size of the software home on the existing node. Click the Install button to continue the process.
• #9: Cluster Node Addition Progress The Cluster Node Addition Progress window displays the progress of the process to add a node. Notice that a log file with the details of the node addition session has been created in the Oracle Central Inventory directory with the name addNodeActions<timestamp>.log. The log file contains environment variables that were received from the command line or by running various scripts to obtain them for each OUI window that has occurred. It also shows detailed progress of each step in the node addition process. OUI is invoked on every node in the cluster. The existing nodes must have the OUI utility update their local Central Oracle Inventory to reflect the new nodes that are being added.
• #10: Running Configuration Scripts on Selected Nodes Next, OUI displays a dialog box indicating that you must run several scripts as the root user on selected nodes. Notice that each script has a different name, is located in a different directory, and is run on a different node. Do not run these scripts in parallel. When the final script has been run, click the OK button to close the dialog box. The output of the root scripts is as follows: [root@host03]# /u01/app/oraInventory/orainstRoot.sh Changing permissions of /u01/app/oraInventory to 770. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete [root@host01]# /u01/app/crs/install/rootaddnode.sh clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Attempting to add 1 new nodes to the configuration
• #11: Running Configuration Scripts on Selected Nodes (continued) Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 3: host03 host03-priv host03 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. /u01/app/crs/bin/srvctl add nodeapps -n host03 -A host03-vip/255.255.254.0/eth0 [root@host03]# /u01/app/crs/root.sh WARNING: directory '/u01/app' is not owned by root Checking to see if Oracle CRS stack is already configured /etc/oracle does not exist. Creating it now. OCR LOCATIONS = /dev/sda1,/dev/sdb1 OCR backup directory '/u01/app/crs/cdata/ORL_Cluster' does not exist. Creating now Setting the permissions on OCR backup directory Setting up Network socket directories Oracle Cluster Registry configuration upgraded successfully The directory '/u01/app' is not owned by root. Changing owner to root clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: host01 host01-priv host01 node 2: host02 host02-priv host02 clscfg: Arguments check out successfully. NO KEYS WERE WRITTEN. Supply -force parameter to override. -force is destructive and will destroy any previous cluster configuration. Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 30 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds.
• #12: End of Node Addition On the “End of Installation” window, click the Exit button. Then click the Yes button in the Exit window to exit the OUI application. Node addition is now complete.
• #13: Completing OUI Interactive Node Addition For step 5, run the ONS configuration utility to add the new node to the ONS configuration. The default port is 6251 for remote communication. The port number that is being used for the existing nodes of the cluster can be obtained by using the ocrdump command to generate a text file containing the values of OCR. The resulting text file can then be searched for the string pattern DATABASE.ONS_HOSTS.<hostname>.PORT to find the port numbers. It is not a requirement for all nodes to use the same port number. A different port can be specified for the new node than the ones used by other nodes. If <CRS_home>/opmn/conf/ons.config contains the useocr=on entry, the ocrdump command should be used to verify ports because each node can be distinct. If the useocr=off entry (not recommended) is present, the port numbers will be defined in this file under the remoteport setting. If OCR is configured to use a different port number from the ons.config file, errors will result. For step 6, perform integrity checks on the cluster to verify a successful node addition.
• #14: Adding a Node with the Silent Option The addNode.sh script can be invoked with the silent option so that no graphical windows are displayed. The DISPLAY environment variable must be set; otherwise, the script will fail to run successfully. For an existing cluster containing the host01 and host02 nodes, a new node host03 will be added as follows: [crs@host01]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={host03}" "CLUSTER_NEW_PRIVATE_NODE_NAMES={host03-priv}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={host03-vip}" Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 4095 MB Passed Oracle Universal Installer, Version 11.1.0.6.0 Production Copyright (C) 1999, 2007, Oracle. All rights reserved. Performing tests to see whether nodes host02,host03 are available ............................................................... 100% Done. ------------------------------------------------------------------------------------------------- Cluster Node Addition Summary
• #15: Adding a Node with the Silent Option (continued) Global Settings Source: /u01/app/crs New Nodes Space Requirements New Nodes host03 /u01: Required 712MB : Available 13.29GB Installed Products Product Names Oracle Clusterware 11.1.0.6.0 Sun JDK 1.5.0.1.1 Installer SDK Component 11.1.0.6.0 Oracle One-Off Patch Installer 11.1.0.6.0 Oracle Universal Installer 11.1.0.6.0 Cluster Verification Utility Files 11.1.0.6.0 Oracle Notification Service 11.1.0.5.0 Cluster Verification Utility Common Files 11.1.0.6.0 Oracle Clusterware RDBMS Files 11.1.0.6.0 Bali Share 1.1.18.0.0 Oracle Extended Windowing Toolkit 3.4.47.0.0 Buildtools Common Files 11.1.0.6.0 Oracle RAC Required Support Files-HAS 11.1.0.6.0 SQL*Plus Required Support Files 11.1.0.6.0 XDK Required Support Files 11.1.0.6.0 Agent Required Support Files 10.2.0.3.1 Parser Generator Required Support Files 11.1.0.6.0 Precompiler Required Support Files 11.1.0.6.0 Platform Required Support Files 11.1.0.6.0 Oracle Core Required Support Files 11.1.0.6.0 Oracle Globalization Support 11.1.0.6.0 Perl Interpreter 5.8.3.0.4 Oracle Ice Browser 5.2.3.6.0 Oracle JFC Extended Windowing Toolkit 4.2.36.0.0 Oracle Help For Java 4.2.9.0.0 LDAP Required Support Files 11.1.0.6.0 SSL Required Support Files for InstantClient 11.1.0.6.0
• #16: Adding a Node with the Silent Option (continued) SSL Required Support Files for InstantClient 11.1.0.6.0 Oracle Net Required Support Files 11.1.0.6.0 RDBMS Required Support Files for Instant Client 11.1.0.6.0 RDBMS Required Support Files 11.1.0.6.0 Enterprise Manager Minimal Integration 11.1.0.6.0 Oracle Locale Builder 11.1.0.6.0 Oracle Globalization Support 11.1.0.6.0 HAS Common Files 11.1.0.6.0 Cluster Ready Services Files 11.1.0.6.0 Required Support Files 11.1.0.6.0 ------------------------------------------------------------------ Instantiating scripts for add node (Tue Oct 07 21:45:40 EDT 2008) . 1% Done. Instantiation of add node scripts complete Copying to remote nodes (Tue Oct 07 21:45:43 EDT 2008) ............................................................................................... 96% Done. Home copied to new nodes Saving inventory on nodes (Tue Oct 07 21:45:58 EDT 2008) . 100% Done. Save inventory complete Warning: A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system. To register the new inventory, run the script at /u01/app/oraInventory/orainstRoot.sh with root privileges on the host03 node. If you do not register the inventory, you may not be able to update or patch the products that you installed. The following configuration scripts need to be executed as the root user in each cluster node: #!/bin/sh #Root script to run /u01/app/oraInventory/orainstRoot.sh #On nodes host03 /u01/app/crs/install/rootaddnode.sh #On nodes host01 /u01/app/crs/root.sh #On nodes host03 To execute the configuration scripts, perform the following steps: 1. Open a terminal window. 2. Log in as root.
• #17: Adding a Node with Silent Option (continued) 3. Run the scripts in each cluster node. The cluster node addition of /u01/app/crs was successful. Check /tmp/silentInstall.log for more details. [root@host03]# /u01/app/oraInventory/orainstRoot.sh Changing permissions of /u01/app/oraInventory to 770. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete [root@host01]# /u01/app/crs/install/rootaddnode.sh clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Attempting to add 1 new nodes to the configuration Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 3: host03 host03-priv host03 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. /u01/app/crs/bin/srvctl add nodeapps -n host03 -A host03-vip/255.255.254.0/eth0 [root@host03]# /u01/app/crs/root.sh Note: The output is almost identical to the root.sh output shown on the earlier notes page that is titled “Running Configuration Scripts on Selected Nodes.”
• #18: Completing OUI Silent Node Addition For step 3, run the ONS utility to add the new node to the ONS configuration. The default port is 6251, but could have been changed to another value. For step 4, perform integrity checks on the cluster to verify a successful node addition.
• #19: Removing a Node from the Cluster On each node in the cluster, Oracle Central Inventory on that node contains information about all the nodes of the cluster. The binary OCR and voting disk also contain information about each node of the cluster. Therefore, to remove a node from the cluster properly, a procedure must be followed. You cannot simply remove the node. The procedure to remove a node from the cluster involves running scripts such as rootdelete.sh and rootdeletenode.sh, along with the OUI utility. Some scripts must be run on a node remaining in the cluster, and other scripts must be run on the node to be deleted. Some scripts are run as the clusterware owner user and others as the root user.
• #20: Deleting a Node from the Cluster Deleting a node from the cluster is a multiple-step process. Some commands are run on the node to be deleted and other commands are run on an existing node of the cluster. Some commands are run by the root user and other commands are run by the Oracle Clusterware software owner’s account. When passing arguments to a command, sometimes the existing node is passed, sometimes the node to be removed is passed, and other times a complete list of remaining nodes is passed as an argument. This requires special attention to detail to avoid making mistakes during the process. For step 1, verify the location of the Oracle Clusterware home. This directory should be consistent on all nodes. For step 2, remove the stored network configuration. This step may not be necessary if the interfaces were installed with the –global option. If the interfaces were installed with the –global option, the following error message is returned: [crs@host01 ~]# oifcfg delif -node host03 PROC-4: The cluster registry key to be operated on does not exist. PRIF-11: cluster registry error For step 3, perform an ocrdump command to dump OCR to the current directory. Using any text editor, locate the port number for the ONS process of the node to be deleted. Note: This procedure assumes that the node to be removed can be accessed. A different procedure must be followed if you cannot execute commands on the node to be removed.
• #21: Deleting a Node from the Cluster (continued) For step 4, remove the ONS daemon configuration from a node that will remain in the cluster using the port number obtained in step 3 as follows: [crs@host01 ~]$ racgons remove_config host03:6251 racgons: Existing key value on host03 = 6251. racgons: host03:6251 removed from OCR. For step 5, disable the Oracle Clusterware applications connected as the root user on the node to be deleted as follows: [root@host03 install]# /u01/app/crs/install/rootdelete.sh Getting local node name NODE = host03 Stopping resources. This could take several minutes. Successfully stopped Oracle Clusterware resources Stopping Cluster Synchronization Services. Shutting down the Cluster Synchronization Services daemon.
• #22: Deleting a Node from the Cluster (continued) Shutdown request successfully issued. Waiting for Cluster Synchronization Services daemon to stop Cluster Synchronization Services daemon has stopped Oracle CRS stack is not running. Oracle CRS stack is down now. Removing script for Oracle Cluster Ready services Updating ocr file for downgrade Cleaning up SCR settings in '/etc/oracle/scls_scr' Cleaning up Network socket directories The rootdelete.sh script stops all the Oracle Clusterware processes on the node to be deleted, both wrapper script processes and daemons. It also removes the entries from the /etc/inittab file so that Oracle Clusterware will not attempt to start if the machine is rebooted.
• #23: Deleting a Node from the Cluster (continued) For step 6, disable the Oracle Clusterware application as root while connected to a node that will remain in the cluster. The node number of the node to be removed will be needed for the command. The node number can be obtained with the olsnodes command from any node as follows: [root@host01]$ olsnodes -n host01 1 host02 2 host03 3 After the node number has been obtained, run the rootdeletenode.sh script as root on a node that will remain in the cluster as follows: [root@host01]# /u01/app/crs/install/rootdeletenode.sh host03,3 CRS-0210: Could not find resource 'ora.host03.ons'. CRS-0210: Could not find resource 'ora.host03.vip'. CRS-0210: Could not find resource 'ora.host03.gsd'. PRKO-2112 : Some or all node applications are not removed successfully on node: host03
• #24: Deleting a Node from the Cluster (continued) CRS-0210: Could not find resource 'ora.host03.vip'.CRS-0210: Could not find resource 'ora.host03.ons'.CRS-0210: Could not find resource 'ora.host03.gsd'. CRS nodeapps are deleted successfully clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Value SYSTEM.crs.versions.host03 marked for deletion is not there. Ignoring. Successfully deleted 15 values from OCR. Key SYSTEM.css.interfaces.nodehost03 marked for deletion is not there. Ignoring. Key SYSTEM.crs.versions.host03 marked for deletion is not there. Ignoring. Successfully deleted 13 keys from OCR. Node deletion operation successful. 'host03,3' deleted successfully For step 7, use the runInstaller utility from the node that is to be deleted as the crs user to remove the node from the node list. Use the –local option as follows: [crs@host03]$ /u01/app/crs/oui/bin/runInstaller \ -updateNodeList ORACLE_HOME=/u01/app/crs \ "CLUSTER_NODES={host03}" CRS=TRUE -local Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 3007 MB Passed Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oraInventory 'UpdateNodeList' was successful. Note: Although the OUI graphical utility is being invoked with this command, a graphical session is not needed for this step.
• #25: Deleting a Node from the Cluster (continued) For step 8, the runInstaller utility is used as the crs user on the node to be deleted to deinstall the software from the machine and adjust the Central Oracle Inventory directory for all nodes. The runInstaller utility must be invoked from a session that is capable of displaying graphical windows.
• #26: Deleting a Node from the Cluster (continued) After the runInstaller utility is invoked in step 8, the Welcome window of OUI is displayed and an Inventory window immediately appears. Check the box to select the Oracle Home for the Oracle Clusterware software identified by its name. Then, click the Remove button to continue.
• #27: Deleting a Node from the Cluster (continued) After you click the Remove button, the Confirmation window appears. This window lists the products that are about to be removed along with their dependent components. Verify in the list presented that only the Oracle Clusterware components are being removed, and then click the Yes button to continue.
• #28: Deleting a Node from the Cluster (continued) Now a Warning window appears, which gives you one last chance to abort this destructive process. The Warning window presents the option to remove the <CRS Home> directory or to deinstall and leave the <CRS Home> directory on the server. Click the Yes button to continue.
• #29: Deleting a Node from the Cluster (continued) A progress window momentarily displays the actions as the software is being removed. As soon the software has been removed, the progress window will disappear and you will be returned to the Inventory window, which displays that Oracle Clusterware is no longer installed on this server. Click the Close button to close the OUI utility.
• #30: Deleting a Node from the Cluster (continued) For step 9, use the runInstaller utility from an existing node as the crs user to update the node list of the remaining nodes in the cluster. A graphical session is not necessary for this step although OUI is normally a graphical program. The output is as follows: [crs@host01 bin]$ runInstaller -updateNodeList ORACLE_HOME=/u01/app/crs "CLUSTER_NODES={host01,host02}" CRS=TRUE Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 4095 MB Passed Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oraInventory 'UpdateNodeList' was successful.
• #31: Answer: 2
• #32: Answer: 1