Before you start this practice, make sure that you went through the following steps on your third node: Practice 4-1: 1, 2, 3, 4, 5, 6, 8 .
In this practice, you extend your cluster to the third node.
Note:
• Unless specified otherwise, you are connected on your first node as the grid user
1) Set up the ssh user equivalence for the grid user between your first node and your third node.
using a terminal session.
[grid@host01 ~]$ /home/oracle/labs/less_05/ssh_config.sh Setting up SSH user equivalency.
grid@host02's password:
grid@host03's password:
Checking SSH user equivalency.
host01 host02 host03
[grid@host01 ~]$
2) Make sure that you can connect from your first node to the third one without being prompted for passwords.
[grid@host01 ~]$ ssh host03 date Fri May 4 10:38:10 EDT 2012 [grid@host01 ~]$
3) Make sure that you set up your environment variables correctly for the grid user to point to your grid installation.
[grid@host01 ~]$ . oraenv ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid
$
4) Check your pre-grid installation for your third node using the Cluster Verification Utility.
[grid@host01 ~]$ cluvfy stage -pre crsinst -n host03
Performing pre-checks for cluster services setup
Checking node reachability...
Node reachability check passed from node "host01"
Checking user equivalence...
Oracle University and Mazz Soluciones SRL use only
User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Node connectivity passed for subnet "192.0.2.0" with node(s) host03
TCP connectivity check passed for subnet "192.0.2.0"
Node connectivity passed for subnet "192.168.1.0" with node(s) host03
TCP connectivity check passed for subnet "192.168.1.0"
Interfaces found on subnet "192.0.2.0" that are likely candidates for VIP are:
host03 eth0:192.0.2.103
WARNING:
Interface subnet "192.0.2.0" does not have a gateway defined
Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
host03 eth1:192.168.1.103
Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
host03 eth2:192.168.1.203
Node connectivity check passed
Checking multicast communication...
Checking subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0" passed.
Checking subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0" passed.
Check of multicast communication passed.
Checking ASMLib configuration.
Check for ASMLib configuration passed.
Total memory check passed Available memory check passed Swap space check passed
Oracle University and Mazz Soluciones SRL use only
Free disk space check passed for "host03:/u01/app/11.2.0/grid"
Free disk space check passed for "host03:/tmp"
Check for multiple users with UID value 502 passed User existence check passed for "grid"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "grid" in group "oinstall" [as Primary] passed
Membership check for user "grid" in group "dba" failed Check failed on nodes:
host03
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "make"
Package existence check passed for "binutils"
Package existence check passed for "gcc(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "elfutils-libelf(x86_64)"
Package existence check passed for "elfutils-libelf-devel"
Package existence check passed for "glibc-common"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "glibc-headers"
Package existence check passed for "gcc-c++(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "ksh"
Check for multiple users with UID value 0 passed Current group ID check passed
Oracle University and Mazz Soluciones SRL use only
Starting check for consistency of primary group of root user
Check for consistency of root user's primary group passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
NTP Configuration file check passed
Checking daemon liveness...
Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes
NTP daemon slewing option check passed
NTP daemon's boot time configuration check for slewing option passed
NTP common Time Server Check started...
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Clock time offset check passed
Clock synchronization check using Network Time Protocol(NTP) passed
Core file name pattern consistency check passed.
User "grid" is not part of "root" group. Check passed Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes
File "/etc/resolv.conf" does not have both domain and search entries defined
domain entry in file "/etc/resolv.conf" is consistent across nodes
search entry in file "/etc/resolv.conf" is consistent across nodes
All nodes have one search entry defined in file
"/etc/resolv.conf"
The DNS response time for an unreachable node is within acceptable limit on all nodes
File "/etc/resolv.conf" is consistent across nodes
Time zone consistency check passed
Pre-check for cluster services setup was unsuccessful on all the nodes. $
Oracle University and Mazz Soluciones SRL use only
5) Generate the fixup scripts with
cluvfywith the
–fixupoption.
[grid@host01 ~]$ cluvfy stage -pre crsinst -n host03 –fixup Performing pre-checks for cluster services setup
Checking node reachability...
Node reachability check passed from node "host01"
Checking user equivalence...
User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Node connectivity passed for subnet "192.0.2.0" with node(s) host03
TCP connectivity check passed for subnet "192.0.2.0"
Node connectivity passed for subnet "192.168.1.0" with node(s) host03
TCP connectivity check passed for subnet "192.168.1.0"
Interfaces found on subnet "192.0.2.0" that are likely candidates for VIP are:
host03 eth0:192.0.2.103
WARNING:
Interface subnet "192.0.2.0" does not have a gateway defined
Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
host03 eth1:192.168.1.103
Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
host03 eth2:192.168.1.203
Node connectivity check passed
Checking multicast communication...
Checking subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0" passed.
Checking subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0"...
Oracle University and Mazz Soluciones SRL use only
Check of subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0" passed.
Check of multicast communication passed.
Checking ASMLib configuration.
Check for ASMLib configuration passed.
Total memory check passed Available memory check passed Swap space check passed
Free disk space check passed for "host03:/u01/app/11.2.0/grid"
Free disk space check passed for "host03:/tmp"
Check for multiple users with UID value 502 passed User existence check passed for "grid"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "grid" in group "oinstall" [as Primary] passed
Membership check for user "grid" in group "dba" failed Check failed on nodes:
host03
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "make"
Package existence check passed for "binutils"
Package existence check passed for "gcc(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "elfutils-libelf(x86_64)"
Package existence check passed for "elfutils-libelf-devel"
Package existence check passed for "glibc-common"
Package existence check passed for "glibc-devel(x86_64)"
Oracle University and Mazz Soluciones SRL use only
Package existence check passed for "glibc-headers"
Package existence check passed for "gcc-c++(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "ksh"
Check for multiple users with UID value 0 passed Current group ID check passed
Starting check for consistency of primary group of root user
Check for consistency of root user's primary group passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
NTP Configuration file check passed
Checking daemon liveness...
Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes
NTP daemon slewing option check passed
NTP daemon's boot time configuration check for slewing option passed
NTP common Time Server Check started...
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Clock time offset check passed
Clock synchronization check using Network Time Protocol(NTP) passed
Core file name pattern consistency check passed.
User "grid" is not part of "root" group. Check passed Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes
File "/etc/resolv.conf" does not have both domain and search entries defined
domain entry in file "/etc/resolv.conf" is consistent across nodes
search entry in file "/etc/resolv.conf" is consistent across nodes
All nodes have one search entry defined in file
"/etc/resolv.conf"
Oracle University and Mazz Soluciones SRL use only
The DNS response time for an unreachable node is within acceptable limit on all nodes
File "/etc/resolv.conf" is consistent across nodes
Time zone consistency check passed
Fixup information has been generated for following node(s):
host03
Please run the following script on each node as "root" user to execute the fixups:
'/tmp/CVU_11.2.0.3.0_grid/runfixup.sh'
Pre-check for cluster services setup was unsuccessful on all the nodes.
6) Run the fixup script as directed.
[root@host01 ~]# ssh root@host03 /tmp/CVU_11.2.0.3.0_grid/runfixup.sh root@host03's password:
Response file being used is
:/tmp/CVU_11.2.0.3.0_grid/fixup.response Enable file being used is
:/tmp/CVU_11.2.0.3.0_grid/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_grid/orarun.log
[root@host01 ~]# exit
[grid@host01 ~]$
7) After setting your environment, use the Cluster Verification Utility to make sure that you can add your third node to the cluster.
[grid@host01 ~]$ . oraenv ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid
[grid@host01 ~]$ cluvfy stage -pre nodeadd -n host03
Performing pre-checks for node addition
Checking node reachability...
Node reachability check passed from node "host01"
Checking user equivalence...
User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Oracle University and Mazz Soluciones SRL use only
Check: Node connectivity for interface "eth0"
Node connectivity passed for interface "eth0"
TCP connectivity check passed for subnet "192.0.2.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.0.2.0".
Subnet mask consistency check passed.
Node connectivity check passed
Checking multicast communication...
Checking subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0" passed.
Check of multicast communication passed.
Checking CRS integrity...
Clusterware version consistency passed
CRS integrity check passed
Checking shared resources...
Checking CRS home location...
"/u01/app/11.2.0/grid" is shared
Shared resources check for node addition passed
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Check: Node connectivity for interface "eth0"
Node connectivity passed for interface "eth0"
TCP connectivity check passed for subnet "192.0.2.0"
Check: Node connectivity for interface "eth1"
Node connectivity passed for interface "eth1"
TCP connectivity check passed for subnet "192.168.1.0"
Check: Node connectivity for interface "eth2"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.0.2.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.
Oracle University and Mazz Soluciones SRL use only
Node connectivity check passed
Checking multicast communication...
Checking subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0" passed.
Checking subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0" passed.
Check of multicast communication passed.
Total memory check passed Available memory check passed Swap space check passed
Free disk space check passed for "host03:/u01/app/11.2.0/grid"
Free disk space check passed for "host01:/u01/app/11.2.0/grid"
Free disk space check passed for "host03:/tmp"
Free disk space check passed for "host01:/tmp"
Check for multiple users with UID value 502 passed User existence check passed for "grid"
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "make"
Package existence check passed for "binutils"
Package existence check passed for "gcc(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Oracle University and Mazz Soluciones SRL use only
Package existence check passed for "elfutils-libelf(x86_64)"
Package existence check passed for "elfutils-libelf-devel"
Package existence check passed for "glibc-common"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "glibc-headers"
Package existence check passed for "gcc-c++(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "ksh"
Check for multiple users with UID value 0 passed Current group ID check passed
Starting check for consistency of primary group of root user
Check for consistency of root user's primary group passed
Checking OCR integrity...
OCR integrity check passed
Checking Oracle Cluster Voting Disk configuration...
Oracle Cluster Voting Disk configuration check passed Time zone consistency check passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
NTP Configuration file check passed
Checking daemon liveness...
Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes
NTP daemon slewing option check passed
NTP daemon's boot time configuration check for slewing option passed
NTP common Time Server Check started...
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Clock time offset check passed
Clock synchronization check using Network Time Protocol(NTP) passed
Oracle University and Mazz Soluciones SRL use only
User "grid" is not part of "root" group. Check passed
Checking consistency of file "/etc/resolv.conf" across nodes
File "/etc/resolv.conf" does not have both domain and search entries defined
domain entry in file "/etc/resolv.conf" is consistent across nodes
search entry in file "/etc/resolv.conf" is consistent across nodes
All nodes have one search entry defined in file
"/etc/resolv.conf"
The DNS response time for an unreachable node is within acceptable limit on all nodes
File "/etc/resolv.conf" is consistent across nodes
Pre-check for node addition was successful.
$
8) Add your third node to the cluster from your first node:
[grid@host01 ~]$ cd $ORACLE_HOME/oui/bin
[grid@host01 bin]$ ./addNode.sh -silent
"CLUSTER_NEW_NODES={host03}"
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={host03-vip}"
Performing pre-checks for node addition
Checking node reachability...
Node reachability check passed from node "host01"
Checking user equivalence...
User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Check: Node connectivity for interface "eth0"
Node connectivity passed for interface "eth0"
TCP connectivity check passed for subnet "192.0.2.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.0.2.0".
Subnet mask consistency check passed.
Node connectivity check passed
Oracle University and Mazz Soluciones SRL use only
Checking multicast communication...
Checking subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0" passed.
Check of multicast communication passed.
Checking CRS integrity...
Clusterware version consistency passed
CRS integrity check passed
Checking shared resources...
Checking CRS home location...
"/u01/app/11.2.0/grid" is shared
Shared resources check for node addition passed
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Check: Node connectivity for interface "eth0"
Node connectivity passed for interface "eth0"
TCP connectivity check passed for subnet "192.0.2.0"
Check: Node connectivity for interface "eth1"
Node connectivity passed for interface "eth1"
TCP connectivity check passed for subnet "192.168.1.0"
Check: Node connectivity for interface "eth2"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.0.2.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.
Node connectivity check passed
Checking multicast communication...
Checking subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.0.2.0" for multicast communication with multicast group "230.0.1.0" passed.
Oracle University and Mazz Soluciones SRL use only
Checking subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.1.0" for multicast communication with
Check of subnet "192.168.1.0" for multicast communication with