Installing Oracle 11g RAC on virtual servers using VMware.

Installing Oracle 11g RAC on virtual servers using VMware.

This post describes the installation of Oracle 11g RAC on virtual servers.Hardware
The following hardware was used for the installation:

Operating system	Processor	Memory	HDD’s
Windows XP SP2 (32 bit)	Intel P4 3Ghz	3Gb	1 x 160Gb 7200rpm & 1x 250Gb 7200rpm. Both on Ultra ATA Controllers

You’ll need at least 3Gb of RAM for this installation as the
virtual machines will be configured with 1Gb of RAM each, leaving 1Gb
for the host operating system. The virtual machines further requires a
total of 40Gb of storage.

Software

General

For obvious reason this installation should never be used for
Production or Development purposes. This type of environment is however
extremely helpful if you want to learn and understand Oracle RAC and
you do not have access to traditional hardware resources.

Virtual Machines

I used VMware Server (1.0.3) to facilitate my virtual machines. If you have not done so already download the software here. You’ll need to register to get an activation code, but it is a free product.

Configure your first virtual machine:

Install VMware Server and launch.
Select ‘local host’ and ‘OK’ to close the pop-up window.
Select ‘New Virtual Machine’ from the VMware Server
Console to create the hardware specification for your first virtual
machine.
This will launch the ‘New Virtual Machine Wizard’. ‘Next’ to continue.
Select ‘Custom’ to allow you to customize the virtual machine. ‘Next’ to continue.
I’ve settled for Oracle Enterprise Linux 4.5 (which is a
rebranded version of Red Hat Enterprise Linux ), therefore select
‘Linux’ as the Guest Operating System and ‘Red Hat
Enterprise Linux 4’ as the Version. Oracle Enterprise Linux 5 is
not yet supported on VMware Server. ‘Next’ to continue.
Name your virtual machine and select a location to store the files.
I’ve named the virtual machines: EL4_11gRAC1 and EL4_11gRAC2. We
will be creating 2 virtual machines with shared storage. You need to
create separate directories for your virtual machines. Also create a
separate directory for your shared storage. I’ve created them as
follows:

Virtual Machine 1 E:Virtual MachinesRACrac11grac1

Virtual Machine 2 E:Virtual MachinesRACrac11grac2

Shared Storage E:Virtual MachinesRACrac11gshare
Set Access Rights. Deselect ‘Make this virtual machine private’. ‘Next’ to continue.
Keep the default Startup/Shutdown Options (Run this virtual
machines as the user that powers on the virtual machine).
‘Next’ to continue.
Keep the Processor Configuration default (One). ‘Next’ to continue.
Allocate 1Gb of memory for your virtual machine. ‘Next’ to continue.
Keep the default network type (bridged networking). ‘Next’ to continue.
Keep the default I/O Adapter Type (LSI Logic). ‘Next’ to continue.
Keep the default setting to create a new virtual disk. ‘Next’ to continue.
Keep the default disk type (SCSI). ‘Next’ to continue.
Specify a disk capacity of 20Gb for your local disk. We need enough
space for the Linux installation, Oracle Clusterware software, Oracle
Database software and a staging area. Make sure to deselect
‘Allocate all disk space now’ and select ‘Split disk
into 2Gb files’.
‘Next’ to continue.
Specify a filename for your virtual disk (localdisk.vmdk) and
create it in the directory for your first virtual machine (E:Virtual
MachinesRACrac11grac1). ‘Next’ to continue.
Edit your virtual machine settings to add the shared disks and an
additional network card. Select ‘Edit virtual machine
settings’ from the Commands section on the VMware Server Console.
‘Next’ to continue.
This will bring up a settings window. Select ‘Add’ to
add new hardware. This will launch the ‘Add New Hardware’
wizard.
Select ‘Ethernet Adapter’ to add an additional network
card. We will use this to configure the private interconnect.
‘Next’ to continue.
This time around select ‘Host-only’ as the network type. ‘Next’ to continue.
Launch the New Hardware Wizard again to add the following SCSI disks:

Purpose Size Virtual device node

Database files 3Gb SCSI 1:0

Database files 3Gb SCSI 1:1

Database files 3Gb SCSI 1:2

Oracle Cluster Registry 300Mb SCSI 1:3

Voting Disk 300Mb SCSI 1:4
Select ‘Hard Disk’ to add a new harddisk. ‘Next’ to continue.
Select ‘Create a new virtual disk’. ‘Next’ to continue.
Select ‘SCSI’ as the disk type. ‘Next’ to continue.
Allocate disk space as per the table in step 22. This time around
keep the default options (‘Allocate all disk space now’
selected and ‘Split disk into 2Gb files’ deselected).
‘Next’ to continue.
Give the new disk a descriptive name (datadisk1.vmdk) and save it
to the shared disk folder (E:Virtual MachinesRACrac11gshare ).
‘Next’ to continue.
Once the new disk has been created, select it and click on the
‘Advanced’ button to edit. ‘Next’ to continue.
Change the Virtual device node as per the table in step 22 and
change the Mode to ‘Independent’ and
‘Persistent’. ‘Next’ to continue.
Repeat steps 23 to 29 until all the disks in the table in step 22 have been configured.
Once all the disks have been added edit the virtual machine
configuration file (E:Virtual MachinesRACrac11grac1Red Hat
Enterprise Linux 4.vmx), but first close the Virtual Machine Settings
screen to save the changes to the configuration file.Add the parameters
in bold blue below to the configuration file in order to enable disk
sharing:config.version = “8″
virtualHW.version = “4″
scsi0.present = “TRUE”
scsi0.virtualDev = “lsilogic”
memsize = “768″
scsi0:0.present = “TRUE”
scsi0:0.fileName = “rac1_localdisk.vmdk”
ide1:0.present = “TRUE”
ide1:0.fileName = “auto detect”
ide1:0.deviceType = “cdrom-raw”
floppy0.present = “FALSE”
Ethernet0.present = “TRUE”
displayName = “EL4_11gRAC1″
guestOS = “rhel4″
priority.grabbed = “normal”
priority.ungrabbed = “normal”
disk.locking = “FALSE”
diskLib.dataCacheMaxSize = “0″
diskLib.dataCacheMaxReadAheadSize = “0″
diskLib.dataCacheMinReadAheadSize = “0″
diskLib.dataCachePageSize = “0″
scsi1.sharedBus = “virtual”
scsi1.present = “TRUE”
scsi1:0.present = “TRUE”
scsi1:0.fileName = “E:Virtual Machinesracrac11gsharedatadisk1.vmdk”
scsi1:0.mode = “independent-persistent”
scsi1:0.deviceType = “disk”
scsi1:1.present = “TRUE”
scsi1:1.fileName = “E:Virtual Machinesracrac11gsharedatadisk2.vmdk”
scsi1:1.mode = “independent-persistent”
scsi1:1.deviceType = “disk”
scsi1:2.present = “TRUE”
scsi1:2.fileName = “E:Virtual Machinesracrac11gsharedatadisk3.vmdk”
scsi1:2.mode = “independent-persistent”
scsi1:2.deviceType = “disk”
scsi1:3.present = “TRUE”
scsi1:3.fileName = “E:Virtual Machinesracrac11gshareocrdisk.vmdk”
scsi1:3.mode = “independent-persistent”
scsi1:3.deviceType = “disk”
scsi1:4.present = “TRUE”
scsi1:4.fileName = “E:Virtual Machinesracrac11gsharevotingdisk.vmdk”
scsi1:4.mode = “independent-persistent”
scsi1:4.deviceType = “disk”
scsi1.virtualDev = “lsilogic”
ide1:0.autodetect = “TRUE”
Ethernet1.present = “TRUE”

You have now successfully configured your first virtual machine.
We’ll clone this at a later stage to provide for the second
virtual machine.

Install Oracle Enterprise Linux

You are now ready to install Linux on the virtual machine. If you
have not done so already, download the Oracle Enterprise Linux 4
(Update 5) files from here and unzip. Once unzipped you should have the following cd image files:
Enterprise-R4-U5-i386-disc1.iso
Enterprise-R4-U5-i386-disc2.iso
Enterprise-R4-U5-i386-disc3.iso
Enterprise-R4-U5-i386-disc4.iso

Install Linux:

To start the installation select ‘Edit virtual machine
settings’ from the main VMware Server Console. Select the CD-ROM
drive. Under ‘Connection’ select ‘Use ISO
image’ and browse to and select the ISO image for the Linux
installation disk 1 (Enterprise-R4-U5-i386-disc1.iso).’OK’
to continue.
Select ‘Start this virtual machine’ to start it up.
The virtual machine will now boot from the CD-ROM and start the
Linux installer. Press ‘Enter’ to select to install in
graphical mode. Select ‘Skip’ on the following screen to
skip media testing.
The graphical installer will now launch. ‘Next’ to continue.
Make your language selection. ‘Next’ to continue.
Select your keyboard configuration. ‘Next’ to continue.
Select ‘Manually partition with Disk Druid’. ‘Next’ to continue.
Select ‘Yes’ for all warning messages that indicate the partition table was unreadable.
Partition local disk
Partition the local disk (/dev/sda) as follows:

Mount point Size

/swap 2048Mb

/tmp 1024Mb

/ 4096Mb

/u01 13312Mb

To partition the local disk select the free space on /dev/sda and
select ‘Edit’ to add partitions. Add all partitions as in
the table above to /dev/sda. Do not configure the other disks at this
stage. Select ‘Next’ to continue once done.
Keep the default boot loader configuration. ‘Next’ to continue.
Network configuration
Configure the network as follows:

Hostname Network device Network type IP address

rac1.startoracle.com eth0 Public 192.168.1.50

eth1 Private 10.10.10.50

Please note: Configure the public IP address for your virtual
machines so that they are on the same network (192.168.1.1 in my case)
as your host machine. You can also change the domain name
(startoracle.com in my case) to your own. Remember to make these
changes throughout the rest of the installation.

Tick both network devices to be ‘Active on boot’:

Select each device and select ‘Edit’ to edit advanced settings:
Deselect ‘Configure using DHCP’
Enter the IP address as per the table above.
Enter the Netmask: 255.255.255.0

Repeat for interface eth1:

Set the hostname: rac1.startoracle.com
Set the Gateway: 192.168.1.1
Set the Primary DNS: 192.168.1.1
‘Next’ to continue when done.
Disable the firewall and SELinux. ‘Next’ to continue.
Select additional language support. ‘Next’ to continue.
Select your time zone. ‘Next’ to continue.
Set your root password. ‘Next’ to continue.
Select ‘Customize software packages to be installed’. ‘Next’ to continue.
Package Group Selection
Keep all the default selected package groups and add the following groups:
– Legacy Software Development
– Graphical Internet (to add Firefox)
– Development Tools
– System Tools. Also select ‘Details’ for this group and add the 3 oracleasm packages and the sysstat package.
‘Next’ to continue.
Select ‘Next’ to start the installation.
During
the installation you’ll be prompted to insert the installation
cd’s. When this happens simply right-click on the CD-ROM icon in
the bottom right corner and select the appropriate ISO image file.
Select ‘Ok’ to continue the installation.
Reboot once installation is done.
After the reboot, finalize the setup. ‘Next’ to skip the Welcome screen.
Accept the license agreement. ‘Next’ to continue.
Check the date and time and set if incorrect. ‘Next’ to continue.
Check and set display. ‘Next’ to continue.
Setup a system user (not the oracle user). You can skip this. ‘Next’ to continue.
Additional CDs. No action required. ‘Next’ to continue.
Select ‘Next’ to complete the installation.
Ensure that you are using the SMP kernel
To ensure you are using this kernel open a terminal window as the root user and enter the following command: uname -rIf the following output is displayed then the correct kernel is used:

If any other kernel is used edit the /boot/grub/grub.conf file and set
the default option to boot the SMP kernel. The first option is 0 and is
usually set as the default boot option, the second option is 1 etc.
Each kernel boot option starts with the ‘title’ line.
At this point power down the virtual machine, edit the virtual
machine settings to reset the CD-ROM to ‘Use physical
drive’ and ‘Auto detect’. Restart the virtual machine.
Install VMware tools in the virtual machine. This is necessary so
that the time between the host and virtual machines stay synchronized.
If the time between the RAC nodes are out of sync installation to the
remote node will fail.To install VMware tools, log in as root in the
virtual machine and select ‘Install VMware Tools…’
from the VM menu in the VMware Server Console.

Confirm installation. ‘Install’ to continue.

This will mount an image on the cd drive with the installation package.

Open the cd folder and double-click the VMwareTools package to install.

Open a terminal window and execute: /usr/bin/vmware-config-tools.pl
to configure VMware Tools. Select the screen resolution during
configuration.Configure VMware Tools to start every time you login. On
the ‘Applications’ menu go to ‘Preferences’,
‘More Preferences’ and launch ‘Sessions’. Add: /usr/bin/vmware-toolbox to Startup Programs.

Log out and back in as root.Make sure to tick the ‘Time
synchronization between the virtual machine and the host operating
system’ box. This will add the following line to the Virtual
Machine configuration file: tools.syncTime = “TRUE”

Minimize the ‘VMware Tools Properties’ window.
Edit /boot/grub/grub.conf and add the following options to the kernel /boot/ line of your default (SMP) kernel:
clock=pit nosmp noapic nolapic
Install the following packages from the indicated cd’s:
libaio-devel-0.3.105-2 (CD 3)
elfutils-libelf-devel-0.97.1-4 (CD 4)
unixODBC-devel-2.2.11-1 (CD 4)

Create disk partitions for shared disks
Prepare the following disks that will host the shared database files,
the Oracle Cluster Registry and the Clusterware voting disk:

Purpose	Size	Virtual device node	Disk
Database files	3Gb	SCSI 1:0	/dev/sdb
Database files	3Gb	SCSI 1:1	/dev/sdc
Database files	3Gb	SCSI 1:2	/dev/sdd
Oracle Cluster Registry	300Mb	SCSI 1:3	/dev/sde
Voting Disk	300Mb	SCSI 1:4	/dev/sdf

Use the following command to list all disks & partitions: fdisk –l
This will list all disks (including /dev/sda which we’ve already configured).

Repeat the following steps for all disks in the table above:
Open a terminal window as the root user and enter the following command (replacing <disk> with the correct disk name): fdisk <disk>

[root@rac1 ~]# fdisk /dev/sdb
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-391, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-391, default 391):
Using default value 391
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

The key sequence after entering the fdisk <disk> command is:
n [return] p [return] 1 [return] [return] [return] w [return]

Prepare raw disks
Bind the OCR and Voting Disk
partitions to raw devices. To do this edit the file:
/etc/sysconfig/rawdevices and add the following lines:
/dev/raw/raw1 /dev/sde1
/dev/raw/raw2 /dev/sdf1

Restart the raw devices service for the new settings to take effect:
[root@rac1 ~]# service rawdevices restart
Assigning devices:
/dev/raw/raw1 –> /dev/sde1
/dev/raw/raw1: bound to major 8, minor 65
/dev/raw/raw2 –> /dev/sdf1
/dev/raw/raw2: bound to major 8, minor 81When
you restart an Oracle Enterprise Linux 4 system, ownership and
permissions for raw devices will revert to the root owner. To prevent
this from happening edit the raw devices section of the
/etc/udev/permissions.d/50-udev.permissions file to look like the
following:
Create users and groups
In Oracle 11g Oracle recommends a new system group (osasm) if you want
to separate the ASM administration function from your DBA’s.
Likewise they recommend creating a separate group (crs) for Clusterware
administration. I will however create a single software owner (oracle)
for all Oracle software.To create the necessary groups and users
execute the following from the command line as the root user:
# groupadd -g 501 oinstall
# groupadd -g 502 dba
# useradd -g oinstall -G dba -s /bin/ksh oracle
# passwd oracle
Configuring kernel parameters and shell limits
Append the following to the /etc/sysctl.conf file as the root user:
kernel.sem = 250 32000 100 128
kernel.shmmax = 536870912
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 262144

You can issue the following command for these settings to take immediate effect: /sbin/sysctl –p

Add the following entries to the /etc/hosts file as the root user:

Public	192.168.1.50	rac1.startoracle.com	rac1
	192.168.1.60	rac2.startoracle.com	rac2
VIP	192.168.1.51	rac1-vip.startoracle.com	rac1-vip
	192.168.1.61	rac2-vip.startoracle.com	rac2-vip
Inter-connect	10.10.10.50	rac1-priv.startoracle.com	rac1-priv
	10.10.10.60	rac2-priv.startoracle.com	rac2-priv

The /etc/hosts file should now contain the following entries:
EL36
Restart the network services to pick up the new settings with the following command: # service network restart

As root create the following oracle directories and change the ownership to oracle:
# mkdir -p /u01/app/oracle/product/11.1.0/crs
# mkdir /u01/app/oracle/product/11.1.0/db_1
# mkdir /u01/Stage11g
# chown -R oracle:oinstall /u01/app
# chown oracle:oinstall /u01/Stage11g
Set shell limits for the Oracle user
As the root user add the following lines to the /etc/security/limits.conf file:
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536Add the following as the second last lines in the /etc/pam.d/login file if it does not exist:
session required pam_limits.so

Append the following to the /etc/profile file:
if [ $USER = “oracle” ]; then
if [ $SHELL = “/bin/ksh” ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
Install the ASMLib package
Download the ASM Lib package from the Oracle Technology Network.
Download the package named ‘oracleasmlib-2.0.2-1.i386.rpm’
under the section ‘Intel IA32 (x86) Architecture’ and
install it as the root user with the following command: # rpm -i oracleasmlib-2.0.2-1.i386.rpm
Clone the virtual machine
At this point shutdown the virtual machine and copy all the files from
the virtual machine’s directory to the directory for the second
virtual machine. Once this is complete add the second virtual machine
as follows:
Select the ‘Home’ tab in the VMware Server Console.

Select ‘Open Existing Virtual Machine’ and browse to the
cloned virtual machine directory and select the file ‘Red Hat
Enterprise Linux 4.vmx’. The cloned virtual machine will now show
up in the VMware Server Console.

Rename the new virtual machine:
To rename the virtual machine select ‘Edit virtual machine
settings’ and select the ‘Options’ tab. Change the
virtual machine name (EL4_11gRAC2) and click ‘OK’ to
continue.

Start the new virtual machine (while leaving the old one powered off)
and log in as root. When prompted if you want to create a new UUID for
the virtual machine select ‘Create’ and ‘OK’ to
continue.

Upon startup you will be asked to remove the 2 “old”
network cards and add “new” network cards. Select
‘Remove Configuration’ to remove the network cards.
You’ll need to do this twice.

Select ‘Configure’ to configure the first network card.

Enter the IP configuration for the second virtual machine’s first
network card (Private) as per the table in step 36. ‘OK’ to
continue.

Select ‘Configure’ again to configure the second network
card and enter the IP configuration for the second virtual
machine’s second network card (Public) as per the table in step
36. ‘OK’ to continue.

Login as the root user and update the network settings.
From the ‘Applications’ menu select ‘System
Settings’ and ‘Network’ to edit network settings.
For each network device do the following:
Select the device and Select ‘Edit’.

Make sure the IP address is correct for each device – eth0 (the bridged
network connection) should be on the public network and eth1 (the
host-only network connection) should be on the private network. If the
settings are incorrect update the IP address, subnet mask and default
gateway.

Select the ‘Hardware Device’ tab and click
‘Probe’ to get the new MAC address for the device. Select
‘OK’ to continue.

Select the ‘DNS’ tab and update the Hostname to
rac2.startoracle.com. From the file menu select ‘Save’ and
‘Quit’.

Restart the virtual machine for the changes to take effect and start
the first virtual machine. Both virtual machines should now be up and
running.Note: To test the network setup, ping the nodes from each other
using the public and private ip addresses (the virtual ip address is
not pingable at this stage).
Enabling SSH User Equivalency
The OUI (Oracle Universal Installer) uses the ssh and scp commands
during installation to run remote commands and copy files to other
nodes in the cluster, therefore you need to setup user equivalency for
the Oracle user on all nodes.Perform the following in order to setup
user equivalency:As the oracle user on node rac1:
$ mkdir ~/.ssh
$ chmod 700 ~/.ssh
$ /usr/bin/ssh-keygen -t rsa
Note: Whenever prompted for a passphrase leave it empty and press return.

As the oracle user on node rac2:
$ mkdir ~/.ssh
$ chmod 700 ~/.ssh
$ /usr/bin/ssh-keygen -t rsa

As the oracle user on node rac1:
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ scp ~/.ssh/authorized_keys rac2:/home/oracle/.ssh/

Perform the following as the oracle user on node rac1 and then node rac2:
ssh rac1 date
ssh rac2 date
ssh rac1.startoracle.com date
ssh rac2.startoracle.com date
ssh rac1-priv date
ssh rac2-priv date
ssh rac1-priv.startoracle.com date
ssh rac2-priv.startoracle.com date
Setup the ASM disksConfigure the Oracle ASM driver
As the root user on both nodes run the following:
# service oracleasm configure

Ouput:
[root@rac1 etc]# service oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets (’[]’). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: [ OK ]
Loading module “oracleasm”: [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]

Stamp the ASM disks
As the root user on node rac1 (or rac2, but not both) run the following:
# service oracleasm createdisk DATA1 /dev/sdb1
# service oracleasm createdisk DATA2 /dev/sdc1
# service oracleasm createdisk RECO1 /dev/sdd1

Output:
[root@rac1 etc]# service oracleasm createdisk DATA1 /dev/sdb1
Marking disk “/dev/sdb1″ as an ASM disk: [ OK ]
[root@rac1 etc]# service oracleasm createdisk DATA2 /dev/sdc1
Marking disk “/dev/sdc1″ as an ASM disk: [ OK ]
[root@rac1 etc]# service oracleasm createdisk RECO1 /dev/sdd1
Marking disk “/dev/sdd1″ as an ASM disk: [ OK ]

Verify the newly configured ASM disks
As the root user on both nodes run the following:
# service oracleasm scandisks
# service oracleasm listdisks

Output:
[root@rac1 etc]# service oracleasm scandisks
Scanning system for ASM disks: [ OK ]
[root@rac1 etc]# service oracleasm listdisks
DATA1
DATA2
RECO1

You are now ready to install Oracle Clusterware.

Install Oracle Clusterware

Copy the Oracle Clusterware and Database Software zip files downloaded earlier to the staging area (/u01/Stage11g) on node rac1. Use pscp to copy the downloaded files from your Windows host machine to the virtual machine:

Alternatively download the zip files from your virtual machine. If your
network is setup correctly you should be able to connect to the
internet from within your virtual machine.
Unzip the Oracle 11g software in your staging area (/u01/Stage11g) as the oracle user:
$ unzip /u01/Stage11g/linux_11gR1_clusterware.zip
$ unzip /u01/Stage11g/linux_11gR1_database.zip
Install the cvuqdisks package
It can be found in the /u01/Stage11g/clusterware/rpm directory. Install on both nodes as the root user: # rpm -i cvuqdisk-1.0.1–1.rpm
Run the Cluster Verification Utility (CVU or Cluvfy) as the oracle
user to verify all pre-installation tasks for Oracle Clusterware are
completed.
Go to your Clusterware staging directory (/u01/Stage11g/clusterware)
and run the following command as the oracle user on node rac1: $ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2The Clufvy output will be similar to the following:
$ pwd
/u01/Stage11g/clusterware

$ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2
Performing pre-checks for cluster services setup
Checking node reachability…
Node reachability check passed from node “rac1″.
Checking user equivalence…
User equivalence check passed for user “oracle”.
Checking administrative privileges…
User existence check passed for “oracle”.
Group existence check passed for “oinstall”.
Membership check for user “oracle” in group “oinstall” [as Primary] passed.
Administrative privileges check passed.
Checking node connectivity…
Node connectivity check passed for subnet “192.168.1.0″ with node(s) rac2,rac1.
Node connectivity check passed for subnet “10.10.10.0″ with node(s) rac2,rac1.
Interfaces found on subnet “10.10.10.0″ that are likely candidates for VIP:
rac2 eth1:10.10.10.60
rac1 eth1:10.10.10.50
Interfaces found on subnet “192.168.1.0″ that are likely
candidates for a private interconnect:rac2 eth0:192.168.1.60
rac1 eth0:192.168.1.50
Node connectivity check passed.
Checking system requirements for ‘crs’…
Total memory check failed.
Check failed on nodes:
rac2,rac1
Free disk space check passed.
Swap space check passed.
System architecture check passed.
Kernel version check passed.
Package existence check passed for “make-3.80″.
Package existence check passed for “binutils-2.15.92.0.2″.
Package existence check passed for “gcc-3.4.5″.
Package existence check passed for “libaio-0.3.105″.
Package existence check passed for “libaio-devel-0.3.105″.
Package existence check passed for “libstdc++-3.4.5″.
Package existence check passed for “elfutils-libelf-devel-0.97″.
Package existence check passed for “sysstat-5.0.5″.
Package existence check passed for “libgcc-3.4.5″.
Package existence check passed for “libstdc++-devel-3.4.5″.
Package existence check passed for “unixODBC-2.2.11″.
Package existence check passed for “unixODBC-devel-2.2.11″.
Package existence check passed for “glibc-2.3.4-2.19″.
Group existence check passed for “dba”.
Group existence check passed for “oinstall”.
User existence check passed for “nobody”.
System requirement failed for ‘crs’
Pre-check for cluster services setup was unsuccessful on all the nodes.

The Clufvy pre-check will fail, because the ‘Total memory
check’ failed, even though we have the recommended 1Gb RAM
allocated to each instance. Make sure all other checks passed.
Launch the OUI for Oracle Clusterware
Prior to installing open a terminal window as the root user on node rac1 and type the following:
# xhost +Open
a terminal window as the oracle user on node rac1 and go to the
clusterware staging directory (/u01/Stage11g/clusterware) and start the
installer with the following command:
$ ./runInstaller

This will launch the OUI (Oracle Universal Installer) for Oracle Clusterware 11g.
Welcome. Select ‘Next’ to continue.
Specify Inventory directory and credentials
Change the inventory directory from the default of
/u01/app/oraInventroy to /u01/app/oracle/oraInventory. Keep the default
(oinstall) Operating System group name. ‘Next’ to continue.
Specify Home Details
Change the path to /u01/app/oracle/product/11.1.0/crs. ‘Next’ to continue.
Product-Specific Prerequisite Checks
The OUI will now verify that the environment meets all the
requirements. All pre-requisite steps should complete successfully.
‘Next’ to continue.
Specify Cluster Configuration
Specify all nodes in the cluster. Initially only node rac1 will be listed.
Select ‘Add’ and add the details for node rac2. Enter all details. ‘OK’ to continue.
After you’ve added node rac2 the Cluster configuration should look as below. ‘Next’ to continue.
Specify Network Interface Usage
Select the eth0 interface name and ‘Edit’ to correct the Interface Type.
Change the eth0 Interface Type to Public. ‘Ok’ to continue.
Verify that eth0 (192.168.1.0) is now your public network and eth1
(10.10.10.0) is your private network. ‘Next’ to continue.
Specify Oracle Cluster Registry (OCR) Location
Select External Redundancy and specify the first raw device
(/dev/raw/raw1) setup earlier as the OCR location. ‘Next’
to continue.
Specify Voting Disk Location
Select External Redundancy and specify the second raw device
(/dev/raw/raw2) setup earlier as the Voting Disk location.
‘Next’ to continue.
Summary
Review the summary and select ‘Install’ to start the installation.
Install
Monitor the progress of the installation.
Configuration Scripts
Once the installation is complete you will be prompted to run 2
configuration scripts as the root user. Open a terminal window and
execute these scripts as the root user on each node. Select
‘Ok’ to continue after these scripts have completed
successfully on both nodes. Do not run the scripts simultaneously on
both nodes.

The output for the configuration scripts will look similar to below:
[root@rac1 ~]# /u01/app/oracle/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oracle/oraInventory to 770.
Changing groupname of /u01/app/oracle/oraInventory to oinstall.
The execution of the script is complete
[root@rac1 ~]# /u01/app/oracle/product/11.1.0/crs/root.sh
WARNING: directory ‘/u01/app/oracle/product/11.1.0′ is not owned by root
WARNING: directory ‘/u01/app/oracle/product’ is not owned by root
WARNING: directory ‘/u01/app/oracle’ is not owned by root
WARNING: directory ‘/u01/app’ is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
The directory ‘/u01/app/oracle/product/11.1.0′ is not owned by root. Changing owner to root
The directory ‘/u01/app/oracle/product’ is not owned by root. Changing owner to root
The directory ‘/u01/app/oracle’ is not owned by root. Changing owner to root
The directory ‘/u01/app’ is not owned by root. Changing owner to root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
Now formatting voting device: /dev/raw/raw2
Format of 1 voting devices complete.
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
rac1
Cluster Synchronization Services is inactive on these nodes.
rac2
Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.
Configuration Assistants
The configuration assistants will now kick into gear. They all need to
complete successfully in order for the installation to be successful.
End of Installation
Once the configuration assistants complete below screen will display.
‘Exit’ to complete the installation and exit the OUI.
Checking the status of Clusterware
To check the status of your installation issue the following commands as the root user:
# /u01/app/oracle/product/11.1.0/crs/bin/crsctl check crs
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthyTo get more detailed information issue the following command as the root user: # /u01/app/oracle/product/11.1.0/crs/bin/crs_stat -t
Name Type R/RA F/FT Target State Host
———— ———– —- —- —— —— —-
ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip application 0/0 0/0 ONLINE ONLINE rac1
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip application 0/0 0/0 ONLINE ONLINE rac2

Install Oracle Database and ASM Software

As root change ownership of the oracle directories to the oracle user on both nodes:
# chown oracle:oinstall /u01/app/oracle
# chown oracle:oinstall /u01/app/oracle/product
# chown oracle:oinstall /u01/app/oracle/product/11.1.0
To start the OUI change to the database directory in your staging
area (/u01/Stage11g/database) and launch the OUI as the oracle user
with the following command: $ ./runInstaller
Welcome
‘Next’ to skip the Welcome screen.
Select Installation Type
Keep the default installation type (Enterprise Edition). ‘Next’ to continue.
Install Location
Keep the default Oracle Base location (/u01/app/oracle) and default
Oracle Home location (/u01/app/oracle/product/11.1.0/db_1).
‘Next’ to continue.
Specify Hardware Cluster Installation Mode
Keep the default ‘Cluster Installation’ selection and select both nodes. ‘Next’ to continue.
Product-Specific Prerequisite Checks
The OUI will now verify that the environment meets all the
requirements. All pre-requisite steps should complete successfully.
Select ‘Next’ to continue.
Select Configuration Option
Select ‘Install Software Only’. We will use the DBCA
(Database Configuration Assistant) at a later stage to configure ASM
and create a database. ‘Next’ to continue.
Privileged Operating System Groups
Keep the default options (dba, oinstall and oinstall). ‘Next’ to continue.
Summary
Review the summary and select ‘Install’ to start the installation.
Install
Monitor the progress of the installation.
Configuration Script
Once the installation is complete you will be prompted to run a script
as the root user. Open a terminal window and execute the script as the
root user on each node. Select ‘Ok’ to continue after the
script has been run successfully on both nodes.
End of Installation
Once the installation is complete select ‘Exit’ to complete the installation and exit the OUI.

Create Oracle ASM instances

Launch the DBCA (Database Configuration Assistant) to configure ASM
and create a RAC database.Open a terminal window as the oracle user.
From the /u01/app/oracle/product/11.1.0/db_1/bin directory launch the
dbca with the following command: $ ./dbca
Welcome
Keep the default selection (Oracle RAC database). ‘Next’ to continue.
Operations
Select ‘Configure ASM’. ‘Next’ to continue.
Node Selection
Select all the nodes and ‘Next’ to continue.
Create ASM Instance
Select a SYS pasword for the ASM instance. Keep the default type of
parameter file (IFILE) to create. ‘Next’ to continue.
Select ‘Ok’ to confirm creation of the ASM instances.
Select ‘Yes’ to let the DBCA create default listeners.
ASM Disk Groups
Select ‘Create New’ to create new ASM disk groups.
Enter
‘DATA’ as the first disk group name. Keep the default
redundancy settings (Normal) and select the 2 data disks (DATA1 and
DATA2). ‘Ok’ to continue.
The DATA disk group should now be mounted. This will be the
location for all our database files. Select ‘Create new’
again to add another disk group.
Enter ‘RECO’ as the disk group name. This time select
‘External’ for redundancy and select the remaining disk
(RECO1). ‘OK’ to continue.
All the disk groups are now created. ‘Finish’ to complete.
Select ‘Yes’ to perform the database creation.

Create Oracle RAC Database

Operations
Select ‘Create a Database’. ‘Next’ to continue.
Node Selection
Select all the nodes and ‘Next’ to continue.
Database Templates
Keep the default setting (General Purpose or Transaction Processing). ‘Next’ to continue.
Database Identification
Enter ‘rac’ (without the quotes) as the global database name. ‘Next’ to continue.
Management Options
Keep the default settings (‘Configure Enterprise Manager’
and ‘Configure Database Control for local management’
selected, but ‘Enable Alert Notifications’ and
‘Enable Daily Disk Backup to Recovery Area’ deselected).
‘Next’ to continue.
Database Credentials
Select ‘Use the same Administrative Password for All Accounts’ and enter a password. ‘Next’ to continue.
Storage Options
Select ‘ASM’ for storage. ‘Next’ to continue.
ASM Disk Groups
Select both disk groups (DATA and RECO). ‘Next’ to continue.
Database File Locations
Keep the default (Use Oracle-Managed Files). Make sure
‘+DATA’ is entered as the Database Area. ‘Next’
to continue.
Select ‘Specify Flash Recovery Area’ and enter
‘+RECO’ as the Flash Recovery Area. Select ‘Enable
archiving’ and click on the ‘Edit Archive Mode
Parameters’ button.
Edit Archive Mode Parameters
Make sure the Flash Recovery Area (RECO) is shown as the archiving
destination. If not change it to ‘+RECO/’ (without the
quotes). ‘OK’ and then ‘Next’ to continue.
Database Content
Select ‘Sample Schemas’ if you want to install them. I did not. ‘Next’ to continue.
Initialization Parameters
Reduce the ‘Memory Size’ to 250Mb and select ‘Use
Automatic Memory Management’ Leave all other settings as is.
‘Next’ to continue.
Security Settings
Keep the default, enhanced 11g security settings. ‘Next’ to continue.
Automatic Maintenance Tasks
Make sure ‘Enable automatic maintenance tasks’ is seleced. ‘Next’ to continue.
Database Storage
Review the storage options for all files. ‘Next’ to continue.
Creation Options
Select ‘Generate database creation scripts’ if you want to
review these at a later stage. ‘Finish’ to review the
installation.
Summary
Select ‘Ok’ to close the review page and ‘Finish’ to start the installation.
The DBCA will first generate the database creation scripts if you
selected this. A message will be displayed once this is done.
Acknowledging the message will start the database creation.
Once database creation is done a summary screen will be displayed. ‘Exit’ to exit the OUI.
As the oracle user edit the /etc/oratab file on both nodes.Replace
the database name with the instance name for the rac database i.e.
replace the rac keyword with rac1 or rac2 depending on the node.
Furthermore add details for your clusterware home to this file. This
will enable you to set the Clusterware home using the oraenv script.
Once edited the /etc/oratab file should contain the following:
On node RAC1:
+ASM1:/u01/app/oracle/products/11.1.0/db_1:N
rac1:/u01/app/oracle/products/11.1.0/db_1:N
crs:/u01/app/oracle/products/11.1.0/crs:N
On node RAC2
+ASM2:/u01/app/oracle/products/11.1.0/db_1:N
rac2:/u01/app/oracle/products/11.1.0/db_1:N
crs:/u01/app/oracle/products/11.1.0/crs:N
Setup the environment for the Oracle user
As the oracle user create the file /home/oracle/.profile on all nodes and add the following lines:
umask 022
set -o vi
export PS1=”[${USER}@`hostname -s`]$ “
export ORACLE_BASE=/u01/app/oracle
export ORACLE_SID=`hostname -s`
export ORAENV_ASK=NO
. oraenv
unset ORAENV_ASK