OS patching on Solaris with VCS/SVM Environment

Here I am providing step by step patching plan for VCS cluster setup environment.

General Note:
1. Download latest patch bundle from oracle site
2. Take full OS backup
3. Take all the configuration screenshot backup as well as explorer (Save it to your Local "Laptop")
4. Check server console access.
5. Check the server's hardware health.
6. Application and Database team contacts details.

Pre-Requisite:
I. Download patch bundle from oracle site and copy to server /var/tmp/ (10_Recommended.zip).
II. Make necessary free space on /,/opt and /var FS.

Pre-Implementation Plan:
1. Take all the configuration screenshot backup as well as explorer (Save it to your Local "Laptop").
Here is the list of general commands -
2. Snooze the servers alerts.

These are the steps to patch VCS cluster servers:
Ensure Application / Databases are brought down (If it is running) Check the server where the cluster groups are running:

[root@tpt01]# hastatus -sum | grep -i online
B ClusterService tpt01 Y N ONLINE
B nbu_sg tpt01 Y N ONLINE
[root@tpt01]#

If the cluster service groups are running in patching node, switch over to another node This command will failover all the running cluster service groups to another node and stop the cluster on current node (from where we are executing the command)

[root@tpt01]# hastop -local -evacuate

Pre- reboot the server before doing patching activity and check if the server booting without any issue.

[root@tpt01]# sync;sync;sync
[root@tpt01]# init 6

Unzip the patch bundle:

[root@tpt01]# /usr/local/bin/unzip 10_Recommended.zip

Move the VCS startup files (to prevent cluster from start while reboot):

[root@tpt01]# mv /etc/rc3.d/S99vcs /etc/rc3.d/bake_S99vcs
[root@tpt01]# mv /etc/llthosts /etc/back_llthosts
[root@tpt01]# mv /etc/llttab /etc/back_llttab
[root@tpt01]# mv /etc/gabtab /etc/back_gabtab

Stop the cluster on current node (Patching node):

[root@tpt01]# hastop -local

Compare below three outputs to identify the rootdisk and rootmirror disk before detaching the secondary rootmirror disk:

[root@tpt01]# prtconf -pv | grep -i bootpath
bootpath: '/pci@9,600000/SUNW,qlc@2/fp@0,0/disk@w2100001862f7d137,0:a'
[root@tpt01]#

[root@tpt01]# metastat d10
d10: Mirror
Submirror 0: d11
State: Okay
Submirror 1: d12
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 75505920 blocks (36 GB)

d11: Submirror of d10
State: Okay
Size: 75505920 blocks (36 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s0 0 No Okay Yes
d12: Submirror of d10
State: Okay
Size: 75505920 blocks (36 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes
Device Relocation Information:
Device Reloc Device ID
c1t0d0 Yes id1,ssd@n2000001862f7d137
c1t1d0 Yes id1,ssd@n2000001862f7f113
[root@tpt01]#

[root@tpt01]# echo | format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w2100001862f7d137,0
1. c1t1d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w2100001862f7f113,0

Detach secondary root mirror disk from root disk (For fail back the change if not success) Validate which are the MD device to detach since this server we have only / and swap FS on root disk. (Mainly we have consider /,/var,/opt and /export/home if we have these FS are separately : To confirm swap device name:

[root@tpt01]# swap -l
swapfile dev swaplo blocks free
/dev/md/dsk/d20 85,20 16 16790384 15817232
[root@tpt01]#

To confirm root device name:

[root@tpt01]# df -h /
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d10 35G 29G 6.2G 83% /
[root@tpt01]#

Detach secondary root mirror disk:

[root@tpt01]# metadetach d10 d12
[root@tpt01]# metadetach d20 d22

Run fsck in secondary disk:

[root@tpt01]# fsck -y /dev/md/rdsk/d12

Mount the secondary disk on /mnt:

[root@tpt01]# mount /dev/dsk/c1t1d0s0 /mnt

Take back of system and vfstab files:

[root@tpt01]# cd /mnt/etc
[root@tpt01]# cp system system.29JAN2014
[root@tpt01]# cp vfstab vfstab.29JAN2014

Edit vfstab and add the entries:

[root@tpt01]# vi vfstab
/dev/dsk/c1t1d0s0 /dev/rdsk/c1t1d0s0 / ufs 1 no -
/dev/dsk/c1t1d0s1 - - swap - no -

Comment out the root information in /etc/system file (to prevent system to boot without MD device:

[root@tpt01]# vi system
*rootdev:/pseudo/md@0:0,10,blk
[root@tpt01]# sync;sync;sync

Un mount the mnt and boot the system with secondary disk:

[root@tpt01]# umount /mnt

Shutdown and bring into OBP Prompt:
Note: It is a good practice to test secondary mirror disk . Reboot the system using the secondary mirror disk to make sure it is booting up properly:

[root@tpt01]#shutdown -i0 -g0 -y
ok>devalias
ok>boot disk1

Shutdown the system again, then boot the primary disk into single-user mode for patching:

[root@tpt01]# shutdown -i0 -g0 -y
ok>boot -s

Go to the patch directory:

[root@tpt01]# cd /var/tmp/patches
[root@tpt01]# cd 10_Recommended 
[root@tpt01]# ./installpatchset --s10patchset

Setup ………………………..

Once patch installation done then reboot the server:

[root@tpt01]# init 6

Do a health check and find the new patch level:

[root@tpt01]# uname –a

Revoke the configuration as like earlier:

[root@tpt01]# mv /etc/rc3.d/bake_S99vcs /etc/rc3.d/S99vcs
[root@tpt01]# mv /etc/back_llthosts /etc/llthosts
[root@tpt01]# mv /etc/back_llttab /etc/llttab
[root@tpt01]# mv /etc/back_gabtab /etc/gabtab

Start the VCS cluster:

[root@tpt01]# hastart

Determine if the upgrade was successful.  Attached it back after few days: [root@tpt01]#metattach d10 d12
[root@tpt01]#metattach d20 d22
If not success reboot the server with secondary disk (roll-back) procedure. Posted dedicated post for revert back the OS patching