Thursday, February 27, 2014

0

How to fix network subnet mask aka netmask issue on solaris

8:08 AM  

Here I am explaining what I encounter network issues in my environment
about network subnet mask aka netmask. I have faced issue with network subnet mask while reboot the server.

Pre-Request:
I. Take ifconfig -a and netstat -rn output prior to reboot the server
II. Collect Explorer output if you missed to take above outputs

 This info will help you to refer netmask values.
Netmask Quick Reference:
Dotted Decimal ValueHex Value
255.0.0.0FF000000
255.255.0.0FFFF0000
255.255.255.0FFFFFF00
255.255.255.128FFFFFF80
255.255.255.192FFFFFFC0
255.255.255.224FFFFFFE0
255.255.255.240FFFFFFF0
255.255.255.248FFFFFFF8
255.255.255.252 *FFFFFFFC *
Netmask Netmask (binary)CIDR Notes
255.255.255.25511111111.11111111.11111111.11111111/32Host (single addr)
255.255.255.25411111111.11111111.11111111.11111110/31Unuseable
255.255.255.25211111111.11111111.11111111.11111100/302 useable
255.255.255.24811111111.11111111.11111111.11111000/296 useable
255.255.255.24011111111.11111111.11111111.11110000/2814 useable
255.255.255.22411111111.11111111.11111111.11100000/2730 useable
255.255.255.19211111111.11111111.11111111.11000000/2662 useable
255.255.255.12811111111.11111111.11111111.10000000/25126 useable
255.255.255.011111111.11111111.11111111.00000000/24"Class C" 254 useable
255.255.254.011111111.11111111.11111110.00000000/232 Class C's
255.255.252.011111111.11111111.11111100.00000000/224 Class C's
255.255.248.011111111.11111111.11111000.00000000/218 Class C's
255.255.240.011111111.11111111.11110000.00000000/2016 Class C's
255.255.224.011111111.11111111.11100000.00000000/1932 Class C's
255.255.192.011111111.11111111.11000000.00000000/1864 Class C's
255.255.128.011111111.11111111.10000000.00000000/17128 Class C's
255.255.0.011111111.11111111.00000000.00000000/16"Class B"
255.254.0.011111111.11111110.00000000.00000000/152 Class B's
255.252.0.011111111.11111100.00000000.00000000/144 Class B's
255.248.0.011111111.11111000.00000000.00000000/138 Class B's
255.240.0.011111111.11110000.00000000.00000000/1216 Class B's
255.224.0.011111111.11100000.00000000.00000000/1132 Class B's
255.192.0.011111111.11000000.00000000.00000000/1064 Class B's
255.128.0.011111111.10000000.00000000.00000000/9128 Class B's
255.0.0.011111111.00000000.00000000.00000000/8"Class A"
254.0.0.011111110.00000000.00000000.00000000/7
252.0.0.011111100.00000000.00000000.00000000/6
248.0.0.011111000.00000000.00000000.00000000/5
240.0.0.011110000.00000000.00000000.00000000/4
224.0.0.011100000.00000000.00000000.00000000/3
192.0.0.011000000.00000000.00000000.00000000/2
128.0.0.010000000.00000000.00000000.00000000/1
0.0.0.000000000.00000000.00000000.00000000/0IP space
0.0.0.0 00000000.00000000.00000000.00000000 /0 IP space
Before reboot the server my configuration was like this referred in explorer file: 
[root@tpt01 ~]#ifconfig aggr420001
aggr420001: flags=201000843 mtu 1500 index 65
        inet 192.168.10.37 netmask ffffff00 broadcast 192.168.10.255
        ether 0:14:4f:87:50:b4
[root@tpt01 ~]#ifconfig aggr430001
aggr430001: flags=201000843 mtu 1500 index 65
        inet 192.168.10.39 netmask ffffff00 broadcast 192.168.10.255
        ether 0:14:4f:87:50:b4
[root@tpt01 ~]#ifconfig aggr440001
aggr440001: flags=201000843 mtu 1500 index 65
        inet 192.168.10.53 netmask ffffff00 broadcast 192.168.10.255
        ether 0:14:4f:87:50:b4
[root@tpt01 ~]#ifconfig aggr490001
aggr490001: flags=201000843 mtu 1500 index 65
        inet 192.168.10.61 netmask ffffff00 broadcast 192.168.10.255
        ether 0:14:4f:47:f9:14
[root@tpt01 ~]#ifconfig aggr520001
aggr520001: flags=201000843 mtu 1500 index 65
        inet 192.168.10.71 netmask ffffff00 broadcast 192.168.10.255
        ether 0:14:4f:47:f9:14
[root@tpt01 ~]#ifconfig aggr474001
aggr474001: flags=201000843 mtu 1500 index 65
        inet 192.168.10.89 netmask ffffff00 broadcast 192.168.10.255
        ether 0:14:4f:87:60:b4
[root@tpt01 ~]#reboot
When I rebooted, I can see one of my NIC (aggr474001) failed to plumb as well as after reboot my network configuration is showing wrong for couple of NIC's.
Here I gave only issue part of screenshot to understand.
Sun Fire T200, No Keyboard
Copyright (c) 1998, 2011, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.30.4.d, 32640 MB memory available, Serial #71578664.
Ethernet address 0:14:5f:77:f9:a0, Host ID: 8447f300.


Boot device: disk  File and args: 
NOTICE: nxge1: xcvr addr:0x0c - link is up 1000 Mbps full duplex
NOTICE: nxge3: xcvr addr:0x0a - link is up 1000 Mbps full duplex
NOTICE: nxge0: xcvr addr:0x0d - link is up 1000 Mbps full duplex
NOTICE: nxge2: xcvr addr:0x0b - link is up 1000 Mbps full duplex
NOTICE: nxge1: xcvr addr:0x0c - link is up 1000 Mbps full duplex
Failed to plumb IPv4 interface(s): 474001
Hostname: tpt01
How to identify wrong netmask values: 
[root@tpt01 ~]# ifconfig -a | egrep -i "ff000000|ffff0000"
        inet 127.0.0.1 netmask ff000000 
        inet 192.168.10.37 netmask ff000000 broadcast 192.255.255.255
        inet 192.168.10.39 netmask ff000000 broadcast 192.255.255.255
        inet 192.168.10.53 netmask ff000000 broadcast 192.255.255.255
        inet 192.168.10.61 netmask ffff0000 broadcast 192.168.255.255
        inet 192.168.10.71 netmask ffff0000 broadcast 192.168.255.255
[root@tpt01 ~]#
To fix the above issues, I have done these are the steps:
Check whether all the configuration entries are ok and make sure there is no duplicate or wrong entry verifying by below configuration files.
I.e  for aggr420001 netmask entry is missing and aggr490001 everything is right but it took wrong netmask as well as duplicate file for 474001.
Let me explain how I fixed these issues.

 Current status: 
[root@tpt01 ~]# cat /etc/hostname.aggr420001
tpt01-420
[root@tpt01 ~]# cat /etc/hosts | grep -i tpt01-420
192.168.10.37    tpt01-420
[root@tpt01 ~]# cat /etc/hosts | grep -i 192.168.10.37
192.168.10.37    tpt01-420
[root@tpt01 ~]# cat /etc/netmasks | grep -i 192.168.10.37
[root@tpt01 ~]# cat /etc/hostname.aggr490001
tpt01-490
[root@tpt01 ~]# cat /etc/hosts | grep -i tpt01-490
192.168.10.61    tpt01-490
[root@tpt01 ~]# cat /etc/hosts | grep -i 192.168.10.61
192.168.10.61    tpt01-490
[root@tpt01 ~]# cat /etc/netmasks | grep -i 192.168.10.61
192.168.10.61 255.255.255.0
[root@tpt01 ~]# ls -ltr /etc/hostname.*|grep -i 474001
-rw-r--r--   1 root     root          13 Jul 28  2012 /etc/hostname.474001
-rw-r--r--   1 root     root          15 Feb 25 14:45 /etc/hostname.aggr474001
[root@tpt01 ~]#
To fix the issue added netmask entry in /etc/netmasks file: 
[root@tpt01 ~]# cat /etc/hostname.aggr420001
tpt01-420
[root@tpt01 ~]# cat /etc/hosts | grep -i tpt01-420
192.168.10.37    tpt01-420
[root@tpt01 ~]# cat /etc/hosts | grep -i 192.168.10.37
192.168.10.37    tpt01-420
[root@tpt01 ~]# cat /etc/netmasks | grep -i 192.168.10.37
192.168.10.37 255.255.255.0
[root@tpt01 ~]#
To fix the issue added IP address with Classless Inter-Domain Routing (CIDR) in /etc/hostname.aggr490001 instead of host-aggr name: 
[root@tpt01 ~]# cat /etc/hostname.aggr490001
192.168.10.61/24
[root@tpt01 ~]# cat /etc/hosts | grep -i tpt01-490
192.168.10.61    tpt01-490
[root@tpt01 ~]# cat /etc/hosts | grep -i 192.168.10.61
192.168.10.61    tpt01-490
[root@tpt01 ~]# cat /etc/netmasks | grep -i 192.168.10.61
192.168.10.61 255.255.255.0
[root@tpt01 ~]#
There are two entry as per the above output so deleted /etc/hostname.474001 file because as per my resent explorer output showed in above the file should be hostname.aggr474001 since it is VLAN aggr IP. 
[root@tpt01 ~]# rm /etc/hostname.474001
[root@tpt01 ~]# ls -ltr /etc/hostname.*|grep -i 474001
-rw-r--r--   1 root     root          15 Feb 25 14:45 /etc/hostname.aggr474001
[root@tpt01 ~]#
After taken all the above steps it got fixed, we have rebooted and checked same, below is the after reboot output screenshots.
[root@tpt01 ~]#reboot
Sun Fire T200, No Keyboard
Copyright (c) 1998, 2011, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.30.4.d, 32640 MB memory available, Serial #71578664.
Ethernet address 0:14:5f:77:f9:a0, Host ID: 8447f300.

Boot device: disk  File and args: 
NOTICE: nxge1: xcvr addr:0x0c - link is up 1000 Mbps full duplex
NOTICE: nxge3: xcvr addr:0x0a - link is up 1000 Mbps full duplex
NOTICE: nxge0: xcvr addr:0x0d - link is up 1000 Mbps full duplex
NOTICE: nxge2: xcvr addr:0x0b - link is up 1000 Mbps full duplex
NOTICE: nxge1: xcvr addr:0x0c - link is up 1000 Mbps full duplex
Hostname: tpt01
[root@tpt01 ~]# ifconfig -a | egrep -i "192.168.10.37|192.168.10.39|192.168.10.53|192.168.10.61|192.168.10.71"
        inet 192.168.10.37 netmask ffffff00 broadcast 192.168.10.255
        inet 192.168.10.39 netmask ffffff00 broadcast 192.168.10.255
        inet 192.168.10.53 netmask ffffff00 broadcast 192.168.10.255
        inet 192.168.10.61 netmask ffffff00 broadcast 192.168.10.255
        inet 192.168.10.71 netmask ffffff00 broadcast 192.168.10.255
[root@tpt01 ~]#
--> Same like above steps done for all the five IP's. Hope this article will help you when you encounter these type of issue. 

 Thanks for reading this post.

Read More

Monday, February 24, 2014

0

How to fix a corrupted wtmpx file on solaris

11:55 AM  , ,


If the wtmpx files is corrupted then we could not able to get login details , like who all are logged in earlier . To fix this issue we need to perform below tasks.
Some of the key points to fix this issue

1). This issue one of the audit compliment in enterprise level
2). Lost all the earlier logged in details
3). This will not capture future logging info until unless fix the issue
4). It will useful when the major issue happen

These are procedure to  fix the corrupted wtmpx file:

How to determine whether wtmpx file is corrupted or not :

 Actually today our date is Jan 24th but last command output says it is Aug 13 which is wrong, it mean wtmpx file got corrupted.
[root@tpt01 /var/adm]# uname -a
SunOS tpt01 5.10 Generic_147147-26 sun4u sparc SUNW,SPARC-Enterprise
[root@tpt01 /var/adm]# date
Fri Jan 24 15:04:08 EST 2014
[root@tpt01 /var/adm]# last | more
tptuser1 sshd 192.168.10.11 Fri  Aug 13 17:05  still logged in
tptuser2 pts/12 192.168.10.15 Fri  Aug 13 16:56  still logged in
tptuser3 sshd 192.168.10.16 Fri  Aug 13 16:56 - 16:59 (00:02)
tptuser4 pts/8 192.168.10.18 Fri  Aug 13 16:52 - 16:59 (00:06)
tptuser5 sshd 192.168.10.22 Fri  Aug 13 16:52 - 16:56 (00:03)
tptuser6 sshd 192.168.10.25 Fri  Aug 13 16:48 - 16:48 (00:00)
tptuser7 sshd 192.168.10.16 Fri  Aug 13 16:48 - 16:48 (00:00)
tptuser8 pts/12 192.168.10.18 Fri  Aug 13 16:30 - 16:34 (00:04)
tptuser9 sshd 192.168.10.22 Fri  Aug 13 16:30 - 16:34 (00:04)
tptuser1 pts/16 192.168.10.25 Fri  Aug 13 16:17 still logged in
[root@tpt01 /var/adm]#
Here is what I performed to fix this issue, in case you encounter similar type of issue.
1) Backup wtmpx log file
[root@tpt01 /]# cp -p /var/adm/wtmpx /var/adm/wtmpx.backup
2) use wtmpfix command to recover the corrupted file and save it in temporary file
This is the output I got: 
[root@tpt01 /]# /usr/lib/acct/wtmpfix /var/adm/wtmpx > /var/adm/wtmpx.tmp
/var/adm/wtmpx: offset 164816832: nonzero pid or status in EMPTY record
/var/adm/wtmpx: offset 164816832: suspecting misaligned records, repositioning
/var/adm/wtmpx: offset 17197884: starting re-scan
/var/adm/wtmpx: offset 175267380: now recognizing aligned records again
/var/adm/wtmpx: offset 526582320: EOF reached after recovering from corruption in the middle of the file.  This file may need to be repaired by hand.
[root@tpt01/]#
3) Check temporary file 
[root@tpt01 /var/adm]# last -f wtmpx.tmp | more
tptuser1 sshd 192.168.10.11  Fri  Jan 24 15:03 - 15:03 (00:00)
tptuser2 pts/12 192.168.10.15  Fri  Jan 24 14:30 - 14:30 (00:00)
tptuser3 sshd 192.168.10.16  Fri  Jan 24 14:28 - 14:29 (00:01)
tptuser4 pts/8 192.168.10.18  Fri  Jan 24 14:16 - 14:16 (00:00)
tptuser5 sshd 192.168.10.22  Fri  Jan 24 14:16 - 14:16 (00:00)
tptuser6 sshd 192.168.10.25  Fri  Jan 24 14:13  still logged in
tptuser7 sshd 192.168.10.16  Fri  Jan 24 14:13 - 14:16 (00:02)
tptuser8 pts/12 192.168.10.18  Fri  Jan 24 13:29 - 13:29 (00:00)
tptuser9 sshd 192.168.10.22  Fri  Jan 24 13:29 - 13:29 (00:00)
tptuser1 pts/16 192.168.10.25  Fri  Jan 24 13:20 - 13:20 (00:00)
[root@tpt01 /var/adm]#
4) If above output command is shows the correct date and time then the issue got fixed so move the temporary file back to original location. 
[root@tpt01 /var/adm]#mv /var/adm/wtmpx.tmp /var/adm/wtmpx
5). Check the original file again to confirm : 
[root@tpt01 /]# last | more
tptuser1 sshd 192.168.10.11  Fri  Jan 24 15:03 - 15:03 (00:00)
tptuser2 pts/12 192.168.10.15  Fri  Jan 24 14:30 - 14:30 (00:00)
tptuser3 sshd 192.168.10.16  Fri  Jan 24 14:28 - 14:29 (00:01)
tptuser4 pts/8 192.168.10.18  Fri  Jan 24 14:16 - 14:16 (00:00)
tptuser5 sshd 192.168.10.22  Fri  Jan 24 14:16 - 14:16 (00:00)
tptuser6 sshd 192.168.10.25  Fri  Jan 24 14:13  still logged in
tptuser7 sshd 192.168.10.16  Fri  Jan 24 14:13 - 14:16 (00:02)
tptuser8 pts/12 192.168.10.18  Fri  Jan 24 13:29 - 13:29 (00:00)
tptuser9 sshd 192.168.10.22  Fri  Jan 24 13:29 - 13:29 (00:00)
tptuser1 pts/16 192.168.10.25  Fri  Jan 24 13:20 - 13:20 (00:00)
[root@tpt01 /]#
Thanks for reading this article...

Read More

Tuesday, February 18, 2014

0

OS patching on Solaris with VCS/SVM Environment

10:01 AM  , ,











Here I am providing step by step patching plan for VCS cluster setup environment.

 General Note:
1. Download latest patch bundle from oracle site
2. Take full OS backup
3. Take all the configuration screenshot backup as well as explorer (Save it to your Local "Laptop")
4. Check server console access.
5. Check the server's hardware health.
6. Application and Database team contacts details.

 Pre-Requisite:
I. Download patch bundle from oracle site and copy to server /var/tmp/ (10_Recommended.zip).
II. Make necessary free space on /,/opt and /var FS.

 Pre-Implementation Plan:
1. Take all the configuration screenshot backup as well as explorer (Save it to your Local "Laptop").
Here is the list of general commands -
2. Snooze the servers alerts.

 These are the steps to patch VCS cluster servers:
Ensure Application / Databases are brought down (If it is running) Check the server where the cluster groups are running:
[root@tpt01]# hastatus -sum | grep -i online
B ClusterService tpt01 Y N ONLINE
B nbu_sg tpt01 Y N ONLINE
[root@tpt01]#
If the cluster service groups are running in patching node, switch over to another node This command will failover all the running cluster service groups to another node and stop the cluster on current node (from where we are executing the command) 
[root@tpt01]# hastop -local -evacuate
Pre- reboot the server before doing patching activity and check if the server booting without any issue. 
[root@tpt01]# sync;sync;sync
[root@tpt01]# init 6
Unzip the patch bundle:
[root@tpt01]# /usr/local/bin/unzip 10_Recommended.zip
Move the VCS startup files (to prevent cluster from start while reboot): 
[root@tpt01]# mv /etc/rc3.d/S99vcs /etc/rc3.d/bake_S99vcs
[root@tpt01]# mv /etc/llthosts /etc/back_llthosts
[root@tpt01]# mv /etc/llttab /etc/back_llttab
[root@tpt01]# mv /etc/gabtab /etc/back_gabtab
Stop the cluster on current node (Patching node): 
[root@tpt01]# hastop -local
Compare below three outputs to identify the rootdisk and rootmirror disk before detaching the secondary rootmirror disk: 
[root@tpt01]# prtconf -pv | grep -i bootpath
bootpath: '/pci@9,600000/SUNW,qlc@2/fp@0,0/disk@w2100001862f7d137,0:a'
[root@tpt01]#

[root@tpt01]# metastat d10
d10: Mirror
Submirror 0: d11
State: Okay
Submirror 1: d12
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 75505920 blocks (36 GB)

d11: Submirror of d10
State: Okay
Size: 75505920 blocks (36 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s0 0 No Okay Yes
d12: Submirror of d10
State: Okay
Size: 75505920 blocks (36 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes
Device Relocation Information:
Device Reloc Device ID
c1t0d0 Yes id1,ssd@n2000001862f7d137
c1t1d0 Yes id1,ssd@n2000001862f7f113
[root@tpt01]#

[root@tpt01]# echo | format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w2100001862f7d137,0
1. c1t1d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w2100001862f7f113,0
Detach secondary root mirror disk from root disk (For fail back the change if not success) Validate which are the MD device to detach since this server we have only / and swap FS on root disk. (Mainly we have consider /,/var,/opt and /export/home if we have these FS are separately : To confirm swap device name: 
[root@tpt01]# swap -l
swapfile dev swaplo blocks free
/dev/md/dsk/d20 85,20 16 16790384 15817232
[root@tpt01]#
To confirm root device name: 
[root@tpt01]# df -h /
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d10 35G 29G 6.2G 83% /
[root@tpt01]#
Detach secondary root mirror disk: 
[root@tpt01]# metadetach d10 d12
[root@tpt01]# metadetach d20 d22
Run fsck in secondary disk: 
[root@tpt01]# fsck -y /dev/md/rdsk/d12
Mount the secondary disk on /mnt: 
[root@tpt01]# mount /dev/dsk/c1t1d0s0 /mnt
Take back of system and vfstab files: 
[root@tpt01]# cd /mnt/etc
[root@tpt01]# cp system system.29JAN2014
[root@tpt01]# cp vfstab vfstab.29JAN2014
Edit vfstab and add the entries: 
[root@tpt01]# vi vfstab
/dev/dsk/c1t1d0s0 /dev/rdsk/c1t1d0s0 / ufs 1 no -
/dev/dsk/c1t1d0s1 - - swap - no -
Comment out the root information in /etc/system file (to prevent system to boot without MD device: 
[root@tpt01]# vi system
*rootdev:/pseudo/md@0:0,10,blk
[root@tpt01]# sync;sync;sync
Un mount the mnt and boot the system with secondary disk: 
[root@tpt01]# umount /mnt
Shutdown and bring into OBP Prompt:
Note: It is a good practice to test secondary mirror disk . Reboot the system using the secondary mirror disk to make sure it is booting up properly: 
[root@tpt01]#shutdown -i0 -g0 -y
ok>devalias
ok>boot disk1
Shutdown the system again, then boot the primary disk into single-user mode for patching: 
[root@tpt01]# shutdown -i0 -g0 -y
ok>boot -s
Go to the patch directory: 
[root@tpt01]# cd /var/tmp/patches
[root@tpt01]# cd 10_Recommended 
[root@tpt01]# ./installpatchset --s10patchset

Setup ………………………..
Once patch installation done then reboot the server: 
[root@tpt01]# init 6
Do a health check and find the new patch level: 
[root@tpt01]# uname –a
Revoke the configuration as like earlier: 
[root@tpt01]# mv /etc/rc3.d/bake_S99vcs /etc/rc3.d/S99vcs
[root@tpt01]# mv /etc/back_llthosts /etc/llthosts
[root@tpt01]# mv /etc/back_llttab /etc/llttab
[root@tpt01]# mv /etc/back_gabtab /etc/gabtab
Start the VCS cluster: 
[root@tpt01]# hastart
Determine if the upgrade was successful.  Attached it back after few days: [root@tpt01]#metattach d10 d12
[root@tpt01]#metattach d20 d22
If not success reboot the server with secondary disk (roll-back) procedure. Posted dedicated post for revert back the OS patching

Read More

Friday, February 14, 2014

0

Fail-back plan for solaris OS patching with VCS/SVM Environment.

7:23 AM  , ,


If the patching activity failed due to some unexpected reason (unfortunately ).Then we have to perform below task to bring the server into normal / operational status .

If the patching was not success, then reboot the server with secondary disk and roll-back the change.

Note : Now we have to consider secondary disk as a primary disk.

Boot the server from secondary disk with multiuser mode: 
OK>boot disk1
Check whether we have any old configured metadb: 
[root@tpt01]#metadb -i
These three below (Clear DB, Copy VTOC & Create DB ) steps are optional but good practice to do this.
Clear old metadb configuration: 
[root@tpt01]#metadb -d /dev/dsk/c1t1d0s7 /dev/dsk/c1t0d0s7
Copy the VTOC from secondary disk to primary disk (old / Corrupted disk): 
[root@tpt01]#prtvtoc /dev/rdsk/c1t1d0s2 | fmthard –s - /dev/rdsk/c1t0d0s2
SVM root mirroring steps. Create metadb newly with three replicas: 
[root@tpt01]#metadb -afc3 /dev/dsk/c1t1d0s7 /dev/dsk/c1t0d0s7
Create MD device for / FS: 
[root@tpt01]#metainit -f d11 1 1 c1t1d0s0
[root@tpt01]#metainit -f d12 1 1 c1t0d0s0
[root@tpt01]#metainit d10 -m d11
[root@tpt01]#metaroot d10
Shutdown and boot the server with secondary disk: 
[root@tpt01]#init 5
OK>boot disk1
Attach secondary submirror to main meta mirror: 
[root@tpt01]#metattach d10 d12
If we have dedicated file system like for i.e /var , /opt , /export/home then we have to perform same steps like above. As per my current setup, I have only / and /swap FS. For root file system I have already completed the mirroring let me do for /swap device. Create MD device for / swap FS: 
[root@tpt01]#metainit -f d21 1 1 c1t1d0s1
[root@tpt01]#metainit -f d22 1 1 c1t0d0s1
[root@tpt01]#metainit d20 -m d21
[root@tpt01]#metattach d20 d22
Check whether syncing is happening: 
[root@tpt01] metastat | grep -i sync or %
Edit vfstab and add the MD entries: Note : when you run metaroot command it will automatically add MD device name for /FS on /etc/vfstab, apart from that we have to add for other FS, i.e I am adding entry for /sawp FS. 
Here I am adding for /swap FS: 
[root@tpt01]# vi vfstab
/dev/md/dsk/d10 /dev/md/rdsk/d10 / ufs 1 no -
/dev/md/dsk/d20 - - swap - no -
[root@tpt01]#
Now move back VCS configuration files to earlier state: 
[root@tpt01]# mv /etc/rc3.d/bake_S99vcs /etc/rc3.d/S99vcs
[root@tpt01]# mv /etc/back_llthosts /etc/llthosts
[root@tpt01]# mv /etc/back_gabtab /etc/gabtab
Start VCS cluster service: 
[root@tpt01]# hastart
Check cluster status: 
[root@tpt01]# hastatus -sum | grep -i online
B ClusterService tpt01 Y N ONLINE
B nbu_sg tpt02 Y N ONLINE
[root@tpt01]#
To update the eeprom value to boot the server into secondary disk persistently. 
Check current status : 
[root@tpt01]# eeprom | grep -i boot-device
boot-device=/pci@9,600000/SUNW,qlc@2/fp@0,0/disk@w2100001862f7d137,0:a disk net
[root@tpt01]# eeprom | grep -i nvramrc
use-nvramrc?=true
nvramrc=devalias net /pci@9,700000/network@2
[root@tpt01]#
As per the above current value use-nvramrc is true. It means server will ignore the boot device name and try to boot the server accordingly referred by use-nvramrc values: To change eeprom value: Here I am creating boot device name and updating nvramrc value: 
[root@tpt01]#eeprom "boot-device=rootdisk rootmirror"
eeprom "nvramrc=devalias rootdisk /pci@9,600000/SUNW,qlc@2/fp@0,0/disk@w2100001862f7f113,0:a devalias rootmirror /pci@9,600000/SUNW,qlc@2/fp@0,0/disk@w2100001862f7d137,0:a"
[root@tpt01]#

Read More