Wednesday, July 2, 2014

0

VXVM Disk group import failed

9:40 AM  ,


I have recently encounter DG import failed issue on one of my environment and finally resolved as below.
While reboot the server I have faced below issue, not able to get the OS prompt due to DG import failed. When I checked the server I came to know some of the FS not able to mount due to the DG not imported.

 Steps to resolve the problem:
Identify issue from OS level:
[root@ivmtst]# svcs -xv | more
svc:/system/filesystem/local:default (local file system mounts)
 State: maintenance since Jan 10, 2014  02:05:45 PM EDT  
Reason: Start method exited with $SMF_EXIT_ERR_FATAL.
   See: http://sun.com/msg/SMF-8000-KS
   See: /var/svc/log/system-filesystem-local:default.log
Impact: 39 dependent services are not running:
        svc:/application/psncollector:default
        svc:/system/sysidtool:net
        svc:/network/rpc/bind:default
        svc:/network/nfs/nlockmgr:default
        svc:/network/nfs/server:default
        svc:/milestone/multi-user-server:default
        svc:/system/basicreg:default
        svc:/system/zones:default
        svc:/network/nfs/status:default
        svc:/network/nfs/cbd:default
        svc:/network/nfs/mapid:default
        svc:/application/sthwreg:default
        svc:/application/stosreg:default
        svc:/milestone/multi-user:default
        svc:/system/xprtld:default
        svc:/system/vxdbdctrl:default
        svc:/system/vxvm/vxvm-recover:default
        svc:/system/iscsitgt:default
        svc:/network/inetd:default
        svc:/system/fmd:default
        svc:/system/fpsd:default
        svc:/system/sysidtool:system
See the SVC Logs for more details to troubleshoot: 
[root@ivmtst]# tail -10 /var/svc/log/system-filesystem-local:default.log
UX:vxfs mount: ERROR: V-3-20002: Cannot access /dev/vx/dsk/IVDG/TVOL50: No such file or directory
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
UX:vxfs mount: ERROR: V-3-20002: Cannot access /dev/vx/dsk/IVDG/TVOL90: No such file or directory
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
UX:vxfs mount: ERROR: V-3-20002: Cannot access /dev/vx/dsk/IVDG/TVOL20: No such file or directory
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
UX:vxfs mount: ERROR: V-3-20002: Cannot access /dev/vx/dsk/IVDG/TVOL30: No such file or directory
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
WARNING: /sbin/mountall -l failed: exit status 1
[ Jan 10 02:05:45 Method "start" exited with status 95 ]
[root@ivmtst]#
Checking DG status: 
[root@ivmtst]# vxdisk -eo alldgs list | grep -i IVDG
emc07_178b1 auto:cdsdisk - (IVDG) online thinrclm c1t5000090001112D9Cd34s2 tprclm
emc07_17b6 auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9Cd34s2 tprclm
emc07_18b7 auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9Cd64s2 tprclm
emc07_19b8 auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9Cd70s2 tprclm
emc07_177a auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9C8164s2 tprclm
emc07_19b9 auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9Cd96s2 tprclm
emc07_19b0 auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9Cd207s2 tprclm
[root@ivmtst]#
As per above output DG in deported state so trying to import the DG but import failed:
[root@ivmtst]# vxdg -Cf import IVDG
Jan 10 02:05:45 vxvm:vxconfigd: V-5-1-0 Disk Group IVDG has a mix of standard and cloned disks: Trying to import the disk group from the standard disks: emc07_177a
Jan 10 02:05:45 vxvm:vxconfigd: V-5-1-0 No enabled configuration copies found
Jan 10 02:05:45 vxvm:vxconfigd: V-5-1-0 Disk group import of IVDG failed with error 182 – Disk group has no valid configuration copies
VxVM vxdg ERROR V-5-1-10978 Disk group IVDG: import failed:
Disk group has no valid configuration copies
[root@ivmtst]#
Identify which are the disks part of IVDG disk group: 
[root@ivmtst]# vxdisk -eo alldgs list | grep -i IVDG
emc07_178b1 auto:cdsdisk emc07_178b1 IVDG online thinrclm c1t5000090001112D9Cd34s2 tprclm
emc07_17b6 auto:cdsdisk emc07_17b6 IVDG online thinrclm c1t5000090001112D9Cd34s2 tprclm
emc07_18b7 auto:cdsdisk emc07_18b7 IVDG online thinrclm c1t5000090001112D9Cd64s2 tprclm
emc07_19b8 auto:cdsdisk emc07_19b8 IVDG online thinrclm c1t5000090001112D9Cd70s2 tprclm
emc07_177a auto:cdsdisk emc07_177a IVDG online thinrclm c1t5000090001112D9C8164s2 tprclm
emc07_19b9 auto:cdsdisk emc07_19b9 IVDG online thinrclm c1t5000090001112D9Cd96s2 tprclm
emc07_19b0 auto:cdsdisk emc07_19b0 IVDG online thinrclm c1t5000090001112D9Cd207s2 tprclm
[root@ivmtst]#
Steps to resolve the disk group import failed problem.
Update udid ID and off the clone flag: 
[root@ivmtst ]# vxdisk updateudid emc07_178b1
[root@ivmtst ]# vxdisk updateudid emc07_17b6
[root@ivmtst ]# vxdisk updateudid emc07_18b7
[root@ivmtst ]# vxdisk updateudid emc07_19b8
[root@ivmtst ]# vxdisk updateudid emc07_177a
[root@ivmtst ]# vxdisk updateudid emc07_19b9
[root@ivmtst ]# vxdisk updateudid emc07_19b0
[root@ivmtst ]# vxdisk set emc07_178b1 clone=off
[root@ivmtst ]# vxdisk set emc07_17b6 clone=off
[root@ivmtst ]# vxdisk set emc07_18b7 clone=off
[root@ivmtst ]# vxdisk set emc07_19b8 clone=off
[root@ivmtst ]# vxdisk set emc07_177a clone=off
[root@ivmtst ]# vxdisk set emc07_19b9 clone=off
[root@ivmtst ]# vxdisk set emc07_19b0 clone=off
[root@ivmtst]
Import the DG again forcefully : 
[root@ivmtst]# vxdg -Cf import IVDG
Jan 10 02:45:20 vxvm:vxconfigd: V-5-1-0 Selecting configuration database copy from emc07_19b9 from disks: emc07_19b9 emc07_19b0 emc07_19b8emc07_18b7 emc07_17b6
Jan 10 02:45:20 vxvm:vxconfigd: V-5-1-0 Trying to import the disk group IVDG using configuration database copy from emc07_19b9
Jan 10 02:45:20 vxvm:vxconfigd: V-5-1-0 Disk group import of IVDG succeeded.
[root@ivmtst]#
Finally we have imported successfully. 
Clear the SMF service from OS level: 
[root@ivmtst]# svcadm clear svc:/system/filesystem/local:default

[root@ivmtst]# svcs -xv | more
svc:/system/filesystem/local:default (local file system mounts)
 State: offline since Jan 10, 2014  02:45:15 PM EDT 
Reason: Start method is running.
   See: http://sun.com/msg/SMF-8000-C4
   See: /var/svc/log/system-filesystem-local:default.log
Impact: 39 dependent services are not running:
        svc:/application/psncollector:default
        svc:/system/sysidtool:net
        svc:/network/rpc/bind:default
        svc:/network/nfs/nlockmgr:default
        svc:/network/nfs/server:default
        svc:/milestone/multi-user-server:default
        svc:/system/basicreg:default
        svc:/system/zones:default
        svc:/network/nfs/status:default
        svc:/network/nfs/cbd:default
        svc:/network/nfs/mapid:default
        svc:/application/sthwreg:default
        svc:/application/stosreg:default
        svc:/milestone/multi-user:default
        svc:/system/xprtld:default
        svc:/system/vxdbdctrl:default
        svc:/system/vxvm/vxvm-recover:default
        svc:/system/iscsitgt:default
        svc:/network/inetd:default
        svc:/system/fmd:default
        svc:/system/fpsd:default
        svc:/system/sysidtool:system
--More--
[root@ivmtst]#
Now all okay , check the status finally: 
[root@ivmtst]# svcs -xv
[root@ivmtst]#
Thanks for reading this article...:)

Read More