Getting error when adding disk to existing diskgroup (VxVM vxdg ERROR V-5-1-10128 Configuration daemon error 441)

The actual my activity is to increase one of the file system size which is part of VXVM mirror volume, when I try to add disks to existing diskgroup I have faced this issue.

This issue happened due to DG configuration mismatching and this post may help you when your in the same situation.  

Current status of FS / Volume:

[root@ivmprd ~]# df -h /opt/nbuback
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/NDG/LVOL01
                       1.2T   1.1T    64G    95%    /opt/nbuback
[root@ivmprd ~]# vxprint -htg NDG
DG NAME  NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME  STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME  DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME  RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME  RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME  CACHEVOL     KSTATE   STATE
VT NAME  RVG          KSTATE   STATE    NVOLUME
V  NAME  RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME  VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME  PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME  PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME  PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME  PARENTVOL    LOGVOL
SP NAME  SNAPVOL      DCO
EX NAME  ASSOC        VC                       PERMS    MODE     STATE
SR NAME  KSTATE

dg NDG      default      default  20000    1201549779.11.ivmprd

dm USDC_01 store1-usp1_01a1 auto 65536 870216448 -
dm USDC_02 store1-usp1_01a2 auto 65536 870216448 -
dm USDC_03 store1-usp1_01a3 auto 65536 419346688 -
dm USDC_04 store1-usp1_01a4 auto 65536 419346688 -
dm CADC_01 store2-usp2_02b1 auto 65536 870216448 -
dm CADC_02 store2-usp2_02b2 auto 65536 870216448 -
dm CADC_03 store2-usp2_02b3 auto 65536 419346688 -
dm CADC_04 store2-usp2_02b4 auto 65536 419346688 -

v  LVOL01  -         ENABLED  ACTIVE   2537300320 SELECT  -        fsgen
pl LVOL01-03 LVOL01  ENABLED  ACTIVE   2537300320 CONCAT  -        RW
sd USDC_01-01 LVOL01-03 USDC_01 0 870216448 0    store1-usp1_01a1 ENA
sd USDC_02-01 LVOL01-03 USDC_02 0 828223072 870216448 store1-usp1_01a2 ENA
sd USDC_03-01 LVOL01-03 USDC_03 0 419346688 1698439520 store1-usp1_01a3 ENA
sd USDC_02-02 LVOL01-03 USDC_02 828223072 83712 2117786208 store1-usp1_01a2 ENA
sd USDC_03-02 LVOL01-03 USDC_04 0 419346688 2117869920 store1-usp1_01a4 ENA
sd USDC_02-03 LVOL01-03 USDC_02 828306784 83712 2537216608 store1-usp1_01a2 ENA
pl LVOL01-04    LVOL01       ENABLED  ACTIVE   2537300320 CONCAT  -        RW
sd CADC_01-01 LVOL01-04 CADC_01 0  870216448 0        store2-usp2_02b1 ENA
sd CADC_02-01 LVOL01-04 CADC_02 0  828223072 870216448 store2-usp2_02b2 ENA
sd CADC_03-01 LVOL01-04 CADC_03 0  419346688 1698439520 store2-usp2_02b3 ENA
sd CADC_02-02 LVOL01-04 CADC_02 828223072 83712 2117786208 store2-usp2_02b2 ENA
sd CADC_03-02 LVOL01-04 CADC_04 0  419346688 2117869920 store2-usp2_02b4 ENA
sd CADC_02-03 LVOL01-04 CADC_02 828306784 83712 2537216608 store2-usp2_02b2 ENA
[root@ivmprd ~]# vxdisk -g NDG list
DEVICE       TYPE            DISK         GROUP        STATUS
store1-usp1_01a3 auto:cdsdisk    USDC_03  NDG        online
store1-usp1_01a1 auto:cdsdisk    USDC_01  NDG        online
store1-usp1_01a2 auto:cdsdisk    USDC_02  NDG        online
store1-usp1_01a4 auto:cdsdisk    USDC_04  NDG        online
store2-usp2_02b1 auto:cdsdisk    CADC_01  NDG        online
store2-usp2_02b2 auto:cdsdisk    CADC_02  NDG        online
store2-usp2_02b3 auto:cdsdisk    CADC_03  NDG        online
store2-usp2_02b4 auto:cdsdisk    CADC_04  NDG        online
[root@ivmprd ~]#

Now I am planing to increase the file system from 1.2 TB to 1.4TB. 
Scan the disk:

[root@ivmprd ~]# devfsadm
[root@ivmprd ~]# echo |format
Searching for disks...done

c2t50060E8006CF9F35d33: configured with capacity of 199.99GB
c2t50060E8006CFE877d32: configured with capacity of 199.99GB
c4t50060E8006CF9F25d33: configured with capacity of 199.99GB
c4t50060E8006CFE867d32: configured with capacity of 199.99GB

We can able to see the disk from OS , Now scan the disk from VXVM level:

[root@ivmprd ~]# vxdctl enable
[root@ivmprd ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
disk_1       auto:none       -            -            online invalid
store1-usp1_01a3 auto:cdsdisk    USDC_03  NDG        online
store1-usp1_01a1 auto:cdsdisk    USDC_01  NDG        online
store1-usp1_01a2 auto:cdsdisk    USDC_02  NDG        online
store1-usp1_01a5 auto            -            -            nolabel
store1-usp1_01a4 auto:cdsdisk    USDC_04  NDG        online
store2-usp2_02b1 auto:cdsdisk    CADC_01  NDG        online
store2-usp2_02b2 auto:cdsdisk    CADC_02  NDG        online
store2-usp2_02b3 auto:cdsdisk    CADC_03  NDG        online
store2-usp2_02b5 auto            -            -            nolabel
store2-usp2_02b4 auto:cdsdisk    CADC_04  NDG        online
[root@ivmprd ~]#

Setup the disk to VXVM control:

[root@ivmprd ~]# vxdisksetup -i store1-usp1_01a5
prtvtoc: /dev/vx/rdmp/store1-usp1_01a5: Unable to read Disk geometry errno = 0x6
[root@ivmprd ~]# vxdisksetup -i store2-usp2_02b5
prtvtoc: /dev/vx/rdmp/store2-usp2_02b5: Unable to read Disk geometry errno = 0x6
[root@ivmprd ~]#

We got the error above due to disks not labeled, Label the disks using format command , since my environment multipath is enabled so we need to to label each path of one disk but I have labeled all the four paths.
Label the disk using format command:

[root@whms7039 /]# format c2t50060E8006CF9F35d33

c2t50060E8006CF9F35d33: configured with capacity of 199.99GB
selecting c2t50060E8006CF9F35d33
[disk formatted]

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> p

PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> p
Current partition table (default):
Total disk cylinders available: 54611 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm    0 -    34      131.25MB    (35/0/0)       268800
  1       swap    wu   35 -    69      131.25MB    (35/0/0)       268800
  2     backup    wu    0 - 54610      199.99GB    (54611/0/0) 419412480
  3 unassigned    wm    0                0         (0/0/0)             0
  4 unassigned    wm    0                0         (0/0/0)             0
  5 unassigned    wm    0                0         (0/0/0)             0
  6        usr    wm   70 - 54610      199.74GB    (54541/0/0) 418874880
  7 unassigned    wm    0                0         (0/0/0)             0

partition> l
Ready to label disk, continue? y

partition> q

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> q
[root@whms7039 /]# format c4t50060E8006CF9F25d33
selecting c4t50060E8006CF9F25d33
[disk formatted]

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> p

PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 54611 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm    0 -    34      131.25MB    (35/0/0)       268800
  1       swap    wu   35 -    69      131.25MB    (35/0/0)       268800
  2     backup    wu    0 - 54610      199.99GB    (54611/0/0) 419412480
  3 unassigned    wm    0                0         (0/0/0)             0
  4 unassigned    wm    0                0         (0/0/0)             0
  5 unassigned    wm    0                0         (0/0/0)             0
  6        usr    wm   70 - 54610      199.74GB    (54541/0/0) 418874880
  7 unassigned    wm    0                0         (0/0/0)             0

partition> l
Ready to label disk, continue? y

partition> q

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> q

[root@whms7039 /]# format c2t50060E8006CFE877d32

c2t50060E8006CFE877d32: configured with capacity of 199.99GB
selecting c2t50060E8006CFE877d32
[disk formatted]

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> p

PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> p
Current partition table (default):
Total disk cylinders available: 54611 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm    0 -    34      131.25MB    (35/0/0)       268800
  1       swap    wu   35 -    69      131.25MB    (35/0/0)       268800
  2     backup    wu    0 - 54610      199.99GB    (54611/0/0) 419412480
  3 unassigned    wm    0                0         (0/0/0)             0
  4 unassigned    wm    0                0         (0/0/0)             0
  5 unassigned    wm    0                0         (0/0/0)             0
  6        usr    wm   70 - 54610      199.74GB    (54541/0/0) 418874880
  7 unassigned    wm    0                0         (0/0/0)             0

partition> l
Ready to label disk, continue? y

partition> q

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> q
[root@whms7039 /]# format c4t50060E8006CFE867d32
selecting c4t50060E8006CFE867d32
[disk formatted]

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> p

PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 54611 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm    0 -    34      131.25MB    (35/0/0)       268800
  1       swap    wu   35 -    69      131.25MB    (35/0/0)       268800
  2     backup    wu    0 - 54610      199.99GB    (54611/0/0) 419412480
  3 unassigned    wm    0                0         (0/0/0)             0
  4 unassigned    wm    0                0         (0/0/0)             0
  5 unassigned    wm    0                0         (0/0/0)             0
  6        usr    wm   70 - 54610      199.74GB    (54541/0/0) 418874880
  7 unassigned    wm    0                0         (0/0/0)             0

partition> l
Ready to label disk, continue? y

partition> q

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> q
[root@whms7039 /]#

Re-read the VXVM configuration again to take changes:

[root@ivmprd ~]# vxdctl enable
[root@ivmprd ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
disk_1       auto:none       -            -            online invalid
store1-usp1_01a3 auto:cdsdisk    USDC_03  NDG        online
store1-usp1_01a1 auto:cdsdisk    USDC_01  NDG        online
store1-usp1_01a2 auto:cdsdisk    USDC_02  NDG        online
store1-usp1_01a5 auto:none       -            -            online invalid
store1-usp1_01a4 auto:cdsdisk    USDC_04  NDG        online
store2-usp2_02b1 auto:cdsdisk    CADC_01  NDG        online
store2-usp2_02b2 auto:cdsdisk    CADC_02  NDG        online
store2-usp2_02b3 auto:cdsdisk    CADC_03  NDG        online
store2-usp2_02b5 auto:none       -            -            online invalid
store2-usp2_02b4 auto:cdsdisk    CADC_04  NDG        online
[root@ivmprd ~]# vxdisksetup -i store1-usp1_01a5
[root@ivmprd ~]# vxdisksetup -i store2-usp2_02b5
[root@ivmprd ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
disk_1       auto:none       -            -            online invalid
store1-usp1_01a3 auto:cdsdisk    USDC_03  NDG        online
store1-usp1_01a1 auto:cdsdisk    USDC_01  NDG        online
store1-usp1_01a2 auto:cdsdisk    USDC_02  NDG        online
store1-usp1_01a5 auto:cdsdisk    -            -            online
store1-usp1_01a4 auto:cdsdisk    USDC_04  NDG        online
store2-usp2_02b1 auto:cdsdisk    CADC_01  NDG        online
store2-usp2_02b2 auto:cdsdisk    CADC_02  NDG        online
store2-usp2_02b3 auto:cdsdisk    CADC_03  NDG        online
store2-usp2_02b5 auto:cdsdisk    -            -            online
store2-usp2_02b4 auto:cdsdisk    CADC_04  NDG        online
[root@ivmprd ~]#

Set the site tag for those disks. Since each LUN's are coming from different DC:

[root@ivmprd ~]# vxdisk settag site=USDC store1-usp1_01a5
[root@ivmprd ~]# vxdisk settag site=CADC store2-usp2_02b5
[root@ivmprd ~]#

Add the disks to existing DG (NDG):

[root@ivmprd ~]# vxdg -g NDG adddisk USDC_05=store1-usp1_01a5
VxVM vxdg ERROR V-5-1-10128  Configuration daemon error 441
[root@ivmprd ~]#

Above we got error, to resolve this issue we need to FLUSH the DG.
FLUSH the DG and add the disk to existing DG (NDG):

[root@ivmprd ~]# vxdg flush NDG
[root@ivmprd ~]# vxdg -g NDG adddisk USDC_05=store1-usp1_01a5
[root@ivmprd ~]# vxdg -g NDG adddisk CADC_05=store2-usp2_02b5
[root@ivmprd ~]#

[root@ivmprd ~]# vxdisk -g NDG list
DEVICE       TYPE            DISK         GROUP        STATUS
store1-usp1_01a3 auto:cdsdisk    USDC_03  NDG        online
store1-usp1_01a1 auto:cdsdisk    USDC_01  NDG        online
store1-usp1_01a2 auto:cdsdisk    USDC_02  NDG        online
store1-usp1_01a5 auto:cdsdisk    USDC_05  NDG        online
store1-usp1_01a4 auto:cdsdisk    USDC_04  NDG        online
store2-usp2_02b1 auto:cdsdisk    CADC_01  NDG        online
store2-usp2_02b2 auto:cdsdisk    CADC_02  NDG        online
store2-usp2_02b3 auto:cdsdisk    CADC_03  NDG        online
store2-usp2_02b5 auto:cdsdisk    CADC_05  NDG        online
store2-usp2_02b4 auto:cdsdisk    CADC_04  NDG        online
[root@ivmprd ~]#

Cool... After FLUSH we can able to add those disk to existing DG (NDG). Just check the available size:

[root@ivmprd ~]# vxassist -g NDG maxsize
Maximum volume size: 922343424 (450363Mb)
[root@ivmprd ~]# vxassist -g NDG maxgrow LVOL01 layout=mirror
Volume LVOL01 can be extended by 461170688 to: 2998471008 (1464097Mb+352 sectors)
[root@ivmprd ~]#

Grow the volume size and check the status:

[root@ivmprd ~]# vxresize -g NDG LVOL01 +200G
[root@ivmprd ~]# vxtask list
TASKID  PTID TYPE/STATE    PCT   PROGRESS
   188         RDWRBACK/R 06.87% 2537300320/2956730720/2566121824 RESYNC LVOL01 NDG
[root@ivmprd ~]#

Once sync is completed successfully , just check the status:

[root@whms7039 /]# df -h /opt/nbuback
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/NDG/LVOL01
                       1.4T   1.1T   256G    82%    /opt/nbuback
[root@ivmprd ~]#

Thanks for reading this post...

For more info you can refer symantec site, Thanks to symantec for providing this document.

http://www.symantec.com/business/support/index?page=content&id=TECH130463

Read More

VXVM Disk group import failed

I have recently encounter DG import failed issue on one of my environment and finally resolved as below.
While reboot the server I have faced below issue, not able to get the OS prompt due to DG import failed. When I checked the server I came to know some of the FS not able to mount due to the DG not imported.

Steps to resolve the problem:
Identify issue from OS level:

[root@ivmtst]# svcs -xv | more
svc:/system/filesystem/local:default (local file system mounts)
 State: maintenance since Jan 10, 2014  02:05:45 PM EDT  
Reason: Start method exited with $SMF_EXIT_ERR_FATAL.
   See: http://sun.com/msg/SMF-8000-KS
   See: /var/svc/log/system-filesystem-local:default.log
Impact: 39 dependent services are not running:
        svc:/application/psncollector:default
        svc:/system/sysidtool:net
        svc:/network/rpc/bind:default
        svc:/network/nfs/nlockmgr:default
        svc:/network/nfs/server:default
        svc:/milestone/multi-user-server:default
        svc:/system/basicreg:default
        svc:/system/zones:default
        svc:/network/nfs/status:default
        svc:/network/nfs/cbd:default
        svc:/network/nfs/mapid:default
        svc:/application/sthwreg:default
        svc:/application/stosreg:default
        svc:/milestone/multi-user:default
        svc:/system/xprtld:default
        svc:/system/vxdbdctrl:default
        svc:/system/vxvm/vxvm-recover:default
        svc:/system/iscsitgt:default
        svc:/network/inetd:default
        svc:/system/fmd:default
        svc:/system/fpsd:default
        svc:/system/sysidtool:system

See the SVC Logs for more details to troubleshoot:

[root@ivmtst]# tail -10 /var/svc/log/system-filesystem-local:default.log
UX:vxfs mount: ERROR: V-3-20002: Cannot access /dev/vx/dsk/IVDG/TVOL50: No such file or directory
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
UX:vxfs mount: ERROR: V-3-20002: Cannot access /dev/vx/dsk/IVDG/TVOL90: No such file or directory
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
UX:vxfs mount: ERROR: V-3-20002: Cannot access /dev/vx/dsk/IVDG/TVOL20: No such file or directory
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
UX:vxfs mount: ERROR: V-3-20002: Cannot access /dev/vx/dsk/IVDG/TVOL30: No such file or directory
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
WARNING: /sbin/mountall -l failed: exit status 1
[ Jan 10 02:05:45 Method "start" exited with status 95 ]
[root@ivmtst]#

Checking DG status:

[root@ivmtst]# vxdisk -eo alldgs list | grep -i IVDG
emc07_178b1 auto:cdsdisk - (IVDG) online thinrclm c1t5000090001112D9Cd34s2 tprclm
emc07_17b6 auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9Cd34s2 tprclm
emc07_18b7 auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9Cd64s2 tprclm
emc07_19b8 auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9Cd70s2 tprclm
emc07_177a auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9C8164s2 tprclm
emc07_19b9 auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9Cd96s2 tprclm
emc07_19b0 auto:cdsdisk  - (IVDG) online thinrclm c1t5000090001112D9Cd207s2 tprclm
[root@ivmtst]#

As per above output DG in deported state so trying to import the DG but import failed:

[root@ivmtst]# vxdg -Cf import IVDG
Jan 10 02:05:45 vxvm:vxconfigd: V-5-1-0 Disk Group IVDG has a mix of standard and cloned disks: Trying to import the disk group from the standard disks: emc07_177a
Jan 10 02:05:45 vxvm:vxconfigd: V-5-1-0 No enabled configuration copies found
Jan 10 02:05:45 vxvm:vxconfigd: V-5-1-0 Disk group import of IVDG failed with error 182 – Disk group has no valid configuration copies
VxVM vxdg ERROR V-5-1-10978 Disk group IVDG: import failed:
Disk group has no valid configuration copies
[root@ivmtst]#

Identify which are the disks part of IVDG disk group:

[root@ivmtst]# vxdisk -eo alldgs list | grep -i IVDG
emc07_178b1 auto:cdsdisk emc07_178b1 IVDG online thinrclm c1t5000090001112D9Cd34s2 tprclm
emc07_17b6 auto:cdsdisk emc07_17b6 IVDG online thinrclm c1t5000090001112D9Cd34s2 tprclm
emc07_18b7 auto:cdsdisk emc07_18b7 IVDG online thinrclm c1t5000090001112D9Cd64s2 tprclm
emc07_19b8 auto:cdsdisk emc07_19b8 IVDG online thinrclm c1t5000090001112D9Cd70s2 tprclm
emc07_177a auto:cdsdisk emc07_177a IVDG online thinrclm c1t5000090001112D9C8164s2 tprclm
emc07_19b9 auto:cdsdisk emc07_19b9 IVDG online thinrclm c1t5000090001112D9Cd96s2 tprclm
emc07_19b0 auto:cdsdisk emc07_19b0 IVDG online thinrclm c1t5000090001112D9Cd207s2 tprclm
[root@ivmtst]#

Steps to resolve the disk group import failed problem.
Update udid ID and off the clone flag:

[root@ivmtst ]# vxdisk updateudid emc07_178b1
[root@ivmtst ]# vxdisk updateudid emc07_17b6
[root@ivmtst ]# vxdisk updateudid emc07_18b7
[root@ivmtst ]# vxdisk updateudid emc07_19b8
[root@ivmtst ]# vxdisk updateudid emc07_177a
[root@ivmtst ]# vxdisk updateudid emc07_19b9
[root@ivmtst ]# vxdisk updateudid emc07_19b0
[root@ivmtst ]# vxdisk set emc07_178b1 clone=off
[root@ivmtst ]# vxdisk set emc07_17b6 clone=off
[root@ivmtst ]# vxdisk set emc07_18b7 clone=off
[root@ivmtst ]# vxdisk set emc07_19b8 clone=off
[root@ivmtst ]# vxdisk set emc07_177a clone=off
[root@ivmtst ]# vxdisk set emc07_19b9 clone=off
[root@ivmtst ]# vxdisk set emc07_19b0 clone=off
[root@ivmtst]

Import the DG again forcefully :

[root@ivmtst]# vxdg -Cf import IVDG
Jan 10 02:45:20 vxvm:vxconfigd: V-5-1-0 Selecting configuration database copy from emc07_19b9 from disks: emc07_19b9 emc07_19b0 emc07_19b8emc07_18b7 emc07_17b6
Jan 10 02:45:20 vxvm:vxconfigd: V-5-1-0 Trying to import the disk group IVDG using configuration database copy from emc07_19b9
Jan 10 02:45:20 vxvm:vxconfigd: V-5-1-0 Disk group import of IVDG succeeded.
[root@ivmtst]#

Finally we have imported successfully. 
Clear the SMF service from OS level:

[root@ivmtst]# svcadm clear svc:/system/filesystem/local:default

[root@ivmtst]# svcs -xv | more
svc:/system/filesystem/local:default (local file system mounts)
 State: offline since Jan 10, 2014  02:45:15 PM EDT 
Reason: Start method is running.
   See: http://sun.com/msg/SMF-8000-C4
   See: /var/svc/log/system-filesystem-local:default.log
Impact: 39 dependent services are not running:
        svc:/application/psncollector:default
        svc:/system/sysidtool:net
        svc:/network/rpc/bind:default
        svc:/network/nfs/nlockmgr:default
        svc:/network/nfs/server:default
        svc:/milestone/multi-user-server:default
        svc:/system/basicreg:default
        svc:/system/zones:default
        svc:/network/nfs/status:default
        svc:/network/nfs/cbd:default
        svc:/network/nfs/mapid:default
        svc:/application/sthwreg:default
        svc:/application/stosreg:default
        svc:/milestone/multi-user:default
        svc:/system/xprtld:default
        svc:/system/vxdbdctrl:default
        svc:/system/vxvm/vxvm-recover:default
        svc:/system/iscsitgt:default
        svc:/network/inetd:default
        svc:/system/fmd:default
        svc:/system/fpsd:default
        svc:/system/sysidtool:system
--More--
[root@ivmtst]# 

Now all okay , check the status finally:

[root@ivmtst]# svcs -xv
[root@ivmtst]# 

Thanks for reading this article...:)

Read More

DF command shows incorrect usage for VXFS file system

I have faced issue with one of the VXFS file system which is showing incorrect volume usage when I execute “df -h”command. But actually if you do "du -sh" there is no data and volume is empty but still it is showing incorrect value , after digging more I got to know this is known issue. 

Here I am giving details how to calculate actual space when this type of issue occur.

Information from Symantec site (VERITAS) :

1. VxFS will use some of this space for structural data such as the Intent Log, Free Extent Bitmaps, Superblocks, etc. 
2. Due to the dynamic inode allocation functionality on a VXFS, free space is not the same as available space.

This space will be released automatically by Veritas when user needs that space means if you start fill up the volume it will get release automatically.

The right way of identifying the actual free available space we should use “df -b” command.

As per below scenario we are missing around 30 GB size.

[root@iv01]# df -h /ora/data1
Filesystem              size   used  avail capacity  Mounted on
/dev/vx/dsk/ora/data1   300G   500M   269G     1%    /ora/data1
[root@iv01]# du -sh /ora/data1
   0K   /ora/data1
[root@iv01]#

To examine correct value:

[root@iv01]# df -b /ora/data1
Filesystem              avail
/ora/data1     313524224
[root@iv01]#

Calculate KB to GB:

[root@iv01]# bc
313524224/1024/1024
299
[root@iv01]#

We can see through "df -b" command , it is reporting correct value. This is not the issue in VXFS but actually VXFS is keeping configuration date like this method.

Read More