Wednesday, August 27, 2014

Getting error when adding disk to existing diskgroup (VxVM vxdg ERROR V-5-1-10128 Configuration daemon error 441)

The actual my activity is to increase one of the file system size which is part of VXVM mirror volume, when I try to add disks to existing diskgroup I have faced this issue.

This issue happened due to DG configuration mismatching and this post may help you when your in the same situation.  

Current status of FS / Volume:
[root@ivmprd ~]# df -h /opt/nbuback
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/NDG/LVOL01
                       1.2T   1.1T    64G    95%    /opt/nbuback
[root@ivmprd ~]# vxprint -htg NDG
DG NAME  NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME  STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME  DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME  RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME  RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME  CACHEVOL     KSTATE   STATE
VT NAME  RVG          KSTATE   STATE    NVOLUME
V  NAME  RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME  VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME  PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME  PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME  PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME  PARENTVOL    LOGVOL
SP NAME  SNAPVOL      DCO
EX NAME  ASSOC        VC                       PERMS    MODE     STATE
SR NAME  KSTATE

dg NDG      default      default  20000    1201549779.11.ivmprd

dm USDC_01 store1-usp1_01a1 auto 65536 870216448 -
dm USDC_02 store1-usp1_01a2 auto 65536 870216448 -
dm USDC_03 store1-usp1_01a3 auto 65536 419346688 -
dm USDC_04 store1-usp1_01a4 auto 65536 419346688 -
dm CADC_01 store2-usp2_02b1 auto 65536 870216448 -
dm CADC_02 store2-usp2_02b2 auto 65536 870216448 -
dm CADC_03 store2-usp2_02b3 auto 65536 419346688 -
dm CADC_04 store2-usp2_02b4 auto 65536 419346688 -

v  LVOL01  -         ENABLED  ACTIVE   2537300320 SELECT  -        fsgen
pl LVOL01-03 LVOL01  ENABLED  ACTIVE   2537300320 CONCAT  -        RW
sd USDC_01-01 LVOL01-03 USDC_01 0 870216448 0    store1-usp1_01a1 ENA
sd USDC_02-01 LVOL01-03 USDC_02 0 828223072 870216448 store1-usp1_01a2 ENA
sd USDC_03-01 LVOL01-03 USDC_03 0 419346688 1698439520 store1-usp1_01a3 ENA
sd USDC_02-02 LVOL01-03 USDC_02 828223072 83712 2117786208 store1-usp1_01a2 ENA
sd USDC_03-02 LVOL01-03 USDC_04 0 419346688 2117869920 store1-usp1_01a4 ENA
sd USDC_02-03 LVOL01-03 USDC_02 828306784 83712 2537216608 store1-usp1_01a2 ENA
pl LVOL01-04    LVOL01       ENABLED  ACTIVE   2537300320 CONCAT  -        RW
sd CADC_01-01 LVOL01-04 CADC_01 0  870216448 0        store2-usp2_02b1 ENA
sd CADC_02-01 LVOL01-04 CADC_02 0  828223072 870216448 store2-usp2_02b2 ENA
sd CADC_03-01 LVOL01-04 CADC_03 0  419346688 1698439520 store2-usp2_02b3 ENA
sd CADC_02-02 LVOL01-04 CADC_02 828223072 83712 2117786208 store2-usp2_02b2 ENA
sd CADC_03-02 LVOL01-04 CADC_04 0  419346688 2117869920 store2-usp2_02b4 ENA
sd CADC_02-03 LVOL01-04 CADC_02 828306784 83712 2537216608 store2-usp2_02b2 ENA
[root@ivmprd ~]# vxdisk -g NDG list
DEVICE       TYPE            DISK         GROUP        STATUS
store1-usp1_01a3 auto:cdsdisk    USDC_03  NDG        online
store1-usp1_01a1 auto:cdsdisk    USDC_01  NDG        online
store1-usp1_01a2 auto:cdsdisk    USDC_02  NDG        online
store1-usp1_01a4 auto:cdsdisk    USDC_04  NDG        online
store2-usp2_02b1 auto:cdsdisk    CADC_01  NDG        online
store2-usp2_02b2 auto:cdsdisk    CADC_02  NDG        online
store2-usp2_02b3 auto:cdsdisk    CADC_03  NDG        online
store2-usp2_02b4 auto:cdsdisk    CADC_04  NDG        online
[root@ivmprd ~]#
Now I am planing to increase the file system from 1.2 TB to 1.4TB
Scan the disk:
[root@ivmprd ~]# devfsadm
[root@ivmprd ~]# echo |format
Searching for disks...done

c2t50060E8006CF9F35d33: configured with capacity of 199.99GB
c2t50060E8006CFE877d32: configured with capacity of 199.99GB
c4t50060E8006CF9F25d33: configured with capacity of 199.99GB
c4t50060E8006CFE867d32: configured with capacity of 199.99GB
We can able to see the disk from OS , Now scan the disk from VXVM level:
[root@ivmprd ~]# vxdctl enable
[root@ivmprd ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
disk_1       auto:none       -            -            online invalid
store1-usp1_01a3 auto:cdsdisk    USDC_03  NDG        online
store1-usp1_01a1 auto:cdsdisk    USDC_01  NDG        online
store1-usp1_01a2 auto:cdsdisk    USDC_02  NDG        online
store1-usp1_01a5 auto            -            -            nolabel
store1-usp1_01a4 auto:cdsdisk    USDC_04  NDG        online
store2-usp2_02b1 auto:cdsdisk    CADC_01  NDG        online
store2-usp2_02b2 auto:cdsdisk    CADC_02  NDG        online
store2-usp2_02b3 auto:cdsdisk    CADC_03  NDG        online
store2-usp2_02b5 auto            -            -            nolabel
store2-usp2_02b4 auto:cdsdisk    CADC_04  NDG        online
[root@ivmprd ~]#
Setup the disk to VXVM control:
[root@ivmprd ~]# vxdisksetup -i store1-usp1_01a5
prtvtoc: /dev/vx/rdmp/store1-usp1_01a5: Unable to read Disk geometry errno = 0x6
[root@ivmprd ~]# vxdisksetup -i store2-usp2_02b5
prtvtoc: /dev/vx/rdmp/store2-usp2_02b5: Unable to read Disk geometry errno = 0x6
[root@ivmprd ~]#
We got the error above due to disks not labeled, Label the disks using format command , since my environment multipath is enabled so we need to to label each path of one disk but I have labeled all the four paths.
Label the disk using format command:
[root@whms7039 /]# format c2t50060E8006CF9F35d33

c2t50060E8006CF9F35d33: configured with capacity of 199.99GB
selecting c2t50060E8006CF9F35d33
[disk formatted]

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> p

PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> p
Current partition table (default):
Total disk cylinders available: 54611 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm    0 -    34      131.25MB    (35/0/0)       268800
  1       swap    wu   35 -    69      131.25MB    (35/0/0)       268800
  2     backup    wu    0 - 54610      199.99GB    (54611/0/0) 419412480
  3 unassigned    wm    0                0         (0/0/0)             0
  4 unassigned    wm    0                0         (0/0/0)             0
  5 unassigned    wm    0                0         (0/0/0)             0
  6        usr    wm   70 - 54610      199.74GB    (54541/0/0) 418874880
  7 unassigned    wm    0                0         (0/0/0)             0

partition> l
Ready to label disk, continue? y

partition> q

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> q
[root@whms7039 /]# format c4t50060E8006CF9F25d33
selecting c4t50060E8006CF9F25d33
[disk formatted]

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> p

PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 54611 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm    0 -    34      131.25MB    (35/0/0)       268800
  1       swap    wu   35 -    69      131.25MB    (35/0/0)       268800
  2     backup    wu    0 - 54610      199.99GB    (54611/0/0) 419412480
  3 unassigned    wm    0                0         (0/0/0)             0
  4 unassigned    wm    0                0         (0/0/0)             0
  5 unassigned    wm    0                0         (0/0/0)             0
  6        usr    wm   70 - 54610      199.74GB    (54541/0/0) 418874880
  7 unassigned    wm    0                0         (0/0/0)             0

partition> l
Ready to label disk, continue? y

partition> q

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> q

[root@whms7039 /]# format c2t50060E8006CFE877d32

c2t50060E8006CFE877d32: configured with capacity of 199.99GB
selecting c2t50060E8006CFE877d32
[disk formatted]

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> p

PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> p
Current partition table (default):
Total disk cylinders available: 54611 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm    0 -    34      131.25MB    (35/0/0)       268800
  1       swap    wu   35 -    69      131.25MB    (35/0/0)       268800
  2     backup    wu    0 - 54610      199.99GB    (54611/0/0) 419412480
  3 unassigned    wm    0                0         (0/0/0)             0
  4 unassigned    wm    0                0         (0/0/0)             0
  5 unassigned    wm    0                0         (0/0/0)             0
  6        usr    wm   70 - 54610      199.74GB    (54541/0/0) 418874880
  7 unassigned    wm    0                0         (0/0/0)             0

partition> l
Ready to label disk, continue? y

partition> q

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> q
[root@whms7039 /]# format c4t50060E8006CFE867d32
selecting c4t50060E8006CFE867d32
[disk formatted]

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> p

PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 54611 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm    0 -    34      131.25MB    (35/0/0)       268800
  1       swap    wu   35 -    69      131.25MB    (35/0/0)       268800
  2     backup    wu    0 - 54610      199.99GB    (54611/0/0) 419412480
  3 unassigned    wm    0                0         (0/0/0)             0
  4 unassigned    wm    0                0         (0/0/0)             0
  5 unassigned    wm    0                0         (0/0/0)             0
  6        usr    wm   70 - 54610      199.74GB    (54541/0/0) 418874880
  7 unassigned    wm    0                0         (0/0/0)             0

partition> l
Ready to label disk, continue? y

partition> q

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> q
[root@whms7039 /]#
Re-read the VXVM configuration again to take changes:
[root@ivmprd ~]# vxdctl enable
[root@ivmprd ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
disk_1       auto:none       -            -            online invalid
store1-usp1_01a3 auto:cdsdisk    USDC_03  NDG        online
store1-usp1_01a1 auto:cdsdisk    USDC_01  NDG        online
store1-usp1_01a2 auto:cdsdisk    USDC_02  NDG        online
store1-usp1_01a5 auto:none       -            -            online invalid
store1-usp1_01a4 auto:cdsdisk    USDC_04  NDG        online
store2-usp2_02b1 auto:cdsdisk    CADC_01  NDG        online
store2-usp2_02b2 auto:cdsdisk    CADC_02  NDG        online
store2-usp2_02b3 auto:cdsdisk    CADC_03  NDG        online
store2-usp2_02b5 auto:none       -            -            online invalid
store2-usp2_02b4 auto:cdsdisk    CADC_04  NDG        online
[root@ivmprd ~]# vxdisksetup -i store1-usp1_01a5
[root@ivmprd ~]# vxdisksetup -i store2-usp2_02b5
[root@ivmprd ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
disk_1       auto:none       -            -            online invalid
store1-usp1_01a3 auto:cdsdisk    USDC_03  NDG        online
store1-usp1_01a1 auto:cdsdisk    USDC_01  NDG        online
store1-usp1_01a2 auto:cdsdisk    USDC_02  NDG        online
store1-usp1_01a5 auto:cdsdisk    -            -            online
store1-usp1_01a4 auto:cdsdisk    USDC_04  NDG        online
store2-usp2_02b1 auto:cdsdisk    CADC_01  NDG        online
store2-usp2_02b2 auto:cdsdisk    CADC_02  NDG        online
store2-usp2_02b3 auto:cdsdisk    CADC_03  NDG        online
store2-usp2_02b5 auto:cdsdisk    -            -            online
store2-usp2_02b4 auto:cdsdisk    CADC_04  NDG        online
[root@ivmprd ~]#
Set the site tag for those disks. Since each LUN's are coming from different DC:
[root@ivmprd ~]# vxdisk settag site=USDC store1-usp1_01a5
[root@ivmprd ~]# vxdisk settag site=CADC store2-usp2_02b5
[root@ivmprd ~]#
Add the disks to existing DG (NDG):
[root@ivmprd ~]# vxdg -g NDG adddisk USDC_05=store1-usp1_01a5
VxVM vxdg ERROR V-5-1-10128  Configuration daemon error 441
[root@ivmprd ~]#
Above we got error, to resolve this issue we need to FLUSH the DG.
FLUSH the DG and add the disk to existing DG (NDG):
[root@ivmprd ~]# vxdg flush NDG
[root@ivmprd ~]# vxdg -g NDG adddisk USDC_05=store1-usp1_01a5
[root@ivmprd ~]# vxdg -g NDG adddisk CADC_05=store2-usp2_02b5
[root@ivmprd ~]#
[root@ivmprd ~]# vxdisk -g NDG list
DEVICE       TYPE            DISK         GROUP        STATUS
store1-usp1_01a3 auto:cdsdisk    USDC_03  NDG        online
store1-usp1_01a1 auto:cdsdisk    USDC_01  NDG        online
store1-usp1_01a2 auto:cdsdisk    USDC_02  NDG        online
store1-usp1_01a5 auto:cdsdisk    USDC_05  NDG        online
store1-usp1_01a4 auto:cdsdisk    USDC_04  NDG        online
store2-usp2_02b1 auto:cdsdisk    CADC_01  NDG        online
store2-usp2_02b2 auto:cdsdisk    CADC_02  NDG        online
store2-usp2_02b3 auto:cdsdisk    CADC_03  NDG        online
store2-usp2_02b5 auto:cdsdisk    CADC_05  NDG        online
store2-usp2_02b4 auto:cdsdisk    CADC_04  NDG        online
[root@ivmprd ~]#
Cool... After FLUSH we can able to add those disk to existing DG (NDG). Just check the available size:
[root@ivmprd ~]# vxassist -g NDG maxsize
Maximum volume size: 922343424 (450363Mb)
[root@ivmprd ~]# vxassist -g NDG maxgrow LVOL01 layout=mirror
Volume LVOL01 can be extended by 461170688 to: 2998471008 (1464097Mb+352 sectors)
[root@ivmprd ~]#
Grow the volume size and check the status:
[root@ivmprd ~]# vxresize -g NDG LVOL01 +200G
[root@ivmprd ~]# vxtask list
TASKID  PTID TYPE/STATE    PCT   PROGRESS
   188         RDWRBACK/R 06.87% 2537300320/2956730720/2566121824 RESYNC LVOL01 NDG
[root@ivmprd ~]#
Once sync is completed successfully , just check the status:
[root@whms7039 /]# df -h /opt/nbuback
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/NDG/LVOL01
                       1.4T   1.1T   256G    82%    /opt/nbuback
[root@ivmprd ~]#
Thanks for reading this post...

For more info you can refer symantec site, Thanks to symantec for providing this document.

http://www.symantec.com/business/support/index?page=content&id=TECH130463


0 comments:

Post a Comment