Expanding ZFS rpool in Solaris 10

I was really bad at googling the steps to grow ZFS rpool online without using zpool append, so here is my little story how I’ve done it.
Before I begin, please, note that everything said below applies only to the configurations where rpool consumes a whole disk. If there is another partition sitting right after (or with some gap) your rpool, you’re risking to corrupt the data. So don’t blindly use the last cylinder (or “$”) when it’s time to modify the partition table.

  1. Here is the rpool:
  2. # zpool list rpool
    rpool  31.1G  23.6G  7.52G  75%  ONLINE  -
  3. Saving the partition table first so later this information could be used to correctly re-lable (re-partition) the expanded disk. Btw, use “zpool status rpool -v” to find the device name:
  4. # prtvtoc /dev/dsk/c4t60060E80167D3C0000017D3C000010CAd0s2
    * /dev/dsk/c4t60060E80167D3C0000017D3C000010CAd0s2 partition map
    * Dimensions:
    *     512 bytes/sector
    *     512 sectors/track
    *      15 tracks/cylinder
    *    7680 sectors/cylinder
    *    8533 cylinders
    *    8531 accessible cylinders
    * Flags:
    *   1: unmountable
    *  10: read-only
    *                          First     Sector    Last
    * Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
           0      2    00          0  65518080  65518079
           2      5    01          0  65518080  65518079
  5. Behind the scene LUN was expanded. To confirm that luxadm could be quite handy (pay attention to “Unformatted capacity”) field:
  6. # luxadm display /dev/rdsk/c4t60060E80167D3C0000017D3C000010CAd0s2
    DEVICE PROPERTIES for disk: /dev/rdsk/c4t60060E80167D3C0000017D3C000010CAd0s2
      Vendor:               HITACHI
      Product ID:           OPEN-V      -SUN
      Revision:             7006
      Serial Num:           50 17D3C10CA
      Unformatted capacity: 36864.000 MBytes
      Read Cache:           Enabled
        Minimum prefetch:   0x0
        Maximum prefetch:   0x0
      Device Type:          Disk device
  7. Time for the scariest part, i.e. re-lable the disk by installing a new partition table.
    To be able to do that a new disk’s geometry must be somehow conveyed to the format utility and that’s surprisingly easy to achieve. Just run format, select the disk and use “type” option to autoconfigure it:
  8. # format
    Searching for disks...done
           0. c1t0d0 
           1. c4t60060E80167D3C0000017D3C000010CAd0 
    Specify disk (enter its number): 1
    selecting c4t60060E80167D3C0000017D3C000010CAd0
    [disk formatted]
    /dev/dsk/c4t60060E80167D3C0000017D3C000010CAd0s0 is part of active ZFS pool rpool. Please see zpool(1M).
            disk       - select a disk
            type       - select (define) a disk type
            partition  - select (define) a partition table
            current    - describe the current disk
            format     - format and analyze the disk
            repair     - repair a defective sector
            label      - write label to the disk
            analyze    - surface analysis
            defect     - defect list management
            backup     - search for backup labels
            verify     - read and display labels
            save       - save new disk/partition definitions
            inquiry    - show vendor, product and revision
            volname    - set 8-character volume name
            !     - execute , then return
    format> type
            0. Auto configure
            1. Quantum ProDrive 80S
            2. Quantum ProDrive 105S
            3. CDC Wren IV 94171-344
            4. SUN0104
            5. SUN0207
            6. SUN0327
            7. SUN0340
            8. SUN0424
            9. SUN0535
            10. SUN0669
            11. SUN1.0G
            12. SUN1.05
            13. SUN1.3G
            14. SUN2.1G
            15. SUN2.9G
            16. Zip 100
            17. Zip 250
            18. Peerless 10GB
            19. HITACHI-OPEN-V-SUN-7005
            20. SUN300G
            21. other
    Specify disk type (enter its number)[19]: 0
    c4t60060E80167D3C0000017D3C000010CAd0: configured with capacity of 35.99GB
    selecting c4t60060E80167D3C0000017D3C000010CAd0
    [disk formatted]
  9. Notice that now it’s configured with a new capacity.
  10. Don’t leave the format prompt yet since we are not done. Next step is to carve out the partition table. Remember, that I only had two partitions (0 for root and 2 for backup) and your situation might be completely different, so don’t copy/paste rashly.
  11. Use the numbers from the menu to select the partition you’re willing to modify:
  12. format> partition
            0      - change `0' partition
            1      - change `1' partition
            2      - change `2' partition
            3      - change `3' partition
            4      - change `4' partition
            5      - change `5' partition
            6      - change `6' partition
            7      - change `7' partition
            select - select a predefined table
            modify - modify a predefined partition table
            name   - name the current table
            print  - display the current table
            label  - write partition map and label to the disk
  13. For me, it was enough to set the size of all partitions to zero except the two: 0 (tagged as root) and 2 (tagged as backup). As you can see below wm (write-mountable) flag was set only for partition 0, whilst the rest have wu (write-unmountable):
  14. partition> print
    Current partition table (unnamed):
    Total disk cylinders available: 9828 + 2 (reserved cylinders)
    Part      Tag    Flag     Cylinders        Size            Blocks
      0       root    wm       0 - 9827       35.99GB    (9828/0/0) 75479040
      1 unassigned    wu       0               0         (0/0/0)           0
      2     backup    wu       0 - 9827       35.99GB    (9828/0/0) 75479040
      3 unassigned    wu       0               0         (0/0/0)           0
      4 unassigned    wu       0               0         (0/0/0)           0
      5 unassigned    wu       0               0         (0/0/0)           0
      6 unassigned    wu       0               0         (0/0/0)           0
      7 unassigned    wu       0               0         (0/0/0)           0
  15. Label the disk and quit the format tool:
  16. partition> label
    Ready to label disk, continue? yes
    partition> q
    format> q

  17. Finally it’s time to grow ZFS rpool and to confirm we’re golden:
  18. # zpool online -e rpool /dev/dsk/c4t60060E80167D3C0000017D3C000010CAd0s0
    # zpool list rpool
    rpool  35.9G  23.6G  12.3G  65%  ONLINE  -

Good luck and safe expanding your pools.

How to create new thin devices using symcli and add them to a masking view and a storage group

Below are the steps that I used recently to create and present a few thin devices. More precisely:

  • Boot disk – 50GB
  • Two TDEVs 150GB each. One for data and the other for logs.
  1. Create a nested initiator group:
  2. symaccess -sid VMAX-SID create -name IG_SERVERNAME_HBA0 -type initiator -consistent_lun
    symaccess -sid VMAX-SID -name IG_SERVERNAME_HBA0 -type initiator -wwn 50060b0000c30e38 add
    symaccess -sid VMAX-SID create -name IG_SERVERNAME_HBA1 -type initiator -consistent_lun
    symaccess -sid VMAX-SID -name IG_SERVERNAME_HBA1 -type initiator -wwn 50060b0000c30e3a add
    symaccess -sid VMAX-SID create -name IG_SERVERNAME -type initiator -consistent_lun
    symaccess -sid VMAX-SID -name IG_SERVERNAME -type initiator -ig IG_SERVERNAME_HBA0 add
    symaccess -sid VMAX-SID -name IG_SERVERNAME -type initiator -ig IG_SERVERNAME_HBA1 add
  3. Show initiator group’s details:
  4. symaccess -sid VMAX-SID show IG_SERVERNAME -type initiator -detail
  5. Create storage group:
  6. symaccess -sid VMAX-SID create -name SG_SERVERNAME -type storage
  7. Create thin devices and bind them to a pool:
  8. symconfigure -sid VMAX-SID -cmd "create dev count=1, size=50GB, emulation=fba, config=TDEV, binding to pool=Pool1;" prepare
    symconfigure -sid VMAX-SID -cmd "create dev count=1, size=50GB, emulation=fba, config=TDEV, binding to pool=Pool1;" commit
    symconfigure -sid VMAX-SID -cmd "create dev count=2, size=150GB, emulation=fba, config=TDEV, binding to pool=Pool1;" prepare
    symconfigure -sid VMAX-SID -cmd "create dev count=2, size=150GB, emulation=fba, config=TDEV, binding to pool=Pool1;" commit

    Don’t forget to record the device IDs. In my case they were the following: 016B (50GB),016C (150GB) and 005F (150GB) respectively.

  9. Add new devices to our storage group:
  10. Adding disks to the storage group and further masking require additional explanation. As all we know, boot device must be lun 0. But you can’t assign a specific LUN ID to every new device in one go during the masking step since there is only a single option, i.e. -lun, which tells the starting LUN ID. BUT, and that’s important, this option, -lun, would assign the starting LUN ID to a device with the smallest ID. And in my case that would be TDEV 005F and that wasn’t what my task was all about. To overcome that hurdle, I firstly added devices 016B (boot disk) and 016C (data disk) and masked them to a storage group with -lun 0 options. By doing that, I could guarantee that my boot device would be assigned LUN 0 (obviously 016B (hex) is smaller than 016C (hex)). After that all I had to do is to add the third data device to the storage group and it would be automatically assigned the next available LUN ID.

    symaccess -sid VMAX-SID -name SG_SERVERNAME -type storage add devs 016B,016C
  11. Create a new masking view and assign LUN IDs to the devices:
  12. symaccess -sid VMAX-SID create view -name MV_SERVERNAME -sg SG_SERVERNAME -pg PG_NAME -ig IG_SERVERNAME -lun 0
  13. Add third device to the storage group:
  14. symaccess -sid VMAX-SID -name SG_SERVERNAME -type storage add devs 005F
  15. Associate the storage group with a FAST VP policy:
  16. symfast -sid VMAX-SID -fp_name FAST_VP_NAME  associate -sg SG_SERVERNAME
  17. View the details of newly created masking view:
  18. symaccess -sid VMAX-SID show view MV_SERVERNAME

Job done!

How to remove the last dead path in ESX

Feb  9 10:56:27 esx vmkernel: 47:18:55:03.308 cpu8:4225)WARNING: NMP: nmp_DeviceRetryCommand: Device "naa.60060e80056e030000006e030000004b": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
Feb  9 10:56:27 esx vmkernel: 47:18:55:03.308 cpu8:4225)WARNING: NMP: nmp_DeviceStartLoop: NMP Device "naa.60060e80056e030000006e030000004b" is blocked. Not starting I/O from device.
Feb  9 10:56:28 esx vmkernel: 47:18:55:04.323 cpu8:4264)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "naa.60060e80056e030000006e030000004b" - issuing command 0x41027f458540
Feb  9 10:56:28 esx vmkernel: 47:18:55:04.323 cpu8:4264)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "naa.60060e80056e030000006e030000004b" - failed to issue command due to Not found (APD), try again...

If these log lines are familiar and you’ve been desperately trying to remove the last FC path to your storage array, which is by a awry coincedence is also dead, then probably the following command could help you in dealing with the issue:

# esxcli corestorage claiming unclaim -A vmhba1 -C 0 -T 3 -L 33 -d naa.60060e80056e030000006e030000004b -t location
# esxcfg-scsidevs -o naa.60060e80056e030000006e030000004b

Just don’t copy/paste it boldly but instead use your C:T:L and naa.* values appropriately.

Worked for me like a charm.

Enable CIM server on MDS Cisco switch

Want to add your shiny MDS Cisco switch into HTnM (Hitachi Tuning Manager) – that’s very easy. All you have to do is to enable cim, preferably secure cim server, and configure agtw (SAN agent) using HTnM cli. How I did it…

  1. ssh into a switch to configure CIM server. First I copied SSL certificate (self-signed will do well) to configure secure cim server:
  2. # copy scp://username@ssh_host/path_to_ssl_certificate_file bootflash:simserver.pem
    # configure t
    # cimserver certificate bootflash:simserver.pem
    # cimserver enableHttps
    operation cimserver enableHttps():  status: Please restart cimserver for this request to take effect
    # no cimserver enableHttp
    # no feature cimserver
    # feature cimserver
    # show cimserver status
     cimserver is enabled
    # show cimserver httpSStatus
     cimserver Https is enabled
    # show cimserver httpStatus
     cimserver Http is not enabled
  3. Go to the server’s console of your server to do HTnM part.
  4. # /opt/jp1pc/tools/jpcinssetup agtw -inst mds_swicth_name
    Connection Destination ( 1:Brocade(DCFM SMI Agent) or 2:Brocade(SMI Agent for FOS) or 3:Brocade(SMI Agent for EOS) or 4:Cisco(Seed Switch) ) : 4
    IP Address                                     : ip_address
     Secure Communication (Y/N) [N]                : Y
     Symbolic name of the fabric [mds_switch_name] :
     Login ID                                      : htnm
     Login Password                                :
                                          Re-enter :
     Store Version            [2.0]                :
    KAVE05080-I The instance environment is now being created. (servicekey=Switch, inst=mds_switch_name)
    KAVE05081-I The instance environment has been created. (servicekey=Switch, inst=mds_switch_name)
  5. Check /opt/jp1pc/agtw/agent/mds_switch_name/jpcagt.ini to make sure that you have port 5989 and secure option enabled in it./


  7. Start the agent.
  8. # /opt/jp1pc/tools/jpcstart agtw inst=mds_switch_name
    KAVE06007-I The service will now start. (service=Agent Store for SANSwitch, inst=mds_switch_name)
    KAVE06007-I The service will now start. (service=Agent for SANSwitch, inst=mds_switch_name)

Job done. Wait for the next round of data collection and enjoy the graphs and stats from your MDS box.

Hi-Track read timed out

Trying to investigate a nasty problem. Every day during seemingly the same hours, i.e. from 4 to 8 a.m., Hitrack reports that one of the controllers in the array it monitors is not responding. During the second consecutive check the problem resolves and the array stays healthy for the rest of the day. Next day it all starts over. The disk array (AMS2100) has the latest firmware installed and no errors or packet drops have been caught in the network. But Hitrack’s logs clearly show that there is a problem and as soon as it reports “Read timed out (2)” twice in a row the aforementioned alert hits the road. Hope, I could find the answer before my head cracks.

Update. Once we had migrated our HCS (HDvM and HTnM) from Windows to Linux the issue miraculously just gone away. And that wasn’t just a coincidence. We didn’t wiped out our Windows installation straight away but just powered it off. And then we turned it back on, guess what? Yes, the original issues had started to pop up again.

Brocade and Qlogic (SANbox2) switches in one fabric

Below is a quick guide about how to merge Brocade and Qlogic (SANbox2) switches together. I understand, that a more intellegent approach would be to stay away from mixing different vendors in a SAN environment but at times we live with what we’ve been given. As always in our life, there is a trade-off one will have to pay: since in the latest Brocade firmware (6.x) the interoperability mode 1 has been depricated thus you won’t be able to plug a new Brocade switch into a fabric built on Qlogic hardware. Another, and a more serious, issue is that with 6.x I was unable to propogate zone changes from Brocade in native mode to Qlogic switch which was set to interopmode 1. The only way to do that was to swipe out zone’s configuration completely from the Qlogic switch and recreate an ISL. If this a show-stopper then refrain from upgrading your Borcade switch and stay with 5.x firmware version.

Our initial state looked like this (port 3 was plugged to Qlogic):

swd77:admin> switchshow
switchName:     swd77
switchType:     34.0
switchState:    Online   
switchMode:     Interop
switchRole:     Principal
switchDomain:   97 (unconfirmed)
switchId:       fffc61
switchWwn:      10:00:00:05:1e:02:a1:3c
zoning:         OFF
switchBeacon:   OFF

Area Port Media Speed State            Proto
  0   0   id    N2   Online           F-Port  50:05:08:b3:00:92:59:91
  1   1   id    N4   No_Light         
  2   2   id    N2   Online           F-Port  50:06:0b:00:00:38:fd:42
  3   3   id    N2   In_Sync          
  4   4   id    N2   Online           F-Port  50:05:08:b3:00:92:59:99
  5   5   id    N4   No_Light         
  6   6   id    N2   Online           F-Port  50:06:0b:00:00:38:fb:a2
  7   7   id    N4   No_Light         
  8   8   --    N4   No_Module        
  9   9   --    N4   No_Module        
 10  10   --    N4   No_Module        
 11  11   --    N4   No_Module        
 12  12   --    N4   No_Module        
 13  13   --    N4   No_Module        
 14  14   --    N4   No_Module        
 15  15   --    N4   No_Module     

swd77:admin> islshow
No ISL found
swd77:admin> cfgshow
Defined configuration:
 cfg:   default all_wwns
 zone:  all_wwns
                50:05:08:b3:00:92:59:91; 50:06:0b:00:00:38:fd:42; 
                50:05:08:b3:00:92:59:99; 50:06:0b:00:00:38:fb:a2

Effective configuration:
 no configuration in effect

swd77:admin> portcfgshow 3
Area Number:              3
Speed Level:              AUTO
Trunk Port                ON
Long Distance             OFF
VC Link Init              OFF
Locked L_Port             OFF
Locked G_Port             OFF
Disabled E_Port           OFF
ISL R_RDY Mode            OFF
RSCN Suppressed           OFF
Persistent Disable        OFF
NPIV capability           ON

The switches were connected through ports 3 and 0 on the Borcade’s and Qlogic’s sides repsectively and as you’ve already noticed port 3 was in sate “In_Sync”, which is a synonymous for “Bad”. The only way to recover from this state was a manual intervention and forcebly configuring port 3 as a G_Port.

swd77:admin> portcfggport 3,1

Only after doing that, an active ISL link was brought on-line:

swd77:admin> islshow
  1:  3->  0 10:00:00:c0:dd:02:cb:fa  98                 sp:  2.000G bw:  2.000G 

swd77:admin> fabricshow
Switch ID   Worldwide Name           Enet IP Addr    FC IP Addr      Name
 97: fffc61 10:00:00:05:1e:02:a1:3c        >"swd77"
 98: fffc62 10:00:00:c0:dd:02:cb:fa         "SANbox2-8"

So manually configuring port 3 on my brocade switch as a G_Port was the main trick that actually made everything spinning. Keep that in mind and have fun!