unixpowered.com » san

Linux multipathing

somedude — Fri, 27 Nov 2009 22:40:46 +0000

I use MPxIO in Solaris quite often and it works very well for me. This time I needed to test out I/O multipathing in RedHat. What I really needed to do: have a server with two HBA’s manage a mirror which has submirrors on separate SAN’s; so that the server has multiple paths to each submirror. That way, if an HBA goes the server has still connection to both submirrors through the remaining HBA.

Gear used in this “experiment”:

Dell Poweredge server.
Two Qlogic QLA2310 HBA’s.
RHEL Server 5.3 x86.
Two SAN’s presenting one LUN each.

Rough steps I took to get this working:

Make sure device mapper package is installed.
Present two LUN’s from two SAN’s.
Probe HBA’s for presented LUN’s.
Configure multipathing.

First and foremost, make sure qla2xxx driver is loaded. You also have to make sure you have device-mapper-multipath-0.4.7-23.el5 installed. Next, configure multipathing daemon so that it starts on boot:

[root@carbon ~]# chkconfig multipathd on

When that’s done you need to make the system aware of the presented LUN’s. One way to do so is to reboot the server. Another option is to force HBA scan:

[root@carbon ~]# echo "- - -" > /sys/class/scsi_host/host1/scan

During this you should watch /var/log/messages to see if your LUN’s are detected. When done, make multipathd aware of the LUN’s:

[root@carbon ~]# multipath -v2 -d

The above command is a “dry run”. There will be no device map changes committed. You will only be shown device mapper changes that will be made. To commit device map changes run:

[root@carbon ~]# multipath -v2

Once this is done you can see what multipathd is seeing:

[root@carbon ~]# multipath -ll mpath2 (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][enabled] \_ 1:0:3:1 sde 8:64 [active][ready] \_ 2:0:3:1 sdh 8:112 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:2:1 sdd 8:48 [active][ready] \_ 2:0:2:1 sdg 8:96 [active][ready] mpath1 (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][enabled] \_ 1:0:0:1 sdb 8:16 [active][ready] \_ 2:0:4:1 sdi 8:128 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:1:1 sdc 8:32 [active][ready] \_ 2:0:1:1 sdf 8:80 [active][ready]

If everything looks good, you can create configuration file for multipathd. You will need to edit /etc/multipath.conf and depending on your environment, add or modify some parameters. The configuration file contains enough comments and examples to figure out what different parameters mean. When in doubt, consult the man pages.

First, add a blacklist section, which will make certain device exempt from multipathing. I have my internal drives listed in blacklist section:

blacklist { devnode "^sd[a-b].*" devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z]" }

Next, you are going to need device section. This is going to be specific to your SAN. The one below is for EVA5000. I got the parameters from HP’s device mapper package:

device { vendor "HP|COMPAQ" product "HSV1[01]1 \(C\)COMPAQ|HSV[2][01]0|HSV300" path_grouping_policy group_by_prio getuid_callout "/sbin/scsi_id -g -u -s /block/%n" path_checker tur path_selector "round-robin 0" prio_callout "/sbin/mpath_prio_alua /dev/%n" rr_weight uniform failback immediate hardware_handler "0" no_path_retry 12 rr_min_io 100 }

You should also look at defaults section to make sure it is configured for your setup. Again, the parameters in mine are specific to EVA5000:

defaults { udev_dir /dev polling_interval 10 selector "round-robin 0" path_grouping_policy failover getuid_callout "/sbin/scsi_id -g -u -s /block/%n" prio_callout "/bin/true" path_checker tur rr_min_io 100 rr_weight uniform failback immediate no_path_retry 12 user_friendly_names yes bindings_file "/var/lib/multipath/bindings" }

Finally, you will need to specify configuration for the presented LUN’s. This applies to the multipaths section of multipath.conf file:

multipath { wwid 3600508b4001031250000900001490000 alias san1data } multipath { wwid 3600508b400011c300000f00001a90000 alias san2data }

After you are done, restart multipathd and check output of multipath -ll command:

[root@carbon ~]# multipath -ll san2data (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][active] \_ 1:0:3:1 sde 8:64 [active][ready] \_ 2:0:3:1 sdh 8:112 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:2:1 sdd 8:48 [active][ready] \_ 2:0:2:1 sdg 8:96 [active][ready] san1data (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=50][enabled] \_ 1:0:0:1 sdb 8:16 [active][ready] \_ 2:0:4:1 sdi 8:128 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:1:1 sdc 8:32 [active][ready] \_ 2:0:1:1 sdf 8:80 [active][ready]

That should be it. You should test the setup by disabling paths to see if your LUN’s stay up.

Growing mirrored LUN in RedHat

somedude — Fri, 06 Feb 2009 22:52:36 +0000

I was putting a RedHat server onto a SAN and I could not find any clear instructions on how to grow a single mirrored LUN on the fly. Anyway, here are some notes on the process. First the setup: Two LUN’s mirrored across two SAN’s with LVM volume on the top of it. I could have easily just presented another set of mirrored LUN’s, add them to VG and go from there. I wanted to avoid that, as that kind of setup can quickly get out of hand as the number of presented LUN’s grows. If there is a more “sensible” and flexible setup, I would most definitely want to know about it.

For sake of completeness, here are steps to recreate the initial setup I had:

Create a mirror from two LUN’s
Use the mirror as PV
Create a VG using the PV
Create LV on the top of the VG
Make ext3 filesystem on the top of LV and mount it

Here are the actual steps with some output:

[root@ultra /]# mdadm --create /dev/md10 --level=1 --raid-devices=2 /dev/mapper/mpath4 /dev/mapper/mpath5 mdadm: array /dev/md10 started. [root@ultra/]# pvcreate /dev/md10 Physical volume "/dev/md10" successfully created [root@ultra/]# vgcreate testvg /dev/md10 Volume group "testvg" successfully created [root@ultra/]# lvcreate -l+100%FREE -n testlv testvg Logical volume "testlv" created [root@ultra/]# mkfs -t ext3 /dev/testvg/testlv [root@ultra /]# mount /dev/testvg/testlv /tmp/test

Now the resizing part. There might be a few steps but the upshot is that the filesystem can stay mounted and in use. High level overview of steps to take:

Grow the two LUN’s using SAN management software
Fail and remove one of the submirrors
Force the kernel to see the size increase of the submirror
Flush and recreate the multipath device map so multipathing sees the new size
Re-add the submirror to the mirror and let it sync
Repeat 2-4 for the second submirror
Resize the PV
Resize the LV
Resize the filesystem

First, you fail and remove the submirror:

[root@ultra/]# mdadm /dev/md10 -f /dev/mapper/mpath4 -r /dev/mapper/mpath4 mdadm: set /dev/mapper/mpath4 faulty in /dev/md10 mdadm: hot removed /dev/mapper/mpath4
Now, note all paths to the LUN. Kernel sees a separate device at the end of each path to a LUN. In this case they are sdj, sdt, sdg and sdq.

[root@ultra/]# multipath -ll mpath4 mpath4 (3600508b400011c300000f000008d0000) [size=12 GB][features="1 queue_if_no_path"][hwhandler="0"] _ round-robin 0 [prio=100][active] ._ 1:0:3:1 sdj 8:144 [active][ready] ._ 2:0:3:1 sdt 65:48 [active][ready] _ round-robin 0 [prio=20][enabled] ._ 1:0:2:1 sdg 8:96 [active][ready] ._ 2:0:2:1 sdq 65:0 [active][ready]

At this point the problem is to get the kernel to recognize the new size without reboot. After a lot of trying and sifting through man pages I found that blockdev command does the magic. Then I googled “blockdev resize” and I found this confirming my finding. So, the next step is to probe all logical paths to the LUN:

[root@ultra/]# blockdev --rereadpt /dev/sdj [root@ultra/]# blockdev --rereadpt /dev/sdt [root@ultra/]# blockdev --rereadpt /dev/sdg [root@ultra /]# blockdev --rereadpt /dev/sdq

You should see messages in /var/log/messages about kernel seeing new size on all paths. If you were to issue multipath -ll right now you would see that multipathing is still reporting old size. To fix that, flush the device map of the LUN and then recreate it:

[root@ultra/]# multipath -f mpath4 [root@ultra/]# multipath -v2 create: mpath4 (3600508b400011c300000f000008d0000) [size=13 GB][features="0"][hwhandler="0"] _ round-robin 0 [prio=100] ._ 1:0:3:1 sdj 8:144 [ready] ._ 2:0:3:1 sdt 65:48 [ready] _ round-robin 0 [prio=20] ._ 1:0:2:1 sdg 8:96 [ready] ._ 2:0:2:1 sdq 65:0 [ready]

Multipathing should be reporting the new size. Now you are ready to put back the grown submirror and let the whole mirror sync:

[root@ultra/]# mdadm /dev/md10 -a /dev/mapper/mpath4 mdadm: hot added /dev/mapper/mpath4

When the mirror has synced up, repeat the above process for the second submirror and wait for the sync to finish. Time to grow the mirror device itself:

[root@ultra /]# mdadm --grow /dev/md10 --size=max

After the completion /proc/mdstat should report increase in size of /dev/md10. Moving on you need to grow the PV that resides on /dev/md10:

[root@ultra/]# pvresize /dev/md10 Physical volume "/dev/md10" changed 1 physical volume(s) resized / 0 physical volume(s) not resized

And finally, you need to resize the LV:

[root@ultra/]# lvresize -l+100%FREE testvg/testlv Extending logical volume testlv to 13.00 GB Logical volume testlv successfully resized

Of course, don’t forget to grow the filesystem itself:

[root@ultra/]# ext2online /dev/testvg/testlv ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b [root@ultra/]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-rootlv . 132304280 5104976 120478588 5% / /dev/md0 132134 32791 92521 27% /boot none 8202920 0 8202920 0% /dev/shm /dev/mapper/testvg-testlv . 13413488 63516 12668820 1% /tmp/test

That should be it. The sync time for huge volumes is going to be something to keep in mind. The whole setup is clean and neat without clutter. I could have opted to mirror using LVM, but there seems to be a strange requirement for third, log volume. It is possible to keep the log in memory, but that supposedly causes resync on boot.

Growing Solaris Volume Manager mirrors online

somedude — Sat, 08 Nov 2008 14:41:10 +0000

You have a SVM mirror and you need to grow it. Online. All steps are straight forward except for the last one. So, in a nutshell:

Grow sizes of both submirror LUN’s on SAN
Detach the first submirror metadevice
Clear the submirror metadevice
Make the system see new LUN size
Recreate the submirror metadevice, attach it to the mirror and let it sync
Repeat the above for the second submirror
Not so obvious step
Grow the filesystem

You have a mirror metadevice d200 consisting of submirrors d201 and d202. Both submirrors are sliced in such way that all space is allocated in slice 0. At this point both LUN’s should be already grown, but you need to make the system to see the size increase. Here is what d201, one of the submirrors of d200, should look like:

bash-3.00# metastat d201 d201: Submirror of d200 State: Okay Size: 2064384 blocks (1008 MB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

First, detach d201 from its parent and clear it:

bash-3.00# metadetach d200 d201 bash-3.00# metaclear d201

The next step is likely to alter the LUN slice configuration, so make sure you have it noted.

You need to run format command to get the system to see the grown LUN. So, run format then select type and then select Autoconfigure. At this point format should be reporting the new LUN size. Make sure your slice configuration is correct and that all the new space is tacked onto the end of correct slice. In this case, it’s slice 0.

Now recreate d201, attach it to d200 and let the mirror sync. When checking status of d200 note that overall mirror size is still the same but the size of d201 increased:

bash-3.00# metastat d200 d200: Mirror Submirror 0: d202 State: Okay Submirror 1: d201 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 2064384 blocks (1008 MB)

d202: Submirror of d200 State: Okay Size: 2064384 blocks (1008 MB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003460000d0s0 0 No Okay Yes

d201: Submirror of d200 State: Okay Size: 4128768 blocks (2.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

Now repeat the same process with d202. After you are done you will have both submirrors bigger, but the size of overall mirror will still be the same:

bash-3.00# metastat d200 d200: Mirror Submirror 0: d201 State: Okay Submirror 1: d202 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 2064384 blocks (1008 MB)

d201: Submirror of d200 State: Okay Size: 4128768 blocks (2.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003410000d0s0 0 No Okay Yes

d202: Submirror of d200 State: Okay Size: 4161536 blocks (2.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare /dev/dsk/c5t511714A510011C370000C00003460000d0s0 0 No Okay Yes

Now comes the “Not so obvious” step. In order to increase overall mirror size you need to run metattach on the mirror itself:

bash-3.00# metattach d200 bash-3.00# metastat d200 d200: Mirror Submirror 0: d201 State: Okay Submirror 1: d202 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 4128768 blocks (2.0 GB)

This will grow the size of the entire mirror to the size of submirror devices. Now you can grow the filesystem on it without unmounting it. Keep in mind that for the duration of growfs command there will be no writes to the volume.

bash-3.00# growfs -M /export/home/users /dev/md/rdsk/d200 /dev/md/rdsk/d200: 4128768 sectors in 126 cylinders of 128 tracks, 256 sectors 2016.0MB in 63 cyl groups (2 c/g, 32.00MB/g, 15040 i/g) super-block backups (for fsck -F ufs -o b=#) at: 32, 65824, 131616, 197408, 263200, 328992, 394784, 460576, 526368, 592160, 3487008, 3552800, 3618592, 3684384, 3750176, 3815968, 3881760, 3947552, 4013344, 4079136

That’s all there is to it. The “Not so obvious” step is mentioned in the metattach man page. But unless you do this stuff every day, this very useful piece of information can be easily forgotten.

Disabling MPxIO for onboard disks

somedude — Mon, 01 Sep 2008 18:36:21 +0000

I was configuring MPxIO on Sun Fire V490. After issuing stmsboot -e command and rebooting the server, internal disks were under MPxIO’s control.

This was not something I wanted and I needed to disable it. In order to do so I had to edit /kernel/drv/fp.conf file and tell Solaris not to enable MPxIO on internal disks. So I added the following line at the end of fp.conf file:

name="fp" parent="/pci@8,600000/SUNW,qlc@4" port=0 mpxio-disable="yes";

This line tells Solaris to disable MPxIO on port 0 for all devices whose parent device is /pci@8,600000/SUNW,qlc@4. Of course, similar line should be added for all HBA’s and ports you do not want to have under MPxIO’s control. You can get the parent device from your /var/adm/messages file or from device links pointing to the internal disks. Here is a partial list of device links:

<-----------------SNIP---------------->

lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s0 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:a lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s1 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:b lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s2 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:c lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s3 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:d lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s4 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:e lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s5 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:f lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s6 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:g lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s7 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:h

<-----------------SNIP---------------->

Gathering fibre channel info using fcinfo

somedude — Tue, 19 Aug 2008 19:47:25 +0000

I came across fcinfo command reading some man page. So, I decided to see what kind of useful fibre channel information it could give me. It turns out fcinfo knows quite a bit.

We can list all local fibre channel ports:

bash-3.00# fcinfo hba-port HBA Port WWN: 21000003ba16dbd2 OS Device Name: /dev/cfg/c1 Manufacturer: QLogic Corp. Model: 2200 Firmware Version: 2.1.144 FCode/BIOS Version: ISP2200 FC-AL Host Adapter Driver: 1.12 01/01/16 Type: L-port State: online Supported Speeds: 1Gb Current Speed: 1Gb Node WWN: 20000003ba16dbd2 HBA Port WWN: 2100001b320e3853 OS Device Name: /dev/cfg/c3 Manufacturer: QLogic Corp. Model: QLA2462 Firmware Version: 4.0.27 FCode/BIOS Version: QLA2462 Host Adapter Driver(SPARC): 1.24 11/15/06 Type: N-port State: online Supported Speeds: 1Gb 2Gb 4Gb Current Speed: 2Gb Node WWN: 2000001b320e3853 HBA Port WWN: 2101001b322e3853 OS Device Name: /dev/cfg/c4 Manufacturer: QLogic Corp. Model: QLA2462 Firmware Version: 4.0.27 FCode/BIOS Version: QLA2462 Host Adapter Driver(SPARC): 1.24 11/15/06 Type: N-port State: online Supported Speeds: 1Gb 2Gb 4Gb Current Speed: 2Gb Node WWN: 2001001b322e3853

We can take a look what remote ports are seen by particular local fibre channel port, in this case 2100001b320e3853:

bash-3.00# fcinfo remote-port -p 2100001b320e3853 Remote Port WWN: 100000e00216aef3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e01124b88f Remote Port WWN: 50001fe15003b384 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Remote Port WWN: 100000e0022744f3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e0020744f3 Remote Port WWN: 50001fe150216de9 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53 Remote Port WWN: 100000e00228f492 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e00208f492 Remote Port WWN: 50001fe15076b59b Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Remote Port WWN: 50001fe150216ded Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53

Here, we get link statistics for remote fibre channel device whose WWN is 100000e0020744f3:

bash-3.00# fcinfo remote-port -l -p 2100001b320e3853 100000e0020744f3 Remote Port WWN: 100000e0022744f3 Active FC4 Types: SCSI SCSI Target: unknown Node WWN: 100000e0020744f3 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0

We can also get link statistics and SCSI target information for all remote fibre channel devices see on local port whose WWN is 2100001b320e3853:

bash-3.00# fcinfo remote-port -sl -p 2100001b320e3853 Remote Port WWN: 100000e00216aef3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e01124b88f Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: HP Product: MSL6000 Series OS Device Name: /devices/pci@8,600000/SUNW,qlc@1/fp@0,0/sgen@w100000e00216aef3,0 LUN: 1 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/2n Remote Port WWN: 50001fe15003b384 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B4001031250000900000540000d0s2 Remote Port WWN: 100000e0022744f3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e0020744f3 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: HP Product: MSL6000 Series OS Device Name: /devices/pci@8,600000/SUNW,qlc@1/fp@0,0/sgen@w100000e0022744f3,0 LUN: 1 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/1n LUN: 2 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/0n Remote Port WWN: 50001fe150216de9 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B400011C370000C00003210000d0s2 Remote Port WWN: 100000e00228f492 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e00208f492 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/4n LUN: 1 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/3n Remote Port WWN: 50001fe15076b59b Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B4001031250000900000540000d0s2 Remote Port WWN: 50001fe150216ded Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B400011C370000C00003210000d0s2

This command is quite handy when troubleshooting fibre channel. Very cool…

Moving LUN's between hosts using metarecover

somedude — Wed, 02 Apr 2008 07:11:32 +0000

Sometimes you might need to move a LUN between hosts X and Y on a SAN. Solaris has cool command, metarecover, that lets you do just that. First, you will need to stop all processes on X that might be using the given LUN. Then unmount the LUN and present it to the host Y. Now you can use metarecover to recover metadevice information from the LUN. Couple of things to remember:

There is -n switch that will cause dry run, i.e. metarecover will output what it would do – it will not actually perform recovery operation. Also, be careful about metadevice names on host Y and the ones you are trying to recover. I have not tried it so I am not sure what would happen. It would probably bomb out. You could use metarename to rename Y’s existing metadevices and try to eliminate possible conflict before recovery.

root@ultra# metarecover -v c3t600508B400011C370000C00002970000d0s2 -p -d Verifying on-disk structures on c3t600508B400011C370000C00002970000d0s2. The following extent headers were found on c3t600508B400011C370000C00002970000d0s2. Name Seq# Type Offset Length d100 0 ALLOC 205635593 25165825 d110 0 ALLOC 16384 32769 d120 0 ALLOC 49153 32769 d130 0 ALLOC 81922 32769 d140 0 ALLOC 114691 41943041 d150 0 ALLOC 42057732 146800641 d160 0 ALLOC 188858373 2097153 d170 0 ALLOC 190955526 6291457 d180 0 ALLOC 197246983 6291457 d190 0 ALLOC 203538440 2097153 NONE 0 END 251609087 1 NONE 0 FREE 230801418 20807669 Found 10 soft partition(s) on c3t600508B400011C370000C00002970000d0s2. Checking sequence numbers. mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 100 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 25165824 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 205635594 25165824 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 110 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 32768 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 16385 32768 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 120 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 32768 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 49154 32768 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 130 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 32768 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 81923 32768 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 140 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 41943040 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 114692 41943040 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 150 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 146800640 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 42057733 146800640 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 160 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 2097152 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 188858374 2097152 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 170 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 6291456 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 190955527 6291456 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 180 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 6291456 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 197246984 6291456 mp->c.un_type: 5 mp->c.un_size: 136 mp->c.un_self_id: 190 mp->un_status: 5 mp->un_numexts: 1 mp->un_length: 2097152 mp->un_dev: 30933010 mp->un_key: 43 Ext# voff poff Len 0 0 203538441 2097152 The following soft partitions were found and will be added to your metadevice configuration. Name Size No. of Extents d100 25165824 1 d110 32768 1 d120 32768 1 d130 32768 1 d140 41943040 1 d150 146800640 1 d160 2097152 1 d170 6291456 1 d180 6291456 1 d190 2097152 1 WARNING: You are about to add one or more soft partition metadevices to your metadevice configuration. If there appears to be an error in the soft partition(s) displayed above, do NOT proceed with this recovery operation. Are you sure you want to do this (yes/no)? yes c3t600508B400011C370000C00002970000d0s2: Soft Partitions recovered from device. root@ultra#