unixpowered.com » fibre channel

Linux multipathing

somedude — Fri, 27 Nov 2009 22:40:46 +0000

I use MPxIO in Solaris quite often and it works very well for me. This time I needed to test out I/O multipathing in RedHat. What I really needed to do: have a server with two HBA’s manage a mirror which has submirrors on separate SAN’s; so that the server has multiple paths to each submirror. That way, if an HBA goes the server has still connection to both submirrors through the remaining HBA.

Gear used in this “experiment”:

Dell Poweredge server.
Two Qlogic QLA2310 HBA’s.
RHEL Server 5.3 x86.
Two SAN’s presenting one LUN each.

Rough steps I took to get this working:

Make sure device mapper package is installed.
Present two LUN’s from two SAN’s.
Probe HBA’s for presented LUN’s.
Configure multipathing.

First and foremost, make sure qla2xxx driver is loaded. You also have to make sure you have device-mapper-multipath-0.4.7-23.el5 installed. Next, configure multipathing daemon so that it starts on boot:

[root@carbon ~]# chkconfig multipathd on

When that’s done you need to make the system aware of the presented LUN’s. One way to do so is to reboot the server. Another option is to force HBA scan:

[root@carbon ~]# echo "- - -" > /sys/class/scsi_host/host1/scan

During this you should watch /var/log/messages to see if your LUN’s are detected. When done, make multipathd aware of the LUN’s:

[root@carbon ~]# multipath -v2 -d

The above command is a “dry run”. There will be no device map changes committed. You will only be shown device mapper changes that will be made. To commit device map changes run:

[root@carbon ~]# multipath -v2

Once this is done you can see what multipathd is seeing:

[root@carbon ~]# multipath -ll mpath2 (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][enabled] \_ 1:0:3:1 sde 8:64 [active][ready] \_ 2:0:3:1 sdh 8:112 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:2:1 sdd 8:48 [active][ready] \_ 2:0:2:1 sdg 8:96 [active][ready] mpath1 (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][enabled] \_ 1:0:0:1 sdb 8:16 [active][ready] \_ 2:0:4:1 sdi 8:128 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:1:1 sdc 8:32 [active][ready] \_ 2:0:1:1 sdf 8:80 [active][ready]

If everything looks good, you can create configuration file for multipathd. You will need to edit /etc/multipath.conf and depending on your environment, add or modify some parameters. The configuration file contains enough comments and examples to figure out what different parameters mean. When in doubt, consult the man pages.

First, add a blacklist section, which will make certain device exempt from multipathing. I have my internal drives listed in blacklist section:

blacklist { devnode "^sd[a-b].*" devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z]" }

Next, you are going to need device section. This is going to be specific to your SAN. The one below is for EVA5000. I got the parameters from HP’s device mapper package:

device { vendor "HP|COMPAQ" product "HSV1[01]1 \(C\)COMPAQ|HSV[2][01]0|HSV300" path_grouping_policy group_by_prio getuid_callout "/sbin/scsi_id -g -u -s /block/%n" path_checker tur path_selector "round-robin 0" prio_callout "/sbin/mpath_prio_alua /dev/%n" rr_weight uniform failback immediate hardware_handler "0" no_path_retry 12 rr_min_io 100 }

You should also look at defaults section to make sure it is configured for your setup. Again, the parameters in mine are specific to EVA5000:

defaults { udev_dir /dev polling_interval 10 selector "round-robin 0" path_grouping_policy failover getuid_callout "/sbin/scsi_id -g -u -s /block/%n" prio_callout "/bin/true" path_checker tur rr_min_io 100 rr_weight uniform failback immediate no_path_retry 12 user_friendly_names yes bindings_file "/var/lib/multipath/bindings" }

Finally, you will need to specify configuration for the presented LUN’s. This applies to the multipaths section of multipath.conf file:

multipath { wwid 3600508b4001031250000900001490000 alias san1data } multipath { wwid 3600508b400011c300000f00001a90000 alias san2data }

After you are done, restart multipathd and check output of multipath -ll command:

[root@carbon ~]# multipath -ll san2data (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][active] \_ 1:0:3:1 sde 8:64 [active][ready] \_ 2:0:3:1 sdh 8:112 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:2:1 sdd 8:48 [active][ready] \_ 2:0:2:1 sdg 8:96 [active][ready] san1data (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ [size=15G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=50][enabled] \_ 1:0:0:1 sdb 8:16 [active][ready] \_ 2:0:4:1 sdi 8:128 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 1:0:1:1 sdc 8:32 [active][ready] \_ 2:0:1:1 sdf 8:80 [active][ready]

That should be it. You should test the setup by disabling paths to see if your LUN’s stay up.

Disabling MPxIO for onboard disks

somedude — Mon, 01 Sep 2008 18:36:21 +0000

I was configuring MPxIO on Sun Fire V490. After issuing stmsboot -e command and rebooting the server, internal disks were under MPxIO’s control.

This was not something I wanted and I needed to disable it. In order to do so I had to edit /kernel/drv/fp.conf file and tell Solaris not to enable MPxIO on internal disks. So I added the following line at the end of fp.conf file:

name="fp" parent="/pci@8,600000/SUNW,qlc@4" port=0 mpxio-disable="yes";

This line tells Solaris to disable MPxIO on port 0 for all devices whose parent device is /pci@8,600000/SUNW,qlc@4. Of course, similar line should be added for all HBA’s and ports you do not want to have under MPxIO’s control. You can get the parent device from your /var/adm/messages file or from device links pointing to the internal disks. Here is a partial list of device links:

<-----------------SNIP---------------->

lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s0 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:a lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s1 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:b lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s2 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:c lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s3 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:d lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s4 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:e lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s5 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:f lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s6 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:g lrwxrwxrwx 1 root other 70 Jun 12 11:50 c1t0d0s7 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100002037c3e2ef,0:h

<-----------------SNIP---------------->

Gathering fibre channel info using fcinfo

somedude — Tue, 19 Aug 2008 19:47:25 +0000

I came across fcinfo command reading some man page. So, I decided to see what kind of useful fibre channel information it could give me. It turns out fcinfo knows quite a bit.

We can list all local fibre channel ports:

bash-3.00# fcinfo hba-port HBA Port WWN: 21000003ba16dbd2 OS Device Name: /dev/cfg/c1 Manufacturer: QLogic Corp. Model: 2200 Firmware Version: 2.1.144 FCode/BIOS Version: ISP2200 FC-AL Host Adapter Driver: 1.12 01/01/16 Type: L-port State: online Supported Speeds: 1Gb Current Speed: 1Gb Node WWN: 20000003ba16dbd2 HBA Port WWN: 2100001b320e3853 OS Device Name: /dev/cfg/c3 Manufacturer: QLogic Corp. Model: QLA2462 Firmware Version: 4.0.27 FCode/BIOS Version: QLA2462 Host Adapter Driver(SPARC): 1.24 11/15/06 Type: N-port State: online Supported Speeds: 1Gb 2Gb 4Gb Current Speed: 2Gb Node WWN: 2000001b320e3853 HBA Port WWN: 2101001b322e3853 OS Device Name: /dev/cfg/c4 Manufacturer: QLogic Corp. Model: QLA2462 Firmware Version: 4.0.27 FCode/BIOS Version: QLA2462 Host Adapter Driver(SPARC): 1.24 11/15/06 Type: N-port State: online Supported Speeds: 1Gb 2Gb 4Gb Current Speed: 2Gb Node WWN: 2001001b322e3853

We can take a look what remote ports are seen by particular local fibre channel port, in this case 2100001b320e3853:

bash-3.00# fcinfo remote-port -p 2100001b320e3853 Remote Port WWN: 100000e00216aef3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e01124b88f Remote Port WWN: 50001fe15003b384 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Remote Port WWN: 100000e0022744f3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e0020744f3 Remote Port WWN: 50001fe150216de9 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53 Remote Port WWN: 100000e00228f492 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e00208f492 Remote Port WWN: 50001fe15076b59b Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Remote Port WWN: 50001fe150216ded Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53

Here, we get link statistics for remote fibre channel device whose WWN is 100000e0020744f3:

bash-3.00# fcinfo remote-port -l -p 2100001b320e3853 100000e0020744f3 Remote Port WWN: 100000e0022744f3 Active FC4 Types: SCSI SCSI Target: unknown Node WWN: 100000e0020744f3 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0

We can also get link statistics and SCSI target information for all remote fibre channel devices see on local port whose WWN is 2100001b320e3853:

bash-3.00# fcinfo remote-port -sl -p 2100001b320e3853 Remote Port WWN: 100000e00216aef3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e01124b88f Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: HP Product: MSL6000 Series OS Device Name: /devices/pci@8,600000/SUNW,qlc@1/fp@0,0/sgen@w100000e00216aef3,0 LUN: 1 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/2n Remote Port WWN: 50001fe15003b384 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B4001031250000900000540000d0s2 Remote Port WWN: 100000e0022744f3 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e0020744f3 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: HP Product: MSL6000 Series OS Device Name: /devices/pci@8,600000/SUNW,qlc@1/fp@0,0/sgen@w100000e0022744f3,0 LUN: 1 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/1n LUN: 2 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/0n Remote Port WWN: 50001fe150216de9 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B400011C370000C00003210000d0s2 Remote Port WWN: 100000e00228f492 Active FC4 Types: SCSI SCSI Target: yes Node WWN: 100000e00208f492 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/4n LUN: 1 Vendor: HP Product: Ultrium 2-SCSI OS Device Name: /dev/rmt/3n Remote Port WWN: 50001fe15076b59b Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe15037e759 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B4001031250000900000540000d0s2 Remote Port WWN: 50001fe150216ded Active FC4 Types: SCSI SCSI Target: yes Node WWN: 50001fe11025bb53 Link Error Statistics: Link Failure Count: 0 Loss of Sync Count: 0 Loss of Signal Count: 0 Primitive Seq Protocol Error Count: 0 Invalid Tx Word Count: 0 Invalid CRC Count: 0 LUN: 0 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: Unknown LUN: 1 Vendor: COMPAQ Product: HSV111 (C)COMPAQ OS Device Name: /dev/rdsk/c5t600508B400011C370000C00003210000d0s2

This command is quite handy when troubleshooting fibre channel. Very cool…