Linux multipathing
I use MPxIO in Solaris quite often and it works very well for me. This time I needed to test out I/O multipathing in RedHat. What I really needed to do: have a server with two HBA’s manage a mirror which has submirrors on separate SAN’s; so that the server has multiple paths to each submirror. That way, if an HBA goes the server has still connection to both submirrors through the remaining HBA.
Gear used in this “experiment”:
- Dell Poweredge server.
- Two Qlogic QLA2310 HBA’s.
- RHEL Server 5.3 x86.
- Two SAN’s presenting one LUN each.
Rough steps I took to get this working:
- Make sure device mapper package is installed.
- Present two LUN’s from two SAN’s.
- Probe HBA’s for presented LUN’s.
- Configure multipathing.
First and foremost, make sure qla2xxx driver is loaded. You also have to make sure you have device-mapper-multipath-0.4.7-23.el5 installed. Next, configure multipathing daemon so that it starts on boot:
[root@carbon ~]# chkconfig multipathd on
When that’s done you need to make the system aware of the presented LUN’s. One way to do so is to reboot the server. Another option is to force HBA scan:
[root@carbon ~]# echo "- - -" > /sys/class/scsi_host/host1/scan
During this you should watch /var/log/messages to see if your LUN’s are detected. When done, make multipathd aware of the LUN’s:
[root@carbon ~]# multipath -v2 -d
The above command is a “dry run”. There will be no device map changes committed. You will only be shown device mapper changes that will be made. To commit device map changes run:
[root@carbon ~]# multipath -v2
Once this is done you can see what multipathd is seeing:
[root@carbon ~]# multipath -ll
mpath2 (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][enabled]
\_ 1:0:3:1 sde 8:64 [active][ready]
\_ 2:0:3:1 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 1:0:2:1 sdd 8:48 [active][ready]
\_ 2:0:2:1 sdg 8:96 [active][ready]
mpath1 (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][enabled]
\_ 1:0:0:1 sdb 8:16 [active][ready]
\_ 2:0:4:1 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 1:0:1:1 sdc 8:32 [active][ready]
\_ 2:0:1:1 sdf 8:80 [active][ready]
If everything looks good, you can create configuration file for multipathd. You will need to edit /etc/multipath.conf and depending on your environment, add or modify some parameters. The configuration file contains enough comments and examples to figure out what different parameters mean. When in doubt, consult the man pages.
First, add a blacklist section, which will make certain device exempt from multipathing. I have my internal drives listed in blacklist section:
blacklist {
devnode "^sd[a-b].*"
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
Next, you are going to need device section. This is going to be specific to your SAN. The one below is for EVA5000. I got the parameters from HP’s device mapper package:
device {
vendor "HP|COMPAQ"
product "HSV1[01]1 \(C\)COMPAQ|HSV[2][01]0|HSV300"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
path_checker tur
path_selector "round-robin 0"
prio_callout "/sbin/mpath_prio_alua /dev/%n"
rr_weight uniform
failback immediate
hardware_handler "0"
no_path_retry 12
rr_min_io 100
}
You should also look at defaults section to make sure it is configured for your setup. Again, the parameters in mine are specific to EVA5000:
defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy failover
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/bin/true"
path_checker tur
rr_min_io 100
rr_weight uniform
failback immediate
no_path_retry 12
user_friendly_names yes
bindings_file "/var/lib/multipath/bindings"
}
Finally, you will need to specify configuration for the presented LUN’s. This applies to the multipaths section of multipath.conf file:
multipath {
wwid 3600508b4001031250000900001490000
alias san1data
}
multipath {
wwid 3600508b400011c300000f00001a90000
alias san2data
}
After you are done, restart multipathd and check output of multipath -ll command:
[root@carbon ~]# multipath -ll
san2data (3600508d311100a300000f00001a90000) dm-3 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
\_ 1:0:3:1 sde 8:64 [active][ready]
\_ 2:0:3:1 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 1:0:2:1 sdd 8:48 [active][ready]
\_ 2:0:2:1 sdg 8:96 [active][ready]
san1data (3600508c362d0a1250000900001490000) dm-2 COMPAQ,HSV111 (C)COMPAQ
[size=15G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][enabled]
\_ 1:0:0:1 sdb 8:16 [active][ready]
\_ 2:0:4:1 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 1:0:1:1 sdc 8:32 [active][ready]
\_ 2:0:1:1 sdf 8:80 [active][ready]
That should be it. You should test the setup by disabling paths to see if your LUN’s stay up. Continue Reading
