Multiple iSCSI interfaces with multipathing on CentOS 7
So, what do we have here? CentOS 7 server, with two NICs used for network connections. Another two NICs for separate paths to storage with multipathd to provide failover of storage paths. Here is a simple diagram:
First, the configuration of network interfaces. Here we define bond interface to use for data:
[somedude@dbase023 network-scripts]$ cat ifcfg-bond0
NAME=bond0
BONDING_MASTER=yes
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="mode=balance-rr miimon=100"
NM_CONTROLLED=no
Next we define IP settings for tha bond interface, note that it is a VLAN interface:
[somedude@dbase023 network-scripts]$ cat ifcfg-bond0.2501
NAME=bond0.2501
DEVICE=bond0.2501
ONPARENT=yes
NM_CONTROLLED=no
BOOTPROTO=none
IPADDR=10.151.21.115
NETMASK=255.255.255.0
GATEWAY=10.151.21.1
VLAN="yes"
TYPE="vlan"
DEFROUTE=yes
ONBOOT=yes
…and now the physical interfaces themselves:
[somedude@dbase023 network-scripts]$ cat ifcfg-em1
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
UUID=8b40d73d-43b3-404c-8990-4f382c1f1ea3
DEVICE=em1
ONBOOT=yes
NAME=bond0-slave0
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no
[somedude@dbase023 network-scripts]$ cat ifcfg-em3
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
UUID=4079d2f8-8d05-40bc-b1bc-95ce37dc738d
DEVICE=em3
ONBOOT=yes
NAME=bond0-slave1
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no
As for iSCSI interfaces, it is assumed they are configured with IP settings and LUN is already presented to the server. The following is their configuration. Note that they are also VLAN interfaces:
[somedude@dbase023 network-scripts]$ more ifcfg-p1p1
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=no
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=no
NAME=p1p1
UUID=2925c663-a79f-4ca8-9d23-30699a667ada
DEVICE=p1p1
ONBOOT=yes
NM_CONTROLLED=no
[somedude@dbase023 network-scripts]$ more ifcfg-p1p2
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=no
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=no
NAME=p1p2
UUID=56f5e63a-7e80-4521-8d8b-7a3f8e60c874
DEVICE=p1p2
ONBOOT=yes
NM_CONTROLLED=no
[somedude@dbase023 network-scripts]$ more ifcfg-p1p1.501
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=no
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=no
NAME=p1p1.501
DEVICE=p1p1.501
ONBOOT=yes
IPADDR=10.51.1.115
NETMASK=255.255.255.0
GATEWAY=10.51.1.1
VLAN=yes
TYPE=vlan
[somedude@dbase023 network-scripts]$ more ifcfg-p1p2.502
TYPE=Ethernet
BOOTPROTO=no
DEFROUTE=no
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=no
NAME=p1p2.502
DEVICE=p1p2.502
ONBOOT=yes
IPADDR=10.51.2.115
NETMASK=255.255.255.0
GATEWAY=10.51.2.1
NM_CONTROLLED=no
VLAN=yes
TYPE=vlan
[somedude@dbase023 network-scripts]$ netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.151.21.1 0.0.0.0 UG 0 0 0 bond0.2501
10.51.1.0 0.0.0.0 255.255.255.0 U 0 0 0 p1p1.501
10.51.2.0 0.0.0.0 255.255.255.0 U 0 0 0 p1p2.502
10.151.21.0 0.0.0.0 255.255.255.0 U 0 0 0 bond0.2501
If iSCSI interfaces are on the same subnet, the following settings should be put in place as well:
[root@dbase023 somedude]# more /etc/sysctl.d/90-iscsi.conf
net.ipv4.conf.ens1f0.rp_filter=2
net.ipv4.conf.ens1f0.arp_ignore=1
net.ipv4.conf.ens1f0.arp_announce=2
net.ipv4.conf.ens1f1.rp_filter=2
net.ipv4.conf.ens1f1.arp_ignore=1
net.ipv4.conf.ens1f1.arp_announce=2
Next, iSCSI daemon needs to be configured; This sets number of sessions per interface:
/etc/iscsi/iscsid.conf file and change the following parameter:
node.session.nr_sessions = 2
Before continuing make sure all network interfaces are up. Enable multipathd:
[root@dbase023 somedude]# systemctl enable multipathd && systemctl start multipathd
Create new iSCSI initiators:
[root@dbase023 iscsi]# iscsiadm -m iface -I ISCSI-ens1f1 -o new
[root@dbase023 iscsi]# iscsiadm -m iface -I ISCSI-ens1f0 -o new
Next, give initiators some sane names:
[root@dbase023 iscsi]# iscsiadm -m iface -I ISCSI-ens1f0 -o update -n iface.initiatorname -v iqn.1994-05.com.redhat-dbase023:ens1f0
[root@dbase023 iscsi]# iscsiadm -m iface -I ISCSI-ens1f1 -o update -n iface.initiatorname -v iqn.1994-05.com.redhat-dbase023:ens1f1
Check that ifaces had been created:
[root@dbase023 iscsi]# iscsiadm -m iface -P0
default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
ISCSI-ens1f1 tcp,<empty>,<empty>,<empty>,iqn.1994-05.com.redhat-dbase023:ens1f1
ISCSI-ens1f0 tcp,<empty>,<empty>,<empty>,iqn.1994-05.com.redhat-dbase023:ens1f0
Now, login to the iSCSI target:
[root@dbase023 iscsi]# iscsiadm -m discovery -t st -p 10.50.2.80 -I ISCSI-ens1f1 --login
[root@dbase023 iscsi]# iscsiadm -m discovery -t st -p 10.50.1.80 -I ISCSI-ens1f0 --login
…and verify login results:
[root@dbase023 iscsi]# iscsiadm -m session -P1
Target: iqn.1992-04.com.emc:cx.apm00200602260.b4 (non-flash)
Current Portal: 10.50.2.80:3260,3
Persistent Portal: 10.50.2.80:3260,3
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:c9fce25638f
Iface IPaddress: 10.50.2.115
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 1
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
**********
Interface:
**********
Iface Name: ISCSI-ens1f0
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f0
Iface IPaddress: 10.50.2.115
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 9
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
**********
Interface:
**********
Iface Name: ISCSI-ens1f1
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f1
Iface IPaddress: 10.50.2.115
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 4
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
**********
Interface:
**********
Iface Name: ISCSI-ens1f1
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f1
Iface IPaddress: 10.50.2.115
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 3
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
**********
Interface:
**********
Iface Name: ISCSI-ens1f0
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f0
Iface IPaddress: 10.50.2.115
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 10
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
Target: iqn.1992-04.com.emc:cx.apm00200602260.a4 (non-flash)
Current Portal: 10.50.1.80:3260,2
Persistent Portal: 10.50.1.80:3260,2
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:c9fce25638f
Iface IPaddress: 10.50.1.115
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 2
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
**********
Interface:
**********
Iface Name: ISCSI-ens1f0
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f0
Iface IPaddress: 10.50.1.115
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 8
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
**********
Interface:
**********
Iface Name: ISCSI-ens1f0
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f0
Iface IPaddress: 10.50.1.115
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 7
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
**********
Interface:
**********
Iface Name: ISCSI-ens1f1
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f1
Iface IPaddress: 10.50.1.115
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 6
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
**********
Interface:
**********
Iface Name: ISCSI-ens1f1
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f1
Iface IPaddress: 10.50.1.115
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 5
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
Finally, configure multipathing daemon. Note the blacklist section. WWID’s that should not be under control of multipath daemon should be specified there. device section is specific to the storage attached. In this case, it was EMC Unity.
[somedude@dbase023 network-scripts]$ cat /etc/multipath.conf
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
# The folowing is sda and is an internal disk, so blacklisting
# using WWID since sda is not guaranteed to be the same device
# when hardware config changes
wwid 3600508b1001cec06e06b2e5977bc1b1f
}
multipaths {
multipath {
wwid 36006016004d042002e201460c1b3fc4a
alias oradata
}
}
devices {
device {
vendor "DGC"
product ".*"
product_blacklist "LUNZ"
path_grouping_policy "multibus"
features "1 queue_if_no_path"
prio "alua"
failback immediate
rr_weight "uniform"
no_path_retry queue
retain_attached_hw_handler yes
detect_prio yes
detect_path_checker yes
}
}
That should really conclude the setup. LUNs presented to the server should be available for mounting.