Multiple iSCSI interfaces with multipathing on CentOS 7

So, what do we have here? CentOS 7 server, with two NICs used for network connections. Another two NICs for separate paths to storage with multipathd to provide failover of storage paths. Here is a simple diagram:

Multiple iSCSI interfaces with multipathing

First, the configuration of network interfaces. Here we define bond interface to use for data:

[somedude@dbase023 network-scripts]$ cat ifcfg-bond0
NAME=bond0
BONDING_MASTER=yes
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="mode=balance-rr miimon=100"
NM_CONTROLLED=no

Next we define IP settings for tha bond interface, note that it is a VLAN interface:

[somedude@dbase023 network-scripts]$ cat ifcfg-bond0.2501
NAME=bond0.2501
DEVICE=bond0.2501
ONPARENT=yes
NM_CONTROLLED=no
BOOTPROTO=none
IPADDR=10.151.21.115
NETMASK=255.255.255.0
GATEWAY=10.151.21.1
VLAN="yes"
TYPE="vlan"
DEFROUTE=yes
ONBOOT=yes

…and now the physical interfaces themselves:

[somedude@dbase023 network-scripts]$ cat ifcfg-em1
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
UUID=8b40d73d-43b3-404c-8990-4f382c1f1ea3
DEVICE=em1
ONBOOT=yes
NAME=bond0-slave0
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no
[somedude@dbase023 network-scripts]$ cat ifcfg-em3
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
UUID=4079d2f8-8d05-40bc-b1bc-95ce37dc738d
DEVICE=em3
ONBOOT=yes
NAME=bond0-slave1
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no

As for iSCSI interfaces, it is assumed they are configured with IP settings and LUN is already presented to the server. The following is their configuration. Note that they are also VLAN interfaces:

[somedude@dbase023 network-scripts]$ more ifcfg-p1p1
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=no
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=no
NAME=p1p1
UUID=2925c663-a79f-4ca8-9d23-30699a667ada
DEVICE=p1p1
ONBOOT=yes
NM_CONTROLLED=no
[somedude@dbase023 network-scripts]$ more ifcfg-p1p2
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=no
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=no
NAME=p1p2
UUID=56f5e63a-7e80-4521-8d8b-7a3f8e60c874
DEVICE=p1p2
ONBOOT=yes
NM_CONTROLLED=no
[somedude@dbase023 network-scripts]$ more ifcfg-p1p1.501
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=no
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=no
NAME=p1p1.501
DEVICE=p1p1.501
ONBOOT=yes
IPADDR=10.51.1.115
NETMASK=255.255.255.0
GATEWAY=10.51.1.1
VLAN=yes
TYPE=vlan
[somedude@dbase023 network-scripts]$ more ifcfg-p1p2.502
TYPE=Ethernet
BOOTPROTO=no
DEFROUTE=no
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=no
NAME=p1p2.502
DEVICE=p1p2.502
ONBOOT=yes
IPADDR=10.51.2.115
NETMASK=255.255.255.0
GATEWAY=10.51.2.1
NM_CONTROLLED=no
VLAN=yes
TYPE=vlan
[somedude@dbase023 network-scripts]$ netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.151.21.1     0.0.0.0         UG        0 0          0 bond0.2501
10.51.1.0       0.0.0.0         255.255.255.0   U         0 0          0 p1p1.501
10.51.2.0       0.0.0.0         255.255.255.0   U         0 0          0 p1p2.502
10.151.21.0     0.0.0.0         255.255.255.0   U         0 0          0 bond0.2501

If iSCSI interfaces are on the same subnet, the following settings should be put in place as well:

[root@dbase023 somedude]# more /etc/sysctl.d/90-iscsi.conf
net.ipv4.conf.ens1f0.rp_filter=2
net.ipv4.conf.ens1f0.arp_ignore=1
net.ipv4.conf.ens1f0.arp_announce=2

net.ipv4.conf.ens1f1.rp_filter=2
net.ipv4.conf.ens1f1.arp_ignore=1
net.ipv4.conf.ens1f1.arp_announce=2

Next, iSCSI daemon needs to be configured; This sets number of sessions per interface:

/etc/iscsi/iscsid.conf file and change the following parameter:
node.session.nr_sessions = 2

Before continuing make sure all network interfaces are up. Enable multipathd:

[root@dbase023 somedude]# systemctl enable multipathd && systemctl start multipathd

Create new iSCSI initiators:

[root@dbase023 iscsi]# iscsiadm -m iface -I ISCSI-ens1f1 -o new
[root@dbase023 iscsi]# iscsiadm -m iface -I ISCSI-ens1f0 -o new

Next, give initiators some sane names:

[root@dbase023 iscsi]# iscsiadm -m iface -I ISCSI-ens1f0 -o update -n iface.initiatorname -v iqn.1994-05.com.redhat-dbase023:ens1f0
[root@dbase023 iscsi]# iscsiadm -m iface -I ISCSI-ens1f1 -o update -n iface.initiatorname -v iqn.1994-05.com.redhat-dbase023:ens1f1

Check that ifaces had been created:

[root@dbase023 iscsi]# iscsiadm -m iface -P0
default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
ISCSI-ens1f1 tcp,<empty>,<empty>,<empty>,iqn.1994-05.com.redhat-dbase023:ens1f1
ISCSI-ens1f0 tcp,<empty>,<empty>,<empty>,iqn.1994-05.com.redhat-dbase023:ens1f0

Now, login to the iSCSI target:

[root@dbase023 iscsi]# iscsiadm -m discovery -t st -p 10.50.2.80 -I ISCSI-ens1f1 --login
[root@dbase023 iscsi]# iscsiadm -m discovery -t st -p 10.50.1.80 -I ISCSI-ens1f0 --login

…and verify login results:

[root@dbase023 iscsi]# iscsiadm -m session -P1
Target: iqn.1992-04.com.emc:cx.apm00200602260.b4 (non-flash)
    Current Portal: 10.50.2.80:3260,3
    Persistent Portal: 10.50.2.80:3260,3
        **********
        Interface:
        **********
        Iface Name: default
        Iface Transport: tcp
        Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:c9fce25638f
        Iface IPaddress: 10.50.2.115
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 1
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE

        **********
        Interface:
        **********
        Iface Name: ISCSI-ens1f0
        Iface Transport: tcp
        Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f0
        Iface IPaddress: 10.50.2.115
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 9
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE

        **********
        Interface:
        **********
        Iface Name: ISCSI-ens1f1
        Iface Transport: tcp
        Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f1
        Iface IPaddress: 10.50.2.115
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 4
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE

        **********
        Interface:
        **********
        Iface Name: ISCSI-ens1f1
        Iface Transport: tcp
        Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f1
        Iface IPaddress: 10.50.2.115
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 3
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE

        **********
        Interface:
        **********
        Iface Name: ISCSI-ens1f0
        Iface Transport: tcp
        Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f0
        Iface IPaddress: 10.50.2.115
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 10
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE
Target: iqn.1992-04.com.emc:cx.apm00200602260.a4 (non-flash)
    Current Portal: 10.50.1.80:3260,2
    Persistent Portal: 10.50.1.80:3260,2
        **********
        Interface:
        **********
        Iface Name: default
        Iface Transport: tcp
        Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:c9fce25638f
        Iface IPaddress: 10.50.1.115
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 2
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE

        **********
        Interface:
        **********
        Iface Name: ISCSI-ens1f0
        Iface Transport: tcp
        Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f0
        Iface IPaddress: 10.50.1.115
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 8
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE

        **********
        Interface:
        **********
        Iface Name: ISCSI-ens1f0
        Iface Transport: tcp
        Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f0
        Iface IPaddress: 10.50.1.115
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 7
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE

        **********
        Interface:
        **********
        Iface Name: ISCSI-ens1f1
        Iface Transport: tcp
        Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f1
        Iface IPaddress: 10.50.1.115
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 6
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE

        **********
        Interface:
        **********
        Iface Name: ISCSI-ens1f1
        Iface Transport: tcp
        Iface Initiatorname: iqn.1994-05.com.redhat-dbase023:ens1f1
        Iface IPaddress: 10.50.1.115
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 5
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE

Finally, configure multipathing daemon. Note the blacklist section. WWID’s that should not be under control of multipath daemon should be specified there. device section is specific to the storage attached. In this case, it was EMC Unity.

[somedude@dbase023 network-scripts]$ cat /etc/multipath.conf
blacklist {
            devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
            # The floowing is sda and is an internal disk, so blacklisting
            # using WWID since sda is not guaranteed to be the same device
            # when hardware config changes
            wwid 3600508b1001cec06e06b2e5977bc1b1f
}

multipaths {
            multipath {
                        wwid                             36006016004d042002e201460c1b3fc4a
                        alias                             oradata
            }
}

devices {
            device {
                    vendor "DGC"
                    product ".*"
                    product_blacklist "LUNZ"
                    path_grouping_policy "multibus"
                    features "1 queue_if_no_path"
                    prio "alua"
                    failback immediate
                    rr_weight "uniform"
                    no_path_retry queue
                    retain_attached_hw_handler yes
                    detect_prio yes
                    detect_path_checker yes
            }
}

That should really conclude the setup. LUNs presented to the server should be available for mounting.