* You are viewing the archive for the ‘networking’ Category
Well, this is just cool. Everyone has been there; sitting at a Cisco switch console wondering how long the unmarked UTP plugged into port 17 is… Thankfully Cisco IOS might be able to tell you:
core# test cable-diagnostics tdr interface gigabitethernet0/17
Unfortunately it’s not available on all switches. I dug this up on here Cisco site. And here is the command reference… Continue Reading
Bonding Ethernet interfaces in Linux is pretty straight forward. There is bunch of articles out there on it already, but since this is where I keep some of my notes, I decided to write a post on it. Plus I do not have to bother with Google and I can come straight here for instructions.
This was done on Poweredge running CentOS 5.2. Here are things that need to be done to make this happen:
The following is /etc/modprobe.conf file. To get the OS to load bonding module on boot, you will need to add the alias bond0 bonding line. You can also pass some options to the bonding module. In this case I wanted the driver to check for link loss every 100ms. I also wanted the bond0 interface to perform adaptive load balancing, hence mode=6. Adaptive load balancing does not require any configuration on the switch side. If you choose a different mode, you might have to do additional configuration on the switch.
[root@bigfoot etc]# cat /etc/modprobe.conf
alias eth0 e1000
alias eth1 e1000
alias bond0 bonding
alias scsi_hostadapter qla1280
alias scsi_hostadapter1 megaraid_mbox
alias scsi_hostadapter2 ata_piix
options bond0 miimon=100 mode=6
Next, you need to set up configuration files for physical interfaces to be included in the bond0 interface. In my case bond0 consists of eth0 and eth1. Configuration files for both interfaces are identical except for DEVICE= lines.
[root@bigfoot etc]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82541GI Gigabit Ethernet Controller
DEVICE=eth0
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
ONBOOT=yes
USERCTL=no
The last step is to configure bond0 interface itself:
[root@bigfoot etc]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
IPADDR=192.168.11.200
NETMASK=255.255.255.0
NETWORK=192.168.11.0
ONBOOT=yes
USERCTL=no
That is all. You can now do either /etc/init.d/networking restart or reboot the box.
This time I actually ran into a problem, where the physical interfaces were not being “enslaved” properly:
May 4 11:17:40 bigfoot kernel: ADDRCONF(NETDEV_UP): bond0: link is not ready
May 4 11:17:40 bigfoot kernel: bonding: bond0: Adding slave eth0.
May 4 11:17:40 bigfoot kernel: bonding: bond0: enslaving eth0 as an active interface with a down link.
May 4 11:17:40 bigfoot kernel: bonding: bond0: link status definitely up for interface eth0.
May 4 11:17:40 bigfoot kernel: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
May 4 11:17:40 bigfoot kernel: bonding: bond0: Adding slave eth1.
May 4 11:17:40 bigfoot kernel: bonding: bond0: enslaving eth1 as an active interface with a down link.
May 4 11:17:45 bigfoot kernel: bonding: bond0: Removing slave eth0
May 4 11:17:45 bigfoot kernel: bonding: bond0: Warning: the permanent HWaddr of eth0 - 00:11:43:D8:AF:63 - is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts.
May 4 11:17:45 bigfoot kernel: bonding: bond0: releasing active interface eth0
May 4 11:17:47 bigfoot kernel: bonding: bond0: Adding slave eth0.
May 4 11:17:48 bigfoot kernel: bonding: bond0: Warning: failed to get speed and duplex from eth0, assumed to be 100Mb/sec and Full.
May 4 11:17:48 bigfoot kernel: bonding: bond0: enslaving eth0 as an active interface with an up link.
I have never had this problem before and quick googlage revealed that I am not alone. I came across this guy who had the same problem. He also links to the solution. Basically it seems Xen is causing the issue and to fix it you will need to edit /etc/xen/xend-config.sxp and force the network device to be used for network bridge in Xen:
(network-script 'network-bridge netdev=bond0')
Once I had that in place everything worked as advertised. Oh, and for thorough documentation check out Documentation included with kernel source. The file is called bonding.txt. Here is an online version of it. Continue Reading
Link aggregation takes a bunch of network interfaces and creates a big pipe out of them.
Aggregation also provides redundancy. If all interfaces but one go down, the server will remain connected to the network.
Before starting make sure that:
The following will create aggr1 interface with bge1 as one of its members:
bash-3.00# dladm create-aggr -d bge1 1
Next plumb the aggregate interface, configure an IP address on it and bring it up:
bash-3.00# ifconfig aggr1 plumb 192.168.1.5 netmask 255.255.255.0 up
At this point you can list aggregations:
bash-3.00# dladm show-aggr
key: 1 (0x0001) policy: L4 address: 0:3:ba:56:7f:ba (auto)
device address speed duplex link state
bge1 0:3:ba:56:7f:ba 0 Mbps unknown down standby
Now add bge0 as a second member of aggr1 aggregation interface and list aggregate interfaces:
bash-3.00# dladm add-aggr -d bge0 1
bash-3.00# dladm show-aggr
key: 1 (0x0001) policy: L4 address: 0:3:ba:56:7f:ba (auto)
device address speed duplex link state
bge1 0:3:ba:56:7f:ba 0 Mbps unknown down standby
bge0 0:3:ba:56:7f:b9 1000 Mbps full up attached
To keep the configuration persistent across reboots, create /etc/hostname.aggr1 with appropriate content and remove any hostname.* files pertaining to the interfaces that are now members of aggr1.
To have link aggregation working properly you need to have the switch to which server is connected to properly configured with LACP.
Another thing to consider is load balancing policy for outgoing traffic. You can load balance on layers 2,3 and 4. Load balancing policy can be changed using dladm command. Here is a quick example that will modify load balancing policy to combination of L3 and L4:
bash-3.00# dladm modify-aggr -P L3,L4 1
bash-3.00# dladm show-aggr -L
key: 1 (0x0001) policy: L3,L4 address: 0:3:ba:56:7f:ba (auto)
LACP mode: off LACP timer: short
device activity timeout aggregatable sync coll dist defaulted expired
bge1 passive short yes no no no no no
bge0 passive short yes no no no no no
And finally, command that will allow you to see utilisation of individual links within aggregation. Note the %ipkts column, I did not have LACP turned on on the switch at that time:
bash-3.00# dladm show-aggr -s
key: 1 ipackets rbytes opackets obytes %ipkts %opkts
Total 2723785 2287233197 1481682 710633551
bge1 618712 115674760 870443 636559150 22.7 58.7
bge0 2105073 2171558437 611239 74074401 77.3 41.3
More info on link aggregation is here. Continue Reading
Finally, I got tired of remembering which network interfaces is configured on my Netra test box. So I do not have to remember which interface to plug cable into I configured IPMP on the box. IPMP provides link redundancy among multiple network interfaces in multipathing group. IPMP is not meant to be full fledged load balancing solution, though it will spread outgoing traffic across the interfaces.
I have put my two hme interfaces into a multipathing group. The group has a failover IP address assigned to it. Initially this address will be assigned to hme0. If hme0 fails, the address will automatically move to the other interface in the failover group.
First I edited /etc/hostname.hme0:
unreal-hme0 netmask + broadcast + deprecated -failover group unrealgrp1 up addif unreal netmask + broadcast + failover up
This configures physical hme0 with IP address of 192.168.11.6, which will not fail over and puts hme0 in unrealgrp1 multipathing group. It will additionally configure a virtual IP address of 192.168.11.5 which will failover when hme0 link goes down. Deprecated means the IP address 192.168.11.5 will not be used as source address for any outgoing packets.
Then I edited /etc/hostname.hme1:
unreal-hme1 netmask + broadcast + deprecated -failover group unrealgrp1 up
Similarly, hme1 will be configured with IP address of 192.168.11.7 and as a member of unrealgrp1 multipathing group. Again, 192.168.11.7 is marked as deprecated so it will not be used for outgoing packets. Finally I made sure my hosts file is correct:
bash-3.00# cat /etc/hosts
127.0.0.1 localhost
192.168.11.5 unreal loghost
192.168.11.6 unreal-hme0
192.168.11.7 unreal-hme1
And here is the result:
bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
hme0: flags=9040843 mtu 1500 index 2
inet 192.168.11.6 netmask ffffff00 broadcast 192.168.11.255
groupname unrealgrp1
ether 8:0:20:d9:ac:c
hme0:1: flags=1000843 mtu 1500 index 2
inet 192.168.11.5 netmask ffffff00 broadcast 192.168.11.255
hme1: flags=19040803 mtu 1500 index 3
inet 192.168.11.7 netmask ffffff00 broadcast 192.168.11.255
groupname unrealgrp1
ether 8:0:20:d9:ac:d
Essentially IP address 192.168.11.5 “floats” among interfaces. If I were to unplug hme0, 192.168.11.5 would fail over to hme1. Failure is detected on link loss. There are some tunable parameters in /etc/default/mpathd.
This is all I needed. Of course, there is much more to IPMP: you can setup test system, that your system will test for reachability, detection of interfaces missing on boot, etc. Sun has much more info on it here. Continue Reading
I had a Sun box attached to a switch I had not control over and I needed to jumpstart it. Unfortunately, I was getting nowhere. After crosschecking everything in my setup the only thing I did not try was to force speed and duplex on the Sun box to see if it would start booting.
I wondered if it would be possible to do so in Open Boot prompt. So I fired up the most logical man page in the given situation – the boot man page. Sure enough it is possible to force speed and duplex from ok prompt:
{1} ok boot net:speed=100,duplex=full - install
And off it went happily booting… Continue Reading
