Red Hat, Inc.

Global Professional Services BOND/VLAN/Xen Network Configuration

Presented to: The Community Prepared by: Mark Nielsen, GPS Senior Consultant Red Hat, Inc.

Copyright 2006 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks or registered trademarks of Red Hat, Inc. in the US and other countries. Linux is a registered trademark of Linus Torvalds. WHP145916US 10/06

About

This document is an overview of the current network configuration in the Lab I work in. This network configuration consists of 2 twisted-pair NICs bonded, and 2 fiber NICs bonded together with VLAN tagging enabled. This is a document originally written by Red Hat contractor Calvin Smith at the U.S. Census Bureau's Bowie Computer Center. I've modified it to match my network configuration and have added the Xen networking piece.

Scope
The current scope of the document focuses only on server-side configuration options that need to be done, with the assumption that any necessary configuration that needs to be done on the switch(es) involved has already been done. The documentation was written and tested against RHEL5 running on Dell 2950 servers. In this documentation, NICs eth0 and eth1 (copper) will be bonded together to form a new channel bonding interface using bond mode=1 (active-backup) and NICs eth2 and eth3 (fiber) will be bonded together to form a new channel bonding interface using the 802.3ad Dynamic link aggregation (4) mode. The IP addresses and possibly other details will need to be changed to reflect the configuration that is being set up in your particular environment.

Planning

1. Ensure console, or out-of-band access is available. Since the network on the servers is being reconfigured, it is important that out-of-band access to the server is available and in fact it is recommended that the following steps are performed on the console. 2. Identify the NICs that will be used in the channel bonding interface and ensure that the switches those NICs are connected to are properly configured to support channel bonding since the channel bonding interface may rapidly move a MAC address from NIC to NIC. Consult with a network technician if unsure. For the purposes of this documentation, the NICs that will be configured will be eth2 and eth3. 3. Determine what mode of bonding is desired for the channel bonding interface. In my example, we have network switches and NICs that support the 802.3ad DLA mode. 4. If Cluster Suite has already been configured, ensure that cman (the Cluster Suite manager daemon) is disabled. Performing these network configurations while Cluster Suite is running will result in a STONITH operation and your server will be fenced.

'mode=' (Specifies one of 7 policies allowed), for example: mode=0 (balance-rr) Round-robin policy: Transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance. mode=1 (active-backup) Active-backup policy: Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond's MAC address is externally visible on only one port (network adapter) to avoid confusing the switch. This mode provides fault tolerance. The primary option affects the behavior of this mode. mode=2 (balance-xor) XOR policy: Transmit based on [(source MAC address XOR'd with

Copyright 2006 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks or registered trademarks of Red Hat, Inc. in the US and other countries. Linux is a registered trademark of Linus Torvalds. WHP145916US 10/06

destination MAC address) modulo slave count]. This selects the same slave for each destination MAC address. This mode provides load balancing and fault tolerance. mode=3 (broadcast) Broadcast policy: transmits everything on all slave interfaces. This mode provides fault tolerance. Mode=4 (802.3ad) IEEE 802.3ad Dynamic link aggregation. Creates aggregation groups that share the same speed and duplex settings. Utilizes all slaves in the active aggregator according to the 802.3ad specification. Pre-requisites: 1. Ethtool support in the base drivers for retrieving the speed and duplex of each slave. 2. A switch that supports IEEE 802.3ad Dynamic link aggregation. Most switches will require some type of configuration to enable 802.3ad mode. mode=5 (balance-tlb) Adaptive transmit load balancing: channel bonding that does not require any special switch support. The outgoing traffic is distributed according to the current load (computed relative to the speed) on each slave. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed receiving slave. Prerequisite: Ethtool support in the base drivers for retrieving the speed of each slave. mode=6 (balance-alb) Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the slaves in the bond such that different peers use different hardware addresses for the server. The first four modes are the most commonly used. The network environment I work in utilizes mode=4 as well as mode=1.

Procedure
This is a quick explanation how VLAN Tagging and NIC Bonding work for the servers we have setup at my project. I've bonded eth0 and eth1 (copper), and eth2 and eth3 (fiber) in my example.

NIC Bonding
Add the following lines to /etc/modprobe.conf:
aliasbond0bonding aliasbond1bonding optionsbondingmax_bonds=2

There currently exists a bug (https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=229643) at the time of this writing. That bug prevents the second bond from ever coming up. You can either apply the patch, or simply add max_bonds=2 as I have done in my modprobe.conf. The max_bonds option will cause sysfs to actually create both bonds whether you actually configure them or not. All other bonding options have been moved out of modprobe.conf and in to your bond configuration
Copyright 2006 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks or registered trademarks of Red Hat, Inc. in the US and other countries. Linux is a registered trademark of Linus Torvalds. WHP145916US 10/06

files, as seen below. Edit /etc/sysconfig/network-scripts/ifcfg-eth0 it should look something like this:
DEVICE=eth0 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none HWADDR=<macaddr>

Edit /etc/sysconfig/network-scripts/ifcfg-eth1 it should look something like this:

DEVICE=eth1 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none HWADDR=<macaddr>

bond0bonddevice

No IP address will be assigned to the bond0 device.

Create a new file /etc/sysconfig/network-scripts/ifcfg-bond0 it should look like this:

DEVICE=bond0 BOOTPROTO=none ONBOOT=yes TYPE=Ethernet BONDING_OPTS=mode=1miimon=100

Repeat this configuration for bond1, which consists of eth2 and eth3 (the fiber). Just create the same files for ifcfg-bond1, ifcfg-eth2, and ifcfg-eth3 replacing bond0 with bond1. In my network configuration, I also have a different bonding option for the LACP. /etc/sysconfig/network-scripts/ifcfg-bond1:
DEVICE=bond1 BOOTPROTO=none ONBOOT=yes TYPE=Ethernet BONDING_OPTS=mode=4 miimon=100

bond0:0 - private non-VLAN tagged, bonded interface

This will be one virtual interface without a VLAN tag. Create a new file /etc/sysconfig/network-scripts/ifcfg-bond0:0 it should look like this:

Copyright 2006 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks or registered trademarks of Red Hat, Inc. in the US and other countries. Linux is a registered trademark of Linus Torvalds. WHP145916US 10/06

DEVICE=bond0:0 ONBOOT=yes TYPE=Ethernet BOOTPROTO=Static NETMASK=255.255.255.0 NETWORK=192.168.0.0 IPADDR=192.168.0.100

At the project I'm on, this is a private interface that is used for Cluster Suite communications and GFS fencing through DRAC (Dell Remote Access Control).

bond1.48 - public VLAN tagged and bonded interface

This will be a virtual interface with a VLAN tag of 48. Your VLAN set-up is most likely different so just replace 48 with the VLAN tag of your network. i.e. bond1.50 would be the bonded interface for VLAN 50. Create a new file /etc/sysconfig/network-scripts/ifcfg-bond1.48 it should look like this:
DEVICE=bond1.48 ONBOOT=yes TYPE=Ethernet BOOTPROTO=static VLAN=yes NETMASK=255.255.255.0 NETWORK=192.168.48.0 IPADDR=192.168.48.100

Default gateway
Ensure that the default gateway in this configuration is recorded in the /etc/sysconfig/network file otherwise it may not work properly. Once done, it should look something like:
NETWORKING=yes HOSTNAME=<server'shostname> GATEWAY=192.168.48.1

You have now setup bonding and VLAN tagging. You will need to restart networking to make your changes active.
servicenetworkrestart

Testing

Verify bonding interface is up and running ifconfig -a

Verify configuration (RHEL 5 is using sysfs, so check out /sys/class/net/)

Copyright 2006 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks or registered trademarks of Red Hat, Inc. in the US and other countries. Linux is a registered trademark of Linus Torvalds. WHP145916US 10/06

Test interface failover

Copyright 2006 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks or registered trademarks of Red Hat, Inc. in the US and other countries. Linux is a registered trademark of Linus Torvalds. WHP145916US 10/06

Xen Networking
Now we'll take this one step further and throw Xen down on top of all this. Unfortunately, the default configuration scripts for the Xen networking don't seem to understand the complexity of my network environment. Thanks to some intense research, however, I was able to find a script that nearly fit my needs. With some slight modification it worked like a charm. The script is located at http://lists.xensource.com/archives/html/xen-users/2006-04/msg00186.html and was written by Christopher G. Stach II. Thank you, sir! Here is the critical part of the script which includes the changes I've made for my network: /etc/xen/scripts/network-bridge-bond-vlan
op_start () { create_bridge xenbr0 add_to_bridge xenbr0 vif0.0 add_to_bridge2 xenbr0 bond1.48 transfer_addrs bond1.48 xenbr0 transfer_routes bond1.48 xenbr0 } op_stop () { transfer_routes xenbr0 bond1.48 ip link set xenbr0 down brctl delbr xenbr0 }

As you can see, the only significant change I made was to re-name the bond device to my particular bond, which also has a VLAN applied to it. The next change you need to make is within /etc/xen/xend-config.sxp to call this script. You should see the following section:
#Tobridgenetworktraffic,likethis: # #dom0:fakeeth0>vif0.0+ #| #bridge>realeth0>thenetwork #| #domU:fakeeth0>vifN.0+ # #use # #(networkscriptnetworkbridge) (networkscriptnetworkbridgebondvlan)

Again, all I've done is changed the default script that is called to the one I've modified for my environment. This sets up the initial bridge to my default VLAN for dom0. This is the VLAN I use to deploy my domU systems. Then, once installed, I move my Xen VMs (domU) to their destination VLANs.

Copyright 2006 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks or registered trademarks of Red Hat, Inc. in the US and other countries. Linux is a registered trademark of Linus Torvalds. WHP145916US 10/06

Here's where things get tough:

I have a RHEL 5 Cluster Suite setup using GFS. My Xen scripts (/etc/xen/) and my Xen images (/var/lib/xen/images) are all served from the clustered (GFS) file system on the SAN. This will allow me to treat my Xen VMs as a service with the ability to migrate them throughout the cluster using Conga. We also have the VMs residing on a VLAN that dom0 is not part of. This provides us separation between dom0 and domU and any of the VMs within domU to be separate from each other. Basically I can completely control which VLAN I want any given VM to be on, and the network team can constrain those VLANs any way they wish. Update: I've since moved my Xen images off the /var/lib/xen/images/ file system and now serve the root file systems to the VMs as separate logical volumes. This was done to prevent any performance issues I should have expected in my original architecture. The original plan was to have the VM roots as flat files on a GFS partition across the bare metal cluster. That would have introduced a lot of unnecessary overhead for the VMs. All I really need to do is have a LUN presented to all the domain-0 boxes. I slice that up using LVM, but never mount any of those LVMs on domain-0. Visit this page for clustered clvmd: http://sources.redhat.com/cluster/faq.html#clvmd_clustered . Yeah, that seems redundant, but trust me, read that! Now I just install to the LVM rather than a GFS flat file skipping all the overhead of GFS. This works pretty well, the only issue I've had so far was unusual. Cluster Suite on domain-0 started the same VM on 2 different servers.... yeah, had some FS corruption! Nothing a quick fsck couldn't take care of ;)

Chicken and egg!

Our Xen networking script is on GFS, but the way CMAN works creates a dilemma. If xend is chkconfigured ON in the run level you are going to, cman reads the network-bridge script. If this network-bridge script is on the GFS partition, how can you read it until CMAN comes up? Solution: you have to store a local copy on each node's /etc/xen/scripts/ local mount. This allows CMAN to come up properly, it will also create your bridge at this time. When GFS starts, the /etc/xen/ directory is re-mounted so you'll have all your xen VM configurations available. (I've heard there's a fix in the works for this.)

VMs as Services
Follow the presentation here: http://sourceware.org/cluster/conga/cookbook/VMs_as_services/ to configure your VMs as services. (An additional note is to set NTP so if a VM migrates to another node, you won't see a lot of errors).

Multiple VLANs
Now we add the secret sauce to create the additional VLANs which our VMs will bridge to. I have many different VLANs, so it is possible to have more than just the one I'm going to document. You

Copyright 2006 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks or registered trademarks of Red Hat, Inc. in the US and other countries. Linux is a registered trademark of Linus Torvalds. WHP145916US 10/06

just simply add more configuration files for any additional VLANs you want. Create /etc/sysconfig/network-scripts/bond1.20
DEVICE=bond1.20 ONBOOT=yes BOOTPROTO=none VLAN=yes

This is simply an interface which will provide VLAN tagging by dom0 for the Xen bridge I will create for the domU systems I want on VLAN 20. There are 3 commands that need to be run to set those bridges up. There are several options available to you for the automation of this. I chose to create a chkconfig-able service. I wrote up a simple script and put it in /etc/init.d/ and have it set to come on as one of the last services . I ran in to quite a few issues trying to have the network-bridge script create these additional bridges, so I did not place them there. You could also put these commands in /etc/rc.d/rc.local if you want. This is all standard sys V style (which RHEL uses), so if you have a different variant you'll have to figure out the best place for your system.
/usr/sbin/brctladdbrxenbr1 /usr/sbin/brctladdifxenbr1bond1.20 /sbin/ifconfigxenbr1up

You'll need to have an option to stop the service if you create this as a sys V style script:
/sbin/ifconfigxenbr1down /usr/sbin/brctldelbrxenbr1

If you have more than one bridge (I have 3) then you just simply adjust the script by adding the same lines for your second bridge. NOTE: When you initially create a VM, Xen networking magicians inside your system will determine which bridge to use. I saw it do some really odd things if I didn't have a bridge based on my default route (tried to call it xenbr48), so I leave my default bridge on VLAN 48 in place. (Again, that 48 number is just my local configuration, use whatever number you have for your default VLAN.) Then when I create a new VM, it uses my default xenbr0 and I can install from my Red Hat Satellite Server which is on that VLAN. Once my VM is installed, I modify my network configuration, shut down the VM and modify the default bridge in /etc/xen/myvm. It should have a line in the configuration file that looks somewhat like this:
vif=['mac=00:12:34:45:78:90,bridge=xenbr1',]

You, of course, will need to change the bridge= part to point to whichever bridge you wish to have the VM communicate on. My xenbr1 correlates to my VLAN 20, so my network configuration for eth0 inside my VM will have an IP address that corresponds to that VLAN (i.e. 192.168.20.100). Start the VM back up, and voila! It should be using VLAN 20 (as tagged by dom0). Wash, rinse, repeat for each VM/VLAN/bond configuration you need to implement. Questions? Feel free to email me at mnielsen@redhat.com.

Additional notes:

Copyright 2006 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks or registered trademarks of Red Hat, Inc. in the US and other countries. Linux is a registered trademark of Linus Torvalds. WHP145916US 10/06

If you're running iptables, make sure you open up all required ports (use iptables logging if you aren't sure which ports you need to open). If you're using SELinux in enforcing mode, you'll need to do some research. There are some bugs and other issues in making it all work. Again, check your logs!

Additional links:
some great multiple VLAN info here: http://felipe-alfaro.org/blog/2006/07/21/xen-network-configuration-and-multiple-vlans/ http://felipe-alfaro.org/blog/2006/08/21/xen-network-configuration-and-multiple-vlans-ii/ some VLAN bridge info: http://renial.net/weblog/2007/02/27/xen-vlan/ Awesome Red Hat Magazine article! MUST READ: http://www.redhatmagazine.com/2007/08/23/automated-failover-and-recovery-of-virtualizedguests-in-advanced-platform/

Copyright 2006 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks or registered trademarks of Red Hat, Inc. in the US and other countries. Linux is a registered trademark of Linus Torvalds. WHP145916US 10/06