Xen3 and a Virtual Network
Introduction
By default, Xen creates a bridge to which it attaches a physical network interface. To keep the host or priviledged system (dom0) working, it creates a virtual interface for it and attaches this to the bridge.
This works quite transparently on a workstation, but has issues on a server which also serves as a router and firewall. Standard answer, and best practice according to other Xen users, is to hide the physical network interfaces for dom0 and expose them in a domU and have it handle the routing and firewalling.
A Virtual Network
For my purpose, I would like to run a Xen guest incidentally on my SOHO server. This has some implications, especially when using a firewall. Packets coming from a standard bridge pass the firewall tables differently and will never be masqueraded. Unless you hack extra rules using ebtables, which is what iptables is for routed traffic.
My requiremenst were actually not to have a physical NIC renumbered, renamed and attached to a bridge, but to have a virtual network on my server which will be treated as a DMZ.
The Brouter Setup
To accomplish this, and to have the domUs on my DMZ masqueraded to the net, I tried a brouter setup. And it works wonderfully well.
A brouter is a combination of a bridge an a router. In this configuration, the interfaces attached to it do not have an IP address, but the bridge does. Sticking with the descriptions used in the Xen manual, it is a switch and network interface in one, attached to dom0 and to which the domUs are connected.
So the domUs are bridged to each other, but the connection to dom0 is routed. Hence the name 'brouter'.
It looks like this:
The startup procedure is in essence as follows:
- At the start of Xend
- Start a bridge
- Configure it with an ip address as if it was a normal network interface
- Set the proper route to it and restart the firewall.
- At the start of the domU
- Attach the vif to the bridge.
- Start the domU
To accomplish this, I use a special script to be started when xend is started, but still use the regular (bridging) script for starting the domUs.
The Configuration
my /etc/xen/xend-config.sxp
has contains the following settings for the network (disable any others):
## Use the following if the domU's are on their own virtual net # and dom0 does all routing etc. This is actually a brouter (network-script 'network-virtual bridgeip="10.98.98.1/24" brnet="10.98.98.0/24"') (vif-script vif-bridge)
and my domU configuration looks as follows:
disk = [ 'file:/var/lib/xen/images/MyDomU/hda,hda1,w', 'file:/var/lib/xen/images/MyDomU/swap,hda2,w'] memory = 128 kernel = "/var/lib/xen/images/MyDomU/vmlinuz-xen" ramdisk = "/var/lib/xen/images/MyDomU/initrd-xen" name = "MyDomU" root = "/dev/hda1" vif = ['mac=aa:cc:00:00:00:01']
You will notice the absence of any network settings in this configuration file. That is because I use the regular SUSE configuration system in my domU to set these.
The Script
Maybe it would be logical to show the script before the configuratione, but due to its size I thought it better to put it at the end of the page.
The script is called network-virtual
and should be located in /etc/xen/scripts
#!/bin/sh #============================================================================ # Default Xen network start/stop script. # Xend calls a network script when it starts. # The script name to use is defined in /etc/xen/xend-config.sxp # in the network-script field. # # This script creates a bridge (default xenbr${vifnum}), gives it an IP address # and the appropriate route. Then it starts the SuSEfirewall2 which should have # the bridge device in the zone you want it. # # If all goes well, this should ensure that networking stays up. # However, some configurations are upset by this, especially # NFS roots. If the bridged setup does not meet your needs, # configure a different script, for example using routing instead. # # Usage: # # vnet-brouter (start|stop|status) {VAR=VAL}* # # Vars: # # bridgeip Holds the ip address the bridge should have in the # the form ip/mask (10.0.0.1/24). # brnet Holds the network of the bridge (10.0.0.1/24). # # vifnum Virtual device number to use (default 0). Numbers >=8 # require the netback driver to have nloopbacks set to a # higher value than its default of 8. # bridge The bridge to use (default xenbr${vifnum}). # # start: # Creates the bridge # Gives it the IP address and netmask # Adds the routes to the routing table. # # stop: # Removes all routes from the bridge # Removes any devices on the bridge from it. # Deletes bridge # # status: # Print addresses, interfaces, routes # #============================================================================ dir=$(dirname "$0") . "$dir/xen-script-common.sh" . "$dir/xen-network-common.sh" findCommand "$@" evalVariables "$@" vifnum=${vifnum:-0} bridgeip=${bridgeip:-10.6.7.1/24} brnet=${brnet:-10.6.7.0/24} bridge=${bridge:-xenbr$vifnum} ## # link_exists interface # # Returns 0 if the interface named exists (whether up or down), 1 otherwise. # link_exists() { test -d "/sys/class/net/$1" } # Usage: create_bridge bridge create_bridge () { local bridge="$1" # Don't create the bridge if it already exists. if [ ! -d "/sys/class/net/$bridge/bridge" ]; then brctl addbr "$bridge}" brctl stp "$bridge" off brctl setfd "$bridge" 0 fi ip link set ${bridge} up } # Usage: add_to_bridge bridge dev add_to_bridge () { local bridge="$1" local dev="$2" # Don't add $dev to $bridge if it is already on a bridge. if ! brctl show | grep -wq "$dev"; then brctl addif "$bridge" "$dev" fi } # Usage: show_status dev bridge # Print address config and routes. show_status () { local dev="$1" local bridge="$2" echo '============================================================' ip addr show "$dev" ip addr show "$bridge" echo ' ' brctl show "$bridge" echo ' ' ip route echo '============================================================' } op_start () { if [ "$bridge" = "null" ] ; then return fi create_bridge "$bridge" if link_exists "$bridge"; then ip addr add "$bridgeip" dev "$bridge" ip link set "$bridge" up fi if [ "$antispoof" = "yes" ] ; then antispoofing fi rcSuSEfirewall2 start } op_stop () { if [ "$bridge" = "null" ]; then return fi if ! link_exists "$bridge"; then return fi ip link set "$bridge" down ip addr del "$bridgeip" dev "$bridge" ##FIXME: disconnect the interfaces from the bridge 1st brctl delbr "$bridge" rcSuSEfirewall2 start } case "$command" in start) op_start ;; stop) op_stop ;; status) show_status "$netdev" "$bridge" ;; *) echo "Unknown command: $command" >&2 echo 'Valid commands are: start, stop, status' >&2 exit 1 esac