Installing XenServer and Virtual Hosts in a Routed Network

A few days ago, we had to set up a XenServer Host, running on one of Hetzners dedicated servers. There are plenty of howto’s around, but since we happened to run into a few problems with routed networking and Domain Name Resolution during installation, that weren’t documented in Hetzners Wiki, i decided to provide a blog post for self-reference and as a way to contribute a (hopefully helpful) basic guide on how to install XenServer on a dedicated server – in our Case a Hetzner Root Server.
References:

  • When i talk about the host system or simply the host, i am referring to the dedicated server itself.
  • When i talk about the guest system, the guest or simply the VM, i refer to the virtual machines running on the Host system.

IP-Addresses and Networks
By default, Hetzners servers come with a single IP-address. Make sure to write down your network configuration from your originally installed base system (or boot up your dedicated server into a rescue system). You need that information during the installation of your base system! Also, when you plan on running externally accessible guest systems on your XenServer, you need to order additional IPs (you can either order a single IP or order a whole subnet with Hetzner). Keep in mind that Hetzners network setup delivers traffic only to the mac-address of the physical interface, so we have to do some trickery to set up the network for our VMs later on.
Downloading the installation image

First of all, download the current XenServer installation image: http://xenserver.org/open-source-virtualization-download.html. We decided to install our host system, using LARA (Hetzners remote console, providing KVM-over-IP). To mount the ISO installation image in LARA, click on the “Interfaces” tab in LARAs login page, then go to the “Virtual Media” link. Here you can enter the details of how you want to mount your image:
Mounting the ISO file and installing the base system

To access the virtual media interface please click on the “Interfaces” tab and then go to the “Virtual Media” link. Here you can enter the details of the location of the ISO file: You can mount it either via the SAMBA/CIFS networking protocol from your backup space or from your own server (as long as it is reachable via SAMBA/CIFS), or mount it directly in your LARA (although this isn’t officially supported by Hetzner).
Mount your ISO via SAMBA/CIFS:

Under Interfaces > Virtual Media > Choose “Image on Windows Share”:
Share Host/IP: <username>.your-backup.de
Share Name: backup
Image File with Path: <file_name>
User Name: <username>
Password: <password>
<user name> is the username of your backup space, which you can found under “Backup” in Hetzners robot.your-server.de
<file_name> is the name of your ISO installation image
<password> is the password that you have setup in your backup space.

Mount your ISO directly in LARA:

In your LARA, click on the floppy symbol on the top right and navigate to your local ISO installation image.
You can verify under Interfaces > Virtual Media > “Virtual Media Active Image” if your ISO is correctly mounted.

After rebooting your server, you should see the installation wizard coming up. It’s pretty easy from here, just follow through the the installation process, no biggies here…
Checking your raid configuration
To monitor your LSI raid controller with megactl, install MegaCLI RPM (available from http://download.hetzner.de/tools/LSI/ and other sources) on your base system:

[root@XenServer-Host ~]# unzip 8.07.14_MegaCLI.zip
[root@XenServer-Host ~]# rpm -ivh Linux/MegaCli-8.07.14-1.noarch.rpm
...
[root@XenServer-Host ~]#

Verify the installation by checking your raid status:

[root@XenServer-Host ~]# /opt/MegaRAID/MegaCli/MegaCli LDInfo -Lall -Aall
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 1.818 TB
Sector Size         : 512
Is VD emulated      : No
Mirror Data         : 1.818 TB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives    : 2
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No
Exit Code: 0x00
[root@XenServer-Host ~]#

Updating the base system
Updates are available from http://xenserver.org/open-source-virtualization-download.html. Citrix provides an official YUM repo, but for some reason it’s empty, so that leaves us with installing updates manually. Make sure to install all updates in the right order and reboot your host system when necessary!

For example:
[root@XenServer-Host ~]# mkdir XS62ESP1014
[root@XenServer-Host ~]# cd XS62ESP1014
[root@XenServer-Host ~]# wget http://downloadns.citrix.com.edgesuite.net/akdlm/9708/XS62ESP1014.zip
[root@XenServer-Host ~]# unzip XS62ESP1014.zip
[root@XenServer-Hoste ~]# xe patch-upload file-name=XS62ESP1014.xsupdate
[root@XenServer-Host ~]# xe patch-apply uuid=4fc82e62-b938-407d-a2c6-68c8922f3ec2  host-uuid=099a87af-0c80-4ea2-a406-f8a1bdf1e2a1
[root@XenServer-Host ~]# cd ..
[root@XenServer-Host ~]# rm -Rf XS62ESP1014
[root@XenServer-Host ~]# reboot

Configuring a network interface for your guest systems
We ordered a dedicated Subnet for  our guest systems, and added the first IP as an alias device to xenbr0:

[root@XenServer-Host ~]# cat /etc/sysconfig/network-scripts/ifcfg-xenbr0:1
DEVICE=xenbr0:1
ONBOOT=yes
BOOTPROTO=none
NETMASK=<netmask>
IPADDR=<first_subnet_ip>
[root@XenServer-Host ~]# ifup xenbr0:1
[root@XenServer-Host ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr <hwaddress>
          UP BROADCAST RUNNING PROMISC  MTU:1500  Metric:1
          RX packets:38850 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26088 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4238740 (4.0 MiB)  TX bytes:3740798 (3.5 MiB)
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2083 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2083 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1065579 (1.0 MiB)  TX bytes:1065579 (1.0 MiB)
xenbr0    Link encap:Ethernet  HWaddr <hwaddress>
          inet addr:<host_ip_address>  Bcast:<broadcast>  Mask:<netmask>
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:45603 errors:0 dropped:0 overruns:0 frame:0
          TX packets:31564 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5305346 (5.0 MiB)  TX bytes:4642674 (4.4 MiB)
xenbr0:1  Link encap:Ethernet  HWaddr <hwaddress>
          inet addr:<first_subnet_ip> Bcast:<broadcast>  Mask:<netmask>
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
[root@XenServer-Host ~]#

 Activate package forwarding
With routed setups (like with Hetzner, as mentioned earlier in this post), we need to set up package forwarding!

[root@XenServer-Host ~]# cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
#net.ipv6.conf.all.forwarding=1
# Controls proxy arp
net.ipv4.conf.default.proxy_arp = 0
# Turn off redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.lo.send_redirects = 0
net.ipv4.conf.xenbr0.send_redirects = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 0
# Enable broadcast echo Protection
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
# Don.t send Redirect Messages
net.ipv4.conf.all.send_redirects = 0
# ARP replies only on single interface with corresponding IP address
net.ipv4.conf.all.arp_filter = 1
# Disable *tables rules for bridge traffic to increase performance
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-arptables = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 1
# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536
# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 4294967295
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 268435456
# Controls the percentage of system memory reserved for dirty pages
vm.dirty_ratio = 5
# Mimimum priority of kernel message to log
kernel.printk = 4 4 1 4
# Maximum io_events: 2048 tapdisks, 32k headroom
fs.aio-max-nr = 1048576
# Panic on soft lockups.
kernel.softlockup_panic = 1
[root@XenServer-Host ~]# sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.lo.send_redirects = 0
net.ipv4.conf.xenbr0.send_redirects = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.arp_filter = 1
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
kernel.sysrq = 1
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 4294967295
kernel.shmall = 268435456
vm.dirty_ratio = 5
kernel.printk = 4 4 1 4
fs.aio-max-nr = 1048576
kernel.softlockup_panic = 1
[root@XenServer-Host ~]#

Configuring the firewall
To allow our subnet to communicate outside the host system, we need to update some firewall settings. We learned during the installation of our guest system, that we failed with the netinstall routine: when it came to choosing a debian mirror, the installation wizard failed to resolve all available mirrors, which was quite strange since we had network connectivity. Basically, we could connect to and from the base installation with ssh to any external ip – proving that our routing setup worked perfectly fine. However, we could not resolve any domain names. All we got when trying to ping or resolve a domain name was a not-so-helpfull error message stating “bad domain name”.  After some troubleshooting, we found out that XenServer firewalls network communication to and from the guest systems: Per default, only communication on ports 22, 80 and 443 (ssh, http, https) are allowed. Any other communication, including DNS (Port 53) is blocked, therefore causing Debian netinstall to fail.
To fix this, we added a few iptables rules:

[root@XenServer-Host ~]# iptables -I RH-Firewall-1-INPUT 3 -s <subnet>/28 -j ACCEPT
[root@XenServer-Host ~]#
[root@XenServer-Host ~]# iptables -nvL --line-numbers
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    19091 2967K RH-Firewall-1-INPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    11971 1722K RH-Firewall-1-INPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0
Chain OUTPUT (policy ACCEPT 21439 packets, 3530K bytes)
num   pkts bytes target     prot opt in     out     source               destination
Chain RH-Firewall-1-INPUT (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1     2073 1065K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
2      163 10568 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 255
3     6079  898K ACCEPT     all  --  *      *       <subnet>/28     0.0.0.0/0
4        0     0 ACCEPT     esp  --  *      *       0.0.0.0/0            0.0.0.0/0
5        0     0 ACCEPT     ah   --  *      *       0.0.0.0/0            0.0.0.0/0
6        0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            224.0.0.251         udp dpt:5353
7        0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:631
8        1    40 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:631
9    18922 2496K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
10       0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW udp dpt:694
11    1090 54556 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
12     123  6188 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:80
13      57  2796 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:443
14    2554  156K REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited
[root@XenServer-Host ~]#

Providing ISO image files for guest system installation

Finally, we want to install a guest system on our XenServer host. Because we can’t simply mount an installation image within XenCenter, we need to create a ISO repository first:

[root@XenServer-Host ~]# mkdir -p /var/opt/xen/iso_import
[root@XenServer-Host ~]# xe sr-create name-label="MyISORepository" type=iso device-config:location=/var/opt/xen/iso_import/ device-config:legacy_mode=true content-type=is

And, for instance, provide a Debian netinstall image here:

[root@XenServer-Host ~]# cd /var/opt/xen/iso_import
[root@XenServer-Host ~]# wget http://cdimage.debian.org/debian-cd/7.7.0/amd64/iso-cd/debian-7.7.0-amd64-netinst.iso

Keep in mind that your root file system only has 4 GB available, therefore try to provide only one installation image at a time.
That’s it!
You should now have a running, properly configured XenServer with the capability of installing and running multiple virtual machines in it. Even though this is how we got it up and running on Hetzner hardware, i don’t claim that this is the only (or best) way to do it. If you have any questions, concerns or better guides on this topic, feel free to share it in the comments!
Additional Links & Sources
http://support.citrix.com/article/CTX121169
http://techblog.conglomer.net/adding-different-subnet-ips-to-xenserver/
http://www.jansipke.nl/using-xenserver-in-a-routed-ip-network/
http://wiki.hetzner.de/index.php/LSI_RAID_Controller
http://wiki.hetzner.de/index.php/XenServer_5.6_Installation_per_LARA

Kommentare

  1. In addition to the links Mike posted for saving up on public IP addresses, theres another possibility to add connectivity to multiple VMs:
    instead of NATting the VMs on the Host you could also install a router guest, then use an private IP on your XenServer Host and the public IP (with NAT for Host and other guests) on the router.
    Could be any OS of choice that supports NAT, but you might also use a specific router/security OS like vyatta or pfsense.
    i would prefer this setup to the ones in Mike's links, as it prevents the XenHost from being accessible globally on the Internet. To secure it even more you could run a vpn endpoint on your router guest to access the XenHost.
    Citrix in general is actually really slow on some updates. both hearthbleed and shellshock took weeks to be "hot"fixed in 2014, so i would rather add that extra layer of security.
    If your router VM is down you can still restart it from the Hetzner KVM-over-IP interface. (and if everything goes down you can add your public ip back to the xenserver.)

  2. Hi JJ,
    In this setup, you would need an additional subnet. If you only have one public IP address available, you would need to set up a local network, like in these examples:
    http://stackoverflow.com/questions/21500233/how-to-use-the-same-ip-for-dom0-and-one-domu-in-xen
    http://www.x83.net/setting-a-local-network-on-xenserver/
    Hope that helps :)

  3. Hello,
    very nice tutorial.
    I have only one doubt, I need obligatorily to order an additional subnet? or I can work using the only one single ip address and route traffic to guests os?
    Thanks in advance.
    Best regards