Virt-install&Virt-manager at Xen 4.0 ( pvops) Dom0 on top Ubuntu Karmic Koala Server

March 26, 2010

The final target of this post is to demonstrate how flexible is Ubuntu Karmic Koala Server environment. Build Xen 4.0 and the most recent stable pvops kernel ( with DRM support). Loading Xen Instance followed by install Ubuntu’s Libvirt environment, providing utility virt-install for creating PV and HVM DomUs, virt-manager completely functional to manage any DomU been built via virt-install. Change-set 21112 from xen-4.0-testing.hg back ported to Xen 4.0 to support grub2 notation ‘(hd0,1)’ specific for Ubuntu 10.04 aka Lucid Lynx.

First install required packages:

apt-get install libcurl4-openssl-dev \
xserver-xorg-dev \
python2.6-dev \
mercurial gitk \
build-essential \
libncurses5-dev \
uuid-dev gawk \
gettext texinfo bcc \
bridge-utils iasl \
dpkg-dev debhelper fakeroot

Second – build Xen from source

# cd /usr/src

At this point creare grub2-fix.patch1 as raw content of Changeset 21112 pygrub: Fix Grub2 support for Ubuntu 10.04

gunzip xen-4.0.0.tar.gz
tar -xvf xen-4.0.0.tar
cd xen-4.0.0
patch -p1 -s -i ../grub2-fix.patch1
make xen
make tools
make install-xen
make install-tools PYTHON_PREFIX_ARG=

Set up xend,xendomains to run as services

update-rc.d xend defaults 20 21
update-rc.d xendomains defaults 21 20

Now checkout and build pvops kernel

cd /usr/src
git clone
git:// linux-2.6-xen
cd linux-2.6-xen
git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x

Build kernel in Debian way

make menuconfig # Enabled the dom0 support and the Xen backends
make -j(x)number_of_kernels
chmod g-s /usr/src -R
make deb-pkg
dpkg -i ../linux-image**.deb
update-initramfs -c -k

One more notice regarding /etc/modules editing :-
In case configuring kernel (1)
<M> Xen /dev/xen/evtchn device
echo “xen-evtchn” >> /etc/modules
In case configuring kernel (2)
<*> Xen /dev/xen/evtchn device
/etc/modules shouldn’t be touched.

With config bellow there is no need to touch /etc/modules at all

At this point kernel is installed

Tuning results Xen packages to be placed into /usr/local/lib/python2.6/dist-packages
due to Changeset 19594 in xen-3.4-testing.hg. Otherwise, Xen packages would go to /usr/lib/python2.6/site-packages, which is not default location for python 2.6 on Ubuntu 9.10 ( vs F12 ). Thus you won’t be able to start xend in Dom0.
Add entry to /boot/grub/grab.cfg:-
menuentry “Xen 4.0 / Ubuntu 9.10 kernel pvops” {
insmod ext2
set root=(hd1,8)
multiboot (hd1,8)/boot/xen.gz
module (hd1,8)/boot/vmlinuz- dummy=dummy root=/dev/sdb8 ro console=tty0
module (hd1,8)/boot/initrd.img-
Reboot system and install libvirt environment :-
Two steps in procedure bellow make Xen default Hypervisor on Ubuntu 9.10.
First is commenting out (xend-unix-server yes) in /etc/xen/xend-config.sxp ,
second export variable export VIRSH_DEFAULT_CONNECT_URI=”xen:///”
in root’s .bashrc.
Then run:-
# apt-get install ubuntu-virt-server ubuntu-virt-mgmt
Now you should be able to install Ubuntu 10.04 Lucid Server HVM via virt-install :-

# virt-install -n LucidSRV -r 1024 --hvm --vnc -f /dev/sda8 -c /home/boris/Lucid.iso --debug

Create following XML LucidPVG-def.xml

<domain type='xen' id='1'>
<clock offset='utc'/>
<disk type='block' device='disk'>
<driver name='phy'/>
<source dev='/dev/sda8'/>
<target dev='xvda' bus='xen'/>
<interface type='bridge'>
<mac address='00:16:3e:3f:79:01'/>
<source bridge='eth0'/>
<script path='/etc/xen/scripts/vif-bridge'/>
<target dev='vif1.0'/>
<console type='pty' tty='/dev/pts/0'>
<source path='/dev/pts/0'/>
<target port='0'/>
<input type='mouse' bus='xen'/>
<graphics type='vnc' port='5900' autoport='yes'/>

Then define and start via virt-manager Lucid RC PV DomU
# virsh define LucidPVG-def.xml

Set up Xen 3.4.3 Dom0 via xenified kernel on top of Ubuntu 9.10 Server

March 8, 2010

Following bellow is build Xen Host on top Ubuntu 9.10 Server.It is based on direct clone via “hg” and build Xen 3.4.3-rc4 from , applying Andrew Lyon’s 2.6.31 patch set V12 to vanilla kernel along with tuning patched kernel via “make menuconfig” and building aka Suse xenified kernel in Ubuntu 9.10 environment.Notice that forward porting of original patch set was done by Jan Beulich for kernel on OpenSuse 11.2. Built in patched kernel open-source NVIDIA and Radeon HD drivers work just fine on Ubuntu 9.10, providing Gnome Desktop Environment right away.

# cd /usr/src/build/
# wget

4.Untar linux- and apply patches.

# cd /usr/src/build/
# bzcat linux- | tar xf -
# mkdir -p xen-patches-2.6.31-12
# cd xen-patches-2.6.31-12
# bzcat ../xen-patches-2.6.31-12.tar.bz2 | tar xf -
# cd ../linux-

Run script

for P in `ls ../xen-patches-2.6.31-12/6*.patch1 | sort `
patch -p1 -s -i $P
if [ $? = 0 ]; then
echo $P applied
echo "Error processing "$P
exit 1

Build kernel :-

# make menuconfig

Processor type and features ->
[*] Symmetric multi-processing support
[ ] Support sparse irq numbering
[*] Enable MPS table
[*] Enable Xen compatible kernel
[*] Single-depth WCHAN output
Processor family (Generic-x86-64) --->
[ ] Configure Maximum number of SMP Processors and NUMA Nodes
(64) Maximum number of CPUs
Preemption Model (No Forced Preemption (Server)) --->
[ ] Machine Check Exception

Device Drivers -> XEN
[*] Privileged Guest (domain 0)
<*> Backend driver support
<*> Block-device backend driver
<*> Block-device tap backend driver
<*> Block-device tap backend driver 2
>*> Network-device backend driver
(8) Maximum simultaneous transmit requests (as a power of 2)
[ ] Pipelined transmitter (DANGEROUS)
< > Network-device loopback driver
<*> PCI-device backend driver
PCI Backend Mode (Virtual PCI) --->
[ ] PCI Backend Debugging
< > TPM-device backend driver
<M> SCSI backend driver
<M> USB backend driver
<M> Block-device frontend driver
<M> Network-device frontend driver
<M> Network-device frontend driver acceleration for Solarflare NICs
<M> SCSI frontend driver
<M> USB frontend driver
[*] Taking the HCD statistics (for debug)
<*> User-space granted page access driver
<*> Framebuffer-device frontend driver

# make -j(x)(number_of_cores)
# make modules_install install
# mkinitramfs -o /boot/initrd-

Different option is to install pvops enabled kernel via download :-

# git clone git:// linux-2.6-xen
# cd linux-2.6-xen
# git checkout origin/xen/stable -b xen/stable

Notice :-

# git checkout origin/xen/master -b xen/master

will result building the most recent 2.31.6 version of pvops kernel. Branch above , actually is not required to be checked out . It’s default branch of “git clone”. Difference in meantime is “acpi” branch not merged yet to xen/stable. However, APIC rework has been committed for

# make menuconfig
# make
# make modules_install install
# mkinitramfs -o /boot/initrd-

With the most recent i was able to enable DRM for Radeon and get positive dmesg report on F12 and Ubuntu Karmic Koala Server.

To test xenified aka Suse kernel setup Xen 3.4.3 on to top of Ubuntu Karmic Server. First – install on Ubuntu Karmic Server all packages required for Xen build along with Ubuntu-Desktop ( via tasksel). Of course, Xen Host doesn’t require GUI. Install Ubuntu-Desktop just make work comfortable and faster.

apt-get install libcurl4-openssl-dev \
xserver-xorg-dev \
python2.6-dev \
mercurial gitk \
build-essential \
libncurses5-dev \
uuid-dev gawk \
gettext texinfo bcc

Second – build Xen from source

# cd /usr/src
# hg clone
# cd xen-3.4-testing.hg
Set in
PYTHON = python
# make install-xen
# make install-tools
# make install-stubdom

Tuning results Xen packages to be placed into /usr/local/lib/python2.6/dist-packages due to Changeset 19594 in xen-3.4-testing.hg. Otherwise, Xen packages would go to /usr/lib/python2.6/site-packages, which is not default location for python 2.6 on Ubuntu 9.10 ( vs F12 ). Thus you won’t be able to start xend in Dom0. Same thing happens when building Xen Unstable on Ubuntu 9.10 Server.
Add entry to /boot/grub/grab.cfg:-

menuentry "Xen 3.4 / Ubuntu 9.10 kernel xenified" {
insmod ext2
set root=(hd1,8)
multiboot (hd1,8)/boot/xen-3.4.gz
module (hd1,8)/boot/vmlinuz-
module (hd1,8)/boot/initrd-

and rebooted into xen host.Tested OSOL 133 PV DomU,F12 PV DomU, Ubuntu Kamic HVM and PV DomUs sharing same image device.

1.Gentoo-xen-kernel dom0 Xen patches

The Book of Xen (Review)

March 5, 2010

This is the best book i’ve ever read in my life about architecture of Xen Hypervisor, concepts of paravirtualization, dom0,domU, performance advantages of PV guests.
I would say that core theoretical background of Xen Hypervisor is written pretty clear. It brings comprehension of Xen to persons, which even wouldn’t be willing to work on this topic too much. The last is very important from my standpoint.
I also believe that Xen 4.0 + PVOps kernel based Jeremy’s git branch xen/stable would bring Xen far ahead KVM in about one year. “Kernel to support Dom0” problem is pretty much solved. Xen Hypervisor itself doesn’t require any additional comments.
Regarding Sun xVM 3.4.2 i would add that “virt-install” was always considered by Sun as core installation technology, “virsh & virt-manager ” as a tool for domain management. Sun was
focused on Libvirt interface to Xen on OpenSolaris ( former Nevada) from the very beginning ( SNV_75)

Completely functional virsh (Libvirt 0.1.7-15) & GRUB2 Support at Xen 3.4.3 (pvops on top of F12

March 3, 2010

Recent pulling GRUB2 supporting CSs into xen-3.4-testing.hg immediately makes sense to build xen 3.4.3-2.fc12.src.rpm to be able to set up current version of 3.4.3 Xen Hypervisor on Fedora 12 working pretty smoothly with Libvirt packages ( vs Xen 4.0) and the at same time supporting GRUB2 PV DomUs , say Ubuntu 9.10 Server or Debian Squeeze. Download xen-3.4.2-2.fc13.src.rpm

# rpm -iv xen-3.4.2-2.fc13.src.rpm
# hg clone
# cp -R xen-3.4-testing.hg xen-3.4.3
# cd xen-3.4.3
# git clone git:// qemu-xen.git
# cp qemu-xen.git.tar.gz /root/rpmbuild/SOURCES/qemu-xen.git.xen-3.4.3-rc3.tar.gz
# cd ..
# cp xen-3.4.3.tar.gz /root/rpmbuild/SOURCES/xen-3.4.3.tar.gz

Place also localqemu-xen.patch into SOURCES :

[root@fedora12a SOURCES]# cat localqemu-xen.patch
--- 2010-03-03 13:51:45.103040705 +0300
+++ 2010-03-03 13:53:55.051793198 +0300
@@ -122,8 +122,8 @@

# Specify which qemu-dm to use. This may be `ioemu' to use the old
# Mercurial in-tree version, or a local directory, or a git URL.
-# CONFIG_QEMU ?= ../qemu-xen.git
+CONFIG_QEMU ?= ../qemu-xen.git

QEMU_TAG := xen-3.4.3-rc3

Now cd ../SPECS and modify xen.spec as follows :

Version: 3.4.3
. . .
Source36: qemu-xen.git.xen-3.4.3-rc3.tar.gz
. . .
Patch16: localqemu-xen.patch
. . . .
%patch16 -p1

# rpmbuild -ba ./xen.spec
# cd ../RPMS/x86_64
Expected result

[root@fedora12a x86_64]# ls -l *.rpm
-rw-r--r-- 1 root root 918248 2010-03-03 17:34 xen-3.4.3-2.fc12.x86_64.rpm
-rw-r--r-- 1 root root 1079572 2010-03-03 17:34 xen-debuginfo-3.4.3-2.fc12.x86_64.rpm
-rw-r--r-- 1 root root 237716 2010-03-03 17:34 xen-devel-3.4.3-2.fc12.x86_64.rpm
-rw-r--r-- 1 root root 854192 2010-03-03 17:34 xen-doc-3.4.3-2.fc12.x86_64.rpm
-rw-r--r-- 1 root root 3023336 2010-03-03 17:34 xen-hypervisor-3.4.3-2.fc12.x86_64.rpm
-rw-r--r-- 1 root root 168848 2010-03-03 17:34 xen-libs-3.4.3-2.fc12.x86_64.rpm
-rw-r--r-- 1 root root 4244188 2010-03-03 17:34 xen-runtime-3.4.3-2.fc12.x86_64.rpm

Now install current Xen 3.4.3 Hypervisor on Fedora12 and proceed as usual.

yum install xen-3.4.3-2.fc12.x86_64.rpm \
xen-debuginfo-3.4.3-2.fc12.x86_64.rpm \
xen-devel-3.4.3-2.fc12.x86_64.rpm \
xen-doc-3.4.3-2.fc12.x86_64.rpm \
xen-hypervisor-3.4.3-2.fc12.x86_64.rpm \
xen-libs-3.4.3-2.fc12.x86_64.rpm \

Install Libvirt

# yum install libvirt libvirt-python python-virtinst \
virt-manager virt-viewer

PVOPS kernel has been installed via Jeremy Fitzhardinge
Git Repository to support Xen 3.4.3 Dom0.

# git clone git:// linux-2.6-xen
# cd linux-2.6-xen
# git checkout origin/xen/stable -b xen/stable
# make menuconfig ( setup Xen Dom0 support)
# make -j(x)(number_of_kernels)
# make modules_install install

Reboot in Xen environment

host : fedora12a
release :
version : #1 SMP Sun Feb 28 11:42:51 MSK 2010
machine : x86_64
nr_cpus : 2
nr_nodes : 1
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 2999
hw_caps : bfebfbff:20100800:00000000:00000140:0008e3fd:00000000:00000001:00000000
virt_caps : hvm
total_memory : 8191
free_memory : 4105
node_to_cpu : node0:0-1
node_to_memory : node0:4105
xen_major : 3
xen_minor : 4
xen_extra : .3-rc4-pre
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : Tue Mar 02 19:22:22 2010 +0000 19913:6063c16aeeaa
cc_compiler : gcc version 4.4.3 20100127 (Red Hat 4.4.3-4) (GCC)
cc_compile_by : root
cc_compile_domain :
cc_compile_date : Wed Mar 3 17:31:45 MSK 2010
xend_config_format : 4

Now we have built environment allows install OSOL133 via virsh ( been built against Libvirt 0.1.7) and the time have Ubuntu Karmic Koala HVM and PV DomUs installed at Xen 3.4.3 Dom0 and sharing the same image device. View :-
Set up OSOL PV Guests (build<=131) via virsh on Xen 3.4.3 Dom0 on top of Ubuntu 9.10 Server
Set up Ubuntu Karmic Server (Grub2) PV Guest at Xen 4.0-rc4 (2.6.32 pvops) on top of Fedora12