Set up Oneiric PVHVM at Xen 4.1.2 Ubuntu Oneiric Dom0 (3.1.0-030100-generic)

This post is sample of utilizing optimized paravirtualized PVHVM drivers (also called PV-on-HVM drivers) with Xen fully virtualized HVM guests running Ubuntu 3.1 kernels at Xen 4.1.2 Dom0. Xen PVHVM drivers completely bypass the Qemu emulation and provide much faster disk- and network IO performance. First thing I had to do it was rebuild the recent Ubuntu kernel for precise Ubuntu-3.1.0-3 ( follow [1]) with CONFIG_XEN_PLATFORM_PCI=y. As result following debian packages gets created.

root@boris-System-P5Q3:/mnt/root/precise# ls -l
total 54196
-rw-r--r-- 1 root root 484 2011-11-02 17:44 config.flavour.pvhvm
-rw-r--r-- 1 root root 4496134 2011-11-02 17:53 linux-doc_3.1.0-2.3_all.deb
-rw-r--r-- 1 root root 11586896 2011-11-02 17:53 linux-headers-3.1.0-2_3.1.0-2.3_all.deb
-rw-r--r-- 1 root root 966630 2011-11-02 18:29 linux-headers-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb
-rw-r--r-- 1 root root 37869094 2011-11-02 18:29 linux-image-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb
-rw-r--r-- 1 root root 63952 2011-11-02 17:53 linux-source-3.1.0_3.1.0-2.3_all.deb
-rw-r--r-- 1 root root 382274 2011-11-02 17:55 linux-tools-3.1.0-2_3.1.0-2.3_amd64.deb
-rw-r--r-- 1 root root 112522 2011-11-02 17:53 linux-tools-common_3.1.0-2.3_all.deb
drwxr-xr-x 27 root root 4096 2011-11-02 17:55 source

I’ve updated only “git clone” to get 3.1 kernel.

git clone git://kernel.ubuntu.com/ubuntu/ubuntu-precise.git source

There is another attractive option – just add driver xen-platform-pci to initrd.img as post installation procedure. Then reboot with xen_platform_pci=1. I choosed kernel rebuild option following core development [2]
Regardless [1] is written for Oneiric , it works for Precise kernel with minimal change above.Details may be viewed in Rebuilding 3.1 Linux Kernel in Ubuntu’s style
Next install regular HVM domain via profile :-

kernel ='/usr/lib64/xen/boot/hvmloader'
builder='hvm'
device_model = '/usr/lib64/xen/bin/qemu-dm'
name = "OneiricPV-HVM"
memory = 2048
vcpus=2
pae=1
acpi=1
apic=1
vif = [ 'type=ioemu, mac=00:16:3f:03:01:14, bridge=virbr0, model=e1000' ]
disk = [ 'phy:/dev/sda7,hda,w', 'phy:/dev/loop0,hdc:cdrom,r' ]
# disk = [ 'phy:/dev/sda7,hda,w']
boot='d'
xen_platform_pci=0
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
sdl=0
vnc=1
vncpasswd=''
stdvga=0
serial='pty'
tsc_mode=0
usb=1
usbdevice='tablet'


Then three debian packages (result of standard Ubuntu 3.1 kernel rebuild):

linux-headers-3.1.0-2_3.1.0-2.3_all.deb
linux-headers-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb
linux-image-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb

have been scp’ed to Oneiric HVM DomU and new 3.1 Ubuntu kernel has been installed.

sudo dpkg -i linux-headers-3.1.0-2_3.1.0-2.3_all.deb \
linux-headers-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb \
linux-image-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb

Following xen front end’s drivers were included into /boot/initrd.img-3.1.0-2-pvhvm

root@boris-HVM-domU:~# echo "xen-blkfront" >> /etc/initramfs-tools/modules
root@boris-HVM-domU:~# echo "xen-netfront" >> /etc/initramfs-tools/modules
root@boris-HVM-domU:~# vi /etc/initramfs-tools/modules
root@boris-HVM-domU:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-3.1.0-2-pvhvm

Domain has been shutdown and restarted with updated profile,having
xen_platform_pci=1
Runtime profile

kernel ='/usr/lib64/xen/boot/hvmloader'
builder='hvm'
device_model = '/usr/lib64/xen/bin/qemu-dm'
name = "OneiricPV-HVM"
memory = 2048
vcpus=2
pae=1
acpi=1
apic=1
vif = [ 'mac=00:16:3f:03:01:14, bridge=virbr0, model=e1000' ]
# disk = [ 'phy:/dev/sda7,hda,w', 'phy:/dev/loop0,hdc:cdrom,r' ]
disk = [ 'phy:/dev/sda7,hda,w']
boot='c'
xen_platform_pci=1
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
sdl=0
vnc=1
vncpasswd=''
stdvga=0
serial='pty'
tsc_mode=0
usb=1
usbdevice='tablet'

This time HVM domU gets loaded ( vs using regular kernel) and dmesg report contains following messages which are the fair of success PVHVM domain loading :-

[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.1.0-2-pvhvm (root@boris-System-P5Q3) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #3 SMP Wed Nov 2 15:15:13 MSK 2011 (Ubuntu 3.1.0-2.3-pvhvm 3.1.0)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.1.0-2-pvhvm root=UUID=bff48eed-20df-4abf-b788-935c75ea6226 ro quiet splash vt.handoff=7
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009e000 (usable)
[ 0.000000] BIOS-e820: 000000000009e000 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 000000003f800000 (usable)
[ 0.000000] BIOS-e820: 00000000fc000000 - 0000000100000000 (reserved)
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] DMI 2.4 present.
[ 0.000000] DMI: Xen HVM domU, BIOS 4.1.2 10/20/2011
[ 0.000000] Hypervisor detected: Xen HVM
[ 0.000000] Xen version 4.1.
[ 0.000000] Xen Platform PCI: I/O protocol version 1
[ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
[ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
[ 0.000000] You might have to change the root device
[ 0.000000] from /dev/hd[a-d] to /dev/xvd[a-d]



Managing domains via xm ( Xen 4.1.2)

# apt-get install python-lxml
# xm new oneiric7.hvm
# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 7199 4 r----- 9276.7
OneiricPV-HVM 2048 2 0.0
# xm start OneiricPV-HVM
# vncviewer localhost:0

Several useful commands per http://wiki.xen.org/xenwiki/XenLinuxPVonHVMdrivers

# dmesg | egrep -i 'xen|front'
[ 0.000000] DMI: Xen HVM domU, BIOS 4.1.2 10/20/2011
[ 0.000000] Hypervisor detected: Xen HVM
[ 0.000000] Xen version 4.1.
[ 0.000000] Xen Platform PCI: I/O protocol version 1
[ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
[ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
[ 0.000000] ACPI: RSDP 00000000000ea020 00024 (v02 Xen)
[ 0.000000] ACPI: XSDT 00000000fc0134f0 0003C (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: FACP 00000000fc0132d0 000F4 (v04 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: DSDT 00000000fc003440 0FE05 (v02 Xen HVM 00000000 INTL 20100528)
[ 0.000000] ACPI: APIC 00000000fc0133d0 000D8 (v02 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: HPET 00000000fc0134b0 00038 (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] Booting paravirtualized kernel on Xen HVM
[ 0.000000] Xen HVM callback vector for event delivery is enabled
[ 0.087915] Xen: using vcpuop timer interface
[ 0.087922] installing Xen timer for CPU 0
[ 0.176052] installing Xen timer for CPU 1
[ 0.631112] xen/balloon: Initialising balloon driver.
[ 0.631125] xen-balloon: Initialising balloon driver.
[ 0.636079] Switching to clocksource xen
[ 0.647494] xen: --> pirq=16 -> irq=8 (gsi=8)
[ 0.647569] xen: --> pirq=17 -> irq=12 (gsi=12)
[ 0.647619] xen: --> pirq=18 -> irq=1 (gsi=1)
[ 0.647665] xen: --> pirq=19 -> irq=6 (gsi=6)
[ 0.647719] xen: --> pirq=20 -> irq=4 (gsi=4)
[ 0.647792] xen: --> pirq=21 -> irq=7 (gsi=7)
[ 0.731857] xen: --> pirq=22 -> irq=28 (gsi=28)
[ 0.731861] xen-platform-pci 0000:00:03.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
[ 0.833848] xen: --> pirq=23 -> irq=23 (gsi=23)
[ 1.255228] blkfront device/vbd/768 num-ring-pages 1 nr_ents 32.
[ 1.258186] blkfront: xvda: flush diskcache: enabled
[ 1.274037] Initialising Xen virtual ethernet driver.
# ethtool -i eth0
driver: vif
version:
firmware-version:
bus-info: vif-0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
# cat /proc/partitions
major minor #blocks name
202 0 20480000 xvda
202 1 19439616 xvda1
202 2 1 xvda2
202 5 1037312 xvda5

At this point domain may be loaded via pygrub as PV guest :-

<domain type='xen' id='3'>
<name>OneiricPV</name>
<description></description>
<memory>2097152</memory>
<currentMemory>2097152</currentMemory>
<vcpu>2</vcpu>
<bootloader>/usr/lib/xen-default/bin/pygrub</bootloader>
<bootloader_args></bootloader_args>
<os>
<type>linux</type>
<kernel></kernel>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<disk type='block' device='disk'>
<driver name='phy'/>
<source dev='/dev/sda7'/>
<target dev='xvda' bus='xen'/>
</disk>
<interface type='bridge'>
<mac address='00:16:3e:35:61:47'/>
<source bridge='br0'/>
<script path='/etc/xen/scripts/vif-bridge'/>
<target dev='vif3.0'/>
</interface>
<console type='pty' tty='/dev/pts/2'>
<source path='/dev/pts/2'/>
<target type='xen' port='0'/>
</console>
<input type='mouse' bus='xen'/>
<graphics type='vnc' port='5900' autoport='yes' keymap='en-us'/>
</devices>
</domain>

# virsh define OneiricPV.xml
# virsh start OneiricPV



References
1. http://blog.avirtualhome.com/2011/10/28/how-to-compile-a-new-ubuntu-11-10-oneiric-kernel/
2. http://xen.1045712.n5.nabble.com/PATCH-xen-remove-XEN-PLATFORM-PCI-config-option-td4833154.html