Rebuilding 3.1 & 3.0 Linux Kernel in Ubuntu style to support PV-on-HVM

November 6, 2011

*********************************************************************************************************************
Stock kernel for Oneiric (3.0.X.Y) has same Xen configuration as 3.1. I have filed bug against 3.1 at Launchpad regarding kernel configuration fix,however, in meantime i had to pass through procedure bellow with intend to benefit from PV-on-HVM feature providing by Xen 4.1.X Hypervisor and properly built 3.0 or 3.1 linux kernel. I follow [1] step by step updating config.flavour.pvhvm as required to perform kernel rebuild. In meantime status of bug at Launchpad is “Triaged”.There is another attractive option – just add driver xen-platform-pci.ko to initrd.img as post installation procedure. Then reboot with xen_platform_pci=1. I choosed kernel rebuild as option following core development [2]
**********************************************************************************************************************

A) To rebuild 3.1 :-
$ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-precise.git source
B) To rebuild 3.0 :-
$ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-oneiric.git source
$ cd source
$ git tag|grep Ubu|sort -V
Ubuntu-3.1.0-1.0
Ubuntu-3.1.0-1.1
Ubuntu-3.1.0-2.2
Ubuntu-3.1.0-2.3
Checkout branch and give it name "pvhvm"
$ git checkout Ubuntu-3.1.0-2.3 -b pvhvm
$ cp debian.master/config/amd64/config.flavour.generic debian.master/config/amd64/config.flavour.pvhvm
$ fakeroot debian/rules clean
$ debian/rules updateconfigs
$ debian/rules editconfigs
*************************************************************
$ Do you want to edit config: amd64/config.flavour.pvhvm? [Y/n]
*************************************************************
#
# Config options for config.flavour.pv automatically generated by splitconfig.pl
#
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_FB_SYS_COPYAREA=y
CONFIG_FB_SYS_FILLRECT=y
CONFIG_FB_SYS_FOPS=y
CONFIG_FB_SYS_IMAGEBLIT=y
CONFIG_INTEL_IDLE=y
CONFIG_NR_CPUS=256
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_VIRTIO=m
CONFIG_VIRTIO_BLK=m
CONFIG_VIRTIO_NET=m
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_RING=m
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_XEN_FBDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_PLATFORM_PCI=y
CONFIG_XEN_XENBUS_FRONTEND=y






Make sure you have updated config.flavour.pvhvm before proceed.
****************
Backup config
****************

$ cp debian.master/config/amd64/config.flavour.pvhvm ../.
$ git reset --hard
$ git clean -df
To pick up number of previous version.
$ ls debian.master/abi
3.1.0-2.2
$ cp debian.master/abi/3.1.0-2.2/amd64/generic debian.master/abi/3.1.0-2.2/amd64/pvhvm
$ cp debian.master/abi/3.1.0-2.2/amd64/generic.modules debian.master/abi/3.1.0-2.2/amd64/pvhvm.modules

***************
Restore config
***************

cp ../config.flavour.pvhvm debian.master/config/amd64/

File: debian.master/etc/getabis

Search for the line:
getall amd64 generic server virtual
Change it in:
getall amd64 generic server virtual pvhvm

File: debian.master/rules.d/amd64.mk

Search for the line:
flavours = generic server virtual
Change it in:
flavours = generic server virtual pvhvm

File: debian.master/control.d/vars.pv
This files does not exist and in order to make the compilation process aware of our own flavor we want to compile we need to create it.

$ cp debian.master/control.d/vars.generic debian.master/control.d/vars.pvhvm
$ git add .
$ git commit -a -m "pv Modifications"
Checkout branch "work" for build
$ git checkout -b work
$ fakeroot debian/rules clean
$ skipabi=true skipmodule=true fakeroot debian/rules binary-indep
$ skipabi=true skipmodule=true fakeroot debian/rules binary-perarch
$ skipabi=true skipmodule=true fakeroot debian/rules binary-pvhvm

If all went OK, then
~/KERNEL02/source$ ls -l ../
total 54208
-rw-rw-r– 1 boris boris 606 2011-11-06 16:39 config.flavour.pvhvm
-rw-r–r– 1 boris boris 4496196 2011-11-06 16:52 linux-doc_3.1.0-2.3_all.deb
-rw-r–r– 1 boris boris 11586658 2011-11-06 16:52 linux-headers-3.1.0-2_3.1.0-2.3_all.deb
-rw-r–r– 1 boris boris 965564 2011-11-06 17:19 linux-headers-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb
-rw-r–r– 1 boris boris 37880092 2011-11-06 17:19 linux-image-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb
-rw-r–r– 1 boris boris 63952 2011-11-06 16:52 linux-source-3.1.0_3.1.0-2.3_all.deb
-rw-r–r– 1 boris boris 382278 2011-11-06 16:53 linux-tools-3.1.0-2_3.1.0-2.3_amd64.deb
-rw-r–r– 1 boris boris 112558 2011-11-06 16:52 linux-tools-common_3.1.0-2.3_all.deb
drwxr-xr-x 27 boris boris 4096 2011-11-06 16:53 source

Changes done to kernel been rebuilt .

--- xenconf.3.1.0-030100-generic 2011-11-06 11:03:39.225828270 +0400
+++ xenconf.3.1.0-2-vnc 2011-11-06 11:04:58.621827223 +0400
@@ -16,19 +16,20 @@
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=m
CONFIG_HVC_XEN=y
CONFIG_XEN_WDT=m
-CONFIG_XEN_FBDEV_FRONTEND=m
+CONFIG_XEN_FBDEV_FRONTEND=y
CONFIG_XEN_BALLOON=y
-# CONFIG_XEN_SELFBALLOONING is not set
+CONFIG_XEN_SELFBALLOONING=y
+CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=m
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
-CONFIG_XEN_XENBUS_FRONTEND=m
+CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GRANT_DEV_ALLOC=m
-CONFIG_XEN_PLATFORM_PCI=m
+CONFIG_XEN_PLATFORM_PCI=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=y
CONFIG_XEN_PCIDEV_BACKEND=m

References
1. http://blog.avirtualhome.com/2011/10/28/how-to-compile-a-new-ubuntu-11-10-oneiric-kernel/
2. http://xen.1045712.n5.nabble.com/PATCH-xen-remove-XEN-PLATFORM-PCI-config-option-td4833154.html


Set up Oneiric PVHVM at Xen 4.1.2 Ubuntu Oneiric Dom0 (3.1.0-030100-generic)

November 2, 2011

This post is sample of utilizing optimized paravirtualized PVHVM drivers (also called PV-on-HVM drivers) with Xen fully virtualized HVM guests running Ubuntu 3.1 kernels at Xen 4.1.2 Dom0. Xen PVHVM drivers completely bypass the Qemu emulation and provide much faster disk- and network IO performance. First thing I had to do it was rebuild the recent Ubuntu kernel for precise Ubuntu-3.1.0-3 ( follow [1]) with CONFIG_XEN_PLATFORM_PCI=y. As result following debian packages gets created.

root@boris-System-P5Q3:/mnt/root/precise# ls -l
total 54196
-rw-r--r-- 1 root root 484 2011-11-02 17:44 config.flavour.pvhvm
-rw-r--r-- 1 root root 4496134 2011-11-02 17:53 linux-doc_3.1.0-2.3_all.deb
-rw-r--r-- 1 root root 11586896 2011-11-02 17:53 linux-headers-3.1.0-2_3.1.0-2.3_all.deb
-rw-r--r-- 1 root root 966630 2011-11-02 18:29 linux-headers-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb
-rw-r--r-- 1 root root 37869094 2011-11-02 18:29 linux-image-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb
-rw-r--r-- 1 root root 63952 2011-11-02 17:53 linux-source-3.1.0_3.1.0-2.3_all.deb
-rw-r--r-- 1 root root 382274 2011-11-02 17:55 linux-tools-3.1.0-2_3.1.0-2.3_amd64.deb
-rw-r--r-- 1 root root 112522 2011-11-02 17:53 linux-tools-common_3.1.0-2.3_all.deb
drwxr-xr-x 27 root root 4096 2011-11-02 17:55 source

I’ve updated only “git clone” to get 3.1 kernel.

git clone git://kernel.ubuntu.com/ubuntu/ubuntu-precise.git source

There is another attractive option – just add driver xen-platform-pci to initrd.img as post installation procedure. Then reboot with xen_platform_pci=1. I choosed kernel rebuild option following core development [2]
Regardless [1] is written for Oneiric , it works for Precise kernel with minimal change above.Details may be viewed in Rebuilding 3.1 Linux Kernel in Ubuntu’s style
Next install regular HVM domain via profile :-

kernel ='/usr/lib64/xen/boot/hvmloader'
builder='hvm'
device_model = '/usr/lib64/xen/bin/qemu-dm'
name = "OneiricPV-HVM"
memory = 2048
vcpus=2
pae=1
acpi=1
apic=1
vif = [ 'type=ioemu, mac=00:16:3f:03:01:14, bridge=virbr0, model=e1000' ]
disk = [ 'phy:/dev/sda7,hda,w', 'phy:/dev/loop0,hdc:cdrom,r' ]
# disk = [ 'phy:/dev/sda7,hda,w']
boot='d'
xen_platform_pci=0
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
sdl=0
vnc=1
vncpasswd=''
stdvga=0
serial='pty'
tsc_mode=0
usb=1
usbdevice='tablet'


Then three debian packages (result of standard Ubuntu 3.1 kernel rebuild):

linux-headers-3.1.0-2_3.1.0-2.3_all.deb
linux-headers-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb
linux-image-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb

have been scp’ed to Oneiric HVM DomU and new 3.1 Ubuntu kernel has been installed.

sudo dpkg -i linux-headers-3.1.0-2_3.1.0-2.3_all.deb \
linux-headers-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb \
linux-image-3.1.0-2-pvhvm_3.1.0-2.3_amd64.deb

Following xen front end’s drivers were included into /boot/initrd.img-3.1.0-2-pvhvm

root@boris-HVM-domU:~# echo "xen-blkfront" >> /etc/initramfs-tools/modules
root@boris-HVM-domU:~# echo "xen-netfront" >> /etc/initramfs-tools/modules
root@boris-HVM-domU:~# vi /etc/initramfs-tools/modules
root@boris-HVM-domU:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-3.1.0-2-pvhvm

Domain has been shutdown and restarted with updated profile,having
xen_platform_pci=1
Runtime profile

kernel ='/usr/lib64/xen/boot/hvmloader'
builder='hvm'
device_model = '/usr/lib64/xen/bin/qemu-dm'
name = "OneiricPV-HVM"
memory = 2048
vcpus=2
pae=1
acpi=1
apic=1
vif = [ 'mac=00:16:3f:03:01:14, bridge=virbr0, model=e1000' ]
# disk = [ 'phy:/dev/sda7,hda,w', 'phy:/dev/loop0,hdc:cdrom,r' ]
disk = [ 'phy:/dev/sda7,hda,w']
boot='c'
xen_platform_pci=1
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
sdl=0
vnc=1
vncpasswd=''
stdvga=0
serial='pty'
tsc_mode=0
usb=1
usbdevice='tablet'

This time HVM domU gets loaded ( vs using regular kernel) and dmesg report contains following messages which are the fair of success PVHVM domain loading :-

[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.1.0-2-pvhvm (root@boris-System-P5Q3) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #3 SMP Wed Nov 2 15:15:13 MSK 2011 (Ubuntu 3.1.0-2.3-pvhvm 3.1.0)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.1.0-2-pvhvm root=UUID=bff48eed-20df-4abf-b788-935c75ea6226 ro quiet splash vt.handoff=7
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009e000 (usable)
[ 0.000000] BIOS-e820: 000000000009e000 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 000000003f800000 (usable)
[ 0.000000] BIOS-e820: 00000000fc000000 - 0000000100000000 (reserved)
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] DMI 2.4 present.
[ 0.000000] DMI: Xen HVM domU, BIOS 4.1.2 10/20/2011
[ 0.000000] Hypervisor detected: Xen HVM
[ 0.000000] Xen version 4.1.
[ 0.000000] Xen Platform PCI: I/O protocol version 1
[ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
[ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
[ 0.000000] You might have to change the root device
[ 0.000000] from /dev/hd[a-d] to /dev/xvd[a-d]



Managing domains via xm ( Xen 4.1.2)

# apt-get install python-lxml
# xm new oneiric7.hvm
# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 7199 4 r----- 9276.7
OneiricPV-HVM 2048 2 0.0
# xm start OneiricPV-HVM
# vncviewer localhost:0

Several useful commands per http://wiki.xen.org/xenwiki/XenLinuxPVonHVMdrivers

# dmesg | egrep -i 'xen|front'
[ 0.000000] DMI: Xen HVM domU, BIOS 4.1.2 10/20/2011
[ 0.000000] Hypervisor detected: Xen HVM
[ 0.000000] Xen version 4.1.
[ 0.000000] Xen Platform PCI: I/O protocol version 1
[ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
[ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
[ 0.000000] ACPI: RSDP 00000000000ea020 00024 (v02 Xen)
[ 0.000000] ACPI: XSDT 00000000fc0134f0 0003C (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: FACP 00000000fc0132d0 000F4 (v04 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: DSDT 00000000fc003440 0FE05 (v02 Xen HVM 00000000 INTL 20100528)
[ 0.000000] ACPI: APIC 00000000fc0133d0 000D8 (v02 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: HPET 00000000fc0134b0 00038 (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] Booting paravirtualized kernel on Xen HVM
[ 0.000000] Xen HVM callback vector for event delivery is enabled
[ 0.087915] Xen: using vcpuop timer interface
[ 0.087922] installing Xen timer for CPU 0
[ 0.176052] installing Xen timer for CPU 1
[ 0.631112] xen/balloon: Initialising balloon driver.
[ 0.631125] xen-balloon: Initialising balloon driver.
[ 0.636079] Switching to clocksource xen
[ 0.647494] xen: --> pirq=16 -> irq=8 (gsi=8)
[ 0.647569] xen: --> pirq=17 -> irq=12 (gsi=12)
[ 0.647619] xen: --> pirq=18 -> irq=1 (gsi=1)
[ 0.647665] xen: --> pirq=19 -> irq=6 (gsi=6)
[ 0.647719] xen: --> pirq=20 -> irq=4 (gsi=4)
[ 0.647792] xen: --> pirq=21 -> irq=7 (gsi=7)
[ 0.731857] xen: --> pirq=22 -> irq=28 (gsi=28)
[ 0.731861] xen-platform-pci 0000:00:03.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
[ 0.833848] xen: --> pirq=23 -> irq=23 (gsi=23)
[ 1.255228] blkfront device/vbd/768 num-ring-pages 1 nr_ents 32.
[ 1.258186] blkfront: xvda: flush diskcache: enabled
[ 1.274037] Initialising Xen virtual ethernet driver.
# ethtool -i eth0
driver: vif
version:
firmware-version:
bus-info: vif-0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
# cat /proc/partitions
major minor #blocks name
202 0 20480000 xvda
202 1 19439616 xvda1
202 2 1 xvda2
202 5 1037312 xvda5

At this point domain may be loaded via pygrub as PV guest :-

<domain type='xen' id='3'>
<name>OneiricPV</name>
<description></description>
<memory>2097152</memory>
<currentMemory>2097152</currentMemory>
<vcpu>2</vcpu>
<bootloader>/usr/lib/xen-default/bin/pygrub</bootloader>
<bootloader_args></bootloader_args>
<os>
<type>linux</type>
<kernel></kernel>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<disk type='block' device='disk'>
<driver name='phy'/>
<source dev='/dev/sda7'/>
<target dev='xvda' bus='xen'/>
</disk>
<interface type='bridge'>
<mac address='00:16:3e:35:61:47'/>
<source bridge='br0'/>
<script path='/etc/xen/scripts/vif-bridge'/>
<target dev='vif3.0'/>
</interface>
<console type='pty' tty='/dev/pts/2'>
<source path='/dev/pts/2'/>
<target type='xen' port='0'/>
</console>
<input type='mouse' bus='xen'/>
<graphics type='vnc' port='5900' autoport='yes' keymap='en-us'/>
</devices>
</domain>

# virsh define OneiricPV.xml
# virsh start OneiricPV



References
1. http://blog.avirtualhome.com/2011/10/28/how-to-compile-a-new-ubuntu-11-10-oneiric-kernel/
2. http://xen.1045712.n5.nabble.com/PATCH-xen-remove-XEN-PLATFORM-PCI-config-option-td4833154.html


Follow

Get every new post delivered to your Inbox.