Disabling checksum offloading at install OSOL 2009.06 PV DomU on Xen 3.4.1 Ubuntu 9.04 Dom0 ( with xenified aka Suse kernel)

July 19, 2009

Regardless bug with failure checksum offloading was fixed in SNV_81 , i.e. a while ago. OpenSolaris 2009.06 PV DomU install at Xen 3.4.1 Linux Dom0 appears still to be affected. Wrong checksum may be easily captured via tcpdump on the LAN due to DHCPDISCOVER request issued by OSOL installer is a broadcast request. Following bellow procedure involves Solaris Kernel module debugger to patch OSOL’s (SNV_111b) kernel at booting up to succeed with dhcp lease and to be able to proceed with initial install, what appears to be critical point for some users. View Opensolaris domU unable to get dhcp lease. File /etc/system gets updated via root terminal session before standard reboot to make the patch for kernel permanent. Posting is devoted first of all to Linux users , who are not aware of “kmdb” syntax and it’s involvement into boot process.
Modified installation profile. (-kd) key is added to extra line :-

name = 'OSOL0906PV'
memory = '2048'
disk = [ 'phy:/dev/loop0,xvdc:cdrom,r','phy:/dev/sda7,xvda,w']
vif = [ 'bridge = eth0' ]
bootloader = '/usr/local/bin/pygrub'
kernel = '/platform/i86xpv/kernel/amd64/unix'
ramdisk = '/boot/amd64/x86.microroot'
extra = '/platform/i86xpv/kernel/amd64/unix -kd - nowin -B install_media=cdrom'

Start install

root@ServerJaunty:/home/boris/nevada# xm create -c osol.install
Using config file "./osol.install".
Started domain osol.install (id=6)
Loading kmdb...
Welcome to kmdb
Loaded modules: [ unix krtld genunix ]

Make input at the prompt :-
[0]> ::bp xnf`_init
[0]> :c

v3.4.1-rc7 chgset 'Wed Jul 15 13:26:50 2009 +0100 19696:22ef8e900a6d'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Hostname: opensolaris
Remounting root read/write
Probing for device nodes ...
Loaded modules: [ scsi_vhci crypto mac neti xpv_psm ufs mpt zfs s1394 uhci hook
ip usba specfs md random sctp arp emlxs sockfs xpv_uppc ]
kmdb: stop at xnf`_init
kmdb: target stopped at:
xnf`_init: pushq %rbp

Make input at the prompt :-
[0]> xnf_cksum_offload/W 0
xnf`xnf_cksum_offload: 0x1 = 0x0
[0]> :c

Preparing live image for use
Done mounting Live image
USB keyboard
1. Albanian 23. Lithuanian
2. Belarusian 24. Latvian
3. Belgian 25. Macedonian
4. Brazilian 26. Malta_UK
5. Bulgarian 27. Malta_US
6. Canadian-Bilingual 28. Norwegian
7. Croatian 29. Polish
8. Czech 30. Portuguese
9. Danish 31. Russian
10. Dutch 32. Serbia-And-Montenegro
11. Finnish 33. Slovenian
12. French 34. Slovakian
13. French-Canadian 35. Spanish
14. Hungarian 36. Swedish
15. German 37. Swiss-French
16. Greek 38. Swiss-German
17. Icelandic 39. Traditional-Chinese
18. Italian 40. TurkishQ
19. Japanese-type6 41. TurkishF
20. Japanese 42. UK-English
21. Korean 43. US-English
22. Latin-American
To select the keyboard layout, enter a number [default 43]:
1. Arabic
2. Chinese - Simplified
3. Chinese - Traditional
4. Czech
5. Dutch
6. English
7. French
8. German
9. Greek
10. Hebrew
11. Hungarian
12. Indonesian
13. Italian
14. Japanese
15. Korean
16. Polish
17. Portuguese - Brazil
18. Russian
19. Slovak
20. Spanish
21. Swedish
To select desktop language, enter a number [default is 6]:
User selected: English
Configuring devices.
Mounting cdroms
Reading ZFS config: done.
opensolaris console login: jack
Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008
jack@opensolaris:~$ Jul 19 03:18:29 opensolaris in.routed[786]: route --> nexthop is not directly connected
jack@opensolaris:~$ pfexec ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet netmask ff000000
xnf0: flags=1004843 mtu 1500 index 2
inet netmask ffffff00 broadcast
ether 0:16:3e:74:95:f
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
xnf0: flags=2004841 mtu 1500 index 2
inet6 fe80::216:3eff:fe74:950f/10
ether 0:16:3e:74:95:f
jack@opensolaris:~$ cp .Xclients .vnc/xstartup
jack@opensolaris:~$ vncserver

Open an other terminal window and connect via vncviewer to DomU
Go through normal install
At the end drop to root shell and add to /etc/system one line :-

set xnf:xnf_cksum_offload = 0

Then click “REBOOT” button

Loading OSOL 2009.06 DomU

Setup Solaris Nevada (build 117) PV DomU at Xen 3.4.1 Ubuntu 9.04 Dom0 via backport CS 19889 libfsimage: Support for zfs version 16

July 14, 2009

Actually, CS 19889 is in Xen 3.5 Unstable. However, attempt to install mentioned DomU at Xen 3.5 Dom0 via standard pygrub profile:-

root@ServerXen341:/home/boris/nevada# cat snv117.install
name = 'Solaris117PV'
memory = '2048'
disk = [ 'phy:/dev/loop0,6:cdrom,r','phy:/dev/sdb3,0,w']
vif = [ 'bridge = eth1' ]
bootloader = '/usr/local/bin/pygrub'
kernel = '/boot/platform/i86xpv/kernel/amd64/unix'
ramdisk = '/boot/amd64/x86.miniroot'
extra = '/platform/i86xpv/kernel/amd64/unix - nowin -B install_media=cdrom'
root@ServerXen341:/home/boris/nevada# xm create -c snv117.install

aborts with message cannot find kernel “/boot/platform/i86xpv/kernel/amd64/unix”. Setup bellow has been tested with xenified (aka Suse) kernel and 2.6.31-rc1 pvops enabled kernel been built via rebase/master branch of Jeremy Fitzhardinge git repository. The root cause of the problem is ZFS v.16 utilized by SNV_117 DomU for it’s image device unsupported by Xen 3.4.1. Place “raw” content of CS 19889 into file /usr/src/zfs16.patch1.
Next step :-

# cd /usr/src
# hg clone http://xenbits.xensource.com/xen-3.4-testing.hg
# cd xen-3.4-testing.hg
Set in Config.mk
PYTHON = python
# patch -p1 -s -i ../zfs16.patch1
# make xen
# make tools
# make stubdom
# make install-xen
# make install-tools
# make install-stubdom

Modify corresponding Grub entry and reboot Xen Host.
Install SNV_117 DomU via profile above. To be sure install would be OK use only primary partitions as image devices for SNV_117 DomU (ZFS based) or the first one in extended partion on x86 architecture.
Symptom of wrong partition choice could be message showing up when attempt to load via standard pygrub profile :-

NOTICE: zfs_parse_bootfs: error 2
Cannot mount root on rpool/57 fstype zfs
panic[cpu0]/thread=fffffffffbc60c20: vfs_mountroot: cannot mount root
fffffffffbc94a00 genunix:vfs_mountroot+350 ()
fffffffffbc94a30 genunix:main+f0 ()
fffffffffbc94a40 unix:_locore_start+80 ()
skipping system dump - no dump device configured

VNC Setup on SNV_117 DomU is the same as on OSOL 2009.06 DomU.

cat /etc/X11/gdm/custom.conf
# GDM Custom Configuration file.
# overrides: /usr/share/gdm/defaults.conf
# AllowRoot=true
# AllowRemoteRoot=true

Services restart:-

svcadm disable xvnc-inetd gdm
svcadm enable xvnc-inetd gdm

Vinagre remote console :-

Standard VNC console :-

Set up Xen 3.4.3 Dom0 via xenified kernel on top Ubuntu 9.10 Server

July 7, 2009

Following bellow is build Xen Host on top Ubuntu 9.10 Server.It is based on direct clone via “hg” and build Xen 3.4.3-rc4 from xensource.com , applying Andrew Lyon’s 2.6.31 patch set V12 to vanilla kernel along with tuning patched kernel via “make menuconfig” and building aka Suse xenified kernel in Ubuntu 9.10 environment.Notice that forward porting of original patch set was done by Jan Beulich for kernel on OpenSuse 11.2. Built in patched kernel open-source NVIDIA and Radeon HD drivers work just fine on Ubuntu 9.10, providing Gnome Desktop Environment right away.

# cd /usr/src/build/
# wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.31-12.tar.bz2

4.Untar linux- and apply patches.

# cd /usr/src/build/
# bzcat linux- | tar xf -
# mkdir -p xen-patches-2.6.31-12
# cd xen-patches-2.6.31-12
# bzcat ../xen-patches-2.6.31-12.tar.bz2 | tar xf -
# cd ../linux-

Run script

for P in `ls ../xen-patches-2.6.31-12/6*.patch1 | sort `
patch -p1 -s -i $P
if [ $? = 0 ]; then
echo $P applied
echo "Error processing "$P
exit 1

Build kernel :-

# make menuconfig

Processor type and features ->
[*] Symmetric multi-processing support
[ ] Support sparse irq numbering
[*] Enable MPS table
[*] Enable Xen compatible kernel
[*] Single-depth WCHAN output
Processor family (Generic-x86-64) --->
[ ] Configure Maximum number of SMP Processors and NUMA Nodes
(64) Maximum number of CPUs
Preemption Model (No Forced Preemption (Server)) --->
[ ] Machine Check Exception

Device Drivers -> XEN
[*] Privileged Guest (domain 0)
<*> Backend driver support
<*> Block-device backend driver
<*> Block-device tap backend driver
<*> Block-device tap backend driver 2
>*> Network-device backend driver
(8) Maximum simultaneous transmit requests (as a power of 2)
[ ] Pipelined transmitter (DANGEROUS)
< > Network-device loopback driver
<*> PCI-device backend driver
PCI Backend Mode (Virtual PCI) --->
[ ] PCI Backend Debugging
< > TPM-device backend driver
<M> SCSI backend driver
<M> USB backend driver
<M> Block-device frontend driver
<M> Network-device frontend driver
<M> Network-device frontend driver acceleration for Solarflare NICs
<M> SCSI frontend driver
<M> USB frontend driver
[*] Taking the HCD statistics (for debug)
<*> User-space granted page access driver
<*> Framebuffer-device frontend driver

# make -j(x)(number_of_cores)
# make modules_install install
# mkinitramfs -o /boot/initrd-

Different option is to install pvops enabled kernel via download :-

# git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
# cd linux-2.6-xen
# git checkout origin/xen/stable -b xen/stable

Notice :-

# git checkout origin/xen/master -b xen/master

will result building the most recent 2.31.6 version of pvops kernel. Branch
above , actually is not required to be checked out . It’s default branch of
“git clone”. Difference in meantime is in “acpi” branch not merged yet to xen/stable. However, APIC rework has been committed for

# make menuconfig
# make
# make modules_install install
# mkinitramfs -o /boot/initrd-

To test xenified aka Suse kernel setup Xen 3.4.3 on to top of Ubuntu Karmic Server. First – install on Ubuntu Karmic Server all packages required for Xen build along with Ubuntu-Desktop ( via tasksel). Of course, Xen Host doesn’t require GUI. Install Ubuntu-Desktop just make work comfortable and faster.

apt-get install libcurl4-openssl-dev \
xserver-xorg-dev \
python2.6-dev \
mercurial gitk \
build-essential \
libncurses5-dev \
uuid-dev gawk \
gettext texinfo bcc

Second – build Xen from source

# cd /usr/src
# hg clone http://xenbits.xensource.com/xen-3.4-testing.hg
# cd xen-3.4-testing.hg
Set in Config.mk
PYTHON = python
# make install-xen
# make install-tools
# make install-stubdom

Tuning Config.mk results Xen packages to be placed into /usr/local/lib/python2.6/dist-packages due to Changeset 19594 in xen-3.4-testing.hg. Otherwise, Xen packages would go to /usr/lib/python2.6/site-packages, which is not default location for python 2.6 on Ubuntu 9.04 ( vs F11 ). Thus you won’t be able to start xend in Dom0. Same thing happens when building Xen Unstable on Ubuntu 9.04 Server.
Add entry to /boot/grub/menu.lst:-

title Xen 3.4 / Ubuntu 9.04 kernel
uuid 34d2c0bd-fe30-47e0-990e-4921caf1e845
kernel /boot/xen-3.4.gz
module /boot/vmlinuz- root=/dev/sdb2 ro console=tty0
module /boot/initrd-

and rebooted into xen host.Tested OSOL 133 PV DomU,F12 PV DomU, Ubuntu Kamic HVM and PV DomUs sharing same image device.

1.Gentoo-xen-kernel dom0 Xen patches