Set up RHEL6 (beta) PV Guest at Xen 4.0 (pvops kernel 2.6.32.12) on top of F12 and Ubuntu 9.10 Server

April 27, 2010

The schema bellow would work fine at same Xen 4.0 Dom0 on top of Ubuntu 9.10 Server with several issues to be fixed. Virt-install command line would require –os-type=linux –os-variant=rhel5 , symlink
for pygrub , invoked by default , should be created pointing to /usr/local/bin/pygrub. On Fedora 12 everything is just ready for RHEL 6 PV DomU virt-install.
Create local Apache Mirror :-

# chkconfig httpd on
# service httpd start
# mount -o loop,ro \
RHEL6.0-20100414.0-AP-x86_64-DVD1.iso \
/var/www/html/rhel

Then start virt-install in no graphics mode , vnc mode would cause installer to hang on any system F12 or Ubuntu 9.10 server

# virt-install -n RHEL6 -r 1024 -p --nographics \
-f /dev/sda7 \
-l http://192.168.1.41/rhel --debug

and bring up VNC at DomU at runtime







When install completed DomU will reboot automatically in text mode



Set up VNC at DomU mannually

# yum install vnc-server
# vncserver

Edit ~.vnc/xstartup file and replace line with “xterm”
/usr/bin/gnome-session &
Add to /etc/rc.local

export HOME=/root
vncserver

Now shutdown DomU and start it up via virt-manager.
Open terminal session and issue virsh console RHEL6



Connect via vncviewer to DomU




Virt-install command line at Xen 4.0 (pvops 2.6.32.12) on top of Ubuntu 9.10 Server

# virt-install -n RHEL6 -r 1024 -p --nographics \
--os-type=linux --os-variant=rhel5
-f /dev/sda7 \
-l http://192.168.1.41/rhel --debug

Runtime snapshots on Ubuntu 9.10 Server





Set up Ubuntu 10.04 Server PV DomU at Xen 4.0 Dom0 (pvops 2.6.32.12 kernel) Dom0 on top of Ubuntu 10.04 Server

April 24, 2010

The procedure bellow in general follows Thiago’s Martins submission [1] to xen-devel mailing list. However,sequence of steps has been changed and also option of pvops kernel build via make-kpkg was added , /dev/xen/evtchn device hard linked with with pvops kernel, git checkout procedure is more straight forward and one Change-set from xen-4.0-testing.hg  back ported to Xen 4.0 to support grub2 notation ‘(hd0,1)’ specific for Ubuntu 10.04 aka Lucid Lynx. Several suggestions have been also made regarding to have libvirt ,virtinst and related packages to work with Xen 4.0 on 10.04 Server. The way as Xen 4.0 Dom0 set up worked on Ubuntu 9.10 (04)  Servers doesn’t work any longer.

apt-get install libcurl4-openssl-dev \
xserver-xorg-dev \
mercurial gitk \
build-essential \
libncurses5-dev \
uuid-dev gawk \
gettext texinfo bcc \
libncurses5-dev dpkg-dev debhelper \
iasl texinfo bridge-utils bison flex
apt-get build-dep xen-3.3

Now git clone and checkout branch xen/stable

cd /usr/src
git clone
git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
cd linux-2.6-xen
git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x

Build kernel in Debian way

make menuconfig # Enabled the dom0 support and the Xen backend as usual on Ubuntu 9.10, F12
make -j(x)number_of_kernels
make modules_install install
chmod g-s /usr/src -R
make deb-pkg
dpkg -i ../linux-image*2.6.32.12*.deb
depmod 2.6.32.12
update-initramfs -c -k 2.6.32.12

OR

aptitude install kernel-package fakeroot
export CONCURRENCY_LEVEL=(number_of_cores) +1
make-kpkg clean
fakeroot make-kpkg --initrd --append-to-version=-pv kernel-image kernel-headers
dpkg -i ../linux-image-2.6.32.16-pv_2.6.32.16-pv-10.00.Custom_amd64.deb
update-initramfs -c -k 2.6.32.16-pv

One more notice regarding /etc/modules editing silently skipped in [1]
In case configuring kernel (1)
<M> Xen /dev/xen/evtchn device
echo “xen-evtchn” >> /etc/modules
In case configuring kernel (2)
<*> Xen /dev/xen/evtchn device
/etc/modules shouldn’t be touched.

With config bellow there is no need to touch /etc/modules at all.
Option “userspace grant access device driver is set to” <*> to make virt-install handy at Xen guests
installs



Set up Xen 4.0 Hypervisor
*********************************************************************************************************************
UPDATE 06/09/2010
*********************************************************************************************************************
As of 06/09/2010 checking out xen/stable or xen/stable-2.6.32.x branch will result building 2.6.32.15 kernel. This kernel may work properly only with Xen 4.0.1-rc2-pre ( at the time of writing). So build xen as follows

# cd /usr/src
# hg clone http://xenbits.xensource.com/xen-4.0-testing.hg
# cd xen-4.0-testing.hg
# make xen
# make tools
# make install-xen
# make install-tools PYTHON_PREFIX_ARG=

********************************************************************************************************************

cd /usr/src
At this point creare grub2-fix.patch1
as raw content of http://xenbits.xensource.com/xen-4.0-testing.hg?rev/065efaec5e71
Changeset 21112 pygrub: Fix Grub2 support for Ubuntu 10.04

gunzip xen-4.0.0.tar.gz
tar -xvf xen-4.0.0.tar
cd xen-4.0.0
patch -p1 -s -i ../grub2-fix.patch1
make xen
make tools
make install-xen
make install-tools PYTHON_PREFIX_ARG=

Set up xend,xendomains to run as services

update-rc.d xend defaults 20 21
update-rc.d xendomains defaults 21 20

Reboot in Xen with grub entry

menuentry "Xen 4 / Ubuntu 10.04 kernel 2.6.32.12" {
insmod ext2
set root='(hd1,5)'
multiboot (hd1,5)/boot/xen.gz dummy=dummy dom0_mem=1024M
module (hd1,5)/boot/vmlinuz-2.6.32.12 dummy=dummy nopat root=/dev/sdb5 ro console=tty0
module (hd1,5)/boot/initrd.img-2.6.32.12
}

Verify Xen environment :-

host : ServerLucid
release : 2.6.32.12
version : #7 SMP Wed Apr 28 20:52:50 MSD 2010
machine : x86_64
nr_cpus : 2
nr_nodes : 1
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 2999
hw_caps : bfebfbff:20100800:00000000:00000940:0008e3fd:00000000:00000001:00000000
virt_caps : hvm
total_memory : 7954
free_memory : 2054
node_to_cpu : node0:0-1
node_to_memory : node0:2054
node_to_dma32_mem : node0:1225
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .0
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline :
cc_compiler : gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)
cc_compile_by : root
cc_compile_domain :
cc_compile_date : Fri Apr 23 21:24:30 MSD 2010
xend_config_format : 4

root@ServerLucid:~# brctl show
bridge name bridge id STP enabled interfaces
eth0 8000.000c76e01ec5 no peth0
virbr0 8000.000000000000 yes

Test Xen installation via creation Ubuntu 10.04 Server PV DomU
Create HVM DomU via profile

kernel = "/usr/lib/xen/boot/hvmloader"
builder = 'hvm'
memory = 2048
name = "LucidSRV"
vcpus = 1
vif = [ 'type=ioemu, bridge=eth0' ]
disk = [ 'phy:/dev/sda9,hda,w','phy:/dev/loop0,hdc:cdrom,r' ]
device_model = '/usr/lib/xen/bin/qemu-dm'
vnc=1
boot='d'

Due to CS 21112 PV DomU based on the same image may be loaded via profile

name="LucidPVG"
memory=2048
vcpus=1
bootloader="/usr/local/bin/pygrub"
disk=['phy:/dev/sda9,xvda,w']
vif= ['bridge=eth0']
vfb=['type=vnc,vncunused=1']







Tested OK on final release
—————————————————————————————————————–
UPDATE on 06/01/2010
——————————————————————————————————————
If your major concern is Libvirt you may switch to Set up Libvirt 0.7.7&Xen 4.0 Dom0 on top of Fedora 13
——————————————————————————————————————–
Another option is to set up Libvirt 0.8.0 along with with all related packages via Daniel Baumann PPA
https://launchpad.net/~dnjl/+archive/virtualization. It seems to be working OK with 2.6.32.15 pvops kernel
(xen/stable-2.6.32.x) under Xen 4.0. View for details Set up Libvirt 0.8.1 & Xen 4.0.1 Dom0 (2.6.32.15 pvops) on top of Ubuntu 10.04 Server .
——————————————————————————————————————-

root@ServerLX:~# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D18C7A1DF6E6BED2
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv-keys D18C7A1DF6E6BED2
gpg: requesting key F6E6BED2 from hkp server keyserver.ubuntu.com
gpg: key F6E6BED2: public key "Launchpad ppa" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
root@ServerLX:~# apt-get update
Hit http://us.archive.ubuntu.com lucid Release.gpg
Get:1 http://ppa.launchpad.net lucid Release.gpg [307B]
Ign http://ppa.launchpad.net/dnjl/virtualization/ubuntu/ lucid/main Translation-en_US
Hit http://ppa.launchpad.net lucid Release.gpg
Hit http://security.ubuntu.com lucid-security Release.gpg
Ign http://us.archive.ubuntu.com/ubuntu/ lucid/main Translation-en_US
Ign http://security.ubuntu.com/ubuntu/ lucid-security/main Translation-en_US
Ign http://security.ubuntu.com/ubuntu/ lucid-security/restricted Translation-en_US
Ign http://us.archive.ubuntu.com/ubuntu/ lucid/restricted Translation-en_US
Ign http://ppa.launchpad.net/gwibber-daily/ppa/ubuntu/ lucid/main Translation-en_US
Get:2 http://ppa.launchpad.net lucid Release [57.3kB]
Ign http://security.ubuntu.com/ubuntu/ lucid-security/universe Translation-en_US
Ign http://security.ubuntu.com/ubuntu/ lucid-security/multiverse Translation-en_US
Hit http://security.ubuntu.com lucid-security Release
Ign http://us.archive.ubuntu.com/ubuntu/ lucid/universe Translation-en_US
Hit http://ppa.launchpad.net lucid Release
Ign http://us.archive.ubuntu.com/ubuntu/ lucid/multiverse Translation-en_US
Hit http://security.ubuntu.com lucid-security/main Packages
Hit http://ppa.launchpad.net lucid/main Packages
Hit http://ppa.launchpad.net lucid/main Sources
Hit http://us.archive.ubuntu.com lucid-updates Release.gpg
Hit http://security.ubuntu.com lucid-security/restricted Packages
Hit http://security.ubuntu.com lucid-security/main Sources
Hit http://security.ubuntu.com lucid-security/restricted Sources
Hit http://security.ubuntu.com lucid-security/universe Packages
Ign http://us.archive.ubuntu.com/ubuntu/ lucid-updates/main Translation-en_US
Hit http://ppa.launchpad.net lucid/main Packages
Hit http://security.ubuntu.com lucid-security/universe Sources
Hit http://security.ubuntu.com lucid-security/multiverse Packages
Hit http://security.ubuntu.com lucid-security/multiverse Sources
Ign http://us.archive.ubuntu.com/ubuntu/ lucid-updates/restricted Translation-en_US
Ign http://us.archive.ubuntu.com/ubuntu/ lucid-updates/universe Translation-en_US
Ign http://us.archive.ubuntu.com/ubuntu/ lucid-updates/multiverse Translation-en_US
Hit http://us.archive.ubuntu.com lucid Release
Hit http://us.archive.ubuntu.com lucid-updates Release
Hit http://us.archive.ubuntu.com lucid/main Packages
Hit http://us.archive.ubuntu.com lucid/restricted Packages
Hit http://us.archive.ubuntu.com lucid/main Sources
Hit http://us.archive.ubuntu.com lucid/restricted Sources
Hit http://us.archive.ubuntu.com lucid/universe Packages
Hit http://us.archive.ubuntu.com lucid/universe Sources
Hit http://us.archive.ubuntu.com lucid/multiverse Packages
Hit http://us.archive.ubuntu.com lucid/multiverse Sources
Hit http://us.archive.ubuntu.com lucid-updates/main Packages
Hit http://us.archive.ubuntu.com lucid-updates/restricted Packages
Hit http://us.archive.ubuntu.com lucid-updates/main Sources
Hit http://us.archive.ubuntu.com lucid-updates/restricted Sources
Hit http://us.archive.ubuntu.com lucid-updates/universe Packages
Hit http://us.archive.ubuntu.com lucid-updates/universe Sources
Hit http://us.archive.ubuntu.com lucid-updates/multiverse Packages
Hit http://us.archive.ubuntu.com lucid-updates/multiverse Sources
Fetched 308B in 5s (55B/s)
Reading package lists... Done

root@ServerLX:~# apt-get install virtinst python-virtinst virt-viewer

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
acl bridge-utils libaio1 libvdeplug2 libvirt-bin libvirt0 libxen3 python-libvirt python-urlgrabber qemu-common
qemu-kvm seabios vgabios
Suggested packages:
mol-drivers-macosx openbios-sparc kvm-pxe ubuntu-vm-builder uml-utilities qemu
The following NEW packages will be installed:
acl bridge-utils libaio1 libvdeplug2 libvirt-bin libvirt0 libxen3 python-libvirt python-urlgrabber
python-virtinst qemu-common qemu-kvm seabios vgabios virt-viewer virtinst
0 upgraded, 16 newly installed, 0 to remove and 11 not upgraded.
Need to get 5,451kB of archives.
After this operation, 17.3MB of additional disk space will be used.
Do you want to continue [Y/n]? Y
Get:1 http://ppa.launchpad.net/dnjl/virtualization/ubuntu/ lucid/main libvirt0 0.8.0-2ubuntu0~dnjl2~lucid0 [831kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ lucid/main acl 2.2.49-2 [56.8kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ lucid/main bridge-utils 1.4-5ubuntu2 [33.9kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ lucid/main libaio1 0.3.107-3ubuntu2 [9,608B]
Get:5 http://us.archive.ubuntu.com/ubuntu/ lucid/universe libvdeplug2 2.2.3-3 [13.0kB]
Get:6 http://us.archive.ubuntu.com/ubuntu/ lucid/main libxen3 3.3.0-1ubuntu11 [151kB]
Get:7 http://us.archive.ubuntu.com/ubuntu/ lucid/main python-urlgrabber 3.1.0-5ubuntu1 [57.5kB]
Get:8 http://us.archive.ubuntu.com/ubuntu/ lucid/main seabios 0.5.1-0ubuntu2 [48.2kB]
Get:9 http://us.archive.ubuntu.com/ubuntu/ lucid/main vgabios 0.6c-2ubuntu1 [78.5kB]
Get:10 http://ppa.launchpad.net/dnjl/virtualization/ubuntu/ lucid/main libvirt-bin 0.8.0-2ubuntu0~dnjl2~lucid0 [706kB]
Get:11 http://ppa.launchpad.net/dnjl/virtualization/ubuntu/ lucid/main python-libvirt 0.8.0-2ubuntu0~dnjl2~lucid0 [62.5kB]
Get:12 http://ppa.launchpad.net/dnjl/virtualization/ubuntu/ lucid/main qemu-common 0.12.3+noroms-0ubuntu9.1~dnjl1~lucid0 [30.1kB]
Get:13 http://ppa.launchpad.net/dnjl/virtualization/ubuntu/ lucid/main qemu-kvm 0.12.3+noroms-0ubuntu9.1~dnjl1~lucid0 [2,955kB]
Get:14 http://ppa.launchpad.net/dnjl/virtualization/ubuntu/ lucid/main virtinst 0.500.3-1ubuntu0~dnjl0~lucid0 [356kB]
Get:15 http://ppa.launchpad.net/dnjl/virtualization/ubuntu/ lucid/main python-virtinst 0.500.3-1ubuntu0~dnjl0~lucid0 [1,576B]
Get:16 http://ppa.launchpad.net/dnjl/virtualization/ubuntu/ lucid/main virt-viewer 0.2.1-1ubuntu0~dnjl4~lucid0 [61.9kB]
Fetched 5,451kB in 56s (96.5kB/s)
Selecting previously deselected package acl.
(Reading database ... 147802 files and directories currently installed.)
Unpacking acl (from .../acl_2.2.49-2_amd64.deb) ...
Selecting previously deselected package bridge-utils.
Unpacking bridge-utils (from .../bridge-utils_1.4-5ubuntu2_amd64.deb) ...
Selecting previously deselected package libaio1.
Unpacking libaio1 (from .../libaio1_0.3.107-3ubuntu2_amd64.deb) ...
Selecting previously deselected package libvdeplug2.
Unpacking libvdeplug2 (from .../libvdeplug2_2.2.3-3_amd64.deb) ...
Selecting previously deselected package libxen3.
Unpacking libxen3 (from .../libxen3_3.3.0-1ubuntu11_amd64.deb) ...
Selecting previously deselected package libvirt0.
Unpacking libvirt0 (from .../libvirt0_0.8.0-2ubuntu0~dnjl2~lucid0_amd64.deb) ...
Selecting previously deselected package libvirt-bin.
Unpacking libvirt-bin (from .../libvirt-bin_0.8.0-2ubuntu0~dnjl2~lucid0_amd64.deb) ...
Selecting previously deselected package python-libvirt.
Unpacking python-libvirt (from .../python-libvirt_0.8.0-2ubuntu0~dnjl2~lucid0_amd64.deb) ...
Selecting previously deselected package python-urlgrabber.
Unpacking python-urlgrabber (from .../python-urlgrabber_3.1.0-5ubuntu1_all.deb) ...
Selecting previously deselected package seabios.
Unpacking seabios (from .../seabios_0.5.1-0ubuntu2_all.deb) ...
Selecting previously deselected package vgabios.
Unpacking vgabios (from .../vgabios_0.6c-2ubuntu1_all.deb) ...
Selecting previously deselected package qemu-common.
Unpacking qemu-common (from .../qemu-common_0.12.3+noroms-0ubuntu9.1~dnjl1~lucid0_all.deb) ...
Selecting previously deselected package qemu-kvm.
Unpacking qemu-kvm (from .../qemu-kvm_0.12.3+noroms-0ubuntu9.1~dnjl1~lucid0_amd64.deb) ...
Selecting previously deselected package virtinst.
Unpacking virtinst (from .../virtinst_0.500.3-1ubuntu0~dnjl0~lucid0_all.deb) ...
Selecting previously deselected package python-virtinst.
Unpacking python-virtinst (from .../python-virtinst_0.500.3-1ubuntu0~dnjl0~lucid0_all.deb) ...
Selecting previously deselected package virt-viewer.
Unpacking virt-viewer (from .../virt-viewer_0.2.1-1ubuntu0~dnjl4~lucid0_amd64.deb) ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Setting up acl (2.2.49-2) ...
Setting up bridge-utils (1.4-5ubuntu2) ...
Setting up libaio1 (0.3.107-3ubuntu2) ...
Setting up libvdeplug2 (2.2.3-3) ...
Setting up libxen3 (3.3.0-1ubuntu11) ...
Setting up libvirt0 (0.8.0-2ubuntu0~dnjl2~lucid0) ...
Setting up libvirt-bin (0.8.0-2ubuntu0~dnjl2~lucid0) ...
Adding group `libvirtd' (GID 124) ...
Done.
libvirt-bin start/running, process 2491
Setting up python-libvirt (0.8.0-2ubuntu0~dnjl2~lucid0) ...
Setting up python-urlgrabber (3.1.0-5ubuntu1) ...
Setting up seabios (0.5.1-0ubuntu2) ...
Setting up vgabios (0.6c-2ubuntu1) ...
Setting up qemu-common (0.12.3+noroms-0ubuntu9.1~dnjl1~lucid0) ...
Setting up qemu-kvm (0.12.3+noroms-0ubuntu9.1~dnjl1~lucid0) ...
qemu-kvm start/running
Setting up virt-viewer (0.2.1-1ubuntu0~dnjl4~lucid0) ...
Processing triggers for python-central ...
Setting up virtinst (0.500.3-1ubuntu0~dnjl0~lucid0) ...
Setting up python-virtinst (0.500.3-1ubuntu0~dnjl0~lucid0) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Processing triggers for python-support ...

It’s important to specify “–connect xen:///” inside virt-install command line to succeed with VNC console launching (vs 9.10) . Variable VIRSH_DEFAULT_CONNECT_URI exported via .bashrc in my case caused virt-install to crash.Comment out (xend-unix-server yes) in /etc/xen/xend-config.sxp and export variable

export VIRSH_DEFAULT_CONNECT_URI="xen:///"

in root’s .bashrc. Then reboot Xen Host.

root@ServerLX:~# virsh version
Compiled against library: libvir 0.8.0
Using library: libvir 0.8.0
Using API: Xen 3.0.1
Running hypervisor: Xen 4.0.0









References
1. http://lists.xensource.com/archives/html/xen-users/2010-03/msg00878.html
2.http://www.linuxtoday.com/high_performance/2010043000735OSS
3.http://librenix.com/?inode=14285


ZFS 24 Support via Pygrub & fsimage.so patch for Xen 4.0 Dom0 (pvops 2.6.32.11) on top of F12

April 14, 2010

Finally Mark Johnson submitted to xen-devel mailing list patch for ZFS 24 support at Xen 4.0. Patch mentioned bellow was successfully tested at Xen 4.0 Dom0 with pvops kernel 2.6.32.11 on top of Fedora 12 via modifying Michael’s Young xen-4.0.0.7.fc12.src.rpm and direct patching xen 4.0 official tarball on Ubuntu Karmic Koala Server.
Download from [1] fsimage-zfs-24.patch . Then add it to /root/rpmbuild/SOURCES folder created via install xen-4.0.0-0.7.fc12.src.rpm :-

# cd
# wget http://pasik.reaktio.net/fedora/xen-4.0.0-0.7.fc12.src.rpm
# rpm -iv xen-4.0.0-0.7.fc12.src.rpm

Then activate rpmbuild via:-

# yum install transfig texi2html libaio-devel \
dev86 glibc-devel.i686 \
e2fsprogs-devel gitk mkinitrd iasl
# cd /root/rpmbuild/SPECS
# vi xen.spec

Add processing of newly added patch

Patch18 : fsimage-zfs-24.patch
. . . . . .
%patch18 -p1
. . . . . .

Save and build

# rpmbuild -ba ./xen.spec

When done

# cd ../RPMS/x*
# yum install xen-4.0.0-0.7.fc12.x86_64.rpm \
xen-debuginfo-4.0.0-0.7.fc12.x86_64.rpm \
xen-devel-4.0.0-0.7.fc12.x86_64.rpm \
xen-doc-4.0.0-0.7.fc12.x86_64.rpm \
xen-hypervisor-4.0.0-0.7.fc12.x86_64.rpm \
xen-libs-4.0.0-0.7.fc12.x86_64.rpm \
xen-runtime-4.0.0-0.7.fc12.x86_64.rpm

Now activate service :-

# chkconfig xend on
# chkconfig xendomains on
# chkconfig xenconsoled on
# chkconfig xenstored on

and install libvirt 0.7.1-15

# yum install libvirt libvirt-python python-virtinst virt-manager virt-viewer
# chkconfig libvirtd on

Now checkout and build 2.6.32.11 pvops kernel

# git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
# cd linux-2.6-xen
# git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x
# make menuconfig

and tune kernel to support Xen in Dom0

# make -j(x)number_of_cores
# make modules_install install
# mkinitrd /boot/initrd-2.6.32.11.img 2.6.32.11

Create grub’s entry

title Xen 4.0 Fedora (2.6.32.11 stable)
root (hd0,2)
kernel /xen.gz
module /vmlinuz-2.6.32.11 nopat ro root=/dev/mapper/vg_fedora12a-lv_root LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
module /initrd-2.6.32.11.img

Reboot into Xen environment :-

host : fedora12a
release : 2.6.32.11
version : #13 SMP Wed Apr 7 14:57:48 MSD 2010
machine : x86_64
nr_cpus : 4
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 2833
hw_caps : bfebfbff:20100800:00000000:00000940:0408e3fd:00000000:00000001:00000000
virt_caps : hvm
total_memory : 8158
free_memory : 2054
node_to_cpu : node0:0-3
node_to_memory : node0:2054
node_to_dma32_mem : node0:1192
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .0
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline :
cc_compiler : gcc version 4.4.3 20100127 (Red Hat 4.4.3-4) (GCC)
cc_compile_by : root
cc_compile_domain :
cc_compile_date : Wed Apr 14 16:46:15 MSD 2010
xend_config_format : 4

[root@fedora12a ~]# brctl show
bridge name bridge id STP enabled interfaces
eth0 8000.90e6ba2d11eb no peth0
virbr0 8000.000000000000 yes

Create OSOL 134 PV DomU via profile create_osol134.xml :-

<domain type='xen'>
<name>osol134</name>
<uuid>f80856df-3180-acc5-931d-32190cfe4062</uuid>
<memory>1048576</memory>
<currentMemory>1048576</currentMemory>
<vcpu>2</vcpu>
<bootloader></bootloader>
<os>
<type>linux</type>
<kernel>/home/boris/osol134/unix</kernel>
<initrd>/home/boris/osol134/boot_archive</initrd>
<cmdline>/platform/i86xpv/kernel/amd64/unix - nowin -B install_media=cdrom</cmdline>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>destroy</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<disk type='block' device='disk'>
<driver name='phy'/>
<source dev='/dev/sdb5'/>
<target dev='xvda' bus='xen'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='file'/>
<source file='/root/MyDownloads/osol-dev-134.iso'/>
<target dev='xvdc' bus='xen'/>
<readonly/>
</disk>
<interface type='bridge'>
<mac address='00:16:3e:72:da:33'/>
<source bridge='eth0'/>
<script path='/etc/xen/scripts/vif-bridge'/>
<target dev='vif7.0'/>
</interface>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target port='0'/>
</console>
</devices>
</domain>

and run

# virsh create create_osol134.xml
# virsh console osol134
At this point activate VNC connection to DomU via script

domid=`virsh domid osol134`
ip=`/usr/bin/xenstore-read /local/domain/$domid/guest/ipv4/0/address`
echo $ip
port=`/usr/bin/xenstore-read /local/domain/$domid/guest/vnc/port`
echo $port
/usr/bin/xenstore-read /local/domain/$domid/guest/vnc/passwd
vncviewer $ip:$port

and go through normal install. When done define new domain via profile :

<domain type='xen'>
<name>OS134</name>
<memory>2097152</memory>
<currentMemory>2097152</currentMemory>
<vcpu>2</vcpu>
<bootloader>/usr/bin/pygrub</bootloader>
<os>
<type>linux</type>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<disk type='block' device='disk'>
<driver name='phy'/>
<source dev='/dev/sdb5'/>
<target dev='xvda' bus='xen'/>
</disk>
<interface type='bridge'>
<mac address='00:16:3e:10:93:61'/>
<source bridge='eth0'/>
<script path='/etc/xen/scripts/vif-bridge'/>
<target dev='vif7.0'/>
</interface>
<console type='pty' tty='/dev/pts/2'>
<source path='/dev/pts/2'/>
<target port='0'/>
</console>
</devices>
</domain>

[root@fedora12a OS134]# virsh start OS134
Domain OS134 started

[root@fedora12a OS134]# virsh console OS134
Connected to domain OS134
Escape character is ^]
v4.0.0 chgset ‘unavailable’
SunOS Release 5.11 Version snv_134 64-bit
Copyright 1983-2010 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Hostname: opensolaris134
Reading ZFS config: done.
Mounting ZFS filesystems: (6/6)

opensolaris134 console login: boris
Password:
Last login: Wed Apr 14 13:20:01 on rad/3
Sun Microsystems Inc. SunOS 5.11 snv_134 February 2010
boris@opensolaris134:~$ pfexec ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
xnf0: flags=1004843 mtu 1500 index 2
inet 192.168.1.37 netmask ffffff00 broadcast 192.168.1.255
ether 0:16:3e:10:93:61
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
xnf0: flags=2004841 mtu 1500 index 2
inet6 fe80::216:3eff:fe10:9361/10
ether 0:16:3e:10:93:61





References
1. http://old.nabble.com/-PATCH–RESUBMIT–sync-up-zfs-boot-support-in-pygrub-p28236835.html


Set up OSOL 2010-03 (build 134) PV Guest at Xen 4.0 Dom0 (2.6.32.10 pvops kernel) on top of Ubuntu Karmic Koala Server

April 11, 2010

To perform straight forward install OSOL 134 PV Guest, not using Mark Johnson ZFS 24 Patch for Xen 4.0 , proceed as suggested bellow. As usual copy unix kernel and boot_archive off the disk to Dom0 :

mount -o loop,ro osol-134-dev-x86.iso /mnt
cp /mnt/platform/i86pc/amd64/boot_archive /home/boris/osol134
cp /mnt/platform/i86xpv/kernel/amd64/unix /home/boris/osol134

In my case OSOL 134 PV DomU was able to obtain IP address just once after first boot up configuring SMF . Afterwards it required restarting ( or reenabling) service svc:/network/physical:nwam, what finally brought me to accomplish this procedure as OSOL service , because no analog of /etc/rc.local exists on OSOL or just add /etc/init.d/dhcpdiscover.sh

boris@opensolaris134:/etc/init.d$ cat dhcpdiscover.sh
#!/sbin/sh
# Script
case "$1" in
start)
svcadm restart svc:/network/physical:nwam
;;
restart)
;;
stop)
;;
*)
echo "Usage: $0 {start|stop|restart}"
exit 1
;;
esac
exit 0

and create symlink to run it at start up like on Solaris 10

# cd /etc/rc3.d
# ln -s /etc/init.d/dhcpdiscover.sh /etc/rc3.d/S99dhcpdiscover

Actually, adding script /etc/init.d/dhcpdiscover.sh is quite enough to resolve the issue with broken NWAM. Creating new service, in fact , is not required. It is just a sample of SMF exercise.
Two steps in procedure bellow make Xen default Hypervisor on Ubuntu 9.10. First is commenting out (xend-unix-server yes) in /etc/xen/xend-config.sxp, second in root’s .bashrc.

export VIRSH_DEFAULT_CONNECT_URI=”xen:///”

Then run:-
# apt-get install ubuntu-virt-server ubuntu-virt-mgmt

Create installation XML profile create_osol134.xml as follows :-

<domain type='xen'>
<name>osol134</name>
<uuid>f80856df-3180-acc5-931d-32190cfe4062</uuid>
<memory>1048576</memory>
<currentMemory>1048576</currentMemory>
<vcpu>2</vcpu>
<bootloader></bootloader>
<os>
<type>linux</type>
<kernel>/home/boris/osol134/unix</kernel>
<initrd>/home/boris/osol134/boot_archive</initrd>
<cmdline>/platform/i86xpv/kernel/amd64/unix - nowin -B install_media=cdrom</cmdline>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>destroy</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<disk type='block' device='disk'>
<driver name='phy'/>
<source dev='/dev/sdb5'/>
<target dev='xvda' bus='xen'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='file'/>
<source file='/root/MyDownloads/osol-dev-134.iso'/>
<target dev='xvdc' bus='xen'/>
<readonly/>
</disk>
<interface type='bridge'>
<mac address='00:16:3e:72:da:33'/>
<source bridge='eth0'/>
<script path='/etc/xen/scripts/vif-bridge'/>
<target dev='vif7.0'/>
</interface>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target port='0'/>
</console>
</devices>
</domain>

and run

# virsh create create_osol134.xml
# virsh console osol134
At this point activate VNC connection to DomU via script

domid=`virsh domid osol134`
ip=`/usr/bin/xenstore-read /local/domain/$domid/guest/ipv4/0/address`
echo $ip
port=`/usr/bin/xenstore-read /local/domain/$domid/guest/vnc/port`
echo $port
/usr/bin/xenstore-read /local/domain/$domid/guest/vnc/passwd
vncviewer $ip:$port

and go through normal install. Now define new domain via profile :

<domain type='xen'>
<name>osol34</name>
<uuid>0b9a31cc-13c2-065b-e64d-4ba1a042dedc</uuid>
<memory>1048576</memory>
<currentMemory>1048576</currentMemory>
<vcpu>2</vcpu>
<bootloader></bootloader>
<os>
<type>linux</type>
<kernel>/home/boris/osol134/unix</kernel>
<initrd>/home/boris/osol134/boot_archive</initrd>
<cmdline>/platform/i86xpv/kernel/amd64/unix -B zfs-bootfs=rpool/ROOT/opensolaris,bootpath=/xpvd/xdf@51712:a</cmdline>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>destroy</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<disk type='block' device='disk'>
<driver name='phy'/>
<source dev='/dev/sdb5'/>
<target dev='xvda' bus='xen'/>
</disk>
<interface type='bridge'>
<mac address='00:16:3e:4d:60:e3'/>
<source bridge='eth0'/>
<script path='/etc/xen/scripts/vif-bridge'/>
<target dev='vif8.0'/>
</interface>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target port='0'/>
</console>
</devices>
</domain>

# virsh define osol134-def.xml
# virsh start osol134
# virsh console osol134

In my case after first reboot configuring SMF OSOL 134 stopped obtain IP address via DHCP and required every time commands bellow :-

$ pfexec su -
# svcadm restart svc:/network/physical:nwam
Due /etc/rc.local doesn’t exist on OSOL new service to restart NWAM was
implemented. First i created script /bin/dhcp_again.sh to add as service to SMF

root@opensolaris:~# cat /bin/dhcp_again.sh
svcadm restart svc:/network/physical:nwam

Create file /var/svc/manifest/site/dhcp_again.xml bellow per [1]:

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<!--Script name goes here-->
<service_bundle type='manifest' name='SUNWcsr:dhcp_again'>
<!--Script name goes here too-->
<service
name='site/dhcp_again'
type='service'
version='1'>
<create_default_instance enabled='false' />
<single_instance/>
<!--If your script needs to run after a certain milestone has been met
you can specify that here, otherwise delete this bit.
Replace value for milestone you need to meet-->
<dependency
name='milestone'
grouping='require_all'
restart_on='none'
type='service'>
<service_fmri value='svc:/milestone/network' />
</dependency>
<!--Script to run goes here-->
<exec_method
type='method'
name='start'
exec='/usr/bin/bash /bin/dhcp_again.sh'
timeout_seconds='60' />
<exec_method
type='method'
name='stop'
exec=':kill'
timeout_seconds='60' />
<!--This bit makes it run ONCE and makes sure it is NOT restarted!-->
<property_group name='startd' type='framework'>
<propval name='duration' type='astring' value='transient' />
</property_group>
<template>
<common_name>
<loctext xml:lang='C'>
<!--Script name goes here-->
DHCP_AGAIN
</loctext>
</common_name>
<documentation>
<manpage title='' section=''
manpath='' />
</documentation>
</template>
</service>
</service_bundle>

Next step:-

# cd /var/svc/manifest/site
# svccfg validate dhcp_again.xml
# svccfg import dhcp_again.xml
# svcadm enable dhcp_again
Make sure service has been enabled via command :-
root@opensolaris:~# svcs -a|grep dhcp
online 19:37:20 svc:/site/dhcp_again:default

At this point activate GDM remote login via Xvnc :

root@opensolaris:~# cat /etc/gdm/custom.conf
# Custom Configuration file.
# overrides: /usr/share/gdm/defaults.conf
[daemon]
RemoteGreeter=/usr/lib/gdmgreeter
[security]
DisallowTCP=false
# AllowRoot=true
# AllowRemoteRoot=true
[xdmcp]
Enable=true
[gui]
[greeter]
[chooser]
[debug]
[servers]

Then re-enable services :

svcadm disable xvnc-inetd gdm
svcadm enable xvnc-inetd gdm






Following message at system shutdown is harmless

updating //platform/i86pc/boot_archive
updating //platform/i86pc/amd64/boot_archive
svc.startd: The system is coming down. Please wait.
svc.startd: 95 system services are now being stopped.
Apr 13 21:11:33 opensolaris last message repeated 1 time
Apr 13 21:11:50 opensolaris svc.startd[9]: svc:/site/dhcp_again:default: :kill with no contract
Apr 13 21:11:50 opensolaris svc.startd[9]: site/dhcp_again:default failed: transitioned to maintenance (see 'svcs -xv' for details)
Apr 13 21:11:50 opensolaris syslogd: going down on signal 15
svc.startd: Killing user processes.
Apr 13 21:11:57 The system is down. Shutdown took 9 seconds.
syncing file systems... done

References
1. http://www.hashbang0.com/blog/2010/03/26/making-a-script-run-on-startup-on-opensolaris/


Follow

Get every new post delivered to your Inbox.