Nova libvirt-xen driver fails to schedule instance under Xen 4.4.1 Hypervisor with libxl toolstack

April 13, 2015

UPDATE as of 16/04/2015
For now http://www.slideshare.net/xen_com_mgr/openstack-xenfinal
is supposed to work only with nova networking per Anthony PERARD
  Neutron appears to be an issue.
  Please, view details of troubleshooting and diagnostic obtained (thanks to Ian   Campbell)
http://lists.xen.org/archives/html/xen-devel/2015-04/msg01856.html
END UPDATE

This post is written in regards of two publications done in February 2015
First:   http://wiki.xen.org/wiki/OpenStack_via_DevStack
Second : http://www.slideshare.net/xen_com_mgr/openstack-xenfinal

Both of them are devoted to same problem nova libvirt-xen driver. Second one states that everything is supposed to be fine as far as some mysterious patch will merge mainline libvirt .Both of them don’t work for me generating errors  in  libxl-driver.log even with  libvirt 1.2.14 ( the most recent version as of time of writing).

For better understanding problem been raised up view also https://ask.openstack.org/en/question/64942/nova-libvirt-xen-driver-and-patch-feb-2015-in-upstream-libvirt/

I’ve followed more accurately written second one :-

On Ubuntu 14.04.2

# apt-get update
# apt-get -y upgrade
# apt-get install xen-hypervisor-4.4-amd64
# sudo reboot

$ git clone https://git.openstack.org/openstack-dev/devstack

Created local.conf under devstack folder as follows :-

[[local|localrc]]
HOST_IP=192.168.1.57
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50

FLOATING_RANGE=192.168.10.0/24
FLAT_INTERFACE=eth0
Q_FLOATING_ALLOCATION_POOL=start=192.168.10.150,end=192.168.10.254
PUBLIC_NETWORK_GATEWAY=192.168.10.15

# Useful logging options for debugging:
DEST=/opt/stack
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen

# The default fixed range (10.0.0.0/24) conflicted with an address
# range I was using locally.
FIXED_RANGE=10.254.1.0/24
NETWORK_GATEWAY=10.254.1.1

# Services
disable_service n-net
enable_service n-cauth
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service horizon
disable_service tempest

# This is a Xen Project host:
LIBVIRT_TYPE=xen

Ran ./stack.sh and successfully completed installation, versions of libvirt 1.2.2,1.2.9.1.2.24 have been tested. The first one is default on Trusty, 1.2.9 && 1.2.14 have been built and installed after stack.sh completion. For every version of libvirt been tested new hardware instance of Ubuntu 14.04.2 has been created.

Manual libvirt upgrade was done via :-

# apt-get build-dep libvirt
# tar xvzf libvirt-1.2.14.tar.gz -C /usr/src
# cd /usr/src/libvirt-1.2.14
# ./configure –prefix=/usr/
# make
# make install
# service libvirt-bin restart

root@ubuntu-system:~# virsh –connect xen:///
Welcome to virsh, the virtualization interactive terminal.

Type: ‘help’ for help with commands
‘quit’ to quit

virsh # version
Compiled against library: libvirt 1.2.14
Using library: libvirt 1.2.14
Using API: Xen 1.2.14
Running hypervisor: Xen 4.4.0

Per page 19 of second post

xen.gz command line tuned
ubuntu@ubuntu-system:~/devstack$ nova image-meta cirros-0.3.2-x86_64-uec set vm_mode=HVM
ubuntu@ubuntu-system:~/devstack$ nova image-meta cirros-0.3.2-x86_64-uec delete vm_mode

Attempt to launch instance ( nova-compute is up ) error “No available host found” in n-sch.log from Nova side

The libxl-driver.log reports :-

root@ubuntu-system:/var/log/libvirt/libxl# ls -l
total 32
-rw-r–r– 1 root root 30700 Apr 12 03:47 libxl-driver.log

**************************************************************************************

libxl: debug: libxl_dm.c:1320:libxl__spawn_local_dm: Spawning device-model /usr/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: /usr/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: -xen-domid
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: 2
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: -chardev
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-2,server,nowait
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: -mon
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: -nodefaults
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: -xen-attach
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: -name
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: instance-00000002
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: -vnc
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: 127.0.0.1:1
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: -display
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: none
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: -k
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: en-us
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: -machine
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: xenpv
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: -m
libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: 513
libxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch w=0x7f36cc001990 wpath=/local/domain/0/device-model/2/state token=3/3: register slotnum=3
libxl: debug: libxl_create.c:1356:do_domain_create: ao 0x7f36cc0012e0: inprogress: poller=0x7f36d8013130, flags=i
libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x7f36cc001990 wpath=/local/domain/0/device-model/2/state token=3/3: event epath=/local/domain/0/device-model/2/state
libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x7f36cc001990 wpath=/local/domain/0/device-model/2/state token=3/3: event epath=/local/domain/0/device-model/2/state
libxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch w=0x7f36cc001990 wpath=/local/domain/0/device-model/2/state token=3/3: deregister slotnum=3
libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x7f36cc001990: deregister unregistered
libxl: debug: libxl_qmp.c:696:libxl__qmp_initialize: connected to /var/run/xen/qmp-libxl-2
libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: qmp
libxl: debug: libxl_qmp.c:546:qmp_send_prepare: next qmp command: ‘{
“execute”: “qmp_capabilities”,
“id”: 1
}

libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:546:qmp_send_prepare: next qmp command: ‘{
“execute”: “query-chardev”,
“id”: 2
}

libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:546:qmp_send_prepare: next qmp command: ‘{
“execute”: “query-vnc”,
“id”: 3
}

libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return
libxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch w=0x7f36f284b3e8 wpath=/local/domain/0/backend/vif/2/0/state token=3/4: register slotnum=3
libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x7f36f284b3e8 wpath=/local/domain/0/backend/vif/2/0/state token=3/4: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:657:devstate_watch_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x7f36f284b3e8 wpath=/local/domain/0/backend/vif/2/0/state token=3/4: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:653:devstate_watch_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 2 ok
libxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch w=0x7f36f284b3e8 wpath=/local/domain/0/backend/vif/2/0/state token=3/4: deregister slotnum=3
libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x7f36f284b3e8: deregister unregistered
libxl: debug: libxl_device.c:1023:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge online
libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x7f36f284b470: deregister unregistered
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [-1] exited with error status 1
libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: ip link set vif2.0 name tap5600079c-9e failed
libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x7f36f284b470: deregister unregistered
libxl: error: libxl_create.c:1226:domcreate_attach_vtpms: unable to add nic devices

libxl: debug: libxl_dm.c:1495:kill_device_model: Device Model signaled

 


Qemu-kvm 0.15.0 stable and Spice-Gtk-0.7 on Ubuntu 11.10

August 15, 2011

All packages required are already in PPA “Qemu-kvm 0.15.0 and Spice on Ubuntu 11.10″. Spice-gtk-0.7 has been done for Oneiric as two builds. First one with python, gobject-introspection enabled and –with-gtk=”2.0″(GTK2). Second one –with-introspection and –with-gtk=”3.0″ (GTK3).This builds actually are the core builds in PPA, which bring RH’s technologies to Ubuntu 11.10. Notice, that option –with-audio=”pulse” is also used by both builds and appears to be working fine. Version of Spice Server has been
also upgraded to 0.8.2.
Following bellow is brief description of PPA Qemu-kvm 0.15.0 and Spice on Ubuntu 11.10.
************************************************************************************************************
UPDATE on 08/18/11 I am aware of following builds made at Debian :-
Details about package spice-gtk
Packaging schema at link mentioned above is exactly the same as in spice-gtk-0.7.1-1.fc16.src.rpm
View Rebuild Spice-Gtk-0.7 via Debian Unstable with gobject-introspection enabled
*************************************************************************************************************

1.The most recent qemu-kvm packages were done via “git clone git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git” are Release 0.15.0 as of 08/10/2011.
$ git clone git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
$ cd qemu-kvm
$ git checkout -b stable-0.15 origin/stable-0.15
$ cd ..
Copy to building directory :
$ cp -R qemu-kvm ../BUILD/qemu-kvm-mmddyy
Stable branch is already patched to fix “QXL” issues and to support spicevmc channel

2. Spice-gtk-0.7 has been done for Oneiric as two builds.

GTK2 build performed with
./configure --prefix=/usr --build=x86_64-linux-gnu \
--mandir=/usr/share/man --infodir=/usr/share/info \
--datadir=/usr/share \
--sysconfdir=/etc --localstatedir=/var \
--libexecdir=/usr/bin \
--with-python --disable-static \
--enable-introspection \
--enable-smartcard=no \
--with-gtk="2.0" --with-audio="pulse"

GTK3 build performed with
./configure --prefix=/usr --build=x86_64-linux-gnu \
--mandir=/usr/share/man --infodir=/usr/share/info \
--datadir=/usr/share \
--sysconfdir=/etc --localstatedir=/var \
--libexecdir=/usr/bin \
--disable-static \
--enable-introspection \
--with-gtk="3.0" --enable-smartcard=no --with-audio="pulse"

libspice-protocol-dev 0.8.1-1 is required by both builds

3.Current version of spice-vdagent for Oneiric as KVM guest installs service and *.desktop (file) automatically and requires guest restart for activation or manual gdm restart and starting service.

***********************************************************************
UPDATE on 08/16/11. Please, view https://wiki.ubuntu.com/spice ,
regarding issues raised up on 08/16/11. You might have to import
PUBLIC KEY separately to install PPA.

Would you have problem with adding my PPAs , like this :-
Executing: gpg –ignore-time-conflict –no-options –no-default-keyring –secret-keyring /tmp/tmp.CuZyZy4MSK
–trustdb-name /etc/apt/trustdb.gpg –keyring /etc/apt/trusted.gpg –primary-keyring /etc/apt/trusted.gpg –keyserver hkp://keyserver.ubuntu.com:80/ –recv 5
gpg: “5” not a key ID: skipping
Run “sudo apt-get update” . It will show you value of PUBLIC KEY to be imported . Then run :
$ sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys
$ sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys \
XXXXXXXXXXXXXXX

Where XXXXXXXXXXXXXX is a value been shown by “apt-get update” on your box
$ sudo apt-get update
***********************************************************************
Install software from PPA :-
************************************************************************
$ sudo apt-add-repository ppa:bderzhavets/qemu-git
$ sudo apt-get update
$ sudo apt-get install qemu-kvm qemu qemu-common \
libspice-client libspice-client-dev \
libspicegtk3-client libspicegtk3-client-dev \
spice libspice-protocol-dev spicegtk3-client

$ sudo apt-get install virtinst virt-viewer virt-manager

Disable libvirtd’s apparmor profile :

$ sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/
$ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
$ sudo adduser $USER kvm
$ sudo reboot

The most recent daily builds of Ubuntu 11.10 don’t contain Synaptic Manager. To install
$ sudo apt-get install synaptic