nova file injection的原理和调试过程

file injection代码

file injection原理来讲是比较简单的，在nova boot命令中，有参数--file，是将文件inject到image中

nova boot --flavor 2 --image d96b0e41-8264-41de-8dbb-6b31ce9bfbfc --key-name openstack --security-groups default --file /home/ubuntu/bootfromvolume1.xml=/home/cliu8/images/bootfromvolume.xml testinject19

这里容易混淆的是还有一个参数是user_data，这不是file injection，而是cloud-init使用这里面的东西对系统进行初始化。

在虚拟机启动的时候，会调用到/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py中的spawn函数

def spawn(self, context, instance, image_meta, injected_files,
          admin_password, network_info=None, block_device_info=None):
    disk_info = blockinfo.get_disk_info(CONF.libvirt.virt_type,
                                        instance,
                                        block_device_info,
                                        image_meta)
    self._create_image(context, instance,
                       disk_info['mapping'],
                       network_info=network_info,
                       block_device_info=block_device_info,
                       files=injected_files,
                       admin_pass=admin_password)
    xml = self.to_xml(context, instance, network_info,
                      disk_info, image_meta,
                      block_device_info=block_device_info,
                      write_to_disk=True)

    self._create_domain_and_network(context, xml, instance, network_info,
                                    block_device_info)
    LOG.debug(_("Instance is running"), instance=instance)

其中一步是create image，在_create_image函数中，有下面的逻辑

elif inject_files and CONF.libvirt.inject_partition != -2:
    if booted_from_volume:
        LOG.warn(_('File injection into a boot from volume '
                   'instance is not supported'), instance=instance)
    self._inject_data(
        instance, network_info, admin_pass, files, suffix)

当有文件要inject的时候，并且参数inject_partition不设为-2，则执行file injection过程。

从这里我们可以看出，在参数中应该添加

# cat /etc/nova/nova-compute.conf
[DEFAULT]
compute_driver=libvirt.LibvirtDriver
[libvirt]
virt_type=kvm
inject_partition=-1

在代码中，我们可以看到这个参数是这样解释的：

cfg.IntOpt('inject_partition',
            default=-2,
            help='The partition to inject to : '
                 '-2 => disable, -1 => inspect (libguestfs only), '
                 '0 => not partitioned, >0 => partition number',
           deprecated_name='libvirt_inject_partition',
           deprecated_group='DEFAULT'),

在最新的代码中，我们往往选择用ligguestfs来进行文件注入，因而设为-1

在_inject_data函数中，

disk.inject_data(injection_path,
                 key, net, metadata, admin_pass, files,
                 partition=target_partition,
                 use_cow=CONF.use_cow_images,
                 mandatory=('files',))

(Pdb) p injection_path
'/var/lib/nova/instances/fedc59a9-6ff5-4c23-9595-d877a6796c84/disk'

(Pdb) p target_partition
-1

这里key, net, metadata, admin_pass我们都不注入，所以都为空，只有files有值

这里就调用到/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py的

def inject_data(image, key=None, net=None, metadata=None, admin_password=None,
                files=None, partition=None, use_cow=False, mandatory=()):

在这个函数中首先生成一个对象来操作这个image

fs = vfs.VFS.instance_for_image(image, fmt, partition)

如果能够找到guestfs，则fs为nova.virt.disk.vfs.guestfs.VFSGuestFS

接着调用来初始化环境，该run起来的guestfs的applicance要运行起来，该mount的image要mount上去

fs.setup()

最后进行注入操作

return inject_data_into_fs(fs, key, net, metadata,
                           admin_password, files, mandatory)

在/usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/guestfs.py的setup函数中，

并不是调用guestfish命令，而是直接通过python调用guestfsd的API，可以参考http://libguestfs.org/guestfs.3.html

先要有个handle

self.handle = tpool.Proxy(guestfs.GuestFS(close_on_exit=False))

然后将image作为一个drive添加到appliance

self.handle.add_drive_opts(self.imgfile, format=self.imgfmt)

运行appliance

self.handle.launch()

self.setup_os()

self.handle.aug_init("/", 0)

setup_os的实现很简单，由于我们的参数是-1，则调用setup_os_inspect

def setup_os(self):
    if self.partition == -1:
        self.setup_os_inspect()
    else:
        self.setup_os_static()

在函数setup_os_inspect中

先看有多少个root disk

roots = self.handle.inspect_os()

(Pdb) p roots
['/dev/sda1']

发现只有一个然后调用

self.setup_os_root(roots[0])

在setup_os_root中，先是得到mount point

mounts = self.handle.inspect_get_mountpoints(root)

(Pdb) p mounts
[('/', '/dev/sda1')]

最后self.handle.mount_options("", mount[1], mount[0])

(Pdb) p mount[1]
'/dev/sda1'
(Pdb) p mount[0]
'/'

将/dev/sda1挂载到/下面

至此appliance起来了，image也挂载了, fs.setup()成功，接下来是调用inject_data_into_fs

inject_data_into_fs对不同的注入对象调用不同的函数，对于文件注入调用def _inject_files_into_fs(files, fs):

这里面是一个循环，每个文件都调用def _inject_file_into_fs(fs, path, contents, append=False):

这里面调用了fs.replace_file(path, contents)

其实是写一个文件到一个path

def replace_file(self, path, content):
    LOG.debug(_("Replace file path=%s"), path)
    path = self._canonicalize_path(path)
    self.handle.write(path, content)

(Pdb) p path
u'/home/ubuntu/bootfromvolume1.xml'
(Pdb) p content
"<domain type='kvm'>   <name>bootfromvolume</name>   <uuid>0f0806ab-531d-6134-5def-c5b495529289</uuid>   <memory unit='KiB'>2097152</memory>   <currentMemory unit='KiB'>2097152</currentMemory>   <vcpu placement='static'>1</vcpu>   <os>     <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>     <boot dev='hd'/>   </os>   <features>     <acpi/>     <apic/>     <pae/>   </features>   <clock offset='utc'/>   <on_poweroff>destroy</on_poweroff>   <on_reboot>restart</on_reboot>   <on_crash>restart</on_crash>   <devices>     <emulator>/usr/bin/kvm-spice</emulator>     <disk type='block' device='disk'>       <driver name='qemu' type='raw' cache='none'/>       <source dev='/dev/disk/by-path/ip-16.158.166.197:3260-iscsi-iqn.2010-10.org.openstack:volume-cliu8-test-lun-1'/>       <target dev='vda' bus='virtio'/>       <serial>640a10f7-3965-4a47-9641-002a94526445</serial>       <alias name='virtio-disk0'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>     </disk>     <controller type='usb' index='0'>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>     </controller>     <controller type='pci' index='0' model='pci-root'/>     <controller type='ide' index='0'>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>     </controller>     <interface type='network'>       <mac address='52:54:00:9b:d5:cc'/>       <source network='default'/>       <model type='rtl8139'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>     </interface>        <serial type='pty'>       <target port='0'/>     </serial>     <console type='pty'>       <target type='serial' port='0'/>     </console>     <input type='mouse' bus='ps2'/>     <input type='keyboard' bus='ps2'/>     <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>       <listen type='address' address='0.0.0.0'/>     </graphics>     <video>       <model type='cirrus' vram='9216' heads='1'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>     </video>     <memballoon model='virtio'>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>     </memballoon>   </devices> </domain> "

至此文件注入完毕。

file injection调通过程

上面的过程，理论上很简单，但是真正调通，还是经历了复杂的过程

我的环境是Ubuntu 14.04, apt-get安装的Icehouse

首先是这个错误

2014-07-09 22:50:12.359 7885 WARNING nova.virt.disk.vfs.guestfs [req-b3655a75-1607-4c90-b7cb-cc81e543e769 5df0e89888364c0b80d
47a7a426a9a67 c24c59846a7f44538d958e7548cc74a3] Failed to close augeas aug_close: call launch before using this function
(in guestfish, don't forget to use the 'run' command)
2014-07-09 22:50:12.366 7885 ERROR nova.virt.libvirt.driver [req-b3655a75-1607-4c90-b7cb-cc81e543e769 5df0e89888364c0b80d47a7
a426a9a67 c24c59846a7f44538d958e7548cc74a3] [instance: 7bf0f046-69a7-4c5f-990b-9cbfaf3b0b55] Error injecting data into image
d96b0e41-8264-41de-8dbb-6b31ce9bfbfc (Error mounting /var/lib/nova/instances/7bf0f046-69a7-4c5f-990b-9cbfaf3b0b55/disk with l
ibguestfs (cannot find any suitable libguestfs supermin, fixed or old-style appliance on LIBGUESTFS_PATH (search path: /usr/l
ib/guestfs)))

参考http://libguestfs.org/guestfs-faq.1.html

libguestfs: error: cannot find any suitable libguestfs supermin, fixed or old-style appliance on LIBGUESTFS_PATH

febootstrap-supermin-helper: ext2: parent directory not found

supermin-helper: ext2: parent directory not found

[This issue is fixed permanently in libguestfs ≥ 1.26.]

If you see any of these errors on Debian/Ubuntu, you need to run the following command:

 sudo update-guestfs-appliance

我运行了update-guestfs-appliance，可是还是不管用，那就升级吧

ubuntu 14.04的repository里面的guestfs是1.24，于是我下载了最新的1.26，然后从代码安装

wget http://libguestfs.org/download/1.26-stable/libguestfs-1.26.0.tar.gz

tar xvzf libguestfs-1.26.0.tar.gz

cd libguestfs-1.26.0

apt-get install build-essential

首先用apt-get purge将原来安装的所有的guestfs的package都删除

为了能够编译成功，安装了下面的包

apt-get install libncurses5 libncurses5-dev ocaml ocaml-findlib-wizard ocaml-findlib libfindlib-ocaml libfindlib-ocaml-dev pk e2fslibs e2fslibs-dev gperf flex bison libpcre3 libpcre3-dev libaugeas-dev ncurses-base libcurses-ocaml ncurses-bin ncurses-dev erlang-dev

当然可能还不局限于这些包，在configure, make, make install的过程中，遇到没有的包，就apt-cache search，然后install

在configure libguestfs之前，supermin也需要升级

wget http://libguestfs.org/download/supermin/supermin-5.1.8.tar.gz

tar xvzf supermin-5.1.8.tar.gz

cd supermin-5.1.8

./configure

make

make install

上面的有些包是为安装supermin准备的。

在configure libguestfs之前，要export LIBS="-lncurses"，要不他会找不到tgetent和tgetnum

而且configure的是，会对qemu的版本做检查

在configure.ac中

772 #        AC_MSG_CHECKING([for $QEMU version >= 1])
773 #        if $QEMU -version | grep -sq 'version @<:@1-@:>@'; then
774 #            AC_MSG_RESULT([yes])
775 #        else
776 #            AC_MSG_RESULT([no])
777 #            AC_MSG_FAILURE([$QEMU version must be >= 1.0.])
778 #        fi

这里要求qemu的版本一定是1.x

很不幸

# qemu-system-x86_64 --version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1), Copyright (c) 2003-2008 Fabrice Bellard

我的是2.x，反而报错误说我的版本老了，没办法，只好注释掉

修改了configure.ac，于是需要安装apt-get install autoconf

运行autoconf重新生成configure文件

下面configure libguestfs，需要指定qemu

./configure --with-qemu="qemu-system-x86_64"

中间有任何的错误都必须google，然后让他过，不能忽略，因为libguestfs的工具和包太多，说不定哪个不起作用，都会影响文件注入

make

make install

中间有.so, .a冲突，要把原来的弄掉

cd /

find . -name "dllmlguestfs.so"

cd /usr/lib/ocaml/stublibs/

mv dllmlguestfs.so dllgraphics.so.back

重新make install，直到正常成功结束。

安装成功了，试一把

virt-ls -a ubuntutest.img /home/openstack

似乎成功

guestfish -a ubuntutest.img

里面运行run

似乎也成功

就基本可以了

接下来创建虚拟机，带文件注入

可还是报这个错误

2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1355, in _build_instance
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     filter_properties, bdms, legacy_bdm_in_spec)
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1401, in _reschedule_or_error
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     self._log_original_error(exc_info, instance_uuid)
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     six.reraise(self.type_, self.value, self.tb)
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1396, in _reschedule_or_error
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     bdms, requested_networks)
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2125, in _shutdown_instance
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     requested_networks)
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     six.reraise(self.type_, self.value, self.tb)
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2115, in _shutdown_instance
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     instance, network_info, admin_pass, files, suffix)
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2539, in _inject_data
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     instance=instance)
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     six.reraise(self.type_, self.value, self.tb)
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2533, in _inject_data
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     mandatory=('files',))
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 353, in inject_data
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     fs.setup()
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]   File  /usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/guestfs.py", line 132, in setup
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672]     {'imgfile': self.imgfile, 'e': e})
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] NovaException: Error mounting /var/lib/nova/instances/e92f42dd-4164-4dff-a6a7-d42bd04be672/disk with libguestfs (/usr/local/bin/supermin exited w
ith error status 1.
2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] To see full error messages you may need to enable debugging.

为什么，不明觉厉

用pdb debug一把(在root下)

python -m pdb nova-pdb-compute.py --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf

# cat nova-pdb-compute.py
#! /usr/bin/python
# PBR Generated from u'console_scripts'

import sys
import pdb

from nova.cmd.compute import main

if __name__ == "__main__":
    sys.exit(main())

就是从/usr/bin/nova-compute拷贝过来，前面加上import pdb

单步运行，竟然通过了，想不出什么区别

突然看到

# ps aux | grep nova-compute
root     22745  0.0  0.0  10464   928 pts/3    S+   19:37   0:00 grep --color=auto nova-compute
nova     29488  1.0  0.1 1761032 75100 ?       Ssl  17:25   1:20 /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf

service nova-compute start启动的进程不是以root运行的，而是以nova用户运行的

于是service nova-compute stop

然后在root下运行

/usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf > nova-compute.log 2>&1 &

果然成功了。

可见普通用户运行guestfish还是有问题的，上面的所有操作都是在root下，于是切换到普通用户

cliu8:~/images$ guestfish -a ubuntutest.img

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> run
libguestfs: warning: current user is not a member of the KVM group (group ID 112). This user cannot access /dev/kvm, so libguestfs may run very slowly. It is recommended that you 'chmod 0666 /dev/kvm' or add the current user to the KVM group (you might need to log out and log in again).
libguestfs: error: /usr/local/bin/supermin exited with error status 1.
To see full error messages you may need to enable debugging.
See http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs

哈哈，在普通用户下，果然run不起appliance来

usermod -G kvm -a nova
usermod -G kvm -a cliu8
usermod -G root -a cliu8  
usermod -G root -a nova

将普通用户和nova都添加到kvm, root的group

打开debug，再运行guestfish

cliu8:~/images$ export LIBGUESTFS_DEBUG=1
cliu8:~/images$ export LIBGUESTFS_TRACE=1
cliu8:~/images$ guestfish -a ubuntutest.img
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: create: flags = 0, handle = 0x196ba50, program = guestfish
libguestfs: trace: set_pgroup true
libguestfs: trace: set_pgroup = 0
libguestfs: trace: add_drive "ubuntutest.img"
libguestfs: trace: add_drive = 0

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> run
libguestfs: trace: launch
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: trace: version
libguestfs: trace: version = <struct guestfs_version *>
libguestfs: trace: get_backend
libguestfs: trace: get_backend = "direct"
libguestfs: launch: program=guestfish
libguestfs: launch: version=1.26.0
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=direct
libguestfs: launch: tmpdir=/tmp/libguestfsE2FajZ
libguestfs: launch: umask=0002
libguestfs: launch: euid=1004
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: [00000ms] begin building supermin appliance
libguestfs: [00000ms] run supermin
libguestfs: command: run: /usr/local/bin/supermin
libguestfs: command: run: --build
libguestfs: command: run: --verbose
libguestfs: command: run: --if-newer
libguestfs: command: run: --lock /var/tmp/.guestfs-1004/lock
libguestfs: command: run: --copy-kernel
libguestfs: command: run: -f ext2
libguestfs: command: run: --host-cpu x86_64
libguestfs: command: run: /usr/local/lib/guestfs/supermin.d
libguestfs: command: run: -o /var/tmp/.guestfs-1004/appliance.d
supermin: version: 5.1.8
supermin: package handler: debian/dpkg
supermin: acquiring lock on /var/tmp/.guestfs-1004/lock
supermin: build: /usr/local/lib/guestfs/supermin.d
supermin: build: visiting /usr/local/lib/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/local/lib/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/local/lib/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/local/lib/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/local/lib/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/local/lib/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/local/lib/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: build: 161 packages, including dependencies
supermin: build: 8030 files
supermin: build: 8030 files, after removing unreadable files
supermin: build: 5362 files, after matching excludefiles
supermin: build: 5369 files, after adding hostfiles
supermin: build: 5372 files, after munging
supermin: kernel: picked kernel vmlinuz-3.13.0-27-generic
cp: cannot open '/boot/vmlinuz-3.13.0-27-generic' for reading: Permission denied
supermin: cp -p '/boot/vmlinuz-3.13.0-27-generic' '/var/tmp/.guestfs-1004/appliance.d.383nplf0/kernel': command failed, see earlier errors
libguestfs: error: /usr/local/bin/supermin exited with error status 1, see debug messages above
libguestfs: trace: launch = -1 (error)
><fs> quit

竟然还是不行

最后终于发现

http://libguestfs.org/guestfs-faq.1.html

DOWNLOADING, INSTALLING, COMPILING LIBGUESTFS

Debian and Ubuntu After installing libguestfs you need to do:

 sudo update-guestfs-appliance

On Ubuntu only:

 sudo chmod 0644 /boot/vmlinuz*

You may need to add yourself to the kvm group:

 sudo usermod -a -G kvm yourlogin

真是不看文档，吃亏在眼前啊

赶紧

chmod 0644 /boot/vmlinuz*

然后运行guestfish -a ubuntutest.img，里面run，哈哈成功了

于是可以再nova用户下运行nova-compute了

service nova-compute start

再order一个机器，带文件注入的

成了

# ip netns exec qrouter-26a45e0e-a58a-443b-a972-d62c0c5a1323 ssh -i openstack.pem ubuntu@192.168.0.33
The authenticity of host '192.168.0.33 (192.168.0.33)' can't be established.
ECDSA key fingerprint is 20:f6:58:a9:c0:ab:54:7e:5f:3b:1f:3c:75:c6:36:26.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.33' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.2.0-64-virtual x86_64)

* Documentation:  https://help.ubuntu.com/

  System information as of Thu Jul 10 11:48:11 UTC 2014

  System load:  0.0               Processes:           59
  Usage of /:   4.0% of 19.68GB   Users logged in:     0
  Memory usage: 3%                IP address for eth0: 192.168.0.33
  Swap usage:   0%

  Graph this data and manage this system at:
https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@testinject15:~$ ls
bootfromvolume1.xml