Category Archives: Filesystems

Installing OmniOS KVM/QEMU guest onto a Linux host zvol

I originally wanted to install Solaris x86 as a KVM guest, but had problems getting the network to function. The installation itself was easy enough, but all attempts to get networking to work failed. So I decided to try an Illumos distribution, as that would meet the requirements I was looking for.

Since OpenIndiana is the de facto Illumos distribution, I started with that. Unfortunately, it turned out to have the same problem. So I decided to try OmniOS, only to discover it too had the same problem. At this point, I wasn’t sure if it was my Linux KVM that was at fault, since three operating systems failed with the same problem. However, since I have several other operating systems running as KVM guests on Linux and they all work fine, I was still inclined to think it was the distro.

Neither Solaris x86 or OpenIndiana had any new updates, but my OmniOS distro was a little old – I had downloaded the OmniOS_Text_r151012.iso ISO sometime ago. So I checked and found a newer version – OmniOS_Text_r151014.iso. After downloading and installing it, I had a functional network! Based on this, I can only conclude that the reasons Solaris x86 and OpenIndiana failed are due to internal problems with those distros, and not some problem with Linux KVM. Either that or there is some magic KVM setting I’m not familiar with that is needed to get them to work (and I tried a bunch of them).

Installing OmniOS is pretty straight forward. Aside from answering a couple of questions, the installation is totally automatic. The only thing you need to do after installation is manually configure the network. Here are the commands I used to install OmniOS as a KVM guest under Linux.

Create the zvol:

zfs create -p -V 16G rpool/kvm/omnios11/disk0

Create the VM:

qemu-system-x86_64 -enable-kvm -cpu host -m 8192M -drive format=raw,file=/dev/zvol/rpool/kvm/omnios11/disk0 -cdrom OmniOS_Text_r151014.iso -boot d -smp 2

Start the VM:

qemu-system-x86_64 -enable-kvm -cpu host -m 8192M -drive format=raw,file=/dev/zvol/rpool/kvm/omnios11/disk0 -net nic,model=e1000,netdev=net0 -netdev tap,id=net0 -smp 2

Once the VM is started, login as root with no password and configure networking:

ipadm create-if e1000g0
ipadm create-addr -T static -a e1000g0/v4
route -p add default
echo 'nameserver' >> /etc/resolv.conf
cp /etc/nsswitch.dns /etc/nsswitch.conf

You can find more information on setting up OmniOS here:

Installing FreeBSD KVM/QEMU guest onto a Linux host zvol

This configuration is so simple, it’s hardly worth blogging about. But if for no other reason than to keep the commands I used handy, here it is.

Create the zvol:

zfs create -p -V 16G rpool/kvm/freebsd10/disk0

Create the VM:

qemu-system-x86_64 -enable-kvm -m 4096M -drive format=raw,file=/dev/zvol/rpool/kvm/freebsd10/disk0 -cdrom FreeBSD-10.2-RELEASE-amd64-dvd1.iso -boot d

Start the VM:

qemu-system-x86_64 -enable-kvm -m 4096M -drive format=raw,file=/dev/zvol/rpool/kvm/freebsd10/disk0 -netdev user,id=net0 -device e1000,netdev=net0 -smp 2

That’s it!

If you prefer to use bridge networking, use the below command instead:

qemu-system-x86_64 -enable-kvm -m 4096M -drive format=raw,file=/dev/zvol/rpool/kvm/freebsd10/disk0 -net nic,model=e1000,netdev=net0 -netdev tap,id=net0 -smp 2

For more details on bridge networking, see my previous post.

Installing OS X KVM/QEMU guest onto a Linux host zvol

First off, let me start by saying that most of what you read in this post is information I’ve gathered from other sources. Why then have a blog about the subject? Because for some reason, none of the procedures on the other sites completely worked for me. Using the knowledge learned from other sites, along with a few minor tweaks and trial and error, I finally got a sequence that worked for me. This blog will serve as a reminder to me of what I did should I need to do it again and may also help other people who are in the same situation I was in.

In my quest to get an OS X KVM guest under Linux, I visited many, many sites. I can’t be 100% sure that I’ve included all those sites below, but here is a list of sites I’m positive I’ve frequented in order to get OS X running under KVM. 90% of what I present here is a derivative of their work:

I have tested the procedure below with Mountain Lion, Mavericks and Yosemite. For Mavericks and Yosemite, there are extra steps needed to get networking going, but the steps are manageable.

The first problem I encountered was how to create a usable ISO image from the OS X installers obtained through the App Store. Some of the sites claimed you could simply use dd under Linux, while others said you could use the Mac Disk Utility to extract the image. Yet other sites had long sequences of commands and/or scripts to run that would create the ISO for you. There is even a createinstallmedia command on Mavericks and Yosemite. For whatever reason, none of these worked for me. The installation method either failed altogether or presented me with this error when trying to create the VM:

With Mountain Lion or Mavericks:

Can't find /mach_kernel

With Yosemite:

Can't find /System/Library/Kernels/kernel

Since having an ISO image of OS X is step #1, I do not know how other people have gotten OS X to work under KVM by following the directions on the web for creating an ISO. I’m assuming all those people can’t be wrong and I must be the problem, but since I couldn’t get a working ISO using the procedures they describe, I’ve slightly modified those procedures to get a working ISO. To me, it seems the problem is that the bootloader used to bootstrap the system (more on that later) can’t/isn’t looking for the kernel inside of the package (which makes perfect sense to me). To compensate, I’ve extracted the kernel from the package and put them where the bootloader wants them to be.

Note that you will need a Mac in order to create the ISO images as described below. I’m sure there is a way to achieve the same results using Windows or Linux, but since all of the posts I’ve read used a Mac, and I happen to have a Hackintosh that I recently built handy, I just created the ISO on my Hackintosh. If you have another way to create the ISO or already have a working one, then you can skip the ISO creation stuff.

From poking around the OS X installers and reading other posts, I learned that the kernel name and location changed in Yosemite, so that explains why I get two different errors. The next question is why isn’t the kernel present in the OS X Installer? Examining the installer does indeed show there is no /mach_kernel in the Mountain Lion or Mavericks installer, or /System/Library/Kernels/kernel in the Yosemite installer. However, digging a little deeper, I learned that the kernels do exist in a .pkg file inside of the installer. For Mountain Lion and Mavericks, that package is BaseSystemBinaries.pkg. For Yosemite, it’s Essentials.pkg. After downloading the OS X Installer from the App Store, the packages containing the kernel can be found by:

1) Right-clicking the “Install OS X Mountain Lion|Mavericks|Yosemite” icon in Finder
2) Selecting “Show Package Contents”
3) Opening the resulting “Contents” folder
4) Opening the “SharedSupport” folder
5) Double-clicking the “InstallESD.dmg” icon
6) Opening the “Packages” folder

If you install the Pacifist program from, you can then right-click the correct package (BaseSystemBinaries.pkg or Essentials.pkg) and select “Open With->Pacifist”. From there, you can select the “mach_kernel” file for Mountain Lion or Mavericks and hit “Extract To…” to extract the kernel. For Yosemite, you extract “System/Library/Kernels/kernel” instead.

With the kernel now extracted, I figured I could copy it to the correct place needed by the bootloader and get past the above error. To accomplish this, I slightly modified the script that can be found here: Basically, I inserted a line in the script to copy the kernel into the ISO during creation:

cp -rp ~/Desktop/mach_kernel /Volumes/install_build

Here is the script in its entirety, with my one-line change:

# Mount the installer image:
hdiutil attach /Applications/Install\ OS\ X\ Mountain\ -noverify -nobrowse -mountpoint /Volumes/install_app

# Convert the boot image to a sparse bundle:
hdiutil convert /Volumes/install_app/BaseSystem.dmg -format UDSP -o /tmp/mountain-lion

# Increase the sparse bundle capacity for packages, kernel, etc.:
hdiutil resize -size 8g /tmp/mountain-lion.sparseimage

# Mount the sparse bundle target for further processing:
hdiutil attach /tmp/mountain-lion.sparseimage -noverify -nobrowse -mountpoint /Volumes/install_build

# Remove Package link and replace with actual files:
rm /Volumes/install_build/System/Installation/Packages
cp -rp /Volumes/install_app/Packages /Volumes/install_build/System/Installation/

cp -rp ~/Desktop/mach_kernel /Volumes/install_build/

# Unmount both the installer image and the target sparse bundle:
hdiutil detach /Volumes/install_app
hdiutil detach /Volumes/install_build

# Resize the partition in the sparse bundle to remove any free space:
hdiutil resize -size $(hdiutil resize -limits /tmp/mountain-lion.sparseimage | tail -n 1 | awk '{ print $1 }')b /tmp/mountain-lion.sparseimage

# Convert the sparse bundle to ISO/CD master:
hdiutil convert /tmp/mountain-lion.sparseimage -format UDTO -o /tmp/mountain-lion

# Remove the sparse bundle:
rm /tmp/mountain-lion.sparseimage

# Rename the ISO and move it to the desktop:
mv /tmp/mountain-lion.cdr ~/Desktop/mountain-lion.iso

Be sure to have the kernel you extracted in your Desktop directory, or edit the script accordingly.

After running the script, you will have a mountain-lion.iso under Desktop that is suitable for creating a KVM OS X guest. With a few edits around the operating system names, this script also works with Mavericks and Yosemite, but for Yosemite, the extracted kernel name is “kernel” and needs to be copied to /System/Library/Kernels instead:

cp -rp ~/Desktop/kernel /Volumes/install_build/System/Library/Kernels

Note that once created, you can also burn the ISO to a DVD and restore it later using dd:

dd if=/dev/cdrom of=mountain-lion.iso

This will prevent you from having to repeat this procedure should you need to do it again at a later time.

With the ISO creation done, it is now time to create a zvol. You can do that with the following command:

zfs create -p -V 60G rpool/kvm/mountain-lion/disk0

This will create a 60GB zvol named mountain-lion in the rpool zpool, under dataset kvm/mountain-lion. You can of course substitute the names and size as with your own values.

Next, it’s time to create the actual VM. From what I’ve read, you need to be running kernel version 3.15 or later, as well as QEMU 2.2 or later. I’m not sure if this is true, but I heeded the warning and upgraded both my kernel and QEMU versions.

Before continuing, you need a copy of the Chameleon bootloader. You can find it on some of the above links, or download it here:

Chameleon Bootloader

Assuming you have all the necessary tools installed (KVM, QEMU, etc.), creating the VM is as simple as:

qemu-system-x86_64 \
-enable-kvm -m 4096 \
-cpu core2duo \
-machine q35 \
-usb -device usb-kbd \
-device usb-mouse \
-device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" \
-kernel ./chameleon_svn2534_boot \
-smbios type=2 \
-device ide-drive,bus=ide.2,drive=MacHDD \
-drive id=MacHDD,if=none,format=raw,file=/dev/zvol/rpool/kvm/mountain-lion/disk0 \
-net nic,model=e1000,netdev=net0 \
-netdev tap,id=net0 \
-monitor stdio \
-device ide-drive,bus=ide.0,drive=MacDVD \
-drive id=MacDVD,if=none,snapshot=on,file=./mountain-lion.iso
-smp 2

Note that for Yosemite, you must disable msrs first by running this command as root:

echo 1 > /sys/module/kvm/parameters/ignore_msrs

After running the above qemu-system-x86_64 command, QEMU will start a VNC session that you can connect to in order to finish the installation. I used the TightVNC viewer on Linux, and usage couldn’t be simpler:

vncviewer localhost:5900

This will connect the VNC viewer to localhost port 5900, which is the default. Note that if you have another QEMU instance running, you may need to change the port number, as they increase sequentially with each instance.

Once connected, you will be a the Chameleon boot prompt. I found that I needed to enter the boot option “GraphicsEnabler=No” in order to get a successful installation, so you may need to do the same. At the boot: prompt, type:

boot: GraphicsEnabler=No

After that, the Mac OS X installation should start. Note that for some reason, the installation exits the VNC viewer shortly after starting, and so you have to reconnect a second time. Once connected the second time, it may be a while before the installation splash screen appears. If after 5 or so minutes it doesn’t appear, simply exit QEMU and try again.

Once you’ve reached the splash screen, the first thing you will notice is that the virtual VNC cursor doesn’t quite follow the real cursor. This makes it painful to navigate the menus, so learning to use the keyboard, particularly the Enter and Space keys, will be beneficial. Later on you can get around this issue by using the native Mac Screen Sharing utility instead of the QEMU VNC server.

Now you can follow the usual directions for installing OS X. Don’t forget to partition your zvol using the Disk Utility during installation!

Once the installation is finished, OS X will reboot and you will be back to the Chameleon boot prompt, but this time with two options to boot. The first option is the ISO installer, named “OS X Base Installer”, and the second option is the KVM VM, and is named Mountain Lion, Mavericks or Yosemite. Be sure to select the VM when booting, else the installer will start all over again. If that happens, simply reboot again and choose the right option.

Finish the installation steps, and if all goes well, you should now have an OS X KVM guest running under Linux!

If you installed Mountian Lion, you are all set. If you installed Mavericks or Yosemite, there are a couple of more steps you need to do in order to get networking functional. Apparently, the e1000 driver does not work under Mavericks or Yosemite and so you must install the virtio driver which you can find in the above links or download it here:

Virtual IO Driver

Now the problem is getting the driver to the guest when you have no network. Fortunately, QEMU provides a mechanism for sharing a FAT folder between the host and guest. Unfortunately, the procedure didn’t work for me. After hours of debugging, it turned out that my problem was the procedure didn’t work with the version of QEMU I was using – 2.3.94, but did work with 2.0.0. So I fired up 2.0.0 just to get the driver over to the guest. To create a shared FAT folder, create a directory named VirtIoNetDrv and download the virtio driver there. Then start QEMU 2.0.0 using the same command above, but with the following two additions:

-device ide-drive,bus=ide.4,drive=fatdrive
-drive id=fatdrive,file=fat:ro:VirtIoNetDrv

Also, change the e1000 line to virtio:
For Mavericks:
-net nic,model=virtio,netdev=net0
For Yosemite:
-net nic,model=virtio,netdev=net0,vectors=0

When the OS X guest starts, you will have a drive mounted from which you can copy the virtio driver from. After copying it somewhere to the guest OS, restart the newer version of QEMU as originally described (without the fatdrive arguments, but with the virtio driver instead of e1000). Install the virtio driver and configure networking. That’s it! I imagine this QEMU quirk will be worked out in a future version, and when it does, you won’t have to fire up an older version. It’s also possible that perhaps the way shared FAT folders work in the newer QEMU has changed, and so the procedures on the web for sharing folders does not work.

After installation, you can start the VM by using a slightly simpler QEMU command:

qemu-system-x86_64 \
-enable-kvm -m 4096 \
-cpu core2duo \
-machine q35 \
-usb -device usb-kbd \
-device usb-mouse \
-device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" \
-kernel ./chameleon_svn2534_boot \
-smbios type=2 \
-device ide-drive,bus=ide.2,drive=MacHDD \
-drive id=MacHDD,if=none,format=raw,file=/dev/zvol/rpool/kvm/mountain-lion/disk0 \
-monitor stdio \
-net nic,model=e1000,netdev=net0 \
-netdev tap,id=net0 \
-smp 2

For Mavericks and Yosemite, don’t forget to substitute virtio for e1000, and for Yosemite only, don’t forget to add the vectors=0 option.

Note that you do not need to use the “GraphicsEnabler=No” option to boot – you can simply press Enter. To avoid having to press Enter at all, you can tell OS X to skip the bootloader prompt. To do so, simply create the file /Extra/org.chameleon.boot.plist with these contents:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">

After that, OS X will boot automatically without the need to press Enter at the bootloader.

As mentioned earlier,when you use the VNC provided with QEMU, the virtual mouse pointer doesn’t follow the native mouse pointer, which makes it difficult to use. This can be fixed by enabling screen sharing through OS X and using that instead. But to do that, you will first need to setup bridge networking, which you can read about in my next post.

Installing Windows 7 KVM/QEMU guest onto a Linux host zvol

Though Linux is my desktop of choice, some of my programming tasks require me to run other operating systems. Rather than have multiple machines running various operating systems and/or a single machine with several boot partitions, I have in the past opted for virtualizing these other operating systems using VirtualBox under Linux. That is, until recently, when I switched from using VirtualBox to KVM/QEMU. There’s nothing wrong with VirtualBox, but for what I do, KVM made more sense. And since I’m running ZFS under Linux, using a zvol for the operating systems seemed a natural fit.

Creating a Windows 7 KVM/QEMU guest on a zvol under Linux is easy. First, create the zvol:

zfs create -p -V 60G rpool/kvm/windows7/disk0

This will create a 60GB zvol named windows7 in dataset kvm in the zpool named rpool. You can of course substitute the zpool/dataset names as you see fit.

With the zvol created, insert a copy of Windows 7 into your CD drive and create the guest:

qemu-system-x86_64 -enable-kvm -m 8192M -cpu host -drive file=/dev/zvol/rpool/kvm/windows7/disk0 -cdrom /dev/cdrom -boot d -smp 2

With any luck, qemu will start the Windows 7 installation. Just follow the instructions as you would a normal Windows installation.

Once installation is complete, you can use this command to start the Windows 7 guest:

qemu-system-x86_64 -enable-kvm -m 8192M -cpu host -drive file=/dev/zvol/rpool/kvm/windows7/disk0 -netdev user,id=vlan0 -net nic,model=e1000,netdev=vlan0 -smp 2

This will give your Windows 7 VM a network using DHCP, in which you can communicate from the VM to the outside world, but not the other way around. If you wish to be able to access your VM from the outside, you will need to use bridge networking instead, which you can read about in my future post.

That’s all there is to it!

Note that the -cpu host option is critical – if you forget it, the installation will give you the dreaded BSOD with a STOP 0x0000005D error. And to use the -cpu host option, you must also specify the -enable-kvm option.

If you’d prefer to install to an image file rather than a zvol, you can do that simply by creating the image using QEMU and then substituting the image file for the zvol:

qemu-img create -f qcow2 windows7.img 60G

qemu-system-x86_64 -enable-kvm -m 8192M -cpu host -drive file=windows7.img -cdrom /dev/cdrom -boot d -smp 2

Installing Ubuntu 14.04 to ZFS root pool

It is possible to install Ubuntu 14.04 to a native ZFS root pool. This procedure has been documented elsewhere, and this post is basically a rehash of the post at more as a reminder to myself of how to do this again. You may find it useful as well.

install ubuntu-14.04

apt-get install software-properties-common
apt-add-repository --yes ppa:zfs-native/stable
apt-get update
apt-get install ubuntu-zfs
apt-get install zfs-initramfs
modprobe zfs
dmesg | egrep "SPL|ZFS"

echo '# Create by-id links in /dev for zfs vdev. Needed by grub' > /etc/udev/rules.d/90-zfs-vdev.rules
echo 'KERNEL=="sd*[0-9]", IMPORT{parent}=="ID_*", ENV{ID_FS_TYPE}=="zfs_member", SYMLINK+="$env{ID_BUS}-$env{ID_SERIAL}-part%n"' >> /etc/udev/rules.d/90-zfs-vdev.rules

parted /dev/disk/by-id/scsi-35000c5005627e703
(parted) set 9 bios on

parted /dev/disk/by-id/scsi-35000c5005627d36f
(parted) set 9 bios on

zpool create -f -o ashift=12 rpool scsi-35000c5005627e703-part1 scsi-35000c5005627d36f-part1
zpool export rpool
zpool import -d /dev/disk/by-id rpool
udevadm trigger

zfs create rpool/ROOT
zfs create rpool/ROOT/ubuntu-1
mkdir /mnt/tmp
mount --bind / /mnt/tmp
rsync -avP /mnt/tmp/. /rpool/ROOT/ubuntu-1/.
umount /mnt/tmp

cat /dev/null > /rpool/ROOT/ubuntu-1/etc/fstab

vi /rpool/ROOT/ubuntu-1/etc/default/grub
comment out:
  GRUB_CMDLINE_LINUX="boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu-1"

for d in proc sys dev;do mount --bind /$d /rpool/ROOT/ubuntu-1/$d;done
chroot /rpool/ROOT/ubuntu-1/
for d in proc sys dev;do umount /rpool/ROOT/ubuntu-1/$d;done

for d in /dev/sd[a-b];do grub-install --boot-directory=/rpool/ROOT/ubuntu-1/boot $d;done

init 6