Fixing VMware PVSCSI Kernel-Update Problem [RHEL5]

Recently, I worked on a customer problem which involved using the para-virtualisation drivers that VMware ship as part of their guest tools package for Linux operating systems. The VMware package provides a number of kernel modules, the most significant being their pvscsi (block-storage) and vmxnet3 (network) allowing enhanced performance in a virtual environment- rather than emulating SCSI or an ethernet adaptor such as Intel’s e1000- the choice of what to present to the virtual machine can be configured at any time, but if the drivers/modules aren’t available in the guest operating system, the devices cannot be used.

This is where the problem starts- the Red Hat Enterprise Linux (RHEL) 5 kernel does not have these modules available out of the box, therefore typically an installation of RHEL takes place using emulation to get the system up and running, the tools are then installed (which typically install pre-compiled kernel modules) and the system administrator can then present the para-virtualised devices instead. However, when a new kernel is installed the drivers will not automatically be part of the freshly installed kernel therefore upon a reboot the system will not be able to find it’s disks or network devices (depending on which are being used).

It’s not good enough to simply re-install the tools immediately after a kernel-update because the tools will simply install onto the currently running kernel. Whilst it’s possible to manually select a kernel version to install for, the VMware tools expect to build the modules from source (rather than using the pre-compiled versions) which doesn’t really help our situation- we don’t always want compilers and associated libraries on every system. So, we need a solution where the drivers are automatically present in the new kernel, so after a reboot there’s no issue in the system coming up as expected.

The main problem that needs to be addressed is the pvscsi (block-storage) driver- if your root filesystem resides on a pvscsi device then the system will completely fail to boot. If, at least, pvscsi is available- the system can continue to boot as normal, but the rest of the drivers would need to be reinstalled post-reboot, which as explained earlier is no problem as the tools will install for the current kernel (i.e. the one you just updated to). I focussed my efforts on working to ensure that post-kernel installation there would be an automatic installation of the pvscsi kernel module into the new kernel tree.

The solution I came up with is very simple- I looked through the source of the VMware tools install script and found that the code does the following-

1) Checks to see if there are compatible pre-built kernel modules within the tools
2) If so, it places the modules into /lib/modules/<kernel-version>/misc
3) Executes a ‘depmod’ to scan and refresh the module tree
4) Creates a new initrd with the required modules included for boot

As I know that my customer runs RHEL 5, there will always be a compatible pre-built kernel module available within the tree and due to Red Hat’s extensive effort to keep the kABI/API compatibility across the life-cycle of the major version, we can simply copy the existing pre-built kernel module from the existing kernel tree straight over to the new one and manually create an initrd for the new kernel.

During testing of this code, I wrote the following bash script which takes care of ensuring that the pvscsi module is in the new kernel and a new initrd is in-place:

#!/bin/bash

# Get the latest kernel installed
NEW_VERSION=`ls -tA1 /lib/modules | head -n 1`

# Check if the module is already present, if not- continue
if [ ! -e /lib/modules/$NEW_VERSION/misc/pvscsi.ko ]; then

    # Log output to show version we're fixing
	/bin/logger "Fixing VMware Paravirtual Drivers for reboot. Kernel: $NEW_VERSION"

    # Create the misc directory that VMware tools would
	mkdir /lib/modules/$NEW_VERSION/misc > /dev/null 2>&1

    # Find the current running kernel
	OLD_VERSION=`uname -r`

    # Copy the existing module to new kernel tree
	cp /lib/modules/$OLD_VERSION/misc/pvscsi.ko /lib/modules/$NEW_VERSION/misc > /dev/null 2>&1

    # Rebuild our modules cache
	depmod -a $NEW_VERSION > /dev/null 2>&1

    # Create a new initrd, including the new pvscsi module
	mkinitrd -f --with=pvscsi /boot/initrd-$NEW_VERSION.img $NEW_VERSION > /dev/null 2>&1

else
    # This would occur when module is already present
	/bin/logger "VMware Paravirtual Drivers Fix Installed"
fi
exit 0

Whilst this will fix the new kernel to have the freshly installed VMware pvscsi driver present (and therefore allow a normal boot) we need to look at automating this procedure to avoid accidental reboots without the fix being put in place. With RHEL/rpm-based platforms we can use RPM triggers to execute on a particular event. For example, we can create a ‘dummy-rpm’ which has the above script in that executes every time a particular event, such as a package upgrade/install occurs. Therefore, in our example we would need to create an rpm which simply ran the above code every time the kernel package was installed/upgraded. This is done by the tag ‘%triggerin — kernel’ (short for trigger ‘install’) within the rpm spec file.

If we create a very basic rpm spec file such as the following, we can use rpmbuild to build an rpm for us. Once this is installed on the target system it will simply execute the script code every time there’s a modification to the kernel package:

Name:	    fix-vmware
Version:	0.1
Release:	1%{?dist}
Summary:	Fixes pvscsi.ko to ensure it survives kernel-upgrades with reboot
Packager:	Rhys Oxenham <roxenham@redhat.com>

Group:	    Applications/System
License:	GPLv2+
URL:		http://www.rdoxenham.com
BuildArch:	noarch

%description

This package fixes the VMware paravirtual SCSI driver issue in RHEL5, ensuring
that the pvscsi.ko module is available after a reboot and that the
vmware-config-tools.pl script can execute to re-enable everything else.

%triggerin -- kernel

NEW_VERSION=`ls -tA /lib/modules | head -n 1`

if [ ! -e /lib/modules/$NEW_VERSION/misc/pvscsi.ko ]; then
	/bin/logger "Fixing VMware Paravirtual Drivers for reboot. Kernel: $NEW_VERSION"
	mkdir /lib/modules/$NEW_VERSION/misc > /dev/null 2>&1
	OLD_VERSION=`uname -r`
	cp /lib/modules/$OLD_VERSION/misc/pvscsi.ko /lib/modules/$NEW_VERSION/misc > /dev/null 2>&1
	depmod -a $NEW_VERSION > /dev/null 2>&1
	mkinitrd -f --with=pvscsi /boot/initrd-$NEW_VERSION.img $NEW_VERSION > /dev/null 2>&1
else
	/bin/logger "VMware Paravirtual Drivers Fix Installed"
fi
exit 0

* Wed Apr 4 2012 Rhys Oxenham <roxenham@redhat.com> 0.1-1
- Initial release

For those unfamiliar with building rpm’s, the spec file simply lists how the rpm is built and how the rpm reacts to certain events such as install, remove, upgrade etc. It will also allow for clean removals of packages by maintaining a list of all files the rpm controls. To build an rpm from the above spec file, simply copy the text into a file named fix-vmware.spec and use rpmbuild to create the rpm for you-

(root) $ yum install rpm-build
 $ rpmbuild -bb /path/to/fix-vmware.spec

It will then create a non-arch specific RPM for us, either in your home directory or in /usr/src/redhat if you build using root. We can then install this locally onto the system (or alternatively make this rpm part of your standard build process if you include it in a repository):

(root) $ yum localinstall /path/to/fix-vmware-0.1-1.noarch.rpm --nogpgcheck

Note: You may want to consider signing your rpm to get around the ‘–nogpgcheck’ issue.

For those not wanting to build this themselves, I have a pre-built version available here.

Finally, it’s very important that upon the reboot of the system that the full suite of VMware-tools are reinstalled as per your typical requirements (and over-writing the existing pvscsi module that the script described on this page to ensure the module gets updated over time). The tools will ensure that other modules such as vmxnet3 are installed and that the system can continue on with it’s required workloads. A typical way of doing this would be to create an /etc/rc.d/ boot-up script which runs the following line if the modules cannot be found.

vmware-config-tools.pl -d --clobber-kernel-modules=pvscsi

Any questions, please leave a comment or get in contact :-)

 

10 thoughts on “Fixing VMware PVSCSI Kernel-Update Problem [RHEL5]

  1. Great article. Thanks for the thorough explaination on the issue.

    Dale

  2. If your running a PAE kernel and have a non PAE kernel of the same version installed then it will get confused and put the wrong module in place. Of course that was more a mistake on my part as I originally didn’t need a PAE kernel but installed it for some odd reason. Once I fixed that issue it worked great. Here’s a sample code you can add to /etc/rc.d/rc.local.

    lspci | grep -L VMware
    VMWARE_PRESENT=$?
    if [ $VMWARE_PRESENT != 1 ];then
    if [ ! -f /lib/modules/`uname -r`/misc/vmxnet3.ko ]; then
    /usr/bin/vmware-config-tools.pl -d –clobber-kernel-modules=pvscsi
    service network restart
    fi
    fi

  3. Thanks Rhys. I had written a similar script to do the same thing myself and it recently stopped working. I wasn’t doing the depmod after copying the modules. It used to work ok but after a recent update it stopped working. Adding the depmod fixed it.

    Here is my version, which is based on using VMware’s yum repository (http://packages.vmware.com/tools):

    http://pastebin.com/2H6seJPk

  4. line: NEW_VERSION=`ls -tA /lib/modules | head -n 1`
    should be: NEW_VERSION=`ls -tA1 /lib/modules | head -n 1`

    Add “1″ to produce single column output from ls. If you have a lot of kernels it might produce two columns output on some systems…

  5. Hi,

    This is great thanks, helped a lot.

    Just a query on the last section

    vmware-config-tools.pl -d –clobber-kernel-modules=pvscsi

    is it OK to run this if I’ve install VMWare Tools via YUM?

    If OK then could you give instructions on how to create a boot-up script that only runs if the modules cannot be found.

    Thanks.

  6. Hi Ben,

    Glad it helped… yeah it should be fine if you’ve deployed VMware Tools via yum. The difficulty is that the system won’t boot up if the root disk is sitting on a pvscsi disk, therefore some interaction needs to take place *before* the reboot. If you’re not using pvscsi then it’s not so difficult, I’d recommend doing the following-

    Write a script that on-boot, simply runs the vmware-config-tools.pl script and then places a ‘.vmware_done’ empty file in the /lib/modules directory for *that* kernel. The script should only run if that .vmware_done file cannot be found. So, each time you reboot it will only re-run that script if you’ve updated the kernel.

    Give me a shout at rdoxenham@gmail.com if you need further information, would be glad to assist.

    Cheers
    Rhys

  7. Dude, you just saved me a lot of frustration. I have a few ESX 3.5 hosts to uprgade and wanted to use update manager when I ran into this error message. I found 2 VMWare KB articles making things overly complicated. Then found your instructions and bang the iso is uploading.Thanks.

  8. Thanks Dude,
    It works like charm, you saved lots of time and frustration.. good work.
    Enjoy

Leave a Reply

Your email address will not be published. Required fields are marked *


4 × one =

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>