The week before last I attended the OpenStack Summit in Portland, Oregon. It was a chance for me to get the latest information about one of the most exciting open-source projects for the past few years. I’ve been involved with OpenStack for quite a few months now, mainly deploying environments and writing documentation to aid in my understanding as well as providing it out for others to consume; one of the problems with OpenStack is that it’s very difficult to get started. Whilst I knew that OpenStack had an enormous market-hype with the vast majority of ISVs and hardware vendors jumping on-board, nothing could prepare me for the overwhelming turnout at the summit; the event had actually sold-out, which for an Open Source conference is an achievement. The majority of the sessions were actually overflowing, if you didn’t turn up 20-30 minutes beforehand then you had little chance of getting somewhere to sit. It just goes to show the level of interest in this project, people from all over the world and of all career paths attended as they were either actively involved with OpenStack in some way or knew they had to learn more; the event had a very refreshing buzz about it, people wanted to be there, were passionate about the technology and could really see it going somewhere.

The conference, with the exception of the daily keynotes, was split up into multiple tracks, for the active contributors/developers they had the design summit, but for those of us that wanted to gain an insight into the latest and greatest we found ourselves sitting through presentations covering the widest variety of topics. They also provided hands-on workshops all week, something that I found extremely valuable- for example, as I’m not a networking guy I sometimes find myself confused over complex networking and the concept of virtual networks, the hands-on Quantum lab enabled me to gain a good understanding of it. Even beginners were catered for, there were 101 sessions most days allowing people with very little experience of virtualisation and cloud computing to come away with an understanding of the OpenStack project and where it is headed. As a Red Hat employee myself, putting faces to the names of the colleagues I work with daily was a great opportunity, especially given our recent OpenStack distribution announcement; RDO, our community-supported offering (http://openstack.redhat.com). The ability to network with other OpenStack users (and potential future users) was extremely valuable, receiving feedback about what they wanted to use it for and what features they really wanted to see. In fact, when you look at the attendee list it goes to show the variety of attendees; it seemed to be a mix between the stereotypical LinuxCon attendees and VMworld attendees, a very dynamic environment.

The keynote sessions were extremely useful, the Rackspace keynote being the headline for many was, as expected, really good. The statistic which they keep using is that whilst they’re not reducing the amount of contributions they make, their overall commit percentage continues to decrease; clearly proving the success of the project in the open-source community. It was also good to see that Red Hat is out on top now with the latest Grizzly release and they’re not trying to keep that quiet! Rackspace is using OpenStack in production (no surprises there!) but the way in which they use it is very interesting, they deploy OpenStack on OpenStack for providing “private clouds” on-top of the public cloud, eating their own dog-food at every level. They’ve done this with an extremely high level of reliability and API uptime, so when people say that OpenStack isn’t ready for production, I really do beg to differ. Canonical’s keynote by Mark Shuttleworth was very good too, I think they’re at a set of crossroads, moving from the traditional ‘desktop environment’ to a more strategic cloud play, I’m just not sure they’ve got the resources to actually fulfill what they try and say, despite the fact that OpenStack has been predominantly written to run on Ubuntu (historically, anyway). Aside from the OpenStack “vendors”, organisations such as Best Buy, Bloomberg, Comcast and others presented what they use OpenStack for, how they’ve implemented it at scale and what they’ve learned (and contributed back!), it further goes to show that there’s real interest in many different areas if the use case is right, i.e. scale out, fault tolerant applications.

What interests me is that OpenStack is clearly viewed by many as a threat, you take VMware for example; they are actively contributing to the project to enable their ESXi hypervisor as a compute resource for OpenStack Nova, they’re concerned that when people see the benefits of OpenStack and the fact that it abstracts much of the underlying compute resource, the requirement for ESX will drop. VMware are at risk of becoming irrelevant in the long term where applications are written in different ways, i.e. to be more fault tolerant and not requiring hypervisor-oriented technologies such as HA, they have to try and retain a piece of the pie and leverage the existing investments that organisations have made on their technology. In addition, the next-step of the virtualisation piece is virtual-networking; Quantum provides a virtual network abstraction service with multiple plugins for software and hardware based networking stacks. VMware is active in this space also, providing their Nicira-based NVP plugin for Quantum, this is an emerging technology and will likely be the later piece of the puzzle that gets adopted. VMware and Canonical have just gone to market with a fully-supported offering, this joint venture provides a complete OpenStack environment for customers currently using VMware. As VMware didn’t previously have an operating-system, Ubuntu is able to step in and plug this gap, a potentially strategic opportunity for both organisations.

It’s not just VMware either, the likes of HP, Dell, IBM are all jumping on the OpenStack bandwagon, all with sets of developers contributing upstream as well as to their own product offerings, seeing OpenStack as a way of making money. Many vendors are providing additional components that allow OpenStack to integrate directly into existing environments, bridging the gap between upstream vanilla OpenStack and the modern-day datacenter. The VMware/Canonical offering is too early to tell whether it will be a success or not, but in my opinion they are doing the right thing, I am a firm believer that any vendor looking to provide an OpenStack offering should be open about the partnerships they forge and the additional fringe components that they choose to support; because of the vast support model in the OpenStack trunk it will attract a wide variety of customers with a disprate set of requirements, picking and choosing technologies to support will be extremely important.

OpenStack has opened the doors to a wide variety of new-startups, all providing additional layers or extensions to help integrate and ease the adoption of the product into organisations. Examples of these include Mirantis and SwiftStack. Mirantis is a company based out of Russia, they, more than anyone, impressed me at the summit. They’ve written a tool called Fuel which aims to provide a ground-up management platform for OpenStack; currently to implement OpenStack requires quite a lot of time, knowledge and experience, Fuel is able to configure an array of the underlying bare-metal technologies including networking and storage as well as to provision and manage entire OpenStack environments from a web-interface. If I was in a position to acquire a company, Mirantis would be at the top of my list right now! SwiftStack provide management tools and support for deploying a Swift-based object-storage cloud, their presentations were fascinating and for anyone interested in how Swift works then they’ve written a free book (http://www.swiftstack.com/book/) which I highly recommend. The exhibition room was full of vendors promoting either their distributions, their consultancy/architecture services or their additional add-on components, very rare for an open-source project… even at LinuxCon it’s usually full of the normal open-source companies plus hardware guys, this represented a huge mix.

What last week made me realise is that there’s an enormous opportunity for OpenStack, a lot of the community work has been done for the vendors wishing to pursue an OpenStack strategy, some vendors being in better positions than others to make it a successful venture; integration with existing enterprise technology will be extremely important for vendors to get right. There’s a lot of overlap between the capabilities of some existing products in the industry, however OpenStack, in my opinion, is addressing the problem to the next-generation architectures. There are lots of offerings/flavours/distributions out there, the most important thing about OpenStack-based clouds is interoperability… they must continue to be open, i.e. open API’s, open standards to allow portability between clouds. Whilst I fear that some organisations (especially proprietary vendors) are getting involved in OpenStack because of the hype, it represents a turning point in the way that big corporations think- it’s yet again proving the power of Open Source and what can be achieved when we work together.

Long term, there are many areas in which we can improve OpenStack, there aren’t many organisations out there that are ready to implement an OpenStack environment unless they go greenfield with it; integration is key but the switch from traditional data centers to a fully software-defined environment is a big step to take and this step will take years to fully embrace. It doesn’t mean that traditional enterprise virtualisation will go away either, there will always be a requirement for legacy applications but the convergence of these technologies will be an intriguing concept to watch. Over time I think that OpenStack will become a lot more than just a set of tools to build cloud environments; we see this with the latest tool sets such as the Heat API and nova-baremetal, tools that are implementing features that have been traditionally excluded. It’s an exciting time to be involved with OpenStack, I look forward to seeing what the future can bring for the platform and making it a success.

As I attended about 10 sessions per day (usually 30-40 minutes each), I won’t comment on them all, but some personal favourites to recommend people look into if they’re interested in learning more about what’s coming up in OpenStack:

Orchestration of Fibre Channel in Cinder – The default out of the box block storage configuration in OpenStack is typically iSCSI, whilst there’s plenty of additional options now Fibre Channel support has taken quite some time to make it into the code. The problems are typically around zoning, as far as Nova is concerned this is almost irrelevant as all it has to do is attach the underlying disk to the instance in the same fashion as iSCSI. This technology brings OpenStack closer to enterprise adoption. (http://www.openstack.org/summit/portland-2013/session-videos/presentation/orchestration-of-fibre-channel-technologies-for-private-cloud-deployments)

Ceilometer Metrics -> Metering – So, Ceilometer is a new project within OpenStack, it was introduced in Folsom as an incubated project but has made it into Grizzly as a full component. It enables organisations to implement a flexible chargeback model on pretty much anything they want to plug it into. It vastly extends the very basic quotas and utilisation that Folsom used to provide. There are still some limitations but it’s extremely powerful. (http://www.openstack.org/summit/portland-2013/session-videos/presentation/ceilometer-from-metering-to-metrics)

OpenStack HA with Mirantis – This talk actually turned into a product demonstration/pitch, but it was the one I was most impressed by. This demonstrates Mirantis’ Fuel implementation for managing OpenStack environments from bare-metal. Many organisations keep asking “how can we deploy OpenStack to scale?” or “How can we make OpenStack Highly Available?”. Mirantis attempts to solve the deployment and high availability problems with their tools and ammusingly they say it’s so easy, even a goat can do it! (http://www.openstack.org/summit/portland-2013/session-videos/presentation/standup-ha-openstack-with-open-puppet-manifests-in-under-20-minutes-for-goat)

Deploying and Managing OpenStack with Heat – This talk discusses how the “triple-o” or “OpenStack-on-OpenStack” project uses the Heat API (and nova-baremetal) to deploy entire OpenStack environments automatically, blurring the differences between the cloud layer and the physical world. (http://www.openstack.org/summit/portland-2013/session-videos/presentation/deploying-and-managing-openstack-with-heat)

Software Defined Networking (Scaling in the Cloud) – One of the things I mentioned earlier was not being a networking guy, these sorts of presentations helped me understand how things fit in, where things were going and how virtual networking was solving real world problems with scale. Gone are the days where we provide L2 bridges (+ VLAN tagging) to virtual machines, in the world of software defined networking we can remove a lot of underlying complexity and control it all in software. This is an area that will become extremely important in the future. (http://www.openstack.org/summit/portland-2013/session-videos/presentation/scaling-in-the-cloud-the-hype-and-happenings-of-software-defined-networking)

Note, ALL of the Summit videos are freely available online at: http://www.openstack.org/summit/portland-2013/session-videos/

Cheers,
Rhys

What currently doesn’t work? Well, Thunderbolt hotplug currently isn’t supported; Mac OSX seems to use some form of magic to enable hotplugging and to control the thunderbolt controllers. Whilst the Linux kernel supports the thunderbolt controllers, it only ever works if the adapter is plugged in before the system is booted, I’ve successfully used a Thunderbolt Ethernet adapter. I wouldn’t expect thunderbolt hotplug anytime soon either, which is a shame. There’s also currently a bug with the mini-displayport -> VGA, although HDMI works out of the box. I’ll work on the VGA adapter problem and will update this post if/when it’s working.

Step One (Wireless): The MacBook Pro ships with a Broadcom BCM4331 wireless adapter, there’s an open-source driver available in the kernel (b43) but it requires additional proprietary firmware from Broadcom to use it successfully. I had no end of trouble with this driver, it didn’t support 5GHz/N networks and constantly dropped connectivity – completely unreliable. 

The solution that worked for me and has been rock-solid so far is the proprietary driver direct from Broadcom (wl), it has open-source code to support the driver and provide a standard interface to the binary driver. Broadcom ship this package on their website (http://www.broadcom.com/support/802.11/linux_sta.php) but for reasons unknown to me do not ship the latest code, in addition, via the rpmfusion repositories it’s available as a package for Fedora. The guys over at Ubuntu managed to get access to the latest packages and I was able to build it successfully on Fedora (after reading lots of threads!). The downside is that the compiled module is inserted into each kernel manually after its built and therefore it needs to be built each time there’s a kernel update. I would rather do this and have stable WiFi than use the b43 driver though; perhaps in the future either the b43 or wl driver will be stable enough upstream to use without manual compilation.

Below I’ve outlined the necessary steps for getting the driver working on Fedora 18, note that the package comes as a Debian package that we’ll extract:

$ su -
# yum groupinstall "Development Tools" -y
# yum install binutils -y

# mkdir ~/bcm4331 && cd ~/bcm4331
# wget http://shared.teratan.net/~rdo/bcm4331/wireless-bcm43142-dkms_6.20.55.19-1_amd64.deb
# ar vx wireless-bcm43142-dkms_6.20.55.19-1_amd64.deb
# tar -zxvf data.tar.gz

# cd usr/src/wireless-bcm43142-6.20.55.19
# wget http://shared.teratan.net/~rdo/bcm4331/wl_cfg80211.c
# mv wl_cfg80211.c src/wl/sys/
# make && make install

# depmod -a
# modprobe -r b43 ssb bcma
# modprobe wl

# echo "blacklist bcma" >> /etc/modprobe.d/blacklist.conf
# echo "blacklist ssb" >> /etc/modprobe.d/blacklist.conf
# echo "blacklist b43" >> /etc/modprobe.d/blacklist.conf

# echo "wl" >> /etc/modules-load.d/wireless.conf
# restorecon -v /etc/modules-load.d/wireless.conf

The above commands basically grab the Ubuntu package, over-write some minor changes required to build it on a 3.8+ kernel, compile the module for the local system and install it for the current kernel. In addition it also updates the module “database” and inserts the module with associated dependencies (cfg80211, lib80211 and lib80211_crypt_tkip). Finally, it blacklists the open-source drivers and enables the ‘wl‘ driver to be started on boot up.

The system should now report that the wireless adapter is available and that it’s using the correct driver:

# lspci -k | grep -n2 4331 | tail -n3
42:03:00.0 Network controller: Broadcom Corporation BCM4331 802.11a/b/g/n (rev 02)
43-	Subsystem: Apple Inc. Device 010f
44-	Kernel driver in use: wl

# lsmod | grep wl
wl                   3074693  0 
cfg80211              495993  1 wl
lib80211               13968  2 wl,lib80211_crypt_tkip

At this point I’d recommend rebooting to ensure that the machine comes up successfully and that the drivers are loaded automatically (confirm using the above steps again). You should also not be able to see the ‘b43‘ driver in the list of modules.

Note: I did have some instabilities when using ‘iwconfig‘, I’d recommend that you avoid this tool if you can, ‘ifconfig’ works without problem.

Step Two (Sound/Audio): As the device is very much like any other manufacturers equipment, it utilises commodity components. The Ivybridge CPU provides the standard Intel HD4000 video card and therefore works right out of the box; upstream support has been there for a long time. This is also the same story for a number of other components, the sound card whilst supported requires a slight module modification to get it to work properly. What you may find is that the modules get automatically loaded by Fedora but no sound card is visible.

The module that we’re using to provide audio is ‘snd_hda_intel‘, we need to pass an additional model identifier to it in order to initialise the card:

$ su -
# echo "options snd_hda_intel model=mbp101" >> /etc/modprobe.d/snd_hda_intel.conf
# restorecon -v /etc/modprobe.d/snd_hda_intel.conf

Note that you will either need to reboot or reload the module in order to get sound working; I recommend the former as it will integrate with the rest of your environment without having to restart all of the required services manually. When the machine has come back up, confirm that the module was loaded correctly with the model identifier:

$ cat /sys/module/snd_hda_intel/parameters/model 
mbp101,(null),(null),(null),(null)......

Step Three (Performance): Regardless of the specification you chose to buy, the MacBook Pro is a very capable machine. There are a number of recommendations that I’d like to make in order to maximise the performance of your system. Firstly, any modern machine with plenty of RAM will very rarely need to touch swap space but it certainly still is a requirement, e.g. for hibernation or for various types of workloads. That being said, we should still ask Linux to avoid using the swap space wherever possible:

$ su -
# echo "vm.swappiness=1" >> /etc/sysctl.d/performance.conf
# echo "vm.vfs_cache_pressure=50" >> /etc/sysctl.d/performance.conf
# restorecon -v /etc/sysctl.d/performance.conf

Next, as we’re using non-rotational disks (i.e. SSD/Flash), the ‘noop‘ block scheduler provides a number of performance benefits that we can exploit; it’s essentially no scheduling at all, just basic FIFO (first in, first out), alternatively the ‘deadline‘ scheduler can be used which tries to prioritise reads to enable some sort of read-performance when there is heavy write I/O. My personal preference is ‘noop‘ but your workload may require ‘deadline‘. You can view the current scheduler algorithm by using the following command:

$ cat /sys/block/sda/queue/scheduler
noop deadline [cfq]

In the above example, you can see that I’m currently using the ‘cfq‘ method. It’s easy to make a one-time change to the scheduler by echoing values into that virtual filesystem but to make this persistent we need to add a udev rule:

# echo ACTION=="add|change", KERNEL=="sda", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="noop" >> /etc/udev/rules.d/60-scheduler.rules
# restorecon -v /etc/udev/rules.d/60-scheduler.rules

Note that I have explicity specified the ‘sda’ device here (the internal SSD) and not all sd*, this is because I may want to attach external disks, e.g. via USB, which will be rotational disks and would therefore benefit from the default scheduler. Feel free to adjust the above udev rule to specify ‘deadline‘ if that’s your preference.

Finally, as we’re using a solid state disk it makes sense to implement trim/discard to aid with the wearing of the disk; I’m not going to go into too much detail of why we do this but it will prolong the life of the drive. There are a number of changes to make, firstly your fstab needs to be updated to mount your drives with these options:

$ su -
# vi /etc/fstab

Depending on your partition layout and whether you use encrypted volumes the next steps may be quite different for you. I will, however, assume that you’re using a default partition layout with a separate /home and / mount points. Simply add the parameters ‘discard’ and ‘noatime’ to each mount. For example:

(change)
/dev/mapper/vol0-rootVol    /    ext4    defaults    1 1

(to)
/dev/mapper/vol0-rootVol    /    ext4    defaults,discard    1 1

(repeat for /home)

('i' to edit and 'Esc -> :wq!' to save and quit)

Once that’s complete we need to test your changes:

# mount -o remount /
# mount -o remount /home
# mount | egrep '(on /|/home)'
/dev/mapper/vol0-rootVol on / type ext4 (rw,relatime,seclabel,discard,data=ordered)
/dev/mapper/cryptoVol on /home type ext4 (rw,relatime,seclabel,discard,data=ordered)

If successful, you should see the discard option listed in your output, yours my vary but ensure it has discard listed.

Please note: If you want to increase the performance further and are willing to accept a bit of risk, you can mount your volumes with the ‘noatime‘ option too; this removes the requirement for file updates for just reads… just bear in mind that applications may break! Fedora by default uses ‘relatime‘ which is a nice compromise.

— Additional tweaks coming tomorrow —

As you can see, despite the fact that these options are dropped to the lowest setting possible, I found that the mouse pointers sensitivity was ridiculous. There have been a number of bugs raised previously, e.g. https://bugzilla.redhat.com/show_bug.cgi?id=390131 but was closed years ago. It appears that this bug has shown it’s ugly face again! Until this is fixed, an easy solution to this problem is to modify some of the properties in xinput. Modifications are typically done via a terminal but it’s (of course) easy to make these persist by scripting it and making sure it gets executed upon login (see further below for an example)

To use xinput properly, you need to know which device’s properties you want to modify, a simple ‘xinput’ command can show you all of the devices currently being used (and can be modified)…

[rdo@soundwave ~]$ xinput
⎡ Virtual core pointer                    	id=2	[master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer              	id=4	[slave  pointer  (2)]
⎜   ↳ Logitech USB Laser Mouse                	id=10	[slave  pointer  (2)]
⎣ Virtual core keyboard                   	id=3	[master keyboard (2)]
    ↳ Virtual core XTEST keyboard             	id=5	[slave  keyboard (3)]
    ↳ Power Button                            	id=6	[slave  keyboard (3)]
    ↳ Power Button                            	id=7	[slave  keyboard (3)]
    ↳ Apple, Inc Apple Keyboard               	id=8	[slave  keyboard (3)]
    ↳ Apple, Inc Apple Keyboard               	id=9	[slave  keyboard (3)]
    ↳ Eee PC WMI hotkeys                      	id=11	[slave  keyboard (3)]

So, in my case, I want to modify my ‘Logitech USB Laser Mouse’; to show all of the properties you can use the following:

[rdo@soundwave ~]$ xinput list-props 'Logitech USB Laser Mouse'
Device 'Logitech USB Laser Mouse':
	Device Enabled (131):	1

...(and so on)...

But the ones we’re primarily interested in here are as follows-

• “Device Accel Constant Deceleration”
• “Device Accel Velocity Scaling”

You can have a play with these values, making sure you’re happy with the speed/acceleration of your mouse, specific for your device. I set the following attributes/properties:

$ xinput --set-prop "Logitech USB Laser Mouse" "Device Accel Constant Deceleration" 1.5

$ xinput --set-prop "Logitech USB Laser Mouse" "Device Accel Velocity Scaling" 1

Obviously, this is set dynamically and will not persist after a logout/reboot. So an easy fix is to write a bash script that is called upon login:

[rdo@soundwave ~]$ cat .fix-mouse.sh 
#!/bin/bash
xinput --set-prop "Logitech USB Laser Mouse" "Device Accel Constant Deceleration" 1.5
xinput --set-prop "Logitech USB Laser Mouse" "Device Accel Velocity Scaling" 1

I basically make this a ‘hidden’ file and have GNOME3 execute this script upon login, this is easily done, simply open up a terminal and launch ‘gnome-session-properties’, i.e.

$ gnome-session-properties

You can then click on ‘Add’ (when on the ‘Startup Programs’ tab) and simply set the command path to be your recently created script. My example is shown below…

Any questions, as always, let me know

Typically KVM is designed to run on-top of a bare-metal Linux machine with a CPU that supports virtualisation extensions, i.e. Intel VMX and AMD SVM. This allowed a physical machine to run multiple virtual machines on-top (using associated components such as libvirt and qemu), but there’s a new neat technology known as ‘nested KVM’, in other words, KVM support within a KVM-based guest or a hypervisor within a guest. You may ask the question ‘why do we need this?’… well, in my position I’m often running into situations where I have to carry out product demonstrations or debugging hypervisor environments, having another layer of virtualisation abstraction with nested-KVM is great, especially when on the train or on a plane!

There are, of course, some performance problems with doing this but for debugging or wanting to spin up a test environment with technologies such as Red Hat Enterprise Virtualisation or VMware on a single machine it’s quite a nice solution. So let’s look at how to enable it first, by default it’s usually disabled, at least on my Fedora 16 machine (you can replace Intel with AMD here if you have an AMD-based processor)…

$ cat /sys/module/kvm_intel/parameters/nested 
N

To enable it, we need to make sure the KVM architecture specific module is loaded with the nested option. There are a few options for enabling it, the first way to do this is just update your boot loader to specify the nested option; that way it persists with a reboot or kernel upgrade. Assuming you’re using Fedora with GRUB2, (as root) you need to update the ‘/etc/default/grub‘ file and append ‘kvm-intel.nested=1‘ to the ‘GRUB_CMDLINE_LINUX‘ line. For reference, mine is specified below, remember to replace ‘intel’ with ‘amd’ if required.

# cat /etc/default/grub | grep CMDLINE
GRUB_CMDLINE_LINUX="rd.lvm.lv=vol0/swapVol rd.md=0 rd.dm=0  KEYTABLE=us quiet rd.lvm.lv=vol0/rootVol rhgb rd.luks=0 SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 kvm-intel.nested=1"

Once this is specified, you’ll need to rebuild your GRUB configuration files so that when you next reboot the command line arguments you just specified are loaded, note the GRUB2 configuration location may be different on a non-Fedora machine…

# grub2-mkconfig -o /boot/grub2/grub.cfg

Alternatively, (thanks to Dominic Cleal for the suggestion) this can be modified using the modprobe configuration files, making things slightly easier-

$ echo “options kvm-intel nested=1″ | sudo tee /etc/modprobe.d/kvm-intel.conf

I would now recommend rebooting your machine to verify the changes have been made. Once again, you can re-run the previous command to check this and you should see that the changes to the module have been made.

$ cat /sys/module/kvm_intel/parameters/nested 
Y

And that’s it, you’ve successfully enabled nested-KVM. Next, when you create new virtual machines, e.g. with virt-manager, you will need to ‘require’ vmx or svm to be presented to the virtual machine; that way the guests can make use of the underlying nested-KVM features that have been enabled. This can also be done via direct modification of the libvirt XML definition of a given virtual machine, an example of one of my VM’s is shown below-

<cpu match='exact'>
  <model>Westmere</model>
 <feature policy='require' name='vmx'/>
</cpu>

Any questions please feel free to get in touch, I’d be happy to help out.

Further reading: https://github.com/torvalds/linux/blob/master/Documentation/virtual/kvm/nested-vmx.txt

I take no credit for this solution, but I thought I’d write an easy article on how to change the scroll behaviour on your Linux machine to be the same as your Mac. It’s a very simple procedure; open up a terminal as your local user (which, I hope, isn’t root!) and type the following:

$ echo "pointer = 1 2 3 5 4 6 7 8 9 10 11 12" >> ~/.Xmodmap

After you restart your session, it should have switched the scrolling direction. For the curious amongst you, you’ll notice that the ’4′ and ’5′ are switched around here. You can easily verify that the above has successfully included your changes by simply using the following command:

$ cat ~/.Xmodmap | grep pointer
pointer = 1 2 3 5 4 6 7 8 9 10 11 12

I tested this on my Fedora 16 laptop using GNOME3. Any problems, let me know… have fun!

I use a wide variety of operating systems at home, all services are provided by Linux, e.g. firewall, routing, file-storage and DLNA media. However, I like using a Mac too, I have a late-2009 MacBook Air which I use whilst traveling. Despite all of Lion’s flaws, I really like using it- full-screen apps, gestures and the new Mail.app is really impressive. The specification of this machine really isn’t anything special, the lack of expansion really leaves a lot to be desired but for what I do- it’s plenty. I will certainly be upgrading to the new Ivy Bridge MacBook Air when it comes out, perhaps then I’ll have more than 2GB memory and can run VM’s too(!).

The biggest problem I’ve been having with this machine on Lion (didn’t have it on Snow Leopard) is to do with kernel_task. Instead of splitting all of the underlying kernel operations into their own individual processes (and associated threads) they are all consumed by a single ‘task’ (more of a representation of the underpinnings of the microkernel architecture) that appears in the process list. What I’ve been noticing is that this ‘task’ sometimes goes out of control, consumes CPU resources with the utmost priority. For a long time I wasn’t sure what it was doing, it seemed to be kicking in when I was doing something that was relatively intensive (for a Mac anyway), e.g. YouTube.

A lot of people suggested that ‘rogue kexts’ (kernel drivers/modules) could be causing the problem, perhaps an incompatible module was being started by OS X that was installed when the system was running Snow Leopard. This seemed to make sense as a boot in safe-mode would cause no problems. Suffice to say, I created a Lion Install-USB and re-installed from scratch; guess what… same problem within hours! After a bit of digging around and investigating the kernel_task ‘process’ it was clear to see that it was looping through something continually. After further research I discovered that the kernel will keep looping some very simple tasks, e.g. getting the date, therefore ‘consuming’ (with the highest priority) the majority of the CPU in a bid to cool the system down.

So, it’s all to do with temperature control- you ‘remove’ a large portion of the CPU share from other applications and carry out low-overhead tasks continually until the CPU temperature drops. This sounds like a great solution to cooling, but it’s very intrusive. It’s agressive nature drags the system to a halt in a lot of ways, despite the fact that the CPU in my MacBook Air rarely exceeds 70 degrees (centigrade). Considering the TJ Max of my little 2.13GHz Core2Duo is 85 degrees I’d rather kernel_task not take this invasive action.

Thankfully, this “feature” is built into a kext, in which each model identifier specifies how to control the temperature of the CPU via this invasive action. The simple fix is to remove the entry for your model identifier from this kext- if it “doesn’t know” what to do with your particular model, it won’t take any action. Now, here comes the disclaimer… by taking the same action as I will outline below, I take absolutely no responsibility for any damage or loss caused to you or your property, you do this of your own free will. You’re over-ruling functionality that was designed to prolong the life of your equipment, despite the fact that it’s invasive and very annoying it’s there for a reason. Anyway, on to the fun stuff…

Firstly, you’re going to need the model identifier of your pesky Mac:

$ system_profiler -detailLevel mini | grep "Model Identifier:"
      Model Identifier: MacBookAir2,1

The kext we need to modify is IOPlatformPluginFamily.kext, you can verify it is running by-

sh-3.2# kextstat | grep IOPlatformPluginFamily
   67    3 0xffffff7f81229000 0x7000     0x7000     com.apple.driver.IOPlatformPluginFamily (5.1.0d17) <8 7 6 5 4 3>

Within this kext will be another further kext- ACPI_SMC_PlatformPlugin.kext, in which each recent Macintosh model is listed with relevant instructions that the kernel uses to determine how and when to invoke the control. (Make sure you switch to root now)

sh-3.2# cd /System/Library/Extensions/IOPlatformPluginFamily.kext/
sh-3.2# cd Contents/PlugIns/ACPI_SMC_PlatformPlugin.kext/
sh-3.2# cd Contents/Resources/
sh-3.2# ls | wc -l
49

So, there’s 49 different profiles listed in this kext as of the writing of this. We simply need to move the ‘plist’ file for the model identifier we discovered earlier out of that directory, so in my case I need to move file ‘MacBookAir2_1.plist’ out of this directory (to somewhere safe), but replace my model identifier with your specific one.

sh-3.2# mv MacBookAir2_1.plist /Users/<your username>/

You can simply reboot now for the changes to take effect. We could have removed the entire kext but it’s much safer to remove the plist file for the specific model as (to be honest) I don’t know what the rest of that kernel module does. I’ve been using this ‘fix’ for a few days now and not noticed any problems at all- overall it’s much quicker and I don’t have to worry about it running out of steam. What I would recommend is that you be a bit more careful about the placement of your Mac, you don’t want to be covering the exhaust. The Mac should protect itself if it *does* reach max temperature but I would just be a little more careful.

Note: It’s likely that this kext will get updated in the future by Apple in their update packs and as a result will replace the file you deleted/moved therefore you may have to repeat this process in the future

Let me know how you get on.

This is where the problem starts- the Red Hat Enterprise Linux (RHEL) 5 kernel does not have these modules available out of the box, therefore typically an installation of RHEL takes place using emulation to get the system up and running, the tools are then installed (which typically install pre-compiled kernel modules) and the system administrator can then present the para-virtualised devices instead. However, when a new kernel is installed the drivers will not automatically be part of the freshly installed kernel therefore upon a reboot the system will not be able to find it’s disks or network devices (depending on which are being used).

It’s not good enough to simply re-install the tools immediately after a kernel-update because the tools will simply install onto the currently running kernel. Whilst it’s possible to manually select a kernel version to install for, the VMware tools expect to build the modules from source (rather than using the pre-compiled versions) which doesn’t really help our situation- we don’t always want compilers and associated libraries on every system. So, we need a solution where the drivers are automatically present in the new kernel, so after a reboot there’s no issue in the system coming up as expected.

The main problem that needs to be addressed is the pvscsi (block-storage) driver- if your root filesystem resides on a pvscsi device then the system will completely fail to boot. If, at least, pvscsi is available- the system can continue to boot as normal, but the rest of the drivers would need to be reinstalled post-reboot, which as explained earlier is no problem as the tools will install for the current kernel (i.e. the one you just updated to). I focussed my efforts on working to ensure that post-kernel installation there would be an automatic installation of the pvscsi kernel module into the new kernel tree.

The solution I came up with is very simple- I looked through the source of the VMware tools install script and found that the code does the following-

1) Checks to see if there are compatible pre-built kernel modules within the tools
2) If so, it places the modules into /lib/modules/<kernel-version>/misc
3) Executes a ‘depmod’ to scan and refresh the module tree
4) Creates a new initrd with the required modules included for boot

As I know that my customer runs RHEL 5, there will always be a compatible pre-built kernel module available within the tree and due to Red Hat’s extensive effort to keep the kABI/API compatibility across the life-cycle of the major version, we can simply copy the existing pre-built kernel module from the existing kernel tree straight over to the new one and manually create an initrd for the new kernel.

During testing of this code, I wrote the following bash script which takes care of ensuring that the pvscsi module is in the new kernel and a new initrd is in-place:

#!/bin/bash

# Get the latest kernel installed
NEW_VERSION=`ls -tA1 /lib/modules | head -n 1`

# Check if the module is already present, if not- continue
if [ ! -e /lib/modules/$NEW_VERSION/misc/pvscsi.ko ]; then

    # Log output to show version we're fixing
	/bin/logger "Fixing VMware Paravirtual Drivers for reboot. Kernel: $NEW_VERSION"

    # Create the misc directory that VMware tools would
	mkdir /lib/modules/$NEW_VERSION/misc > /dev/null 2>&1

    # Find the current running kernel
	OLD_VERSION=`uname -r`

    # Copy the existing module to new kernel tree
	cp /lib/modules/$OLD_VERSION/misc/pvscsi.ko /lib/modules/$NEW_VERSION/misc > /dev/null 2>&1

    # Rebuild our modules cache
	depmod -a $NEW_VERSION > /dev/null 2>&1

    # Create a new initrd, including the new pvscsi module
	mkinitrd -f --with=pvscsi /boot/initrd-$NEW_VERSION.img $NEW_VERSION > /dev/null 2>&1

else
    # This would occur when module is already present
	/bin/logger "VMware Paravirtual Drivers Fix Installed"
fi
exit 0

Whilst this will fix the new kernel to have the freshly installed VMware pvscsi driver present (and therefore allow a normal boot) we need to look at automating this procedure to avoid accidental reboots without the fix being put in place. With RHEL/rpm-based platforms we can use RPM triggers to execute on a particular event. For example, we can create a ‘dummy-rpm’ which has the above script in that executes every time a particular event, such as a package upgrade/install occurs. Therefore, in our example we would need to create an rpm which simply ran the above code every time the kernel package was installed/upgraded. This is done by the tag ‘%triggerin — kernel’ (short for trigger ‘install’) within the rpm spec file.

If we create a very basic rpm spec file such as the following, we can use rpmbuild to build an rpm for us. Once this is installed on the target system it will simply execute the script code every time there’s a modification to the kernel package:

Name:	    fix-vmware
Version:	0.1
Release:	1%{?dist}
Summary:	Fixes pvscsi.ko to ensure it survives kernel-upgrades with reboot
Packager:	Rhys Oxenham <roxenham@redhat.com>

Group:	    Applications/System
License:	GPLv2+
URL:		http://www.rdoxenham.com
BuildArch:	noarch

%description

This package fixes the VMware paravirtual SCSI driver issue in RHEL5, ensuring
that the pvscsi.ko module is available after a reboot and that the
vmware-config-tools.pl script can execute to re-enable everything else.

%triggerin -- kernel

NEW_VERSION=`ls -tA /lib/modules | head -n 1`

if [ ! -e /lib/modules/$NEW_VERSION/misc/pvscsi.ko ]; then
	/bin/logger "Fixing VMware Paravirtual Drivers for reboot. Kernel: $NEW_VERSION"
	mkdir /lib/modules/$NEW_VERSION/misc > /dev/null 2>&1
	OLD_VERSION=`uname -r`
	cp /lib/modules/$OLD_VERSION/misc/pvscsi.ko /lib/modules/$NEW_VERSION/misc > /dev/null 2>&1
	depmod -a $NEW_VERSION > /dev/null 2>&1
	mkinitrd -f --with=pvscsi /boot/initrd-$NEW_VERSION.img $NEW_VERSION > /dev/null 2>&1
else
	/bin/logger "VMware Paravirtual Drivers Fix Installed"
fi
exit 0

* Wed Apr 4 2012 Rhys Oxenham <roxenham@redhat.com> 0.1-1
- Initial release

For those unfamiliar with building rpm’s, the spec file simply lists how the rpm is built and how the rpm reacts to certain events such as install, remove, upgrade etc. It will also allow for clean removals of packages by maintaining a list of all files the rpm controls. To build an rpm from the above spec file, simply copy the text into a file named fix-vmware.spec and use rpmbuild to create the rpm for you-

(root) $ yum install rpm-build
 $ rpmbuild -bb /path/to/fix-vmware.spec

It will then create a non-arch specific RPM for us, either in your home directory or in /usr/src/redhat if you build using root. We can then install this locally onto the system (or alternatively make this rpm part of your standard build process if you include it in a repository):

(root) $ yum localinstall /path/to/fix-vmware-0.1-1.noarch.rpm --nogpgcheck

Note: You may want to consider signing your rpm to get around the ‘–nogpgcheck’ issue.

For those not wanting to build this themselves, I have a pre-built version available here.

Finally, it’s very important that upon the reboot of the system that the full suite of VMware-tools are reinstalled as per your typical requirements (and over-writing the existing pvscsi module that the script described on this page to ensure the module gets updated over time). The tools will ensure that other modules such as vmxnet3 are installed and that the system can continue on with it’s required workloads. A typical way of doing this would be to create an /etc/rc.d/ boot-up script which runs the following line if the modules cannot be found.

vmware-config-tools.pl -d --clobber-kernel-modules=pvscsi

Any questions, please leave a comment or get in contact