Unsolvable and annoying VirtualBox slowdown

Since end of December 2016, I was using my personal computer to work from home. All my development environment was living in a VirtualBox CentOS virtual machine. That was working very nicely until March 13th. All of a sudden, the virtual machine became annoying slow. Starting the virtual machine took twice the time as before and everything was taking several seconds to start. It became impossible to start Thunderbird and connect to my corporate email through IMAP.

I tried to restart the virtual machine, reinstall VirtualBox, fiddle with the parameters of the virtual machine, nothing worked! The only workaround that helped a bit and at least allowed me to work was to increase the number of virtual CPUs from 2 to 4. However, even with that, when I was switching windows with Alt-Tab, it was taking several seconds for the window selector to pop up. Typing on the terminal was sluggish, sometimes as if I was connected through SSH on a slow server! But my machine was running locally! Running the unit tests of the main component I was developing used to take 4 minutes and a half on that machine. This bumped to more than 5 minutes, and even 6 minutes last week.

None of the suggestions on forums and nothing I thought about could solve this. I set the power plan to maximum performance so the CPU wasn’t throttled down. I fiddled with virtualization and paravirtualization options, disabled 3D acceleration, enabled it again, absolutely no change. Even the GRUB menu was taking a few seconds to display rather than appearing instantly as before. Maybe this is a CPU issue? I checked the CPU temperature using Speccy: everything seemed normal. VirtualBox was the mainly affected program, the rest was working fine.

This really seemed like a nasty Windows 10 update that screwed everything up. I could attempt a system restore or I could recover the last CloneZilla image I had, but if that was a Windows 10 update, the update would reinstall and thinks would screw up again. I would thus need to downgrade to Windows 7.

The issue persisted more than a month. Sometimes, I tried stuff without success: recreating the virtual machine, copying the VDI files somewhere and putting them back on my SSD (in case of filesystem corruption), checking my NVIDIA settings because the slowdown was mainly affected GUI rendering, etc.

The day before the first slowdown, two things happened: I ran CCleaner and Avast got updated. I tried to restore the registry backup CCleaner made before it cleaned registry: that failed once again. Windows couldn’t restore the registry backup. I tried to disable Avast temporarily: no effect. The hypothesis of a flaky Windows update was the main one, but no update were coming to fix it. Maybe they won’t fix it, because they offer Hyper-V, and they would love people switching from VirtualBox to Hyper-V. But that would involve reinstalling CentOS on a new virtual machine and recreating all my development environment. I could try VMWare Player instead, I think it can migrate a VirtualBox machine, but nothing told me it wouldn’t be affected as well by this issue.

Monday morning, April 17th, I tried all what I could to fix this. I was ready to attempt the system restore if needed. However, it was too late for a simple restore through Windows: I would need to use my CloneZilla image and take the risk Dropbox wouldn’t be smart enough to detect there is an old directory to update and not replace the contents in the cloud with the restored old directory. In case Dropbox screwed up, I backed up my Dropbox folder. I also backed up my virtual machines that I would need to restore after the Clonezilla recovery.

Before doing that, I tried to uninstall CCleaner: no success. I tried to remove NVIDIA Experience: no success. Two days ago, I removed the 3D vision drivers without effect. Then I removed Avast. At this point, I had little hope. I was ready to plug in the external hard drive with the Windows 10 image on it. But before, I rebooted, tried the virtual machine one more time and then, finally, it worked! It wasn’t slow anymore!

I pushed as far as starting my Ubuntu virtual machine and upgrading it from 16.10 to 17.04. This went well, without any issue. I’m writing this post from Ubuntu 17.04, running inside my VirtualBox environment which is finally fixed.

It is not the first time VirtualBox is hindered by an anti-virus program. It happened at work with Symantec Endpoint Connection. I had to downgrade to an old VirtualBox for a few releases until that finally got fixed. VirtualBox or Avast will have to fix something for this system to work again. For now, I am using the anti-virus built into Windows 10. I don’t know if I will retry Avast, reinstall AVG or finally give up and pay for Symantec’s Anti-virus or Kaspersky.


Networking issues on an ultrabook

Since a few weeks, I was experiencing sluggish performance while connecting to some SSH servers. The problem happened when using SSH through the VPN of the company I am working for. This resulted in lags when typing commands on the SSH terminal, which was more and more problematic because of more and more command-line arguments I had to pass and longer and longer file paths.

Too many layers = too many problems

My current working setup is far from simple, because I need to work on a remote virtual machine to have access to some data and run some processing on it. The official connection method using NX doesn’t work well for me, because NX suffers from intermittent keyboard issues, e.g., right alt stopping to work, server acting as if Shift was pressed while it is not, etc. This works correctly for somebody using the mouse rather than the keyboard or able, without significant loss of efficiency, to check every typed character for an eventual error. This is not my case.

I ended up building myself a multi-layer setup as follows:

  1. I am not hooked directly to the cable modem from Videotron. I am rather using a Linksys WRT310N router, and to make things more interesting, I am running DD-WRT, not the official router’s firmware. This rarely caused any issue, though.
  2. Windows running on a computer provided by the company. I am currently working from home using an ultrabook they provided me, with Windows 8.1 on it.
  3. The ultrabook having no Ethernet connectivity, I was using a USB to 100Mbps Ethernet adapter to get a faster and more stable connection than wi-fi.
  4. A Cisco VPN client is needed to access internal resources  of the company.
  5. The machine runs Ubuntu 14.10 in a virtual machine hosted by VirtualBox.
  6. Inside the guest Ubuntu, SSHFS is configured to access my workspace on the virtual machine, so I can use local editors like Emacs.
  7. Inside the guest Ubuntu, I open a terminal and SSH to the virtual machine to run commands there.

Phew! What a list of layers!

Different networking methods

VirtualBox offers several ways to manage networking. I am currently using the first of the three I know about. Here they are.

  1. Bridged. VirtualBox uses a trick I don’t know too much about to clone the host’s network interface and act as if there was a second interface. The guest OS receives its own IP address from my router and thus acts pretty much like an independent machine on the network. This implies that Ubuntu has to establish the VPN connection, but fortunately, there is a Cisco client available. However, when using that method, the Windows host doesn’t have access to internal resources, unless I establish a second VPN connection, on the host side. I am something tempted by the idea of having the router establish the VPN connection. This might be possible with DD-WRT, but this will introduce a security risk: what if I leave the VPN open after my working day or somebody hacks my network?
  2. NAT. VirtualBox acts a bit like an internal router, allocating a private IP to the guest. Requests are translated by VirtualBox to look like if they came from the host. This works correctly and allows the VPN connection to be established by Windows-based official Cisco client, but on the other hand, this introduces a level of indirection: the NAT applied by VirtualBox. Any indirection is subject to hinder performance.
  3. USB. VirtualBox has support for exposing USB devices to guests, so I could, at least in theory, expose my USB to Ethernet interface to Ubuntu. However, without Oracle closed-sourced extensions, I would get only USB1.1 support, which would result in slow or non-functional networking. The extensions are available only for personal or evaluation use, so I cannot use this at work. Even if I solve the licensing issue and get the extensions, with the USB solution, the Windows side would be unable to access networking, so I would loose access to Lync and Outlook. I could work around by turning wi-fi back on, but that starts to be clunky. If I have to go this way, I would be better off using my personal Ubuntu PC rather than a virtual machine.

It seems that 1 is a bit faster than 2, but I am not totally sure, no way to measure scientifically. Even with 1, I was still experiencing sluggish SSH. After switching from 2 to 1, this seemed a bit better, but performance degraded after a few minutes.

Could a better network interface help?

This morning, I tried with a TruLink USB to Gigabit Ethernet interface rather than the 100Mbps one. However, this didn’t go well at all. I got the following issues, all after the other.

  1. VirtualBox partially blocking network. At first, everything seemed to work well. I was able to browse the web and Outlook was working fine. But I soon discovered that Lync wasn’t connecting at all and although Cisco VPN was starting (from Windows, no virtual machine yet), it couldn’t access any internal resource. Trying Windows network diagnostic reported a potential driver issue. I tried to install the driver I found for this adapter, but it just failed; Windows already had the most recent driver built in or installed by IT. I then found out that Windows was connecting to an unknown network using the VirtualBox host-only adapter. So VirtualBox was in the way, partially blocking networking. I had to remove VirtualBox and reinstall it to fix this.
  2. Cisco VPN not working. After reinstalling VirtualBox, I was able to connect to Lync, but VPN was still non-working. It was connecting without issues, but it would allow access to absolutely no local resource. I tried to remove VirtualBox once again, to no avail. I had to remove Cisco VPN client, reboot to be sure everything was clean, reinstall client, test to see if things were back to normal (they were!), reboot once more to be sure, reinstall VirtualBox and test again! Why reboot? Well, if VirtualBox is installed while Cisco VPN is running, network connectivity stops working completely until VirtualBox is removed.
  3. Distracting side issues. During this frustrating troubleshooting, I got several other issues. Firefox took several seconds to start, once again, because I would ideally have to switch to Chrome, transfer my bookmarks from Firefox to Chrome once again, and live with a browser having very weak support for touch screen, at least at the time I am writing. Emacs, which I tried to open and use as a buffer because I wanted to try Ping commands and was always making typos, took at least one minute to launch, and started multiple instances when it finally unstuck. Then the main window of the Cisco VPN client remained open after connection and couldn’t be closed by the X button, Alt-F4 or any other normal way; I would have to live with it on an unused Virtuawin desktop or restart the client. I ended up shutting everything down and rebooting once again.

At least, after all these efforts, I got functional networking and experienced a lot less lags than during the last 2-3 weeks while working from home!

If problems come back, I will probably give up on using this ultrabook and revert to the official laptop provided by the company. The machine is heavier, doesn’t have any reasonable way to output to digital displays (I would either need to purchase a docking station specific to that machine, or try my luck with the mini HDMI port which sometimes works, sometimes not), but it has VGA output, it has Gigabit Ethernet and runs the good old Windows 7 which definitely seems to play nicer with the software tools of my company.

Computer science

The power of i7

Yesterday, I got truly impressed after months of disappointment about performance. I was trying to reinstall Myst IV and have some nostalgic fun with it, but this time, Windows 8.1 decided that it wouldn’t start the autorun program on the DVD. There was no other setup program on the disk, so I had to either downgrade my system or try on an older machine. No help from any forum, only complaints about games failing on Windows 8.1 but working on Windows 8.0! Grrrr…

I got fed up of all these difficulties with Windows 8.1, got tired of reading repetitive non-constructive posts about Microsoft releasing one good Windows version over two and no way to easily roll back to the “good” version, and reached the point of creating a virtual machine running an older version of Windows. For this, I used VirtualBox, which served me very well at Nuance for creating a virtual Ubuntu box.

When I started VirtualBox, I found that my Ubuntu machine was still there. I started it, it was still working, then I decided to upgrade it to 14.10. During the upgrade, I created a second VM and installed Windows XP on it. Yes, Windows XP, no fuss with Windows 7’s bad contrast between selected and unselected items, I was just tired of fighting with Windows. The installation first failed, because I had to configure the VM as IDE and not SATA, but after that it worked like a charm.

Not only two virtual machines were working smoothly in parallel (one installing downloaded packages, one installing Windows XP), but my Core i7 computer was still responsive! I was able to browse the Web without any problem.

In the end, these virtual machines didn’t help much, because when I launched the autorun from Windows XP, I found out I was inserting the second DVD of the game. The autorun of the first DVD failed to start as well on Windows 8.1, but there was a Setup program that started, worked correctly, and the game (patched to 1.03) worked very well. So for this time, no need for a virtual machine, but I will keep it around.