Ubuntu 18.10: a silent release

Usually, upgrading Ubuntu goes well. I run the Upgrade tool which downloads new packages, installs them and then asks me to reboot. Some new versions had minor issues, for example Wayland not fully compatible with my NVIDIA card or an old version of MATE preventing the dist-upgrade, but nothing major, nothing that couldn’t be worked around. This time was different: no sound, and no way to easily get around this.

Cannot dist-upgrade

Sunday, October 21 2018, Ubuntu 18.10 was released since a couple of days. However, after I installed the updates through the Software update tool, I wasn’t proposed to upgrade. I had to dig into the software options and reconfigure the delivery of releases to “every release”, not just the LTS. Then I reran the Software Update tool, and got the option to upgrade. However, clicking on the upgrade button did just… nothing. I tried two or three times: same result.

Before accepting that this time, I would have to go through the clean install route, I searched a bit on Google, and found out that sometimes, the upgrade doesn’t start when updates are still available. But updates were all installed. However, running the following on a terminal found and installed a couple of additional updates.

sudo apt update
sudo apt dist-upgrade

I then retried the Upgrade button, and that worked! Yeah!

The upgrade went well, and then I was offered to reboot, which I did.

Slow and silent

First the new release felt a bit sluggish, then pretty flaky. First time I tried to reboot because I needed to go to Windows, to play Minecraft loaded by Twitch and recorded through OBS, the system hung up waiting for a stop job. I am getting this issue from time to time and when that happens, I have to wait for the 1min30s timeout to elapse before the stop job is forcibly killed. I don’t know what is a stop job, what job is frozen and how I could get rid of this issue for good.

I was in my living room turning off my TV and AV receiver while my new Ubuntu setup finally rebooted, and I didn’t have time to select Windows in GRUB, so it restarted Ubuntu, and trying to restart redid the “stop job” issue! Argh, don’t tell me I’ll get this each time I reboot or shutdown now? I’ll have to wait 1min30s, maybe even 3min, just to power off or reboot. What’s the point of having a SSD if timeouts like this counter its performance benefits?  I got fed up and powered off my PC with the power button. I had to press and hold the button almost ten seconds for the machine to finally turn off, then I had to press the button 2-3 times for it to finally turn on. Maybe I’m heading towards the necessity of replacing my computer CASE, which sucks me to the point of thinking about getting rid of that damned thing and use just a laptop. But that case issue has nothing to do with Ubuntu, I should just have picked a better case, there’s nothing more to it, at least for this post.

After my Minecraft session, which was kind of fruitful, I wanted to check that my Ubuntu installation would boot again and be able to shutdown at least once without the “stop job” issue. I thus rebooted to Ubuntu, but the system froze before showing the welcome screen. I had to press CTRL-ALT-F2 to reach a console, log in, check the syslog, nothing of interest. Then the system finally booted into X. I wanted to switch back to the console to log off, but when coming back to X, it froze again on a black screen, this time no key was working. Then another hard reboot!

Second attempt worked: I reached the desktop. I launched the backup script for my Minecraft world, then the MKV->MP4 batch conversion script. OBS records in MKV, which is more robust against crashes, but VideoStudio doesn’t accept MKV, so I have to turn the recordings into MP4. Fortunately, FFMPEG does it without loss, by just repackaging the MPEG stream into another container. Then I wanted to organize my videos into directories, so I launched one to check it, and found THE thing: no sound!

Checking pavucontrol, I found that my sound card disappeared. Only detected devices were my HDMI port hooked up to my monitor, and a crappy USB Webcam that could serve as very basic and rudimentary mic. I don’t use that for my Minecraft recording; I have an AKG real microphone for that! I tried to reboot to no avail. Trying to run speaker-test just hung, again, needed to press ALT-F4 to shutdown the terminal.

Searching on Google for solutions lead to nothing except old stuff that didn’t work. Some people restored sound by reinstalling ALSA and PulseAudio. Others had to downgrade to the previous kernel. Others edited configuration files, commenting a line that is not there in my case, and adding another line. I tried to reinstall PulseAudio with sudo apt install –reinstall pulseaudio, to no avail. As a last resort, I tried to reboot with the 4.15 kernel of Ubuntu 18.04,

After almost an hour of searching, it was more and more obvious that I needed to give up on Ubuntu 18.10 and either downgrade (which essentially means reinstall) to Ubuntu 18.04, or switch to a new distribution. I was quite annoyed, as preceding upgrades went right, and then the hardware problems start again like in the past. Moreover, the system was freezing for a second each time I hit Tab while in a terminal, before displaying completions.

Can this work at all?

I had to go sleeping (it was past 11 PM), go to work the day after, but I thought about it. First I needed to test if that can work! For this, the simplest solution is to test using a live USB. Monday morning, I had to resist the temptation of testing that before going to work. Doing that would have made me start late and thus finish work late in the evening. So I did my workday first. So on Monday evening, I needed to download the Ubuntu 18.10 ISO; I picked the MATE one since that’s what I’m using now, instead of that GNOME3 thing which works so so. I got the ISO, and used the disk creator tool built into Ubuntu to write it into a USB drive. However, the tool refused to format my drive: too small for the new 2Gb ISO! That old 2Gb key was just a bit too small, which is kind of annoying. As a smart man, I should have extra empty, unused USB keys hanging around, but seems I’m not smart enough for that! So I had to use an existing key.

So I took the Clonezilla key, a 16Gb device, really too large for that small backup tool. I stored Ubuntu on it, and then I installed Clonezilla on the old 2Gb stick. Then I booted off my new Ubuntu medium, the system booted successfully, and I got sound! Ok, so at worst, if I clean install, I should get sound… except maybe if that is an incompatibility with the NVIDIA driver.

Unlocking ALSA

First I ironed out the NVIDIA hypothesis by uninstalling the NVIDIA proprietary graphic driver. After I reboot, I tested and still had no sound, and was back to a default VESA resolution. I thus reinstalled the driver, a bit annoyed that Nouveau cannot even basically handle my graphic card. I don’t expect full 2D/3D acceleration out of Nouveau, but at least, mode switching should work with a card bought in 2013, reaching a 1080p resolution. No, nothing. But at least, there was no incompatibility between the graphic and sound driver.

I then dug further into ALSA, which was still detecting my sound chip. Here is the list of devices it found:

eric@Drake:~$ aplay -L
default
    Playback/recording through the PulseAudio sound server
null
    Discard all samples (playback) or generate zero samples (capture)
pulse
    PulseAudio Sound Server
sysdefault:CARD=PCH
    HDA Intel PCH, ALC887-VD Analog
    Default Audio Device
front:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Front speakers
surround21:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    2.1 Surround output to Front and Subwoofer speakers
surround40:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    4.0 Surround output to Front and Rear speakers
surround41:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    4.1 Surround output to Front, Rear and Subwoofer speakers
surround50:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    5.0 Surround output to Front, Center and Rear speakers
surround51:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    5.1 Surround output to Front, Center, Rear and Subwoofer speakers
surround71:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    7.1 Surround output to Front, Center, Side, Rear and Woofer speakers
iec958:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Digital
    IEC958 (S/PDIF) Digital Audio Output
dmix:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Direct sample mixing device
dmix:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Direct sample mixing device
dsnoop:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Direct sample snooping device
dsnoop:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Direct sample snooping device
hw:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Direct hardware device without any conversions
hw:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Direct hardware device without any conversions
plughw:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Hardware device with all software conversions
plughw:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Hardware device with all software conversions
hdmi:CARD=NVidia,DEV=0
    HDA NVidia, HDMI 0
    HDMI Audio Output
hdmi:CARD=NVidia,DEV=1
    HDA NVidia, HDMI 1
    HDMI Audio Output
hdmi:CARD=NVidia,DEV=2
    HDA NVidia, HDMI 2
    HDMI Audio Output
hdmi:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 3
    HDMI Audio Output
dmix:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Direct sample mixing device
dmix:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Direct sample mixing device
dmix:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Direct sample mixing device
dmix:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Direct sample mixing device
dsnoop:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Direct sample snooping device
dsnoop:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Direct sample snooping device
dsnoop:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Direct sample snooping device
dsnoop:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Direct sample snooping device
hw:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Direct hardware device without any conversions
hw:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Direct hardware device without any conversions
hw:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Direct hardware device without any conversions
hw:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Direct hardware device without any conversions
plughw:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Hardware device with all software conversions
plughw:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Hardware device with all software conversions
plughw:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Hardware device with all software conversions
plughw:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Hardware device with all software conversions

The speaker-test command was hanging, but if I waited enough time after pressing Ctrl-C, it returned. Ok, cool! Running pavucontrol while pseaker-test was playing its noise, I could see it in the PulseAudio applications. Ok, so speaker-test

What if I switch device? I tried speaker-test -D sysdefault:CARD=PCHI got an error because ALSA couldn’t open the device. Searching on Google about that, not specific to Ubuntu, lead to clues: it could be a permission issue. Trying to check /dev/dsp failed: no such file. But I could fiund the following.

eric@Drake:~$ ls -ld /dev/snd/*
drwxr-xr-x  2 root root       60 oct 23 08:11 /dev/snd/by-id
drwxr-xr-x  2 root root      100 oct 23 08:11 /dev/snd/by-path
crw-rw----+ 1 root audio 116,  9 oct 23 08:11 /dev/snd/controlC0
crw-rw----+ 1 root audio 116, 15 oct 23 08:11 /dev/snd/controlC1
crw-rw----+ 1 root audio 116,  8 oct 23 08:11 /dev/snd/controlC2
crw-rw----+ 1 root audio 116,  6 oct 23 08:11 /dev/snd/hwC0D0
crw-rw----+ 1 root audio 116, 14 oct 23 08:11 /dev/snd/hwC1D0
crw-rw----+ 1 root audio 116,  3 oct 23 08:11 /dev/snd/pcmC0D0c
crw-rw----+ 1 root audio 116,  2 oct 23 19:23 /dev/snd/pcmC0D0p
crw-rw----+ 1 root audio 116,  4 oct 23 08:11 /dev/snd/pcmC0D1p
crw-rw----+ 1 root audio 116,  5 oct 23 08:11 /dev/snd/pcmC0D2c
crw-rw----+ 1 root audio 116, 10 oct 23 08:11 /dev/snd/pcmC1D3p
crw-rw----+ 1 root audio 116, 11 oct 23 08:11 /dev/snd/pcmC1D7p
crw-rw----+ 1 root audio 116, 12 oct 23 08:11 /dev/snd/pcmC1D8p
crw-rw----+ 1 root audio 116, 13 oct 23 08:11 /dev/snd/pcmC1D9p
crw-rw----+ 1 root audio 116,  7 oct 23 08:11 /dev/snd/pcmC2D0c
crw-rw----+ 1 root audio 116,  1 oct 23 08:11 /dev/snd/seq
crw-rw----+ 1 root audio 116, 33 oct 23 08:11 /dev/snd/timer

Ok, interesting, only root and users of group audio can access these devices, and thus play sounds. Am I part of the audio group? I tried groups eric and found out I wasn’t. But on my HTPC still running Ubuntu 18.04, I was! Lucky I had that HTPC, otherwise I would have been forced to reboot to the Live USB to check, or boot another machine with it, like my ultrabook.

Ok, so what if I add myself to the group?

sudo usermod -a -G audio eric

I had to relog, and then got sound through ALSA using Speaker-Test! Yeah! But still no PulseAudio! I tried to reboot, not just relog, to no avail. Ok, at least I can reconfigure Audacious to play through ALSA, but I suspect I’ll get trouble for YouTube and Spotify for which I cannot configure audio output.

Unlocking PulseAudio

I tried all sorts of things to debug this. I was able to get some logs out of PulseAudio several ways, but the most useful way was through SystemD. In Ubuntu, PulseAudio is started by a userspace variant of SystemD. There is a command allowing to get the logs: journalctl. I couldn’t figure out a way to filter so I ended up calling journalctl -a -f on one console, and then pulseaudio -k on another to force PulseAudio to restart. Then I was checking the produced log. I ended up finding out errors. The system was not able to communicate through DBus using a certain socket. I started to think that each time the system freezes, this is because PulseAudio tries to emit a sound, has to wait and timeout.

I couldn’t find out how to reconfigure DBus, or understand how PulseAudio and DBus interact enough to troubleshoot this. I was quite stuck, and all I could find was deprecated information. There was a bug report that seemed to affect Ubuntu 18.10, but no solution. It was past 10 PM and I was on this for almost two hours, trying and searching to no avail, more and more pissed off and risking to be so angry that I would end up having trouble to sleep. I was about to give up or reinstall from scratch.

I ended up fed up and decided to completely get rid of PulseAudio using sudo apt purge pulseaudio. After a reboot, sound was WORKING! What??? How come MATE plays sound without PulseAudio? Audacious was working, and VLC as well. But not Firefox for YouTube, and not Spotify.

Before accepting the sacrifice of YouTube and Spotify to avoid a clean install, at least for now, I tried to reinstall PulseAudio. For I really don’t know why, the sound continued working, and PulseAudio displayed my internal audio device now. YouTube and Spotify resumed working. I read posts about some people that got the issue fixed and it came back the next reboot. Ok, let’s reboot and hammer that bug for good, then! I rebooted, and sound was still working. I still don’t fully understand why.

Either something messed into the groups I was member of and PulseAudio got screwed up because I lost permission to use ALSA, either some configuration files needed to be updated but APT decided to keep my current versions. Purging PulseAudio removed the configuration files and reinstalling reverted to sane defaults. At least, sound is working now.

Even better, the system seems more responsive now and didn’t freeze on startup. It really seems that PulseAudio and ALSA had trouble communicating, causing these hangups.

Why not a clean install?

Because my home directory lives on the same partition as my Ubuntu install. Any attempt to put my /home on a separate partition leads to insufficient disk space after a time. I have a 250Gb SSD shared with Windows so I cannot put 50Gb for my /home and 30Gb for just Ubuntu. One simple solution would be to move my /home to an hard drive. As long as Ubuntu is on a SSD, I’ll have a fine boot time. Or I would need a way to use SSD only as a cache and put everything on the hard drive. I could also come back to the dual SSD strategy: one SSD for Windows, one for Ubuntu. I’ll think about it, but at least I don’t have to do anything short term. Maybe I can wait and replace everything, and get a better case to fix the power button hickup at the same time…

If everything had failed, before the clean install, I could have tried to restore my Clonezilla image of my SSD, and that would have got me Ubuntu 18.04 back to normal. In case of failure, then I would have had to clean install 18.04, 18.10 or something else, or just give up on Linux for the moment. At least this is not necessary anymore.

Unsolvable and annoying VirtualBox slowdown

Since end of December 2016, I was using my personal computer to work from home. All my development environment was living in a VirtualBox CentOS virtual machine. That was working very nicely until March 13th. All of a sudden, the virtual machine became annoying slow. Starting the virtual machine took twice the time as before and everything was taking several seconds to start. It became impossible to start Thunderbird and connect to my corporate email through IMAP.

I tried to restart the virtual machine, reinstall VirtualBox, fiddle with the parameters of the virtual machine, nothing worked! The only workaround that helped a bit and at least allowed me to work was to increase the number of virtual CPUs from 2 to 4. However, even with that, when I was switching windows with Alt-Tab, it was taking several seconds for the window selector to pop up. Typing on the terminal was sluggish, sometimes as if I was connected through SSH on a slow server! But my machine was running locally! Running the unit tests of the main component I was developing used to take 4 minutes and a half on that machine. This bumped to more than 5 minutes, and even 6 minutes last week.

None of the suggestions on forums and nothing I thought about could solve this. I set the power plan to maximum performance so the CPU wasn’t throttled down. I fiddled with virtualization and paravirtualization options, disabled 3D acceleration, enabled it again, absolutely no change. Even the GRUB menu was taking a few seconds to display rather than appearing instantly as before. Maybe this is a CPU issue? I checked the CPU temperature using Speccy: everything seemed normal. VirtualBox was the mainly affected program, the rest was working fine.

This really seemed like a nasty Windows 10 update that screwed everything up. I could attempt a system restore or I could recover the last CloneZilla image I had, but if that was a Windows 10 update, the update would reinstall and thinks would screw up again. I would thus need to downgrade to Windows 7.

The issue persisted more than a month. Sometimes, I tried stuff without success: recreating the virtual machine, copying the VDI files somewhere and putting them back on my SSD (in case of filesystem corruption), checking my NVIDIA settings because the slowdown was mainly affected GUI rendering, etc.

The day before the first slowdown, two things happened: I ran CCleaner and Avast got updated. I tried to restore the registry backup CCleaner made before it cleaned registry: that failed once again. Windows couldn’t restore the registry backup. I tried to disable Avast temporarily: no effect. The hypothesis of a flaky Windows update was the main one, but no update were coming to fix it. Maybe they won’t fix it, because they offer Hyper-V, and they would love people switching from VirtualBox to Hyper-V. But that would involve reinstalling CentOS on a new virtual machine and recreating all my development environment. I could try VMWare Player instead, I think it can migrate a VirtualBox machine, but nothing told me it wouldn’t be affected as well by this issue.

Monday morning, April 17th, I tried all what I could to fix this. I was ready to attempt the system restore if needed. However, it was too late for a simple restore through Windows: I would need to use my CloneZilla image and take the risk Dropbox wouldn’t be smart enough to detect there is an old directory to update and not replace the contents in the cloud with the restored old directory. In case Dropbox screwed up, I backed up my Dropbox folder. I also backed up my virtual machines that I would need to restore after the Clonezilla recovery.

Before doing that, I tried to uninstall CCleaner: no success. I tried to remove NVIDIA Experience: no success. Two days ago, I removed the 3D vision drivers without effect. Then I removed Avast. At this point, I had little hope. I was ready to plug in the external hard drive with the Windows 10 image on it. But before, I rebooted, tried the virtual machine one more time and then, finally, it worked! It wasn’t slow anymore!

I pushed as far as starting my Ubuntu virtual machine and upgrading it from 16.10 to 17.04. This went well, without any issue. I’m writing this post from Ubuntu 17.04, running inside my VirtualBox environment which is finally fixed.

It is not the first time VirtualBox is hindered by an anti-virus program. It happened at work with Symantec Endpoint Connection. I had to downgrade to an old VirtualBox for a few releases until that finally got fixed. VirtualBox or Avast will have to fix something for this system to work again. For now, I am using the anti-virus built into Windows 10. I don’t know if I will retry Avast, reinstall AVG or finally give up and pay for Symantec’s Anti-virus or Kaspersky.

Spurious mail delivery errors

A few weeks ago, I started to receive email containing error messages about the delivery of some mails I didn’t send. The contents of such emails looked like spam, but why weren’t they detected by the anti-spam functionality of GMail? Maybe spammers found a new way to send their junk that circumvents current filters. But few weeks later, the annoyance persisted. I was receiving at least one of these emails per day, sometimes several per day. I started to suspect some people hacked into my GMail account and were using it to send spam, but I couldn’t find any trace of these in my “Sent” folder. Maybe they can circumvent it as well. Will I have to change my password just in case? And what would  tell me they wouldn’t hack again?

Friday, April 14th, I got fed up of this. First, do these come from the same sender or group of senders? If it does, I could block these addresses. Otherwise, there is a problem with GMail that would need to be solved eventually, otherwise I would have to switch from GMail to some other email service. Looking at the sender’s address, I found out the message was coming from something @ericbuist.com. Could it be because my mail account from my Web host was misconfigured?

I logged onto my HostPapa cPanel and reached the mail options. I found out that anything @ericbuist.com not corresponding to a valid email account is sent to a default email address. As a result, spammers in need of a fake origin email address can take anything @ericbuist.com in the hope this won’t correspond to a valid address. I thus reconfigured the default route to return an error email instead of redirecting the message. I also found out that besides redirecting traffic to my GMail account, the HostPapa mail service is keeping a copy of the messages. I thus had 250Mb of junk emails there that I deleted to free space. Although the disk space is unlimited on my HostPapa, if every customer abuses it by leaving junk on their account, HostPapa will have to impose quotas at some point.

I didn’t receive other emails about mail delivery failures after that. Unfortunately, this is not the only cause of such problems. Other people had issues with that because they forwarded all their GMail emails to a service sending SMS, and the service went down. They had to disable that forwarding from their GMail accounts. Things get worse when other email addresses are redirected to a central email account. All these can be the cause of spurious emails and thus need to be checked in case of issues.

Bumpy Ableton Live session

Yesterday, I tried upgrading to latest Ableton’s Live, the 9.7.1 version. Everything went well, but I got other issues, not related to Live, that made my work session quite bad and frustrating.

S/PDIF not working great

A month ago, I got a new audio interface: the Focusrite’s Scarlett 18i20. This amazing device provides eight analog audio inputs and 10 outputs. This is far from the advertised 18 inputs and 20 outputs, but these include S/PDIF and an add-on card that plugs into the optical ports of the interface. Anyway, 8 inputs is more than enough for my needs. I have difficulty playing one instrument reliably, so I won’t start playing multiple instruments at the same time, at least not now!

I didn’t have enough long audio jack cables to plug my Novation’s Ultranova (two channels), my Korg’s EMX (two channels) and my Nord’s Drum (1 one channel), so I decided to try hooking my Ultranova through S/PDIF instead. For this, I used a RCA cable I had got somewhere I don’t remember. I plugged the S/PDIF coaxial output of the synthesizer to the appropriate input of the audio interface, then fiddled with MixControl to figure out HOW to enable S/PDIF. Easy, I thought: just set up one entry in the Mix 1 to route S/PDIF L to left channel and S/PDIF R to right channel. The Mix 1 mix was already routed to the two monitor outputs of the interface. With that, I should have obtained sound from my Ultranova into my audio monitors. No, nothing! I verified that the S/PDIF output was enabled from my Ultranova: it was.

I tried, checked many times, searched on the Web, ok, set the sync source to S/PDIF instead of Internal, from MixControl. Did it, no result. I spent at least half an hour trying, checking, trying again, to find that the volume of my Ultranova was turned all the way to minimum. Turning up the volume solved it!

BUT I started to hear cracking sounds from time to time. This happens especially when playing long notes with pad-style sounds. That means S/PDIF doesn’t work well out of my Ultranova, in my audio interface, or that requires a special cable I don’t have. But then WHY is the S/PDIF the exact same shape as an RCA connector?

There is no solution for the moment, except using the analog jacks and not being able to plug my EMX, Ultranova and Drum at the same time.

Jumpy mouse

While trying to work with Ableton’s Live and the MixControl, I had to cope with too small fonts all the times. I ended up using Windows zoom (Windows key plus +). But regularly, the zoom was jumping all around. I figured out that this was the mouse pointer that was regularly moving around without obvious reason. Ah, this is why I am now literally constantly loosing the pointer, forced to bring it back at upper left corner of the screen almost each time I want to click on something! The pointer is really jumping around, I’m not getting crazy! This made working with the mouse a real pain, similar to what I experienced with the old Mac my brother’s wife gave me a year ago. I thought about running Live on that Mac, because many people pretend that Mac’s are more stable for music production, but the machine is way way way too slow for that, I just forgot and never tried!

I ended up trying with another mouse, that seemed to be a bit better, but I realized that the right button was completely non-working!!! Why the hell did I keep this stupid mouse then? I threw it in the thrash can and put back the first one. Then I figured out that putting the mouse on a piece of white paper helped, making it a lot less jumpy.

Windows update restarting computer while I’m using it

Windows 10 sometimes automatically restarts the computer to apply some updates. Up to now, this only happened while the machine was idle. Well yesterday, it happened right in my face, while I was working with Live! I got so pissed off by this that I tried to disable this really bad functionality. I fortunately figured out a way to disable these forced updates. It was relatively easy, although it caused me trouble because my Windows is in French and the procedure was in English. If this procedure doesn’t work and spurious reboots happen too often, this may force me to downgrade to Windows 8 or Windows 7, or switch to Mac and have constant trouble with too small fonts. This could be a dead end case leading me to stop using my computer, at least stop trying to make music with this.

Slower and slower machine

My main computer is on a desk while my music gears are on a table on the opposite wall. I tried to link them together using a long USB cable and a hub, but that failed with crashes from Ableton’s Live. However, my attempts were with the audio interface built into my Ultranova. Maybe I’ll have more luck with my Focusrite, if the cable and hub are stable enough. Why an hub? Well, this is to get a keyboard and mouse next to my music table. I will also transport video through an HDMI cable and get a screen nearby as well.

But for now, I ended up having to use my Lenovo’s IdeaPad Yoga 13 ultrabook for attempts at music production. This worked relatively well, but the machine is starting to be slow since I updated it to Windows 10. Searching on forums gives no result, except other people are experiencing performance problems, sometimes on Windows 10, sometimes on Windows 8.1. Starting Live is now taking almost 45 seconds on this machine. Fortunately, the program is responding correctly for now, until of course I add enough tracks and effects to my Live set to make it choke up like crazy. I guess this will happen if I go far enough in music production.

Difficulties with music production itself

Creating the track I had in mind caused me great trouble. While not super complex, it is not a trivial repeat drum beat. I managed to play it a couple of times, started the recording on my EMX and messed it up completely. I tried again, messed it up again. I cannot play it reliably unless I try 25 times and more. The workaround is to correct notes, but this is quite tedious on the EMX. Tired of this, I tried to record MIDI using my Ultranova as a source and Live as a sequencer. But even from Live, fixing the incorrect notes was a real pain. I experimented with the quantization which also didn’t work correctly.

There is no well-defined workflows and no comprehensive tutorials about music production. All I can find is case-specific pro tips, sometimes involving plugins I don’t want to install yet. I’m just overwhelmed with Live itself, having to constantly check and redo what I am doing, this is not a great time to complicate stuff with plugins.

Conclusion

Although I am having less and less fun with all this for the moment, I feel I can manage to get something good out of it. If I gave up because of difficulties, I would not have been able to get a Ph.D, to keep my job for more than seven years and to create a modded Minecraft map.

An intricate audio puzzle

Since I moved to Windows 8, I am having multiple and increasingly annoying issues with my audio setup for computer-assisted music creation. I am slowly reaching a dead end that will force me to give up on creating music. At best, I will only be able to play some beats for fun with hardware synthesizers: no way to record, mix, apply effects etc., no way to ever come up with a full end-to-end song, just repeated audio patterns.

Initial setup

Here is my configuration:

  • Intel Core i7
  • 16Gb of RAM
  • 240Gb SSD
  • 1Tb HDD
  • M-Audio’s Fast Track Pro
  • (About to become infamous) Ableton’s Live 9.1
  • Infamous Windows 8.1

The M-Audio interface is accepting input from two TRS jacks. I was usually plugging the output of my Korg’s EMX+Kaoss Pad during session recording. The interface also has S/PDIF input that I was using to feed in audio from my Novation’s UltraNova hardware synthesizer. This trick allowed me to record four separate tracks with that otherwise stereo-only interface. Sound is output to two KRK audio monitors.

During a few months, this configuration worked correctly. However, from October 2014, things started to misbehave in multiple random ways. I initially thought this was because of my UltraNova, but problems persist even if I uninstall UltraNova’s drivers and unplug it from USB.

Here are the issues I am facing with the setup:

  • Some recording sessions go well, but after I close Ableton’s Live, turn off my devices and shut down my computer, Windows is stuck in a loop, incapable of completely turning off the machine. Screen remains black, computer fan continues spinning. Problem can happen if I leave the audio interface plugged and turned on, or if I turn it off.
  • Sometimes, sound has an incorrect pitch. There seems to be a mismatch between audio frequencies: Ableton’s Live sends 44.1kHz while audio interface plays at 48kHz. There is no solution, except repeatedly unplugging and plugging the audio interface from a USB port, try in another port, until it works.
  • Sometimes the M-Audio driver goes corrupt and cannot play sound anymore except through ASIO using Live. When this happens, I have to completely uninstall and reinstall the driver.
  • A few weeks ago, my UltraNova started to sound awfully distorted when sound was going through S/PDIF. Worried, I tried hooking up the synthesizer to my home theater AV receiver, through S/PDIF of course, and sound was super clean. Problem is thus caused by the M-Audio interface.
  • The behavior is different but incorrect on Linux: S/PDIF sound goes out distorted unless I lower my UltraNova’s volume to more than the half. Under Windows+Live, distortion happens at that volume level as well. As a result, it seems that some software component is involved in S/PDIF handling, and that component now misbehaves differently on Windows and Linux.

After I started getting distortion through S/PDIF, I felt it was too much for me and tried to change my configuration.

Second setup: another audio interface

My UltraNova offers an onboard audio interface. In order to isolate the M-Audio interface from the overwhelmingly complex equation, I decided to give it a shot. That resulted in the second setup described below.

  • Audio input is now plugged to my UltraNova’s jacks. I ended up plugging a mixer in order to get signals from my EMX+Kaoss Pad as well as a Nord Drum module I recently acquired.
  • My UltraNova sends audio to my computer through USB. As a result, sound from the synthesizer is transferred digitally while sound from my EMX and Nord is sampled by the UltraNova’s onboard chip.

I don’t like this configuration very much because my mixer has an annoying tendency to clip and I end up with a stereo mix of my recording, making it impossible to separate the tracks. I will thus be unable to experiment with mixing using this setup, unless I manually sample each track separately.

But this is not the main issue. Here are the problems:

  • I was able to record a couple of sessions of improvisation using the above configuration. However, a week ago, recorded sound started to be distorted. Even Live’s test tone, played through ASIO, then passing into UltraNova, started to sound distorted!
  • Not only playback is distorted but also recordings are not clean anymore, dirty of distortion. A promising session that could almost have been used as is for a sound track in a future Minecraft video got screwed up by this and is good for throwaway!
  • I got fed up and tried to disable ASIO, using DirectSound instead. That worked, no distortion, but more latency. I can even hear the latency when playing through Live with a MIDI keyboard.
  • Thinking it could be UltraNova’s ASIO driver, I tested with ASIO4ALL instead and got similar issues.
  • Yesterday, Live suddenly died and could not start anymore. For no obvious reasons, I had to reinstall it completely. After that, sound with ASIO worked without distortion, but playback and recording were frequently cutting.

Contradictions, no gos, no solutions!

  • My best friend thinks the issue is caused by Windows 8 and that I should downgrade to Windows 7. However, my machine takes forever, at least one hour, to install Windows, excluding drivers and applications, installation destroys Linux boot loader and it takes me at least half an hour to find out how to restore it because there is no builtin ways in Ubuntu to do so simply. Activation of Windows 7 is likely to fail because I upgraded my license to Windows 8, and I will be left with no solution if that happens, other than trying one crack after the other or installing some piece of crap that would make sure my system’s date is set to something that won’t go past the activation grace period. I just cannot accept to have all messed up file date/times because of a single piece of software.
  • I thought about purchasing a new computer, that would have only Windows 7 and be dedicated to music creation. However, there is no Windows 7 machine anymore; every new system comes with Windows 8. I could try, maybe I would be lucky and the new Windows 8 configuration would work better, but that is a hit or miss without any way to increase chances of success! Probably my best guess would be a custom-made machine with a Windows 7 license for it, if this can at all still be purchased.
  • Some forum posts pretend that Ableton’s Live will work fine on Windows 8 and 8.1, others not, others pretend it worked on Windows 8 and not well on Windows 8.1. Downgrading to Windows 8 is just a non-sense for me, better downgrade to Windows 7 instead. I’m likely to experiment the same activation issues and will loose the same amount of time reinstalling everything and repairing my Ubuntu configuration.
  • Some forum posts suggest that it will never work well on a PC and that I should try with a Mac. Well, I tried on a Mac, and that was the most awful, most frustrating experience through my whole life. The machine was awfully slow and keyboard was not responding in a deterministic way. From this experience, I had to conclude with concerns that the Mac will work well only when used with the mouse, keyboard is just for typing text in fields and emails!!! My visual impairment makes using the mouse a pain for me. Does that mean I am excluded from computer-assisted music creation, unless somebody helps me out all the time? I am slowly but surely reaching that very frustrating conclusion.

For now it somewhat works with non-ASIO configuration and stereo recording, but I know a lot more can be done. Does the problem come from Live, Windows, my motherboard’s USB ports? I just cannot figure out.

I started to investigate about ASIO itself, to try understand how and why it could misbehave. I may have to dig straight into ASIO SDK for that. I investigated about using ASIO with Audacity, because that could help me test without Ableton’s Live and eliminate it from the equation. Even that promises to be major hassle, forcing me to install Visual Studio, SDKs and compile the whole planet to get ASIO support into Audacity!

Each new issue is decreasing my motivation to persist. I have less and less fun playing with that music creation setup and I am often thinking about putting it aside. I’m not to the point of selling the components yet, but if I cannot find out something to unblock the computer part of it, I may come to that. I will at least wait for Windows 10 to come out and give it a shot before selling anything.

I hope some people will read that post and start thinking about concrete solutions, not just “it works for me, so no problem”, “buy a Mac”, “reinstall”, “try on your friend’s computer”. Maybe a Linux-based music creation platform could help, maybe a Linux distribution dedicated to music and providing just the needed components, nothing that can interfere, or something that would make Mac platform faster and more usable.

I am a software developer so I could help in developing the platform, but because Live is closed source, I cannot take what exists and improve over it, I would have to start from something inferior such as Ardour or Audacity, and rebuild/reinvent/rewrite on top of it. This would be a very time consuming experience, and without Ableton’s expertise, I would certainly do it wrong. If all Mac OS X was open source, if it could be compiled from streamlined autoconf-based build processes rather than XCode where small-sized fonts are king, I would be able to open the hood, examine the code and improve the GUI. No, I would have to start from scratch, using just the BSD kernel, and reinvent the WHOLE GUI. Without Apple’s experience, I would certainly do it wrong and messy!

Is Linux overusing hard drives?

Monday, May 18 2015, when I tried to synchronize files between my main computer and my HTPC, I got error messages from Unison telling it was failing for some files. Tired of repeated unexpected failures, I tried copying the files manually. Manual copy was failing too. I quickly noticed that the whole partition I was copying the files to turned into read only. I tried to enable read/write by remounting the partition, that worked, but it reset back to read only after a few minutes. Some Google search later, I was thinking my hard drive was failing. I tried running a self-test using sudo smartctl -t short /dev/sdb and after a few minutes, sudo smartctl -a /dev/sdb. The drive was now failing the short auto-test. Damn, again? Yep, another dead hard drive!

Short term fix

First step was to stop using that beast, so move away all I can before it goes even worse. A failing drive can always become totally unusable, corrupting ALL data on it.

Transfer of my music files from that failing drive (the 1.5Tb one) to another one (the 3Tb one) started to output multiple I/O errors. No, it will fail like this for half of the files and take forever! I aborted that and instead copied my music files from my main PC. At least I would get all the files, not half of them. Fortunately, I was able to transfer most of my ripped DVD and blu-ray disks from my 1.5Tb to my 3Tb drive. I finished the transfer the next day; it ran in the background while I was working from home. I could then unmount the 1.5Tb drive, putting it out of the way for Ubuntu.

I got a new 3Tb drive Friday, May 22 2015 and installed it the morning after. This time, I was able to pick the right drive to remove, because the 1.5Tb was colored green while the 3Tb was red. The new 3Tb is also a red WD model. Unfortunately, things are never smooth when modifying that machine: I had to remove my SSD drive+bracket assembly in order to unscrew the 1.5Tb hard drive. However, after that, I was able to install the new 3Tb drive, put back the SSD assembly and reconnect everything. This went surprising well as opposed to last time where I had trouble finding a way to connect power on the four drives.

The new 3Tb drive seemed to work correctly. I was able to partition it as GPT, create a 3Tb Ext4 partition and mount that partition. I then started a long self-test to make sure it will not fail at me just a couple of days after. The self-test completed during the evening and showed no error.

Tentative prevention

Problem solved? Partly. The thing is that machine runs 24/7 to power a Minecraft server. This makes both hard drives spin non-stop. I would like Ubuntu to stop the drives when they are unused. I moved the Minecraft files to my SSD and will use the hard drives only for media and backup.

No, Ubuntu never ever spins down any hard drive! I tried to set this up with sudo hdparm -S 241 /dev/sdb, no result. Only thing that worked is manually spin down the drive with sudo hdparm -Y /dev/sdb (or /dev/sdg for the other drive). I recently found that gnome-disks has an option to set drive spindown timeout. The spindown setting from gnome-disks was honored once on my main computer, but I need to check if it’s reliable or not.

What if it happens again and again?

Otherwise, it seems I need Windows just to get my hard drives automatically spin down when unused, which is quite a shame! I don’t want to format this HTPC as a Windows machine, because the Minecraft server won’t run smoothly on Windows. I will be stuck with an always-open command prompt window with the server running, unless I search forever to figure out a way to run this as a system service, assuming it is possible.

A colleague at my workplace suggested the use of a Ubuntu virtual machine, but my HTPC doesn’t have enough memory to reliably run a VM and I cannot bump it up more than 4Gb because of motherboard limitation. Well, I could try to stick it 4 4Gb DDR2 modules and see, but I’m not sure the board would accept this at all, even though that could fit physically! If that fails, I would be stuck with useless DDR2 while newer systems use DDR3. What a pain!

I also investigated the possibility of using a RAID to improve reliability of storage. If I put a third 3Tb hard drive, I could configure a RAID5 array of total 6Tb, and even increase to 9Tb with a fourth 3Tb hard drive! The RAID5 splits and mirrors the data in such a way that two drives are involved when accessing files, increasing performance. It also makes sure that if one drive fails, ALL the data can be recovered and the array can be rebuilt by simply removing the failed drive and adding a replacement drive.

I was tempted by this, but that would have forced me to purchase two hard drives and another PSU to have more than four SATA power connectors. I wasn’t sure I wanted to spend more than 300$ just to get this up and running. Moreover, creating the RAID array would have forced me to move all the files away from my current 3Tb drive to combine it with the two new drives, unless I jumped directly to the 4-drive array.

I will instead wait for that machine to die and next system could be a smaller SSD-only HTPC combined with a NAS offering easier drive installation and replacement. I could purchase a dedicated NAS, or build myself a generic computer configured as a NAS. Fortunately, Ubuntu has facilities to configure software RAID, I checked that recently. I’m not sure about the fake RAID using the motherboard, that may or may not work, that may or may not be better than software RAID.

Minor problems stacking up

It seems to happen too often to me. I’m ending up with many different small, minor problems, sometimes not big deals taken alone. But when they sum up, this becomes unbearable, resulting into a bad and exhausting day. Sometimes, the solutions are simple, sometimes not. Here is the most recent stack of such issues.

Spending my time importing modules in Python

For the moment, the only way I have to edit my Python code running on a remote virtual machine is to use Emacs running from that machine. I’m investigating local solutions, but this is just a non-sense chain of complications or requires software free for commercial use that I cannot adopt at work.

One operation that ends up to be frequent is to import a module. You are writing a piece of code and then need to call into a function defined in another module or in the standard Python library.  When this happens, I need to add an instruction to import the module if it is not yet from my current module. Import constructions can appear anywhere in the code, but convention puts the instructions at the beginning of the files. This seems better for code organization and ensures that all imports happen at start of the program rather than at any time during the execution. If a module imported at top of the file is missing, the code will fail fast, as opposed to fail only when a function importing a module is called.

As a result, I am sometimes editing code and need to step at start of file to add an import, then find back where I am and continue. This small interruption in task flow is not a big deal in itself but it becomes more and more painful when it repeats itself tens of times a day, sometimes at each and every successive line of code!

Maybe I went too granular and split my code in too many modules. Am I in a situation where it would be better to have one single huge file with a lot of stuff, rather than splitting my code in multiple files? Well, last Monday, I was at the point of wondering that.

How did I work this out before? When I was programming in Java, I was using Eclipse as my IDE. This program is able to guess the import statements from referred class names and automatically add them at the beginning of the files, without having me go there, loose my position and come back. Probably a Python IDE such as PyDev or PyCharms would do it, but I cannot use them for the moment.

Fortunately, I came up with a very simple trick. In GNU Emacs, you can split the window in two parts by using C-x 2. Both windows first point at the same part of the current buffer, but it is perfectly possible to move the cursor up. This changes the position of the current window, while leaving the cursor unchanged in the second window. I can then add my import statements, switch to the window with original cursor using C-x o, then make it the only window with C-x 1. Something similar can be done using bookmarks, but this requires other keyboard shortcuts harder to remember and each bookmark needs to be named while sometimes they are one-off save/restore scenarios.

Keyboard shortcut confusion

Since I gave up on Virtuawin because its behavior was too inconsistent on Windows 8, I ended up using Desktops from SysInternals which uses different keyboard shortcut than Ubuntu for switching desktops. I thought CTRL-ALT-F1 to F4 would be nice keys, but this ends up being a nightmare. As soon as I came back to Ubuntu after one week working with Desktops, I was screwed up, always making the same mistake of pressing CTRL-ALT-F1, which switched to the console. I then had to press CTRL-ALT-F7 to go back to X. This also happens in VirtualBox virtual machines.

There is no perfect solution for this. My current workaround is to use CTRL-F1 through F4 instead. At least, if I press that in Ubuntu, nothing happens, as opposed to wiping my screen away and forcing me to press CTRL-ALT-F7.

Windows 8 becoming increasingly disturbing

Even Desktops starts to be a nuisance because of Windows 8. As soon as I am on desktops 2, 3 or 4, I need to be careful not to press on the Windows key. If I do it, which I am used to, this switches to Metro which of course replaces my screen with the home screen. Then I am back to desktop 1. Metro is my most common way of starting applications on Windows 8: press Windows key, type a name, press enter. This also works great on Windows 7 and Ubuntu’s Unity.  But this fails with Desktops on Windows 8. There is no way out, other than putting EVERY application on the desktop and spending more than 30 seconds each time I want to start something! I just cannot do that, this is too inefficient, especially knowing that it will take half the time to find out the icons for a sighted person than for me with a visual impairment. I could work around by grouping the icons in some clever way, creating folders, but this remains clunky.

Windows 8 is also causing issues with Lync, the instant messaging software my company is using. Lync works correctly in text mode, but it intermittently fails for voice chat and screen sharing. When this happens, it consistently displays a network error, no matter how hard we try to establish the communication. We have to fall back on alternative ways, like traditional phone. The only solution has been to reinstall Lync.

IT people at my company were of no help. They would have liked me to try out Lync on Windows 8 in the office rather than finding what is going on. However,the Windows 8 ultrabook my company lent me is almost unusable for me unless I hook it to an external monitor and mouse. At the office, I don’t have an HDMI input I could use to plug my mini DisplayPort to HDMI adapter I use at home, and the mini DisplayPort to VGA adapter that could have worked went away a couple of months ago. Somebody borrowed it and never brought it back. Even if I had the adapter, performing this test would be tedious. I would have to wait for a voice chat and when it happens, quickly switch from my regular laptop to the ultrabook and try. This is just non sense stress that may just give no result: it will probably work correctly, showing that the problem comes from VPN, my router, etc.

There is no way out, other than running away from Windows 8. I thus stopped using the ultrabook, at least for the moment, and used the official laptop instead. That machines runs the good old Windows 7 OS. If Lync starts failing with that as well, then I will have to switch routers, try on somebody else’s Internet connection, etc.

SSH connection timing out for no reason

We recently switched from outdated Cisco’s IPSec VPN client to newer SSL-based AnyConnect. This seemed to go smooth, after I uninstalled and reinstalled VirtualBox (seems VirtualBox is interfering with Cisco’s VPN). However, I noticed that SSH started to time out at me when I wasn’t interacting with the shell for a moment. This was forcing me to reconnect, restart what I was working on, sometimes switching to a directory with a long path, sometimes Bash history was working, sometimes not.

My first attempt at working around unreliable connections was to make use of EmacsClient. For this, I started Emacs with the -daemon option, then used emacsclient instead of emacs. That works surprisingly well. With that, Emacs keeps running on the server and doesn’t loose track of unsaved files if connection drops. Of course, I keep the good habit of saving often, which is an additional elementary safety against unreliable connections.

Yesterday, I may have found out a way to get rid of these timeouts. My current hypothesis is that the new VPN client monitors the TCP connections going through the tunnel it establishes and shuts them off if they are inactive. Fortunately, SSH provides a way to keep connections alive: the ServerAliveInterval configuration option. By putting ServerAliveInterval 60 in my .ssh/config for my virtual machine, I am forcing SSH to send a TCP packet every 60 seconds so nor SSH server, nor VPN client, have a reason to kill the connection. This seems to help, but this is not fully tested yet.

The mouse making me mad!

Why does my Razor mouse was so jumpy? I feel I have less problem at the office than at home with the mouse? Will I really need to get myself a Dell mouse like the one I have at the office?  Is it because I am running crazy? Maybe not! I found out this week that my mouse pad is pretty suspect: it has multiple scratches on it that can well screw up the optical-based mouse tracking, and the surface has some patterns. I remember when I worked at my parents’ home having hard time with the mouse. That ended up to be because of the desk! Putting the mouse on top of a piece of white paper, yes, just a plain old piece of paper, nothing more sophisticated, cleared the issue! So I removed that mouse pad and that seemed to help! It is unbelievable how sometimes, simple solutions, almost non-sense stupid ones, can lead us far!

Networking issues on an ultrabook

Since a few weeks, I was experiencing sluggish performance while connecting to some SSH servers. The problem happened when using SSH through the VPN of the company I am working for. This resulted in lags when typing commands on the SSH terminal, which was more and more problematic because of more and more command-line arguments I had to pass and longer and longer file paths.

Too many layers = too many problems

My current working setup is far from simple, because I need to work on a remote virtual machine to have access to some data and run some processing on it. The official connection method using NX doesn’t work well for me, because NX suffers from intermittent keyboard issues, e.g., right alt stopping to work, server acting as if Shift was pressed while it is not, etc. This works correctly for somebody using the mouse rather than the keyboard or able, without significant loss of efficiency, to check every typed character for an eventual error. This is not my case.

I ended up building myself a multi-layer setup as follows:

  1. I am not hooked directly to the cable modem from Videotron. I am rather using a Linksys WRT310N router, and to make things more interesting, I am running DD-WRT, not the official router’s firmware. This rarely caused any issue, though.
  2. Windows running on a computer provided by the company. I am currently working from home using an ultrabook they provided me, with Windows 8.1 on it.
  3. The ultrabook having no Ethernet connectivity, I was using a USB to 100Mbps Ethernet adapter to get a faster and more stable connection than wi-fi.
  4. A Cisco VPN client is needed to access internal resources  of the company.
  5. The machine runs Ubuntu 14.10 in a virtual machine hosted by VirtualBox.
  6. Inside the guest Ubuntu, SSHFS is configured to access my workspace on the virtual machine, so I can use local editors like Emacs.
  7. Inside the guest Ubuntu, I open a terminal and SSH to the virtual machine to run commands there.

Phew! What a list of layers!

Different networking methods

VirtualBox offers several ways to manage networking. I am currently using the first of the three I know about. Here they are.

  1. Bridged. VirtualBox uses a trick I don’t know too much about to clone the host’s network interface and act as if there was a second interface. The guest OS receives its own IP address from my router and thus acts pretty much like an independent machine on the network. This implies that Ubuntu has to establish the VPN connection, but fortunately, there is a Cisco client available. However, when using that method, the Windows host doesn’t have access to internal resources, unless I establish a second VPN connection, on the host side. I am something tempted by the idea of having the router establish the VPN connection. This might be possible with DD-WRT, but this will introduce a security risk: what if I leave the VPN open after my working day or somebody hacks my network?
  2. NAT. VirtualBox acts a bit like an internal router, allocating a private IP to the guest. Requests are translated by VirtualBox to look like if they came from the host. This works correctly and allows the VPN connection to be established by Windows-based official Cisco client, but on the other hand, this introduces a level of indirection: the NAT applied by VirtualBox. Any indirection is subject to hinder performance.
  3. USB. VirtualBox has support for exposing USB devices to guests, so I could, at least in theory, expose my USB to Ethernet interface to Ubuntu. However, without Oracle closed-sourced extensions, I would get only USB1.1 support, which would result in slow or non-functional networking. The extensions are available only for personal or evaluation use, so I cannot use this at work. Even if I solve the licensing issue and get the extensions, with the USB solution, the Windows side would be unable to access networking, so I would loose access to Lync and Outlook. I could work around by turning wi-fi back on, but that starts to be clunky. If I have to go this way, I would be better off using my personal Ubuntu PC rather than a virtual machine.

It seems that 1 is a bit faster than 2, but I am not totally sure, no way to measure scientifically. Even with 1, I was still experiencing sluggish SSH. After switching from 2 to 1, this seemed a bit better, but performance degraded after a few minutes.

Could a better network interface help?

This morning, I tried with a TruLink USB to Gigabit Ethernet interface rather than the 100Mbps one. However, this didn’t go well at all. I got the following issues, all after the other.

  1. VirtualBox partially blocking network. At first, everything seemed to work well. I was able to browse the web and Outlook was working fine. But I soon discovered that Lync wasn’t connecting at all and although Cisco VPN was starting (from Windows, no virtual machine yet), it couldn’t access any internal resource. Trying Windows network diagnostic reported a potential driver issue. I tried to install the driver I found for this adapter, but it just failed; Windows already had the most recent driver built in or installed by IT. I then found out that Windows was connecting to an unknown network using the VirtualBox host-only adapter. So VirtualBox was in the way, partially blocking networking. I had to remove VirtualBox and reinstall it to fix this.
  2. Cisco VPN not working. After reinstalling VirtualBox, I was able to connect to Lync, but VPN was still non-working. It was connecting without issues, but it would allow access to absolutely no local resource. I tried to remove VirtualBox once again, to no avail. I had to remove Cisco VPN client, reboot to be sure everything was clean, reinstall client, test to see if things were back to normal (they were!), reboot once more to be sure, reinstall VirtualBox and test again! Why reboot? Well, if VirtualBox is installed while Cisco VPN is running, network connectivity stops working completely until VirtualBox is removed.
  3. Distracting side issues. During this frustrating troubleshooting, I got several other issues. Firefox took several seconds to start, once again, because I would ideally have to switch to Chrome, transfer my bookmarks from Firefox to Chrome once again, and live with a browser having very weak support for touch screen, at least at the time I am writing. Emacs, which I tried to open and use as a buffer because I wanted to try Ping commands and was always making typos, took at least one minute to launch, and started multiple instances when it finally unstuck. Then the main window of the Cisco VPN client remained open after connection and couldn’t be closed by the X button, Alt-F4 or any other normal way; I would have to live with it on an unused Virtuawin desktop or restart the client. I ended up shutting everything down and rebooting once again.

At least, after all these efforts, I got functional networking and experienced a lot less lags than during the last 2-3 weeks while working from home!

If problems come back, I will probably give up on using this ultrabook and revert to the official laptop provided by the company. The machine is heavier, doesn’t have any reasonable way to output to digital displays (I would either need to purchase a docking station specific to that machine, or try my luck with the mini HDMI port which sometimes works, sometimes not), but it has VGA output, it has Gigabit Ethernet and runs the good old Windows 7 which definitely seems to play nicer with the software tools of my company.

Groovy and Eclipse incompatible in practice

This week, I tried to get my Groovy+Eclipse setup back and running, because I needed to debug a complex script that was outputting 15-20 line stack traces with a lot of useless information about Groovy internals. Finding the exact error message was taking me several seconds and no matter how hard I was trying to fix things, I was getting other and other errors. A step by step debugger was more than welcome.

I looked once more at Groovy web site for the presence of the plugin for Eclipse 4.4. Up to last Monday, release was supposedly imminent. Searching on Google was giving information that the special STS bundle of Eclipse 4.4 was providing Groovy integration. Wow, will I really have to install a special version of Eclipse to get Groovy now? I didn’t want to get STS which will give me Grails, Spring, etc, which I don’t need.

But last Monday, release was there. I tried to install it, but I found out it couldn’t install because of a compatibility problem with some of the features I had in my environment. It is possible that the plugin works only with the original version of 4.4, not the SR1. I tried a couple of times without success, then fell back on the development build. This one installed. I’m not sure it will be stable, but it was better than nothing. However, the same way as in August 2014, I was consistently getting the following error message:

Conflicting module versions. Module [groovy-all is loaded in version 2.2.1 and you are trying to load version 2.3.6

No matter how hard I searched for a solution, there was NO result. I opened my Maven POM file in Eclipse, pulled the dependency hierarchy, searched for Groovy, and found only one Groovy-All dependency. Nothing other than Groovy 2.3.6 was pulled by transitivity from other dependencies. Any search on Google was yielding results about Grails. Some people fixed the issue by removing the GROOVY_SUPPORT container from the build path: well, no GROOVY_SUPPORT in my build path.

I then thought about running a simple Java program in the same project as my Groovy script and output the class-path from the System.getProperties() object. Looking at the JARs in the class-path, I found no Groovy dependency. From my dummy program, I then tried to call my Groovy script. The script is compiled just as a regular class with a main method that I called… with success!

I then found out that you can execute a Groovy script as a regular Java Application in Eclipse: no need to run as a Groovy script. The script runner most likely involves a wrapper with a Groovy 2.2.1 dependency. This wrapper is built into Eclipse’s Groovy plugin. Running directly as a Groovy class avoided the introduction of the Groovy 2.2.1 dependency in the class-path alongside Groovy 2.3.7 so I got passed the hurdle and was able to more easily debug my code! During the process, I updated to Groovy 2.3.7 to match what was used by Groovy Eclipse.

The newest version of the Groovy plugin behaved a bit better, allowing me to use conditional breakpoints a couple of times, as opposed to the last version for which that was consistently failing with compilation errors.

For the complex stack traces, there is a simple solution: the StackTraceUtils class has methods to sanitize stack traces, removing frames related to Groovy internals. I just wrote a small uncaught exception handler and registered that as the default handler, and tada, more compact exception traces! Of course, I have to do it for each of my scripts, but that’s just a one-liner since I put the code registering the default handler in a utility method.

Audio driver conflict causing slow startup

Since at least two months, Windows 8.1 startup is awfully slow. From UEFI boot to login screen, it takes a few seconds, which is perfectly reasonable and what I expect from a Core i7 computer equipped with a SSD. However, after I was typing my password to login, it was taking between 30 to 45 seconds before I could reach the desktop! This was almost wiping out the benefit of the SSD, making the total boot time as long as if I had a regular hard drive with an older CPU.

I was getting annoyed on every startup, but at least, the machine was correctly responding afterwards. But yesterday, that combined with another issue: each and every Metro application is now closing instantly and NOTHING suggested on forum posts can fix the issue; the only solution seems to completely reset my user account.

I thought the slow login time was also linked to my user account and was starting to feel ready for creating myself a clean account and starting from there. Most applications should survive this process, only some settings will be lost. My documents live on a second hard drive which wouldn’t be affected. However, I found out that the slow startup was also experienced for a new user account! This was thus a system issue, probably yet another program that installed malware without me noticing it.

Searching on forums, it seems I would now have to hunt for and install many different anti-virus and anti-spyware programs and regularly create hard drive images. I didn’t have to run into this trouble while I was using Windows 7? What changed in Windows 8.1? Is the system now so unstable that I would need to regularly restore from an hard disk image? This is a real non-sense!

Deeper analysis of the slow startup seemed to link this to CPU! There was little hard disk activity during the long login time and CPU fan seemed to increase speed. I was quite shocked at this. I have a Core i7 CPU that works very well for Ubuntu. Why would Windows 8.1, suddenly, require something more? It was working six months ago, starting quickly. My Nuance ultrabook, which also has a Core i7, starts at normal speed. My personal ultrabook, equipped with a Core i5, starts in a reasonable time as well. Both machines are equipped with Windows 8.1 and newer than this 2012 Core i7, but they are NOT significantly more powerful! Do I really have to face the fact there is a time bomb in Windows, that will trigger after some time and start marking some CPU/motherboard combinations as arbitrarily obsolete? This makes little sense, but I was slowly but surely drifting to that conclusion. I was awfully disappointing because it takes me weeks to shop for new computer parts, check for compatibility with Linux, make sure I won’t get into a dead end with the motherboard, assemble the thing, test, solve issues, etc. I didn’t want to reenter into this just because Microsoft decided I would have to do so!

I didn’t want to perform a clean install, because it is taking too much time. It would break my GRUB configuration while Ubuntu provides no simple way to restore it (each time I need to search more than fifteen minutes, and apply a manual procedure), would require installing multiple drivers each rebooting the machine, and I wasn’t sure Ableton Live would correctly reauthorize. But I was starting to feel ready to try this reformat during the Christmas holidays, because this was becoming too annoying.

I thought about purchasing another system and moving the Windows 8.1 part of my current configuration on it, leaving only Ubuntu on the original machine. That would remove the issue with Windows breaking the GRUB setup at the cost of more money and space.

I also got issue with my audio system. Yesterday, I started Ableton Live and opened up a Live set I got from my friend. I got an error message, because my Ultranova synthesizer was set as the ASIO audio interface but it wasn’t powered up. Instead of turning it on, I switched to the M-Audio ASIO driver, so sound played through my main Fast Track Pro interface.

Things went well… until I closed Live. After that, for the third time, sound stopped working. The M-Audio driver is getting corrupted and stops working. Restarting the machine does nothing; I really had to reinstall the driver.

But after I removed this damned M-Audio driver and restarted the computer, I found out the startup speed was restored to normal! I reinstalled my M-Audio driver, because I need it for ASIO integration, and startup speed remained normal!

I thought about finding an audio interface more suitable for Windows 8, but I found out it makes no sense to replace a working product just because Windows misbehaves with it while it USED to work! I thought about downgrading to Windows 7, but I HATE its low contrast between selected and unselected items and lack of any solution to fix this without disabling Aero. I considered the possibility of installing Live on a dedicated Windows 7 laptop (tired of assembling computers and when purchasing preassembled PCs, you usually get to choice between cheap desktops, reasonable laptops or high-end desktops with too stylish gamer-centric cases). But the new machine would have low storage so I would need some kind of NAS for my systems to access a common storage. That was starting to be endless stacks of problems! I thought about using my Ultranova as sound card for Windows and M-Audio on Linux (because the Focusrite-based audio chip in the Ultranova does not work under Ubuntu) but that was resulting in need for endless reconfiguration (when switching between the two OS).

But today, things are still working. It seems I just must not use the Ultranova ASIO integration in Live and things will continue working. As a workaround for this uncommon issue, I connected the S/PDIF output of my synth to the S/PDIF input of my audio interface using a plain basic RCA cable (nothing fancy), turned on S/PDIF output on my Ultranova and was able to get the digital audio from the S/PDIF input of my audio interface. This has the added benefit of allowing Live to record BOTH from S/PDIF and the main input jacks of the audio interface. Using this new configuration, I was able to record four tracks at a time: the two one from my synth through S/PDIF, and the two outputs of my Korg’s EMX. S/PDIF input has low volume for I don’t know why, but at least this can be worked around in Live.

I hoped that would solve my Metro issue but no, the problem persists. I tried a lot of stuff to solve this without success. I will thus need to reset my user account if I really need these kind of useless Metro apps. At least startup is now normal so I won’t have to format to get this fixed!