Categories
Bug Configuration

Ubuntu 18.10: a silent release

Usually, upgrading Ubuntu goes well. I run the Upgrade tool which downloads new packages, installs them and then asks me to reboot. Some new versions had minor issues, for example Wayland not fully compatible with my NVIDIA card or an old version of MATE preventing the dist-upgrade, but nothing major, nothing that couldn’t be worked around. This time was different: no sound, and no way to easily get around this.

Cannot dist-upgrade

Sunday, October 21 2018, Ubuntu 18.10 was released since a couple of days. However, after I installed the updates through the Software update tool, I wasn’t proposed to upgrade. I had to dig into the software options and reconfigure the delivery of releases to “every release”, not just the LTS. Then I reran the Software Update tool, and got the option to upgrade. However, clicking on the upgrade button did just… nothing. I tried two or three times: same result.

Before accepting that this time, I would have to go through the clean install route, I searched a bit on Google, and found out that sometimes, the upgrade doesn’t start when updates are still available. But updates were all installed. However, running the following on a terminal found and installed a couple of additional updates.

sudo apt update
sudo apt dist-upgrade

I then retried the Upgrade button, and that worked! Yeah!

The upgrade went well, and then I was offered to reboot, which I did.

Slow and silent

First the new release felt a bit sluggish, then pretty flaky. First time I tried to reboot because I needed to go to Windows, to play Minecraft loaded by Twitch and recorded through OBS, the system hung up waiting for a stop job. I am getting this issue from time to time and when that happens, I have to wait for the 1min30s timeout to elapse before the stop job is forcibly killed. I don’t know what is a stop job, what job is frozen and how I could get rid of this issue for good.

I was in my living room turning off my TV and AV receiver while my new Ubuntu setup finally rebooted, and I didn’t have time to select Windows in GRUB, so it restarted Ubuntu, and trying to restart redid the “stop job” issue! Argh, don’t tell me I’ll get this each time I reboot or shutdown now? I’ll have to wait 1min30s, maybe even 3min, just to power off or reboot. What’s the point of having a SSD if timeouts like this counter its performance benefits?  I got fed up and powered off my PC with the power button. I had to press and hold the button almost ten seconds for the machine to finally turn off, then I had to press the button 2-3 times for it to finally turn on. Maybe I’m heading towards the necessity of replacing my computer CASE, which sucks me to the point of thinking about getting rid of that damned thing and use just a laptop. But that case issue has nothing to do with Ubuntu, I should just have picked a better case, there’s nothing more to it, at least for this post.

After my Minecraft session, which was kind of fruitful, I wanted to check that my Ubuntu installation would boot again and be able to shutdown at least once without the “stop job” issue. I thus rebooted to Ubuntu, but the system froze before showing the welcome screen. I had to press CTRL-ALT-F2 to reach a console, log in, check the syslog, nothing of interest. Then the system finally booted into X. I wanted to switch back to the console to log off, but when coming back to X, it froze again on a black screen, this time no key was working. Then another hard reboot!

Second attempt worked: I reached the desktop. I launched the backup script for my Minecraft world, then the MKV->MP4 batch conversion script. OBS records in MKV, which is more robust against crashes, but VideoStudio doesn’t accept MKV, so I have to turn the recordings into MP4. Fortunately, FFMPEG does it without loss, by just repackaging the MPEG stream into another container. Then I wanted to organize my videos into directories, so I launched one to check it, and found THE thing: no sound!

Checking pavucontrol, I found that my sound card disappeared. Only detected devices were my HDMI port hooked up to my monitor, and a crappy USB Webcam that could serve as very basic and rudimentary mic. I don’t use that for my Minecraft recording; I have an AKG real microphone for that! I tried to reboot to no avail. Trying to run speaker-test just hung, again, needed to press ALT-F4 to shutdown the terminal.

Searching on Google for solutions lead to nothing except old stuff that didn’t work. Some people restored sound by reinstalling ALSA and PulseAudio. Others had to downgrade to the previous kernel. Others edited configuration files, commenting a line that is not there in my case, and adding another line. I tried to reinstall PulseAudio with sudo apt install –reinstall pulseaudio, to no avail. As a last resort, I tried to reboot with the 4.15 kernel of Ubuntu 18.04,

After almost an hour of searching, it was more and more obvious that I needed to give up on Ubuntu 18.10 and either downgrade (which essentially means reinstall) to Ubuntu 18.04, or switch to a new distribution. I was quite annoyed, as preceding upgrades went right, and then the hardware problems start again like in the past. Moreover, the system was freezing for a second each time I hit Tab while in a terminal, before displaying completions.

Can this work at all?

I had to go sleeping (it was past 11 PM), go to work the day after, but I thought about it. First I needed to test if that can work! For this, the simplest solution is to test using a live USB. Monday morning, I had to resist the temptation of testing that before going to work. Doing that would have made me start late and thus finish work late in the evening. So I did my workday first. So on Monday evening, I needed to download the Ubuntu 18.10 ISO; I picked the MATE one since that’s what I’m using now, instead of that GNOME3 thing which works so so. I got the ISO, and used the disk creator tool built into Ubuntu to write it into a USB drive. However, the tool refused to format my drive: too small for the new 2Gb ISO! That old 2Gb key was just a bit too small, which is kind of annoying. As a smart man, I should have extra empty, unused USB keys hanging around, but seems I’m not smart enough for that! So I had to use an existing key.

So I took the Clonezilla key, a 16Gb device, really too large for that small backup tool. I stored Ubuntu on it, and then I installed Clonezilla on the old 2Gb stick. Then I booted off my new Ubuntu medium, the system booted successfully, and I got sound! Ok, so at worst, if I clean install, I should get sound… except maybe if that is an incompatibility with the NVIDIA driver.

Unlocking ALSA

First I ironed out the NVIDIA hypothesis by uninstalling the NVIDIA proprietary graphic driver. After I reboot, I tested and still had no sound, and was back to a default VESA resolution. I thus reinstalled the driver, a bit annoyed that Nouveau cannot even basically handle my graphic card. I don’t expect full 2D/3D acceleration out of Nouveau, but at least, mode switching should work with a card bought in 2013, reaching a 1080p resolution. No, nothing. But at least, there was no incompatibility between the graphic and sound driver.

I then dug further into ALSA, which was still detecting my sound chip. Here is the list of devices it found:

eric@Drake:~$ aplay -L
default
    Playback/recording through the PulseAudio sound server
null
    Discard all samples (playback) or generate zero samples (capture)
pulse
    PulseAudio Sound Server
sysdefault:CARD=PCH
    HDA Intel PCH, ALC887-VD Analog
    Default Audio Device
front:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Front speakers
surround21:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    2.1 Surround output to Front and Subwoofer speakers
surround40:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    4.0 Surround output to Front and Rear speakers
surround41:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    4.1 Surround output to Front, Rear and Subwoofer speakers
surround50:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    5.0 Surround output to Front, Center and Rear speakers
surround51:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    5.1 Surround output to Front, Center, Rear and Subwoofer speakers
surround71:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    7.1 Surround output to Front, Center, Side, Rear and Woofer speakers
iec958:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Digital
    IEC958 (S/PDIF) Digital Audio Output
dmix:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Direct sample mixing device
dmix:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Direct sample mixing device
dsnoop:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Direct sample snooping device
dsnoop:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Direct sample snooping device
hw:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Direct hardware device without any conversions
hw:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Direct hardware device without any conversions
plughw:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Hardware device with all software conversions
plughw:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Hardware device with all software conversions
hdmi:CARD=NVidia,DEV=0
    HDA NVidia, HDMI 0
    HDMI Audio Output
hdmi:CARD=NVidia,DEV=1
    HDA NVidia, HDMI 1
    HDMI Audio Output
hdmi:CARD=NVidia,DEV=2
    HDA NVidia, HDMI 2
    HDMI Audio Output
hdmi:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 3
    HDMI Audio Output
dmix:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Direct sample mixing device
dmix:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Direct sample mixing device
dmix:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Direct sample mixing device
dmix:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Direct sample mixing device
dsnoop:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Direct sample snooping device
dsnoop:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Direct sample snooping device
dsnoop:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Direct sample snooping device
dsnoop:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Direct sample snooping device
hw:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Direct hardware device without any conversions
hw:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Direct hardware device without any conversions
hw:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Direct hardware device without any conversions
hw:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Direct hardware device without any conversions
plughw:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Hardware device with all software conversions
plughw:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Hardware device with all software conversions
plughw:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Hardware device with all software conversions
plughw:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Hardware device with all software conversions

The speaker-test command was hanging, but if I waited enough time after pressing Ctrl-C, it returned. Ok, cool! Running pavucontrol while pseaker-test was playing its noise, I could see it in the PulseAudio applications. Ok, so speaker-test

What if I switch device? I tried speaker-test -D sysdefault:CARD=PCHI got an error because ALSA couldn’t open the device. Searching on Google about that, not specific to Ubuntu, lead to clues: it could be a permission issue. Trying to check /dev/dsp failed: no such file. But I could fiund the following.

eric@Drake:~$ ls -ld /dev/snd/*
drwxr-xr-x  2 root root       60 oct 23 08:11 /dev/snd/by-id
drwxr-xr-x  2 root root      100 oct 23 08:11 /dev/snd/by-path
crw-rw----+ 1 root audio 116,  9 oct 23 08:11 /dev/snd/controlC0
crw-rw----+ 1 root audio 116, 15 oct 23 08:11 /dev/snd/controlC1
crw-rw----+ 1 root audio 116,  8 oct 23 08:11 /dev/snd/controlC2
crw-rw----+ 1 root audio 116,  6 oct 23 08:11 /dev/snd/hwC0D0
crw-rw----+ 1 root audio 116, 14 oct 23 08:11 /dev/snd/hwC1D0
crw-rw----+ 1 root audio 116,  3 oct 23 08:11 /dev/snd/pcmC0D0c
crw-rw----+ 1 root audio 116,  2 oct 23 19:23 /dev/snd/pcmC0D0p
crw-rw----+ 1 root audio 116,  4 oct 23 08:11 /dev/snd/pcmC0D1p
crw-rw----+ 1 root audio 116,  5 oct 23 08:11 /dev/snd/pcmC0D2c
crw-rw----+ 1 root audio 116, 10 oct 23 08:11 /dev/snd/pcmC1D3p
crw-rw----+ 1 root audio 116, 11 oct 23 08:11 /dev/snd/pcmC1D7p
crw-rw----+ 1 root audio 116, 12 oct 23 08:11 /dev/snd/pcmC1D8p
crw-rw----+ 1 root audio 116, 13 oct 23 08:11 /dev/snd/pcmC1D9p
crw-rw----+ 1 root audio 116,  7 oct 23 08:11 /dev/snd/pcmC2D0c
crw-rw----+ 1 root audio 116,  1 oct 23 08:11 /dev/snd/seq
crw-rw----+ 1 root audio 116, 33 oct 23 08:11 /dev/snd/timer

Ok, interesting, only root and users of group audio can access these devices, and thus play sounds. Am I part of the audio group? I tried groups eric and found out I wasn’t. But on my HTPC still running Ubuntu 18.04, I was! Lucky I had that HTPC, otherwise I would have been forced to reboot to the Live USB to check, or boot another machine with it, like my ultrabook.

Ok, so what if I add myself to the group?

sudo usermod -a -G audio eric

I had to relog, and then got sound through ALSA using Speaker-Test! Yeah! But still no PulseAudio! I tried to reboot, not just relog, to no avail. Ok, at least I can reconfigure Audacious to play through ALSA, but I suspect I’ll get trouble for YouTube and Spotify for which I cannot configure audio output.

Unlocking PulseAudio

I tried all sorts of things to debug this. I was able to get some logs out of PulseAudio several ways, but the most useful way was through SystemD. In Ubuntu, PulseAudio is started by a userspace variant of SystemD. There is a command allowing to get the logs: journalctl. I couldn’t figure out a way to filter so I ended up calling journalctl -a -f on one console, and then pulseaudio -k on another to force PulseAudio to restart. Then I was checking the produced log. I ended up finding out errors. The system was not able to communicate through DBus using a certain socket. I started to think that each time the system freezes, this is because PulseAudio tries to emit a sound, has to wait and timeout.

I couldn’t find out how to reconfigure DBus, or understand how PulseAudio and DBus interact enough to troubleshoot this. I was quite stuck, and all I could find was deprecated information. There was a bug report that seemed to affect Ubuntu 18.10, but no solution. It was past 10 PM and I was on this for almost two hours, trying and searching to no avail, more and more pissed off and risking to be so angry that I would end up having trouble to sleep. I was about to give up or reinstall from scratch.

I ended up fed up and decided to completely get rid of PulseAudio using sudo apt purge pulseaudio. After a reboot, sound was WORKING! What??? How come MATE plays sound without PulseAudio? Audacious was working, and VLC as well. But not Firefox for YouTube, and not Spotify.

Before accepting the sacrifice of YouTube and Spotify to avoid a clean install, at least for now, I tried to reinstall PulseAudio. For I really don’t know why, the sound continued working, and PulseAudio displayed my internal audio device now. YouTube and Spotify resumed working. I read posts about some people that got the issue fixed and it came back the next reboot. Ok, let’s reboot and hammer that bug for good, then! I rebooted, and sound was still working. I still don’t fully understand why.

Either something messed into the groups I was member of and PulseAudio got screwed up because I lost permission to use ALSA, either some configuration files needed to be updated but APT decided to keep my current versions. Purging PulseAudio removed the configuration files and reinstalling reverted to sane defaults. At least, sound is working now.

Even better, the system seems more responsive now and didn’t freeze on startup. It really seems that PulseAudio and ALSA had trouble communicating, causing these hangups.

Why not a clean install?

Because my home directory lives on the same partition as my Ubuntu install. Any attempt to put my /home on a separate partition leads to insufficient disk space after a time. I have a 250Gb SSD shared with Windows so I cannot put 50Gb for my /home and 30Gb for just Ubuntu. One simple solution would be to move my /home to an hard drive. As long as Ubuntu is on a SSD, I’ll have a fine boot time. Or I would need a way to use SSD only as a cache and put everything on the hard drive. I could also come back to the dual SSD strategy: one SSD for Windows, one for Ubuntu. I’ll think about it, but at least I don’t have to do anything short term. Maybe I can wait and replace everything, and get a better case to fix the power button hickup at the same time…

If everything had failed, before the clean install, I could have tried to restore my Clonezilla image of my SSD, and that would have got me Ubuntu 18.04 back to normal. In case of failure, then I would have had to clean install 18.04, 18.10 or something else, or just give up on Linux for the moment. At least this is not necessary anymore.

Categories
Configuration

Ubuntu 16.04 almost killed my current HTPC setup

Yesterday, I tried to upgrade my HTPC running Ubuntu 14.04 to the new LTS 16.04. That almost went smooth, but some glitches happened at the end and some changes prevented my Minecraft FTB server to start again. The problems are now solved, but I was wondering if I would be able to get this working again.

I had two hopes with this upgrade: get an intermittent awful audio glitch fixed and have the ProjectM visualization work again. From time to time, when I start the playback of a video file, I’m hearing an awful super loud distortion instead of the soundtrack. I then have to restart playback. Usually, that’s enough, sometimes, I have to restart it twice. Fortunately, audio doesn’t go crazy during playback. ProjectM visualization started to fail, I think since Kodi 1.16. It just doesn’t kick in, leaving me a blank screen. At least Kodi doesn’t crash or freeze as some versions of XBMC were doing when ProjectM was unable to access Internet reliably.

CloneZilla failing to start

The week before the upgrade, I wanted to backup the SSD of my HTPC using CloneZilla in case some problems happened. I used an old version I had burned on a CD because I thought this 2009 HTPC wouldn’t boot USB sticks. Well, that old version, although working on my main PC, failed to start on my HTPC. It was simply freezing without any clue of what was happening. Before trying to download the new version and burn it on a CD, I noticed that my external USB hard drive was showing up in the boot up options when pressing F8 at computer startup. I thus tried to boot my CloneZilla USB stick running a more recent version and that worked. I don’t know if my HTPC was always able to boot off USB, maybe this capability got added by a BIOS upgrade. That was a good thing, and allowed me to perform my backup.

Dist-upgrade or clean install?

Several people on forums recommend to perform a clean install, claiming that too much things changed from one version to the other. That may be true in some cases, and that’s probably the safest route, but unfortunately, the clean install doesn’t always detect the drives to mount, requiring time-consuming modifications to /etc/fstab (with copy/pasting of drive UUIDs) and then I would have to figure out what packages were previously installed and reinstall them. I also have a couple of Cron jobs performing automatic backups of my Minecraft worlds that I would need to recreate.

Instead of doing that, I tried to use the Update Manager to perform a dist-upgrade. Unfortunately, by default, the tool won’t go from one LTS to the other. You have to go all the way through 14.10, 15.04, 15.10, then 16.04! Each dist-upgrade would have taken at two hours, making this process a really painful non-sense. Instead, I tried calling update-manager -d and got the option to go from 14.04 to 16.04!

During the installation, I thought that if the power supply of this relatively old system died during the process, the system would probably be unrecoverable, requiring a backup restore or clean install. Aouch! Luckily, no such thing happened.

TeXLive broken

During the dist-upgrade, I got some error messages because the updated TeXLive-related packages couldn’t be configured properly. Why is TeXLive installed on this HTPC? I don’t remember exactly. I don’t need to compile any LaTeX document on this machine so this didn’t seem an issue at all for me. I just asked the installer to ignore the errors and noted down to myself to delete the TeXLive packages after the upgrade to be sure not to run into issues if, for some obscure reasons, I wanted to compile a LaTeX document later on.

Failed dist-upgrade

Unfortunately, the dist-upgrade aborted with an error, no accurate information, just a message telling that the dist-upgrade failed. Argh! The system couldn’t shutdown or reboot anymore, even when running sudo reboot from the command line. I was so frustrated that I considered shutting down this machine, which caused me issues after issues since more than seven years, and never turn it back on again. If I weren’t able to recover from this failure, I could however have restored my CloneZilla image after taking a break from this catastrophic upgrade. In other words, everything wasn’t lost.

I tried pressing the power button a couple of times, the screen became blank and remained blank for a few seconds, then the stupid machine rebooted. At least, the broken Ubuntu installation started up to the GUI. Assuming the main issue was this TeXLive glitch, I opened a Terminal and tried to remove the TeXLive package: sudo apt-get remove texlive. This failed. Apt-get was reporting errors about the TeXLive-related packages that weren’t configured. I tried to remove the package using dpkg, which complained that texlive wasn’t an installed package. I then tried searching for the packages using apt-cache pkgnames tex, and ended up removing tex-commons. That got rid of the incorrectly configured packages and unblocked apt-get.

After this, I ran apt-get update, then apt-get dist-upgrade. That installed a couple of additional packages. Then I ran apt-get autoremove to remove the obsolete packages. This, hopefully, completed the dist-upgrade. I also rebooted to make sure the system could still boot after that.

OpenJDK 8 causing issues

This HTPC is running a Minecraft world my friend and I are sharing. We log less and less often onto that map because my friend plays rarely and I am currently focusing on Agrarian Skies 2 rather than this old FTB Monster pack the map runs on. But I I am considering the possibility of starting a map on FTB Infinity Expert Skyblock pack after I’m done (or completely blocked) with Agrarian Skies 2 and would like to run it on a server with an auto-backup strategy in place and the possibility for friends to join in if they want. I thus wanted to keep the possibility of running Minecraft servers on my HTPC.

Now, when I started the FTB Monster server, I was greeted with a meaningless ConcurrentModificationException. I may be able to retrieve the stack trace, but this is a bit pointless, referring repeatedly to non-sense internal class names. Ok, this is probably broken because of Java 8 and won’t get fixed unless I upgrade the mod pack, which will either force me to start from scratch on a new map, or require hours and hours of work to convert the map, and the map would be quite damaged after the upgrade. In particular, switch to Applied Energistics 2 mod will destroy my logistic network so much that it will require a complete redesign and rebuild. This will be even worse than the switch of Thermal Expansion and IC2 that occurred when I migrated (painfully) from Unleashed to Monster.

Simple solution: run this under OpenJDK 7. That’s simple under Windows, unfortunately… Yep, no available OpenJDK 7 package on apt-get for Ubuntu 16.04! Maybe I could have fiddled something with PPAs or install Oracle’s JDK outside of the apt-get packaging system, but what’s the point of having a packaging system if it requires so many workarounds? I also thought about running the server into a Docker container constructed from an image proposing Java 7, but that’s a bit convoluted and could cause other issues. Who knows if the server will behave well when running in a Docker container? It will probably, but that remains to be tested.

Fortunately, I figured out a way to patch the installation by adding a new JAR to the mods folder. The JAR comes from http://ftb.cursecdn.com/FTB2/maven/net/minecraftforge/lex/legacyjavafixer/1.0/legacyjavafixer-1.0.jar and was recommended by a forum post on http://support.feed-the-beast.com/t/cant-start-crashlanding-server-unable-to-launch-forgemodloader/6028. Installing the JAR fixed the issue and allowed me to start the server!

Totally unexpected, very frustrating

In order to test my Minecraft server, I started the FTB Launcher on my Ubuntu 16.04 main computer. From the launcher, I started the FTB Monster pack: crash. OpenJDK 8, again. I had to apply the JAR patch on my client as well. I did it (instead of fiddling to manually install JDK 7) and that worked. I was able to log on my server and enter my world. However, as soon as I pressed F12 to go full screen, screen went blank and everything was blocked. No way to go out of the game by switching desktop, no way to kill the game window with ALT-F4. I would once again have to go to another machine, SSH into my main computer, kill the JVM, fail, try with kill -9. Instead, I just rebooted the machine, tried with Windows, and that worked. My Minecraft setup was correct. Just the client now requires a different video card or driver to work reliably on Ubuntu, but I changed from onboard Intel HD to a NVIDIA GeForce addon card in 2013 just for that reason. Having to switch back and forth graphic cards from Ubuntu versions to versions is a total non-sense for me.

Kodi is gone

I don’t know exactly how that happened, but Kodi, the new name of XBMC, got removed during the upgrade. Just reinstalling it was simple and enough to fix this. Kodi still works fine, for music and video playback. ProjectM visualization is still broken, though, but that’s not a big deal. I didn’t hear the audio distortion since the upgrade, but it’s too recent to tell if it’s gone for good or not.

Conclusion

For now, I’m not sure it was worth it but at least it didn’t break things. Main functionalities of my HTPC are still there: Minecraft server runs, I was able to listen to YouTube videos, Kodi works for music and videos, SSH is  working properly. I’ll have to see if other surprises are awaiting me.

Categories
PoneyMac

Ubuntu on my Mac: possible but limited

I read quite a lot of wrong and contradictory information when I searched for feasibility of installing Ubuntu on my 2006 MacBook Pro. Part of the problem is that users don’t all understand the pieces of the puzzle involved to boot a Mac into Ubuntu, but the multitude of variants of MacBook Pro, with varying technology in them, causes confusion and contribute to user’s ignorance.

First, Mac doesn’t implement the traditional BIOS used to boot nowadays PCs. They rather implement an Extensible Firmware Interface (EFI), a more modern way of booting computers than the BIOS, dating from the 80s. Newer PCs start to replace the obsolete BIOS with such an EFI, called Unified EFI or more commonly UEFI. No problem, I thought at first, Ubuntu now supports UEFI booting. I UEFI boot onto Ubuntu several times a week. However, Apple introduced several tweaks to the EFI, making it almost unusable for other purposes than booting Mac OS X.

In my particular case, the Mac’s EFI was 32 bits, which means I could EFI boot only 32 bits operating systems. However, although Ubuntu supports UEFI booting, it doesn’t on my Mac, maybe because the Mac’s EFI is too different from a UEFI. Apparently some people successively UEFI booted their Ubuntu installation, either by using unofficial Mac variants of old Ubuntu versions or a kind of patchy program EFI booting and offering the Ubuntu’s ISO as a kind of virtual device. The later requires a version of Ubuntu supporting loopback booting, and that’s again only old versions. Quite annoying, isn’t it? I don’t want to install a 12.04 then dist-upgrade to 14.04, then 14.10!

Note that newer Mac computers have a 64-bit EFI, and it is well possible that it is able to boot Ubuntu and maybe even Windows, in a native way. That means no need for BootCamp anymore to hack the GPT to fake a MBR; both Ubuntu and Windows supports booting from UEFI and GPT!

While I thought Mac’s EFI only supports native EFI booting, I discovered that it has the necessary compatibility module to boot into BIOS-based environment. Maybe I got this CSM added by a firmware upgrade I obtained when installing all available updates. That means a Mac can boot a Ubuntu live DVD in BIOS-based mode. In theory, that would mean it can boot a 64-bit DVD even though it is 32-bit EFI. In practice, no, reason still to be determined. I thus can only boot 32 bit OS.

Moreover, no matter how hard I tried, I wasn’t able to boot Ubuntu off a USB key. I tried installing rEFInd to avoid having to press the infamous defective Option key at boot to get Apple’s boot menu (using an external keyboard), but rEFInd can only boot from the hard drive, ignoring the USB key or telling Apple’s firmware doesn’t support well booting from an external device.

At least, Ubuntu boots from the DVD. In theory, I should be able to install it on a partition on my hard drive and boot from there, but I didn’t do it for the moment.

Categories
PoneyMac

Preliminary results with Ubuntu booting on Mac

After my more and more frustrating failures with Mac OS X, I wanted to do something to give a second life to this apparently dead system. My previous attempts and research about Ubuntu installation on MacBook Pro 2006 have been a real disaster. But this evening’s attempt is a step toward success!

I first downloaded the 32-bit Ubuntu 14.10 ISO. I already had the 64-bit one. Then I burned two DVD disks: one for the 64-bit, one for the 32-bit. It is always handy to have Ubuntu disks; I consider this an essential part of my geek’s toolbox. Because the Mac’s EFI is 32-bit (an unfortunate ill-design given the CPU is 64-bit capable), I tossed a 32-bit Ubuntu DVD in the machine and tried to reboot. This time, Mac OS X stuck at a blue screen, incapable of shutting down or rebooting. Grrr! What a pain in the neck! I forcibly shut this cursed system down by pressing and holding power button.

Then I powered on again and as soon as my external keyboard’s Razer logo turned on, I pressed and held the Alt key. After a tremendous amount of time, almost 45 seconds, I got the choice between the DVD and hard drive. I took the DVD, had to wait again, and finally got Ubuntu booting! Yes, that damn thing can boot Ubuntu 14.10!

I reached the live screen and was about to open a terminal, resolution is correct, audio seems there and system seems responsive. Of course, the DVD drive is just way too slow. I would have to install Ubuntu onto the hard drive to assess performance in a more thorough way.

More than that, I couldn’t find /sys/firmware/efi, which means a non-EFI boot happened. This means the Mac’s firmware DO has CSM, and thus can boot in legacy mode. I still have to try, but I suspect I could give it a 64-bit Ubuntu and bump up memory to 4Gb! I am more confident Ubuntu will accept almost any brand of memory modules while Mac OS X might be more picky and just accept Apple-approved ones.

This is a good preliminary result, something that will allow to sleep better tonight and have a more enjoyable beginning of week!

Categories
Configuration

One SSD for my HTPC

A bit more than a month ago, I successfully transferred my dual boot Windows 8.1 and Ubuntu 14.04 from two 120Gb Solid State Drives (SSD) to a single 240Gb drive.  I got several problems restoring the bootloaders of the two operating systems, thought many times I would have to reinstall, then figured out a way to make them boot.

But what happened to the two drives I removed from my main computer? Well, they sat still on the top of a shelf. But at least one drive will be repurposed: become part of A.R.D.-NAS, my HTPC. Sunday, October 26 2014, I finally got the time and courage to undertake the transfer operation. This time, the software part was pretty smooth, but the hardware part was a uselessly intricate puzzle. During the process, I wondered myself many times about the purpose of generic hardware if it doesn’t fit well together and pestered about the lack of any viable alternatives.

The sacrifice

Well, my NMedia HTPC case has six 3.5″ drive bays. This is quite nice for an HTPC case. This is possible, because I chose an ATX case, to get a motherboard with rear S/PDIF audio connectors rather than just headers accepting brackets I could get nowhere. This case is a bit bulky; I would build off a MicroATX case if I had to start from scratch.

So installing this SSD seemed obvious at start: just add the drive, transfer the Linux boot partition from the hard drive to SSD, remove the original boot partition, setup GRUB on SSD and tada. No, things are rarely as simple. I thought my motherboard had only 4 SATA ports, and they were all used: one 1Tb hard drive, a second 1.5Tb hard drive, a third 3Tb hard drive, then a blu-ray reader/writer. Why so many hard drives? Well, I am ripping and storing all my video disks, even the huge blu-rays, to avoid the need for searching for them on shelves.

Even if I remembered correctly I had six ports on the motherboard (two are free!), my PSU only had four SATA power connectors, so I would not be able to easily and reliably connect all my drives. I could try to find some splitter cables or molex to SATA adapters, but that would add a factor of failure. I could replace my PSU for one with more SATA power cables, but it would also have more molex cables, PCI Express connectors, etc. Unless I went with a more expensive modular PSU, all these cables would have cluttered my case.

Safest and cheapest solution was to sacrifice one of the hard drives, the 1Tb one of course, the smallest. I thus had to move files around to have less than 120Gb of stuff on the hard drive that would be moved to SSD. That process took a lot longer than I thought. My poor HTPC spent the whole Saturday afternoon copying files around! Fortunately, this is machine time so I had plenty of time to experiment music creation with my still new UltraNova synthesizer combined with Ableton’s Live multri-track ability.

Preparation

On Sunday, I first burned the Ubuntu 14.04 ISO on a DVD. Yes, Ubuntu is now large enough to fit only on a DVD. After that, I shut down my Minecraft server running on my HTPC and moved its files to another old PC. I started the server on the old PC and reconfigured the port mapping on my router. This way, if my friend wanted to kill a bit of zombies and creepers while I was installing my SSD, he would be able to do so and I would not be stressed if something bad made my HTPC out of service (like something stuck in the CPU fan breaking it).

I then removed the cover of my HTPC and spent quite a bit of time trying to figure out what was the 1Tb hard drive. Based on the position of the SATA connector on the motherboard, I presumed that was the left-most drive. I thus had to disconnect the drive in the middle bay, and use the freed up power and data connectors to hook up my SSD. I then booted up the machine.

Following picture shows the drive temporarily hooked up.

DSC03934

Then I remembered about my old 22″ LCD that I stopped using after purchasing my Dell touch screen. I went pick it up in my computer room, put it on my kitchen table and plugged it in. This way, I would be able to have the screen right in front of me, keyboard on the table, rather than in front of my 46″ LCD HDTV with the keyboard on my knees.

The SSD and the LCD hooked up, I booted up my HTPC and quickly sticked the Ubuntu DVD in the blu-ray drive. After an incredible amount of time, the machine finally booted up into Live Ubuntu DVD!

Data transfer

After Ubuntu started, I launched GParted and realized that I chose the wrong hard drive. The 1Tb drive containing my Ubuntu setup was disconnected. Oh no! So that means I will have to turn off the machine, connect the right drive and wait once again for this stupidly long, almost five minute, DVD-based boot? No, not this time! Feeling like a cowboy, I decided to try something: drive hot swapping. This is possible with SATA, so let’s see! I thus disconnected the 1.5Tb hard drive, starting with the SATA data cable, then the power cord, then hooked up the 1Tb drive. Hearing the old hard drive coming back to life was kind of thrilling. Everything went well, no cable stuck into my CPU or rear fans, and the PC didn’t freeze like it would do with IDE. The hot swap worked.

After that, this was relatively straightforward. As with my main PC, I used GParted to transfer my Linux boot partition and reconstruct the layout. I fortunately remembered, before, to reset the partition table. If I didn’t do that, the GPT that was on my SSD would have caused booting issues that would have drove me mad! I would probably have ended up reinstalling everything, angry against Ubuntu, the technology and probably the whole human kind. A single step, recreate the msdos partition table from GParted before the transfer, saved me that!

Following picture shows my LCD on which we can see the progress of the transfer.

DSC03935

See how bulky was this setup: HTPC on the floor, case opened, SSD hanging on top. Hopefully it was possible to make this setup clean once again after all this.

DSC03936

The home partition: too big to fit on the SSD

Unfortunately, GParted didn’t want to transfer my home partition to the SSD, because it was obviously too large. I could have shrunk it in order to copy it, but I wanted to avoid altering the hard drive in case something bad happened. I thus instructed GParted to simply create a blank Ext4 partition and used cp to perform the copy. The following terminal session shows how I managed to do it in such a way that all files metadata (timestamps, ownership, permissions) was preserved.

ubuntu@ubuntu:~$ mkdir /media/old-home
mkdir: cannot create directory ‘/media/old-home’: Permission denied
ubuntu@ubuntu:~$ sudo mkdir /media/old-home
ubuntu@ubuntu:~$ sudo fdisk -l /dev/sda

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e2c4d

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *          63    40965749    20482843+  83  Linux
/dev/sda2        40965750    81931499    20482875   83  Linux
/dev/sda3        81931500  1953520064   935794282+   5  Extended
/dev/sda5        81931563  1943286659   930677548+  83  Linux
/dev/sda6      1943286723  1953520064     5116671   82  Linux swap / Solaris
ubuntu@ubuntu:~$ sudo mount -t ext4 /dev/sda5 /media/old-home/
ubuntu@ubuntu:~$ ls /media/old-home/
eric  lost+found  mythtv
ubuntu@ubuntu:~$ sudo fdisk -l /dev/sdg

Disk /dev/sdg: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000d9a0c

   Device Boot      Start         End      Blocks   Id  System
/dev/sdg1            2048    40968191    20483072   83  Linux
/dev/sdg2        40968192    81934335    20483072   83  Linux
/dev/sdg3        81934336   234440703    76253184    5  Extended
/dev/sdg5        81936384    92170239     5116928   82  Linux swap / Solaris
/dev/sdg6        92172288   234440703    71134208   83  Linux
ubuntu@ubuntu:~$ sudo mkdir /media/new-home
ubuntu@ubuntu:~$ sudo mount -t ext4 /dev/sdg6 /media/new-home/
ubuntu@ubuntu:~$ sudo cp -a /media/old-home/* /media/new-home
ubuntu@ubuntu:~$ ls -a /media/new-home/ -l
total 36
drwxr-xr-x  5 root root  4096 Oct 26 19:49 .
drwxr-xr-x  1 root root   100 Oct 26 19:44 ..
drwxr-xr-x 69 1000 1000 12288 Oct 25 22:52 eric
drwx------  2 root root 16384 Sep 26  2009 lost+found
drwxr-xr-x  3  122  130  4096 Jan 24  2011 mythtv

The main idea is to mount both the old and new partitions, then use cp with -a option and root access (with sudo) in order to preserve everything. The operation went smoothly.

The boot loader

Even after copying all Ubuntu-related data from my old hard drive, my SSD was still not bootable. To make booting off the SSD possible, I had to install GRUB. Unfortunately, reinstalling GRUB on Ubuntu is not as simple as it should be. If there is a package doing it, why isn’t it built into Ubuntu’s image? Maybe because for most setups, reinstalling from scratch takes 15 minutes. That’s true, but then how about tweaks to fix mouse pointer too small, make XBMC work with S/PDIF sound, reinstall MakeMKV, etc.? Each step is simple, at least when no unexpected difficulty creeps in, but the sum of small things to tweak makes it long.

So let’s avoid this by running the following!

ubuntu@ubuntu:~$ sudo mkdir /media/ubuntu
ubuntu@ubuntu:~$ sudo mount -t ext4 /dev/sdg1 /media/ubuntu/
ubuntu@ubuntu:~$ sudo mount --rbind /dev /media/ubuntu/dev
ubuntu@ubuntu:~$ sudo mount --rbind /sys /media/ubuntu/sys
ubuntu@ubuntu:~$ sudo mount --rbind /proc /media/ubuntu/proc
ubuntu@ubuntu:~$ sudo chroot /media/ubuntu
root@ubuntu:/# grub-install /dev/sdg
Installing for i386-pc platform.
Installation finished. No error reported.
root@ubuntu:/# update-grub
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-3.13.0-37-generic
Found initrd image: /boot/initrd.img-3.13.0-37-generic
Found linux image: /boot/vmlinuz-3.13.0-36-generic
Found initrd image: /boot/initrd.img-3.13.0-36-generic
Found linux image: /boot/vmlinuz-3.13.0-35-generic
Found initrd image: /boot/initrd.img-3.13.0-35-generic
Found linux image: /boot/vmlinuz-3.2.0-61-generic
Found initrd image: /boot/initrd.img-3.2.0-61-generic
Found linux image: /boot/vmlinuz-3.0.0-17-generic
Found initrd image: /boot/initrd.img-3.0.0-17-generic
Found linux image: /boot/vmlinuz-2.6.38-12-generic
Found initrd image: /boot/initrd.img-2.6.38-12-generic
Found linux image: /boot/vmlinuz-2.6.32-25-generic
Found initrd image: /boot/initrd.img-2.6.32-25-generic
Found linux image: /boot/vmlinuz-2.6.31-21-generic
Found initrd image: /boot/initrd.img-2.6.31-21-generic
Found linux image: /boot/vmlinuz-2.6.28-16-generic
Found initrd image: /boot/initrd.img-2.6.28-16-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
Found Ubuntu 14.04.1 LTS (14.04) on /dev/sda1
done

The main idea here is to create a chroot environment similar to my regular Ubuntu setup, then install GRUB from here. I ran update-grub to make sure any disk identifier would be updated, pointing to the SSD rather than the old hard drive. Unfortunately, a small glitch happened: update-grub detected the Ubuntu setup on my hard drive. To get rid of this, I had to unmount the old hard drive and unplug it! After rerunning update-grub, I got the correct configuration.

Updating mount points

Since I rebuilt the home partition rather than copying it, its UUID changed so I had to update the mount point in /etc/fstab. I thus had to run the following:

root@ubuntu:/# cd /dev/disk/by-uuid/
root@ubuntu:/dev/disk/by-uuid# ls /dev/disk/by-uuid/ -l | grep sdg6
lrwxrwxrwx 1 root root 10 Oct 26 15:37 fb543fcb-908a-463d-bc1f-896f1892e3ad -> ../../sdg6
root@ubuntu:/dev/disk/by-uuid# ls /dev/disk/by-uuid/ -l | grep sdg1
lrwxrwxrwx 1 root root 10 Oct 26 16:11 54f4cbd6-aed0-4b43-91c0-2f8d866f3ee3 -> ../../sdg1

After I figured out the UUIDs, I had to open up /media/ubuntu/etc/fstab with gedit and make sure the mount points were correct. I only had to update the UUID for the /home partition.

Test boot

After all these preparatory steps, it was time for a test! I thus powered off the computer and made sure my old 1Tb hard drive was unplugged and my new SSD was hooked up. I turned on the PC and waited for the forever lasting BIOS POST. Why is it so long to boot a desktop while a laptop almost instantly hands off control to OS? After BIOS handed off control to OS, I got a blank screen with a blinking cursor, nothing else. I tried a second time: same result.

So after all these efforts, do I really have to format and reinstall from scratch? It seems so. Before doing that, I rebooted my machine once again and entered into BIOS setup by hitting the DEL key. Once there, I looked at the hard drives and found that the SSD was hooked up but it was not at SATA port 0.

I turned off the machine and connected the drive into a different port, what seemed to be the first. Looking into the BIOS setup again, my SSD was now at port 0. Ok, let’s try that a last time!

After a blank screen lasting too many seconds for a SSD boot and making me fear for a frustrating reinstall, the Ubuntu logo appeared, and my desktop finally came up! A quick check confirmed me that all the hard drives were present, except of course the disconnected 1Tb one. The SSD was ready to be installed into the machine!

The hardware part

The downside of SSD is that they seem not to fit in any regular desktop cases, only in laptops! This is a very frustrating limitation. Why are these drives all 2.5″ or why cases don’t have 2.5″ bays? When I shopped for my computer case, only high end ones had the 2.5″ bays and that was coming with fancy mechanisms to make the drive removable without plugging any cables, something adding into problem factors. Maybe at the time I am writing this post, some cases with SSD bays are available, but that doesn’t matter; I won’t change my case unless I really need to!

Before installing the SSD, I first removed that old 1Tb drive. I just had to remove four screws from my drive cage and slide the drive out.

DSC03938

DSC03939

To help me install my SSD into my HTPC case, I had a bunch of screws as well as an OCZ bay bracket. Just screwing the drive into the adapter’s tray took me forever, because I had trouble finding screws that fitted, there was a screw in one of the SSD hole I don’t know exactly why and that took me almost five minutes to realize. I then had trouble aligning the screws with the hole, was getting more and more tired and prone to drop screws, etc. At least, the screw I dropped fell on my table so I didn’t have to search it on the floor forever.

Following picture shows the drive in the bracket.

DSC03937

Then I had to screw that assembly into the drive cage of my case. Unfortunately, the upper bays of the cage only offer bottom holes while the SSD adapter has only side screws! I thus had to screw the adapter in one of the bottom bays, which are definitely suited for hard drives with their rubber pads to absorb vibration. None of my screws fitted well. It seems that the SSD adapter has holes smaller than normal while the screws for the drive bay are larger than usual! I got it after more than 15 minutes of attempts. I thought many times I would have to postpone this job and wait for my father to come by with a drill and make some new holes into the SSD bracket or the case.

Following picture shows the drive in the cage.

DSC03941

I don’t know exactly how much time I spent on this installation, but at the end, I was tired and was asking myself if all this would be worth it in the end.

Well after the SSD was screwed and the drive cage back into my HTPC case, I realized I wouldn’t be able to hook up my four SATA drives! No matter what I tried, there was always one drive lacking power. This was because the SATA cable coming out of my power supply unit were too short to accommodate the drive layout I came up with! Ok, I’m at a dead end now.

Before giving up and bringing that beast to a computer store in the hope they would figure out a way to hook the four drives up (maybe with some extension cable I don’t have, or using a new PSU), I remembered that the 1Tb drive I removed was in the middle upper bay which was now empty. My only hope to get the drive powered this day was thus to move one of my hard drives there. Ok, so let’s remove the cage again and play with the screwdriver once more!

I moved my 3Tb drive from the side bay to the upper one and put the drive cage back into my case. I was then able to hook up power. Reaching the SSD drive in the side bay to hook up SATA cables was a bit tricky, but I finally got it. A last check confirmed that all my drives were hooked up, except my blu-ray writer. Ok, just a cable to plug in, and that was it!

Was this all worth it?

After all this hassle, I asked myself this question. When I booted up the machine, it seemed as slow as with the hard drive. What? Maybe the CPU is too slow, after all. But when I reached the desktop and started XBMC, I felt the system was more responsive.

More importantly, the machine became a lot more silent. Since a few weeks, this HTPC was making a lot of noise. I thought it was the CPU fan stressed out by the Minecraft server running on the system, but the 1Tb hard drive was contributing to the noise as well. I suspect it was emitting more and more heat, causing the temperature to raise inside the case and heating up my poor little CPU. The CPU fan was then reacting by spinning like crazy.

Even after I restarted my Minecraft server, the sound didn’t come back. I am still surprised by this effect which I didn’t expect.

This 1Tb hard drive is definitely getting old and emitted some suspect sounds a few times. I am wondering if it would have failed and died if I left it in the machine. This SSD move thus saved me an unexpected reinstall and will help me have a better time with this HTPC.

So yes after all it was worth it!

Categories
Configuration

One SSD instead of two: simpler or not?

My Core i7 machine, named Drake, had two 120Gb SSD drives. I purchased the first one with the machine and put Windows 7 and Ubuntu on it. Then I needed more space to get Mac OS X, so I added a second 120Gb SSD. Mac OS X became a pain, almost unusable because everything was too small. When I reached the point I had to lower screen resolution to get Thunderbird running comfortably, I got rid of Mac OS X. Then Windows 7, upgraded to Windows 8, started to eat up more space so I needed to move Ubuntu to the second SSD.

I ended up with a brittle configuration composed of the ESP (EFI system partition) on the second SSD, Windows 8.1 on the first drive and Ubuntu on the second. I was waiting for a special deal on a 240Gb SSD and finally got one on TigerDirect at the beginning of September 2014. However, purchasing the SSD is only the easy part. Migrating data from two SSD drives to a single one, with Windows 8.1, Ubuntu 14.04 and UEFI in the way, is an incredible source of headache. This page shows how I got it done.

The easy way: reinstall everything

That would have worked ten, maybe even five years ago. Yes, just reinstall Windows, a few drivers, a few programs, put back Ubuntu, perform some settings, fine tune a bit, and enjoy the rebirth of the system, coming back to life and full functionality. Things changed with years, not for good. Now that Microsoft and other hardware manufacturers assume people won’t install by themselves and rather purchase hardware with everything preinstalled and preconfigured, things became more and more time consuming to setup. Just installing Windows 8 takes more than 45 minutes, and although I could obtain a DVD with Windows 8.1, my Windows 8 serial number won’t work with it. I would have had to install Windows 8, then upgrade to Windows 8.1 again!

Then come the drivers. Since I purchased my motherboard before Windows 8 was released, all my motherboard CD has to offer is Windows 7 drivers. So I cannot use the easy auto-install tool performing an unattended setup. I rather have to download every driver separately from Asus, run them, wait, reboot, run the next one, etc. Then there is the NVIDIA driver, requiring 100 Mb of download and yet another installation taking more than five minutes, and yet another reboot. Maybe I chose the wrong motherboard. By sacrificing a few USB ports, S/PDIF audio and maybe some PCI Express slots, maybe I could get something simpler not requiring as many drivers, that would be able to make use of what is prepackaged within Windows. That’s still to be investigated.

Then come the programs. Yes, Ninite can install me many programs automatically but not GNU Emacs, GNU GPG, it won’t configure my Dropbox, resync my Firefox bookmarks, reinitialize my Thunderbird email settings. It won’t link back my Documents, Images, Music and Videos default folders to my data hard drive.

And then come the licenses. How Windows 8.1 activation will behave? Will it happen smoothly, or will Windows decide that this change of SSD is too much and require me to call Microsoft to perform activation by phone, forcing me to exchange, by voice, on a poor channel, tens of nonsensical digits? After Windows 8.1 activation, my DAW, Live from Ableton, also requires authorization. I’m not sure it will reauthorize, since I activated it on my main PC as well as my ultrabook. That means additional hassle.

Bottom line, reinstalling is a pain, and that is just the Windows side. Ubuntu installation is usually smooth, but when a single thing goes bad, it requires hours of Google searches.

This is why I wanted a better way. I was so tired of this tedious process I was considering giving up on this machine and use my ultrabook instead, if data transfer failed. But my ultrabook, with its 128Gb SSD, won’t have enough storage for editing music made of samples or recording/editing Minecraft videos.

Preliminary connection of the new SSD

Before installing the new 240Gb SSD into my system permanently, I wanted to be sure I would be able to transfer my two operating systems (Windows 8.1 and Ubuntu 14.04) and make them boot. I thus only plugged the disk rather than attaching it right away into my case. I fortunately had some free SATA power cables as well as an extra SATA cable and port. That allowed me to connect the new drive without disconnecting the others. This way, it would have been easy to roll back in case of difficulties forcing me to reinstall everything, and then think about another strategy or gather my courage and patience for the full reinstall.

I then booted from a USB stick with a Live installation of Ubuntu 14.04. This was necessary to perform the data transfer on a totally offline, clean, file system.

Before transferring anything on the drive, I ran a SMART self test. For this, I installed smartmontools with apt-get and ran sudo smartctl -t long /dev/sdb. At this time, /dev/sdb was the device of the drive. That took almost an hour, but I could leave this running and do something else.

The self-test found no defects. I learned to do this preliminary step the hard way when I assembled a machine for my parents. The hard drive failed short while I was configuring Windows and I had to RMA it. Performing a self-test may have avoided me a waste of time and some frustration.

The drive being clean from any defect, at least from the point of view of the self test, I moved to the next step: data transfer.

GParted is the king!

A long time ago, my only friend for partitioning and drive transfer was Parition Magic, from PowerQuest, now purchased by Symantec. That time is over, thanks to GParted, a free open source tool that comes with Ubuntu. But that time, my job was pushing GParted to the limits. Here are the operations I needed to perform with it:

  1. Create a GUID Partition Table (GPT) on the new SSD. This is because I want a pure UEFI-based system. But this is not strictly necessary since the drive is far from the 2Tb limit!
  2. Copy the first partition of my second SSD at the beginning of the new drive: this is the ESP.
  3. Copy the first partition of the first SSD: this is the 128Mb system reserved partition of Windows. That copy wasn’t possible, because GParted didn’t know the partition type. I thus left a 128Mb hole declared as Unformatted, to figure out a way out later on. I was hoping Windows could recreate the data on this partition.
  4. Copy the second partition of the first SSD: this was the Windows main partition.
  5. Copy the 40-ish Gb partition of my second SSD at the end of the new drive: this was my home drive from Ubuntu.
  6. Copy the 20-ish Gb partition of my second SSD at the bottom of the free space on new drive: this was my main Ubuntu installation.
  7. Create an extra 20 Gb partition on the new drive in case I would like to give a shot to a new Linux distribution.
  8. Create a 16Gb swap space on the new drive for Ubuntu’s use.
  9. Resize my Windows main partition to take the rest of the space.

Phew!This long sequence gathering pieces from different sources reminds me of infusion crafting in the Thaumcraft mod of Minecraft, where essentias and items are combined together on an altar to craft powerful magical objects.

I hoped that sequence would work, but that failed at step 5. For no obvious reason, GParted wasn’t able to copy my Ubuntu home drive at the end of the new SSD! I had to leave an 8Mb gap and then resize the partition to fill it. I then performed, one by one, the other operations. That was a quite tedious job, because the mouse pointer was too small and impossible to enlarge without a system hack (Ubuntu bug since 11.10! They chose to remove the option to resize mouse pointer rather than fixing the issue.) and sometimes clicking was opening the menu and closing it right away rather than leaving it open.

Following image gives the final layout. Isn’t that great? Not sure at all this is simpler with one drive than with two, after all…

gparted

After this transfer process, I tried to recreate the entries in my UEFI’s NVRAM, using efibootmgr, for Windows and Ubuntu. I then unplugged the SATA cables of my two 120Gb SSD drives from my motherboard and rebooted the PC. I won’t state the exact commands I used here, because that just failed. System wasn’t booting at all.

Fixing Ubuntu

Back to my Ubuntu live USB, after at least five attempts because my motherboard is apparently defective and misses the F8 key from time to time and the need to jump into Setup and change the boot order from there to boot the UEFI USB stick. Boot time with that Asus board is desperately long. Waiting 15 to 20 seconds from power up to boot loader is a shame when knowing it takes less than 1 second on a 300$ laptop! But the laptop lacks storage expandability I need, so I am always stuck on one end or another.

Then comes the fun part. I am pretty surprised there is no easier ways to restore GRUB than the following. I read about boot-repair, but it is just missing, probably yet another PPA to copy/paste and install. Anyway, I ended up getting it to work.

First I found the partition where Ubuntu was installed, /dev/sda5, and mounted it: sudo mkdir /media/ubuntu && sudo mount -t ext4 /dev/sda5 /media/ubuntu. I did the same with my ESP: sudo mkdir /media/efi && sudo mount -t vfat /dev/sda1 /media/efi.

Second step was to establish bindings:

sudo mount –rbind /dev /media/ubuntu/dev
sudo mount –rbind /proc /media/ubuntu/proc
sudo mount –rbind /sys /media/ubuntu/sys
sudo mount –rbind /media/efi /media/ubuntu/boot/efi

That caused some directories inside my Ubuntu mount to mirror exactly the top level directories.

Then I had to chroot into my Ubuntu, using

sudo chroot /media/ubuntu

After all this, system was behaving a bit the same way as if I started a shell on my Ubuntu setup. From this, I tried

sudo upgrade-grub2

That just updated GRUB’s entries, not the EFI one, so didn’t fix the boot.

Then I tried

sudo grub-install

If I remember well, no arguments were necessary, and that fixed my GRUB EFI and added back the Ubuntu entry to NVRAM. This worked only after /boot/efi was correctly referring to my ESP. Note however that for this to work fully, the USB live Ubuntu had to be booted in UEFI mode, not MBR default mode.

A reboot later, I was starting my Ubuntu setup, fully intact and working! Half of the transfer done! Not quite…

Windows was failing to boot and Ubuntu’s update-grub wasn’t detecting Windows anymore. Quite bad.

Windows: desperately dead

Windows, on the other hand, wasn’t booting at all. It was showing a blue screen suggesting me to use the repair tools from Windows DVD. Last time I did this, the tools ran for at least one minute and bailed out, so I had to do a complete refresh which ended up wiping everything and leaving only applications from the Windows store. If I have to choose between such a messed-up repair and a clean install, I would bet for the second option.

Before entering into this reinstall nightmare once again, I tried to recover the reserved partition. For this, I plugged back my Windows 120Gb SSD and booted from my live USB stick to make sure Windows would not kick in and see two copies of itself (one on the old, one on the new SSD). If Windows sees two copies of itself, it changes the disk ID of one copy. If the new drive is changed, everything is messed up and Windows cannot boot anymore, until a refresh is done (and then everything is messed up again!). Back to my live USB, I used DD to transfer the bytes of the old reserved partition to the new one. I also made sure the new /dev/sda2 reserved partition was marked as such in GParted, by modifying the flags. That changed nothing.

The post How to repair the EFI Bootloader in Windows 8 literally saved me hours of work! This gives a procedure that allows to fix the boot loader. The main idea is to log into console from Windows DVD and run bootrec /fixboot command from directory EFI\Microsoft\Boot\ of the ESP, followed by bcdboot  with a couple of arguments, again from the ESP. Luckily, I had my ultrabook, which was quite handy to check the page while I was running the commands on my primary PC.

That solved the issue and allowed me to boot into Windows 8.1! PHEW! Quite a nice step forward.

GRUB not detecting Windows

Now that my machine was able to boot into both Windows and Linux, one could wonder what was missing. Well, I had no easy way to choose which operating system to boot at startup. Originally, GRUB was offering me an option to boot into Windows or Ubuntu. After the transfer, it was only seeing Ubuntu.

I found procedures to manually add an entry for Windows but that involved finding and copy/pasting drive UUID and probably redoing the change on each kernel update. I didn’t want that. Another possibility was to install an alternative EFI boot loader like rEFInd, but these have tendency to display many unwanted icons doing nothing. I got enough trouble with this while fiddling with triple boot (Windows, Linux, Mac OS X).

There was absolutely no way out. People were doing the manual addition of Windows or that was working out of the box. I had to spend more than 45 minutes inspecting the os-prober script and walking through it! By looking at the script and its logs in /var/log/syslog, I manage to find out it was skipping my ESP because the partition was not flagged as Boot! I fixed that from GParted, reran sudo update-grub and tada! GRUB was seeing Windows!

This is NOT the end!

Then I had to proceed with the hardware installation of the new drive. Since I was too impatient to get a SSD, I ended up with an ill-designed system. If I had waited another year before purchasing my Core i7 PC, I would have got a superb case with support for SSD drives. Now I have a CoolerMaster kind of case with only standard 3.5″ drive bays and need to fiddle with SSD brackets. Screwing the SSD drive in this is a painful process of trial and error. Then the assembly doesn’t fit well with the screwless mechanism of the case. This somewhat holds in place, but that’s not smooth installation like a regular 3.5″ drive.

Some more fiddling later, my new SSD was plugged back into my PSU and motherboard, and I got rid of the extra two SATA cables. I stored them away; they will be useful sooner than later, because my two 120Gb SSD won’t remain unused.

I plan to put one of them into my HTPC, which will be another adventure of its own. My HTPC has only four SATA ports, all used up, so I will have to get rid of one hard drive.