Categories
Bug

The down side of a NAS

Recently, I almost lost a bunch of ripped blu-rays, DVDs and downloaded movies and TV series. I thought a RAID5 would preserve that reasonably well, but I didn’t consider carefully enough recovery scenarios in case the whole NAS dies. I learned how a NAS can be expensive and paint myself in a corner, but I also learned more about logical volumes that could greatly improve my Linux installation in the future.

A NAS: a great idea at start

I used to store movies and TV series on separate drives. After I almost filled up 2 3Tb drives with ripped blu-rays, I wanted to do better than just adding another drive. Otherwise, I would end up checking all three drives to find out a given movie or TV series. I thus needed a solution to combine the storage space into a single pool.

One way of doing that is a Redundant Array of Inexpensive Drives (RAID). This can use several drives to improve reliability (mirroring), performance (striping) or both. This also combines the space of multiple drives, resulting into a virtual device with more space. I wanted to go with a RAID5, because that resulted in more space, with the possibility of one drive dying without loosing access to the data. Of course, if a drive of a RAID5 dies, you need to replace it as soon as possible so the array can rebuild and be reliable again, otherwise, if another drive fails, data loss occurs. The larger the storage pool is, the more catastrophic is the data loss!

Unfortunately, at the time I investigated that, there was no user interface in Ubuntu I knew to set up a RAID (that may have changed). One needed to copy/paste obscure commands from web pages. I got tired of such processes and wanted some easier way, other than trying with Windows because I don’t have a Windows license for my HTPC with the storage on it.

I thus wanted to experiment with a Network-Attached Storage (NAS) device. I picked the TS-453Be from QNAP, because I wanted a 4-bay device that is not too expensive. I found a lot of 2-bay devices which would have prevented me from building a RAID5, forum posts suggesting that RAID5 is not great because all data is lost if more than one drive dies, that a mirrored RAID is better, etc. Because I didn’t have 4 drives yet, I started with 2 3Tb drives with the project of expanding to four.

One good side of these devices is their ability to be configured remotely. Instead of hooking the device to an HDMI port (but you can do so if you want to) and interact with it using a screen, you can point a web browser to its interface and configure via a web browser.

Downside is the quantity of settings that made little sense to me, and some still do remain obscure, like the JBOD and iSCSI. It was relatively easy to get a RAID setup, but I quickly noticed that only a portion of the space available on the drives was usable by the volume containing the data. This is because the physical space can be used several ways by the NAS, not just for storing data. One can create snapshots or split the space into multiple logical volumes. Volumes can also host virtual machines. I had an hard time finding how to expand the logical volume with my contents because the option in the UI was hidden away, and it was called “Étendre” in French, which I confused with “Éteindre”, meaning Power off. But everything was there to expand the logical volume, but not shrink it afterwards.

A couple of months later, I found out it was possible to convert the mirrored RAID into a RAID5 after I added two new drives. That gave me 9Tb of storage space, enough for the moment to store my stuff.

Getting more would however be painful, time consuming and expensive, requiring me to swap out each 3Tb drive with larger ones, one drive at a time. Each time a drive sawp would occur, the RAID would have to rebuild from the three remaining drives, and only after the rebuild is complete would I be able to swap another drive. After all drives are swapped, the storage pool would expand based on the size of the smallest drive.

Then what’s the point? Why not copy all the data elsewhere, rebuild a new array and copy the data back? The longer process has the benefit of leaving the storage pool online throughout the whole migration. Data can still be accessed and even, at least as far as I know, modified! A dying drive also doesn’t bring the pool offline. The degraded RAID can still work, until a new drive is added. The rebuild occurs online, while data can still be accessed. I realized that this high availability was overkill for me, but for a business, that would be critical.

An hybrid backup strategy

I didn’t have enough storage space to back up the whole RAID5 array. Instead of solving that, I went with an hybrid strategy.

  • My personal data is already backed up on Dropbox and present on two machines at home. My idea was to make my NAS the second Dropbox client, but I quickly noticed that Dropbox doesn’t run on QTS, the Linux variant installed by QNAP on the NAS. I thus needed to keep my HTPC for that. Dropbox allows me to have an offsite backup, which would be handy if a disaster such as a fire destroyed my home. But thinking more, after such a disaster, getting back my photos, videos, Live sets, etc., would be kind of minor concern.
  • I have hundreds of gigabytes of Minecraft videos I authored and uploaded on YouTube but wanted to keep an archive of these. I ended up uploading these to an Amazon S3 bucket, which is now moved to Glacier. This allows to save the data for a low price but requires more time and some fees to get the data back. I thought the NAS could itself synchronize the MinecraftVideos folder with S3. No, it cannot run S3, not anymore! QNAP switched gear, giving up on S3 in favor of inter-NAS synchronization! That means if I want an automated backup, I would need to buy a second NAS with the same (or higher) storage space, and set it up to be synchronized with the first! For a small business, I can imagine this possible, but for a home user, that looks like overkill.
  • I thought that not backing up the ripped DVDs and blu-rays would not be a big problem since I have the originals. This was a big mistake.
  • More and more recorded videos added up, with no backup. Using my HD PVR from Hauppauge, I recorded several movies from my Videotron set top box.

My backup plan was thus outdated and needed some revisions and improvements.

Noisy box

The NAS, in addition to its lack of integration with Dropbox and S3, was quite noisy. Sometimes, it was quiet, but regularly, it started to make an humming noise, and it wouldn’t stop unless I power it off or tap on it a few times. I searched a long time to solve this. I tried to screw the drives instead of just attaching them with the brackets, but the problem was that one of the drives (a new drive) I put in the NAS was bad and noisy! Switching that with another 3Tb drive I had fixed the issue.

The noisy drive, I moved it into my main computer and it was relatively quiet at first. But it ended up making an annoying humming sound that my mic was picking, reducing the quality of many of my Minecraft videos. At some point, I got fed up and decommsionned that drive, in favor of the last 3Tb drive remaining in my HTPC. My NAS combined with a NVIDIA Shield was pretty much replacing my HTPC which I was more and more thinking about decommisioning.

The disastrous scenario

During 2020 summer, my QNAP NAS suddenly turned off and never powered on. When I was turning the unit on, I was getting a blinking status light and nothing else. At first, I was pissed off, because I wasn’t sure I would be able to fix this and was anticipating delays to get that repaired because of the COVID-19 pandemic. My best hope was to recover the data using a Linux box, then I would decide whether to get this NAS repaired or replace it with a standard Linux PC. Recovery eneded up to be harder than I thought, which made me angry.

When I lost access to all my files, I noticed how time consuming re-ripping the blu-rays and DVD will be. I would need to insert each disc into the blu-ray drive, start MakeMKV, click the button to analyze the disc, wait, wait, wait, then enter the path to store the ripped files to. Even though there is a single field text can be entered in, MakeMKV doesn’t put it in focus, forcing me to locate the super small mouse pointer (at least on my HTPC where I had the blu-ray drive to rip from), click, enter the path, check (and the font was super small), click again and then wait, wait, wait. For one disc, that’s OK. For 30…

I also lost a bunch of movies recorded using my HD PVR. The quantity of recordings increased over time and I didn’t realize none of this was backed up! Backup plans need to evolve and be revised over time.

Recovery attempts

First problem was to move the four drives into a computer that would be able to host them. I didn’t have enough free bays in my main computer, not without removing another drive. I was worried Windows would get screwed up by this and wouldn’t re-establish broken links to my Documents, Music, Videos and Pictures folders. I thus chose to use my old HTPC as a host, but fitting and powering the four drives was a painful procesess. I had to use pretty much all the SATA cables I had, and one broke during the process. I had to unplug the hard drive in a really old PC to get the SATA cable, unplug my DVD drive in my main PC to get the SATA cable, found one other SATA cable in a drawer, etc. I also needed a molex to SATA converter cable because the PSU only had four SATA power cables. I needed to power five drives: the four NAS drives to rebuild the RAID array, and the SSD containing Ubuntu! Because of the pandemic, I wasn’t sure the computer stores in my area were open or not, so my best bet to get new hardware was online, with delays of several days, even for something as simple and stupid as a SATA cable. Using what I have was my best option.

All these efforts were pretty much worthless because I wasn’t able to access the data. We’ll see why later on. All I could do is trigger a SMART long self-test, to at least verify the drives were good. All four drives passed the test. No need to get the NAS fixed or continue recovery attempts if more than one drive had failed that test.

I couldn’t go further without ordering hardware. I started with a four-bay USB to SATA docking station. Finding one was first tricky (one bay, just for Mac, etc.), but I got one and it worked like a charm. However, at first, it caused me issues: plugged the power cable in wrong direction, tested with a defective drive (yes, the noisy 3Tb drive I removed from my system just doesn’t power up anymore!!!), but it ended up working.

I was hoping to put the four NAS drives in there and have Windows see the drives, then I would try using ReclaiMe to get the files back. I also needed to get a large enough drive to hold all the data. I got a 10Tb one, that would do, I thought.

Since I was able to reassemble the block device using MDADM, I explored the idea of dumping that block device to a partition on the new 10Tb drive. For this, I had to plug the 10Tb drive to my HTPC, which already was missing connectors for 5 drives! I used the USB docking station for that. Bad idea: the HTPC is too old, having just USB2, and copying 10Tb through USB2 is a good way to strengthen your patience, and get you annoyed. It would have taken more than 2 days just for that step to finish! There was no solution, unless I got a PCI Express card with a USB3 port or probably better eSATA port, and then I would need to get a eSATA drive enclosure to put the 10Tb drive in! I didn’t like this at all, because that was a lot of waiting for an intermediate result only.

After the failure to create the block device because of USB2 slownesss, I got super fed up and decided to proceed with the repair of my NAS. I contacted their technical support and we figured out that a repair was needed and my warranty was over since a year. I thus needed to pay 300$, plus shipping of the NAS. I felt at that time it was my last hope of recovering the data.

But I tried anyway to get the drives out of my HTPC and plug them into my docking station, so ReclaiMe could analyze them. Well, RelciMe completely failed. It detected the logical volumes on the drives, which looked promising, but instead of making use of the Ext4 file system, it just scanned the drive for data looking like files. It ws thus just able to extract files with no name but some contents, even unsure if the files were complete! That would be unusable garbage, better off just re-ripping all the discs. R-Studio, another tool I tossed at this, also failed miserably, not even able to reassemble the RAID5. I got so fed up that at some point I considered contacting a data recovery company, but I was concerned that it would be too expensive, and I would just get chunks of data named with hash codes like I was about to get with ReclaiMe.

Lasagna file system

QTS makes use of multiple layers when configuring a file system. Each layer provides some features, but all of this is adding complexity at recovery time.

The diagram below summarizes the structure.

3TbHard drive3TbHard drive3TbHard drive3TbHard drive9Tb RAID5 storage poolmdadmDRDB?Volume groupLVM pvcreate/vgcreateLogical partitionLVM lvcreateEXT4 filesystemMoviesTV seriesMusic

First, the physical drives are combined into a RAID5 using the DMRAID module of the Linux kernel. This allows to create a RAID array in software, without any specialized hardware. The MDADM tool can be used to configure or activate a RAID. I was able to activate the RAID and get a block device out of it.

The block device could be used to host a filesystem. Instead, it is formated by QNAP as a DRDB storage pool, at least according to forums I searched on. Some people attempted to mount the DRDB device without success, because QTS uses a forked version of DRDB preventing anything other than QTS to read it! Because of that, only a QNAP NAS can reassemble the RAID5 and get data back!

The DRDB volume is formatted as a physical device for the LVM system. Logical Volume Manager (LVM) allows to split a storage device into multiple logical partitions. Partitions can be resized at runtime and don’t have to use contiguous space on the physical volumes. They can span multiple physical volumes as well. This is something any ordinary Linux distribution supports, as this is part of the mainline Linux kernel! Only caveat is the absence (at the time I am writing) of user interfaces exposing these. One needs to use command lines such as pvcreate, vgcreate and lvcreate to manipulate the logical volumes, but the commands are not as complex as I thought.

I read that QNAP also forked LVM, so I was worred that even if I got past the DRBD layer, I would not cross the LVM one.

Note that when I partitioned my 10Tb drive, I found a LVM partition on it! The assembled RAID apparently was a LVM, so maybe I would have been able to vgimport it and get access to the logical partitions! However, any attempt to do so would have failed or changed the machine id in the volume group, reducing chances my fixed NAS mounts the array and filesystems. My new 10Tb drive was already formatted at the time, with some data on it, so couldn’t use it to back up the full RAID and test, unless I get another drive. I thus decided to stop my attempts there since my NAS was shipped to QNAP at the time of that discovery.

Below the LVM layer, there are logical partitions, at least one large with the files I wanted to recover. The Ext4 native Linux file system is used here. That is used to organize the space into files and folders. Recovering all the data requires handling the Ext4 filesystem to get the full file contents back, not just portions of files with no names.

Full recovery

I got the fixed NAS back and inserted the drives in. The NAS powered up and recognized the drives as if it never got broken. I was thus able to get my files back, everything was there. Because the recovery process was so painful and expensive, I didn’t feel any victory, just a bit of relief that this was over.

While waiting for the fixed NAS, I formatted my 10Tb drive. I experimented with logical voluems, ccreating several partitions on the drive: one for Minecraft Videos, one for movies, one for TV Series, one for a full copy of my Dropbox folder, one for my music files, etc. Using the Ubuntu wiki, it was simple to create the logical volumes and then I started copying files on them. I was ready to transfer the contents of my NAS on the new disk when I got the NAS back.

Even though I recovered my data, I will probably have to re-rip some DVDs that are unreadable by Kodi. The VIDEO_TS structure is causing a lot of headache for pretty much all Linux-based players. VLC seems the most versatile one able to read most DVDs, but sometimes, I needed to use MPlayer, Kaffeine, etc. I remember that almost destroyed my dream of having all my DVDs and blu-rays on hard drives. Of course, Windows with PowerDVD or similar DVD player will work better, but I don’t want Windows on my HTPC, better return back to the sandalone DVD/blu-ray player and spend countless minutes searching for disks. MakeMKV should help solving that, because Kodi can read MKVs without issues. I may be able to convert previously ripped VIDEO_TS into MKV, saving me the trouble of re-ripping the disks.

After that bad experience, I came up with the plan of keeping the NAS as long as it would work, but back up all data on another drive. If the NAS dies a second time, then I would not need to recover any data and would just repurpose the drives, probably in a standard Linux PC.

Lesson learned: a NAS is for cases where you have multiple drives, more than four, if not more than eight, no usual PC can accomodate. It is relatively straightforward to get a standard ATX case that will host six drives, including SSDs, and getting a power supply unit with six SATA connectors is perfectly fine. Having to do it, I would probably explore the route of modular power supplies to reduce cable clutter, but even that is optional.

By trying to save myself some copy/pasting, I ended up with more pain and problems. Having spent at least half a day exploring logical volumes, at worst experimenting in a virtual machine, I would have figured out that my existing HTPC would have been able to combine my existing drives into a storage pool that can expand over time. If the HTPC dies, another Linux PC can import the volume group and things go on. Unfortunately, nothing can prevent disaster caused by failed drives other than back ups.

Other benefits of logical volumes

After I explored logical volumes, I am pretty sure I need them for my next Linux installation, because they will solve a bunch of fundamental issues I am getting again, again and again.

  • Each time I perform an upgrade, I am running into the risk of a catastrophic issue making the whole system unusable. Ubuntu offers no downgrade path. If an upgrade fails, just reinstall from scratch. This is why several people suggest to not dist-upgrade but just reinstall clean, every time. Logical volumes alleviate that through snapshots. Before a dist-upgrade, I could just create a snapshot of the volume holding Linux, upgrade and in case of an issue, just restore the snapshot in a few minutes, no reinstall, no reconfiguration.
  • Supporting upgrades, downgrades, multiple versions, multiple Linux distributions, all of this requires the home directory to be separate from the root file system. But each time I over partition my drive, one partition gets full and I have to either move data around, or restart computer and perform a repartitioning, which is time consuming and a risk for data loss (e.g., power outage while GParted moves data!). Logical voumes solve that, by allowing resizing at runtime. If I need more than 50Gb for my home drive, no problem, just claim some extents from the physical volumes, no need to be continugous space, and the resize occurs at runtime, without any unmount or reboot. I can keep working while the resize occurs. That’s really neat and powerful.
  • Even the classical problem of expanding a drive is easier with logical volumes. LVM can move all data from one physical volume to another, transparently, at runtime, while I can work on the machine. Replacing a too small or end of life SSD is thus easier. Of course, sharing a SSD between Windows and Linux is always painful and problematic process, although perfectly possible. Dual booting Windows and Linux is itself a painful problematic process anyhow.
Categories
Bug

An intricate sounding puzzle

All this story started on Wednesday, October 23rd 2019. When I turned on my subwoofer, I found out it was producing a constant noise, even without audio source playing. I verified the connector at both ends, everything was fine. My first reaction was to think the subwoofer was broken. It was too big for me to carry it over some repair shop, and not sure worth doing that, so I was planning to buy a new one. In the mean time, my sound system would still work, although without the subwoofer.

Following is a picture of the back of the subwoofer, with the connector plugged in.

The back of the subwoofer with connector plugged in
The humming sound my subwoofer was constantly making

The best solution to get a new subwoofer, I though, would be to go to the place I purchased my home theater system, but it was in Chambly, near my parents’ place. It would take weeks before I go back there. I thought about waiting but was tempted to check if I could order a new subwoofer online, on Amazon for example, or find a store in Montreal. However, many stores only sell full kits with everything included (A/V receiver, speakers, subwoofer), or all-in-one sound bars.

Finding the culprit: a ball game

After thinking more about it, before buying the new subwoofer, I wanted to make sure mine was really faulty. A year ago, I got issues with my A/V receiver which started to repeatedly power cycle without any possible solution. I replaced it with a new one but kept the old one in case I could find somebody able to fix it. Although flaky, the old receiver could be used to test my subwoofer. If it produces the noise with the old receiver too, I will know for sure it is broken.

To my surprise, the subwoofer worked flawlessly with the old receiver. Of course, it was just a basic test; I didn’t plug any audio source to verify it would produce sound, but the absence of the constant noise was already a progress. Unfortunately, chances were that the problem came from my A/V receiver again, which would force me to disconnect everything it from it, carry it to a repair shop, wait days, and then spend hours reconnecting everything. The speaker cables were quite hard to plug on this one, too close to each other. I didn’t want to redo this and was thinking about giving up and purchasing a dumb stupid sound bar instead. But I was quite reluctant to give up on my speakers.

At least, I was happy I didn’t order a new subwoofer or, even worse, go into the trouble of carrying one into the subway, from a local store back to my place!

Before going into the trouble of disconnecting the speakers, I tried unplugging devices, to no avail. Even unplugging the A/V receiver didn’t stop the subwoofer from making its constant humming sound! Of course, turning off or unplugging the subwoofer stopped the sound!

After more than one hour of plugging and unplugging, trying to replace the subwoofer cable to no avail, having trouble will stuff falling behind my TV, especially the AM/FM antenna of my A/V receiver and a small router configured as a switch to dispatch Ethernet to my HTPC and NAS, I found out that unplugging an HDMI cable from the A/V receiver stopped the humming sound! What? Which cable is that? A couple of crawling later, I found out: the cable linking my A/V receiver to my cable TV terminal. Trying to replace the cable didn’t change anything. I tried disconnecting the cable TV terminal to no avail. I don’t know why I thought about this but found out that disconnecting the coaxial input cable from the terminal stopped the humming. Of course, that would prevent the cable TV from working!

Next step: replace the cable. Unfortunately, I didn’t have a long or short enough cable. The only cable that worked was a 40 feet (too long) one, but the wires going out of the connector were too long, preventing me from tightening the connector into my terminal’s female port. Even worse, when the 40 feet cable touched the terminal’s connector the first time, the subwoofer briefly hummed. I was pretty sure purchasing and plugging a new long enough cable would produce the same effect as the original cable.

I was stuck, because the problem was coming from the combination of my subwoofer, A/V receiver and cable TV terminal. I could by trial and error replace the subwoofer, then the A/V receiver, to no avail! Contacting manufacturer for any of them would just bounce me the others! Only simple “solution” was to sacrifice one of the devices. Short term, it would be the subwoofer. Longer term, I was mentally preparing myself to give up on cable TV. I prefer a sound system over cable TV because I could get contents from elsewhere like Netflix, if of course I can have it played under Linux. I won’t purchase a Windows license for a 10-year old HTPC! Doing this would be a stupid non-sense, for me at least.

What changed?

But that setup was working before! What, on a sudden, changed, and caused it to fail? Well, it was a workaround for a cable signal issue!

I was getting issues with my cable TV terminal since a couple of day, complaining sporadically about low signal and decided it was enough. Wednesday morning, October 23rd 2019, I examined the connectors at both ends of the coaxial cable, they seemed tight, that should work.

The coaxial cable from the wall’s outlet connects into my power bar offering protection for coaxial input. Another cable goes out of the power bar and connects into my cable TV terminal. That worked like this for years, but it seems that every time somebody in my building is getting hooked up or adds a device, the cable signal weakens for everybody else. I thus thought that maybe I could work around by bypassing the power bar, just hooking the cable directly to the terminal. I tested it, that seemed to work, but just turned on the TV, not the subwoofer.

Then I went working and only later I turned on the whole system to watch a video on YouTube. Then I got the issue with the subwoofer and didn’t think at all it could be linked to the cable terminal.

I ended up, Wednesday night October 30th, reconnecting the cable to my power bar. This looks as follows.

Coaxial cable going in and out of the power bar

But then wait a second, wouldn’t that cause the signal issue to come back? Yes, it will. For now, I didn’t observe it again, but I know it can come back. This just gave me a delay, to think about something better, or to prepare myself for sacrificing the cable TV. I need, for that, to test Netflix on my Linux HTPC, which will be a frustrating experience of trial and error.

Why not contact my cable TV provider?

I thought about it and may end up doing it. The problem is that there is little they can do to improve the signal without accessing a technical room in the basement of my building, and getting the key to that room is problematic these days, because the company managing the building is overwhelmed and not responding to every inquiry.

Moreover, my cable company, Videotron, is moving towards Helix, a new technology combining everything into one device. Helix not only replaces the cable TV box but also the Internet modem and would constrain me to use a specific router I know little about. If I’m lucky, the router will just work as a regular device, offering Ethernet ports and 802.11n, hopefully 802.11ac wi-fi, with standard WPA2 and customizable network name and password. If I’m less lucky, network name will be hard coded and I will have to search forever among similar but slightly different names, and I may be limited to 802.11n 2.5GHz wi-fi (no ac, no 5GHz). If I’m really really unlucky, establishing a wi-fi connection will require a proprietary software which may work just on some devices! The problem is if I switch to Helix to get rid of my problems with my current Illico terminal, I won’t be able to know in advanced if all these quirks exist and will be stuck afterwards; it will be hard to go back. Concurrent telecommunication providers are either not offering cable TV at all, either adopting the same all-in-one modem/router strategy.

Categories
Bug

Misleading broken connector

Wednesday, October 23rd 2019, while troubleshooting an issue with my subwoofer, I discovered that one of my S/PDIF optical cable was disconnected; I found out the loose cable near my A/V receiver. This was previously used to send audio from my HTPC to my A/V receiver, before I got HDMI I/O on my new Yamaha receiver. I left the cable around in case the HDMI audio starts to fail, but it is kind of pointless as if I need to fall back on this, I would have to reconnect the HDMI cable directly on my TV as well. I wanted to either reconnect the cable or remove it altogether. However, I was unable to find back the optical connector on the receiver’s back. All I could find were RCA connectors.

Having a closer look at the back of the receiver, I discovered that one of the input had a broken connector in it. The connector looked like an RCA female but a bit different. The image below shows that connector.

Broken male connector in optical female port

After I discovered that and confirmed this was supposed to be the optical connector, I tried to remove the broken male connector from the female port using a plier. After a couple of attempts, I got the connector part back. It looks as follows.

Broken connector extracted from the port

The broken end of the cable looks as follows.

Optical cable with broken connector

I was tempted to put the broken connector back on the cable.

The “repaired” connector

By screwing the metal part back, it seems the connector is not broken. I didn’t test if it works. Maybe, or maybe it would be unstable.

The complete connector

That’s it. There’s nothing more to this!

Categories
Bug

A dying laptop

My Lenovo Ideapad Yoga 15 is dying.

It happened the second time today, and that’s quite concerning. Suddenly, the machine freezes, keyboard is not responding, the mouse is moving but clicking does nothing. There is no way out other than rebooting, and when I did it, the screen became black with an error message displayed in white on a blue background.

The machine was telling me that there was no boot device or boot failed. What? This Lenovo laptop has a builtin mSATA SSD. I didn’t disassemble the laptop, ever, so why suddenly either the SSD became faulty, or the connection between the SSD and the motherboard is now unstable? Fixing any of these would require disassembling the laptop, with the risk of not being able to put it together again afterwards. The keyboard that needs to be removed to reach the components is clipped, and clips can break when trying to remove.

As the last time, during the holidays, powering off the laptop and booting up again “fixed” it. But who knows if that will last. Maybe it will fail tonight, maybe the next week, I don’t know. And next time it fails, maybe it won’t boot up again.

Besides of that, the battery lasts not really more than 2 hours and the wi-fi is now so clunky that most of the time, I need to plug in a USB to Ethernet or 802.11 adapter.

I’m wondering if that is Windows 10 that is killing this machine. As soon as I upgraded, the laptop became slower. A few months later, the battery life was reduced. If I could make Ableton Live work on Linux, I would attempt a switch to Ubuntu, but nope, no Linux version. All major DAWs have just Windows or Mac builds, no Linux. Things may change in 10-20 years, but that doesn’t matter to me, my passion may not be there at that time.

I’m kind of hopeless now to get any type of working laptop in the 10-20 next years. They all suck. Any candidate, any, I can find bad reports on some forums. People are slowly but surely stopping to use computers and/or migrating to Mac, which I cannot do because of font size issues. Maybe home computing is coming to an end, replaced by “smart” but limited phones and tablets not offering any efficient way of typing text. People are working around by posting meaningless pictures with any description, that’s what I see every day on Facebook: XYZ added on his story, no text, just a picture. I came to the point of disabling Facebook notifications on my smartphone and debating the possibility of closing my Facebook account.

But I want a solution. As a computer scientist, I just cannot give up having a computer, that makes just no sense to me.

Categories
Bug Configuration

Ubuntu 18.10: a silent release

Usually, upgrading Ubuntu goes well. I run the Upgrade tool which downloads new packages, installs them and then asks me to reboot. Some new versions had minor issues, for example Wayland not fully compatible with my NVIDIA card or an old version of MATE preventing the dist-upgrade, but nothing major, nothing that couldn’t be worked around. This time was different: no sound, and no way to easily get around this.

Cannot dist-upgrade

Sunday, October 21 2018, Ubuntu 18.10 was released since a couple of days. However, after I installed the updates through the Software update tool, I wasn’t proposed to upgrade. I had to dig into the software options and reconfigure the delivery of releases to “every release”, not just the LTS. Then I reran the Software Update tool, and got the option to upgrade. However, clicking on the upgrade button did just… nothing. I tried two or three times: same result.

Before accepting that this time, I would have to go through the clean install route, I searched a bit on Google, and found out that sometimes, the upgrade doesn’t start when updates are still available. But updates were all installed. However, running the following on a terminal found and installed a couple of additional updates.

sudo apt update
sudo apt dist-upgrade

I then retried the Upgrade button, and that worked! Yeah!

The upgrade went well, and then I was offered to reboot, which I did.

Slow and silent

First the new release felt a bit sluggish, then pretty flaky. First time I tried to reboot because I needed to go to Windows, to play Minecraft loaded by Twitch and recorded through OBS, the system hung up waiting for a stop job. I am getting this issue from time to time and when that happens, I have to wait for the 1min30s timeout to elapse before the stop job is forcibly killed. I don’t know what is a stop job, what job is frozen and how I could get rid of this issue for good.

I was in my living room turning off my TV and AV receiver while my new Ubuntu setup finally rebooted, and I didn’t have time to select Windows in GRUB, so it restarted Ubuntu, and trying to restart redid the “stop job” issue! Argh, don’t tell me I’ll get this each time I reboot or shutdown now? I’ll have to wait 1min30s, maybe even 3min, just to power off or reboot. What’s the point of having a SSD if timeouts like this counter its performance benefits?  I got fed up and powered off my PC with the power button. I had to press and hold the button almost ten seconds for the machine to finally turn off, then I had to press the button 2-3 times for it to finally turn on. Maybe I’m heading towards the necessity of replacing my computer CASE, which sucks me to the point of thinking about getting rid of that damned thing and use just a laptop. But that case issue has nothing to do with Ubuntu, I should just have picked a better case, there’s nothing more to it, at least for this post.

After my Minecraft session, which was kind of fruitful, I wanted to check that my Ubuntu installation would boot again and be able to shutdown at least once without the “stop job” issue. I thus rebooted to Ubuntu, but the system froze before showing the welcome screen. I had to press CTRL-ALT-F2 to reach a console, log in, check the syslog, nothing of interest. Then the system finally booted into X. I wanted to switch back to the console to log off, but when coming back to X, it froze again on a black screen, this time no key was working. Then another hard reboot!

Second attempt worked: I reached the desktop. I launched the backup script for my Minecraft world, then the MKV->MP4 batch conversion script. OBS records in MKV, which is more robust against crashes, but VideoStudio doesn’t accept MKV, so I have to turn the recordings into MP4. Fortunately, FFMPEG does it without loss, by just repackaging the MPEG stream into another container. Then I wanted to organize my videos into directories, so I launched one to check it, and found THE thing: no sound!

Checking pavucontrol, I found that my sound card disappeared. Only detected devices were my HDMI port hooked up to my monitor, and a crappy USB Webcam that could serve as very basic and rudimentary mic. I don’t use that for my Minecraft recording; I have an AKG real microphone for that! I tried to reboot to no avail. Trying to run speaker-test just hung, again, needed to press ALT-F4 to shutdown the terminal.

Searching on Google for solutions lead to nothing except old stuff that didn’t work. Some people restored sound by reinstalling ALSA and PulseAudio. Others had to downgrade to the previous kernel. Others edited configuration files, commenting a line that is not there in my case, and adding another line. I tried to reinstall PulseAudio with sudo apt install –reinstall pulseaudio, to no avail. As a last resort, I tried to reboot with the 4.15 kernel of Ubuntu 18.04,

After almost an hour of searching, it was more and more obvious that I needed to give up on Ubuntu 18.10 and either downgrade (which essentially means reinstall) to Ubuntu 18.04, or switch to a new distribution. I was quite annoyed, as preceding upgrades went right, and then the hardware problems start again like in the past. Moreover, the system was freezing for a second each time I hit Tab while in a terminal, before displaying completions.

Can this work at all?

I had to go sleeping (it was past 11 PM), go to work the day after, but I thought about it. First I needed to test if that can work! For this, the simplest solution is to test using a live USB. Monday morning, I had to resist the temptation of testing that before going to work. Doing that would have made me start late and thus finish work late in the evening. So I did my workday first. So on Monday evening, I needed to download the Ubuntu 18.10 ISO; I picked the MATE one since that’s what I’m using now, instead of that GNOME3 thing which works so so. I got the ISO, and used the disk creator tool built into Ubuntu to write it into a USB drive. However, the tool refused to format my drive: too small for the new 2Gb ISO! That old 2Gb key was just a bit too small, which is kind of annoying. As a smart man, I should have extra empty, unused USB keys hanging around, but seems I’m not smart enough for that! So I had to use an existing key.

So I took the Clonezilla key, a 16Gb device, really too large for that small backup tool. I stored Ubuntu on it, and then I installed Clonezilla on the old 2Gb stick. Then I booted off my new Ubuntu medium, the system booted successfully, and I got sound! Ok, so at worst, if I clean install, I should get sound… except maybe if that is an incompatibility with the NVIDIA driver.

Unlocking ALSA

First I ironed out the NVIDIA hypothesis by uninstalling the NVIDIA proprietary graphic driver. After I reboot, I tested and still had no sound, and was back to a default VESA resolution. I thus reinstalled the driver, a bit annoyed that Nouveau cannot even basically handle my graphic card. I don’t expect full 2D/3D acceleration out of Nouveau, but at least, mode switching should work with a card bought in 2013, reaching a 1080p resolution. No, nothing. But at least, there was no incompatibility between the graphic and sound driver.

I then dug further into ALSA, which was still detecting my sound chip. Here is the list of devices it found:

eric@Drake:~$ aplay -L
default
    Playback/recording through the PulseAudio sound server
null
    Discard all samples (playback) or generate zero samples (capture)
pulse
    PulseAudio Sound Server
sysdefault:CARD=PCH
    HDA Intel PCH, ALC887-VD Analog
    Default Audio Device
front:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Front speakers
surround21:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    2.1 Surround output to Front and Subwoofer speakers
surround40:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    4.0 Surround output to Front and Rear speakers
surround41:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    4.1 Surround output to Front, Rear and Subwoofer speakers
surround50:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    5.0 Surround output to Front, Center and Rear speakers
surround51:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    5.1 Surround output to Front, Center, Rear and Subwoofer speakers
surround71:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    7.1 Surround output to Front, Center, Side, Rear and Woofer speakers
iec958:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Digital
    IEC958 (S/PDIF) Digital Audio Output
dmix:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Direct sample mixing device
dmix:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Direct sample mixing device
dsnoop:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Direct sample snooping device
dsnoop:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Direct sample snooping device
hw:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Direct hardware device without any conversions
hw:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Direct hardware device without any conversions
plughw:CARD=PCH,DEV=0
    HDA Intel PCH, ALC887-VD Analog
    Hardware device with all software conversions
plughw:CARD=PCH,DEV=1
    HDA Intel PCH, ALC887-VD Digital
    Hardware device with all software conversions
hdmi:CARD=NVidia,DEV=0
    HDA NVidia, HDMI 0
    HDMI Audio Output
hdmi:CARD=NVidia,DEV=1
    HDA NVidia, HDMI 1
    HDMI Audio Output
hdmi:CARD=NVidia,DEV=2
    HDA NVidia, HDMI 2
    HDMI Audio Output
hdmi:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 3
    HDMI Audio Output
dmix:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Direct sample mixing device
dmix:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Direct sample mixing device
dmix:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Direct sample mixing device
dmix:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Direct sample mixing device
dsnoop:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Direct sample snooping device
dsnoop:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Direct sample snooping device
dsnoop:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Direct sample snooping device
dsnoop:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Direct sample snooping device
hw:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Direct hardware device without any conversions
hw:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Direct hardware device without any conversions
hw:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Direct hardware device without any conversions
hw:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Direct hardware device without any conversions
plughw:CARD=NVidia,DEV=3
    HDA NVidia, HDMI 0
    Hardware device with all software conversions
plughw:CARD=NVidia,DEV=7
    HDA NVidia, HDMI 1
    Hardware device with all software conversions
plughw:CARD=NVidia,DEV=8
    HDA NVidia, HDMI 2
    Hardware device with all software conversions
plughw:CARD=NVidia,DEV=9
    HDA NVidia, HDMI 3
    Hardware device with all software conversions

The speaker-test command was hanging, but if I waited enough time after pressing Ctrl-C, it returned. Ok, cool! Running pavucontrol while pseaker-test was playing its noise, I could see it in the PulseAudio applications. Ok, so speaker-test

What if I switch device? I tried speaker-test -D sysdefault:CARD=PCHI got an error because ALSA couldn’t open the device. Searching on Google about that, not specific to Ubuntu, lead to clues: it could be a permission issue. Trying to check /dev/dsp failed: no such file. But I could fiund the following.

eric@Drake:~$ ls -ld /dev/snd/*
drwxr-xr-x  2 root root       60 oct 23 08:11 /dev/snd/by-id
drwxr-xr-x  2 root root      100 oct 23 08:11 /dev/snd/by-path
crw-rw----+ 1 root audio 116,  9 oct 23 08:11 /dev/snd/controlC0
crw-rw----+ 1 root audio 116, 15 oct 23 08:11 /dev/snd/controlC1
crw-rw----+ 1 root audio 116,  8 oct 23 08:11 /dev/snd/controlC2
crw-rw----+ 1 root audio 116,  6 oct 23 08:11 /dev/snd/hwC0D0
crw-rw----+ 1 root audio 116, 14 oct 23 08:11 /dev/snd/hwC1D0
crw-rw----+ 1 root audio 116,  3 oct 23 08:11 /dev/snd/pcmC0D0c
crw-rw----+ 1 root audio 116,  2 oct 23 19:23 /dev/snd/pcmC0D0p
crw-rw----+ 1 root audio 116,  4 oct 23 08:11 /dev/snd/pcmC0D1p
crw-rw----+ 1 root audio 116,  5 oct 23 08:11 /dev/snd/pcmC0D2c
crw-rw----+ 1 root audio 116, 10 oct 23 08:11 /dev/snd/pcmC1D3p
crw-rw----+ 1 root audio 116, 11 oct 23 08:11 /dev/snd/pcmC1D7p
crw-rw----+ 1 root audio 116, 12 oct 23 08:11 /dev/snd/pcmC1D8p
crw-rw----+ 1 root audio 116, 13 oct 23 08:11 /dev/snd/pcmC1D9p
crw-rw----+ 1 root audio 116,  7 oct 23 08:11 /dev/snd/pcmC2D0c
crw-rw----+ 1 root audio 116,  1 oct 23 08:11 /dev/snd/seq
crw-rw----+ 1 root audio 116, 33 oct 23 08:11 /dev/snd/timer

Ok, interesting, only root and users of group audio can access these devices, and thus play sounds. Am I part of the audio group? I tried groups eric and found out I wasn’t. But on my HTPC still running Ubuntu 18.04, I was! Lucky I had that HTPC, otherwise I would have been forced to reboot to the Live USB to check, or boot another machine with it, like my ultrabook.

Ok, so what if I add myself to the group?

sudo usermod -a -G audio eric

I had to relog, and then got sound through ALSA using Speaker-Test! Yeah! But still no PulseAudio! I tried to reboot, not just relog, to no avail. Ok, at least I can reconfigure Audacious to play through ALSA, but I suspect I’ll get trouble for YouTube and Spotify for which I cannot configure audio output.

Unlocking PulseAudio

I tried all sorts of things to debug this. I was able to get some logs out of PulseAudio several ways, but the most useful way was through SystemD. In Ubuntu, PulseAudio is started by a userspace variant of SystemD. There is a command allowing to get the logs: journalctl. I couldn’t figure out a way to filter so I ended up calling journalctl -a -f on one console, and then pulseaudio -k on another to force PulseAudio to restart. Then I was checking the produced log. I ended up finding out errors. The system was not able to communicate through DBus using a certain socket. I started to think that each time the system freezes, this is because PulseAudio tries to emit a sound, has to wait and timeout.

I couldn’t find out how to reconfigure DBus, or understand how PulseAudio and DBus interact enough to troubleshoot this. I was quite stuck, and all I could find was deprecated information. There was a bug report that seemed to affect Ubuntu 18.10, but no solution. It was past 10 PM and I was on this for almost two hours, trying and searching to no avail, more and more pissed off and risking to be so angry that I would end up having trouble to sleep. I was about to give up or reinstall from scratch.

I ended up fed up and decided to completely get rid of PulseAudio using sudo apt purge pulseaudio. After a reboot, sound was WORKING! What??? How come MATE plays sound without PulseAudio? Audacious was working, and VLC as well. But not Firefox for YouTube, and not Spotify.

Before accepting the sacrifice of YouTube and Spotify to avoid a clean install, at least for now, I tried to reinstall PulseAudio. For I really don’t know why, the sound continued working, and PulseAudio displayed my internal audio device now. YouTube and Spotify resumed working. I read posts about some people that got the issue fixed and it came back the next reboot. Ok, let’s reboot and hammer that bug for good, then! I rebooted, and sound was still working. I still don’t fully understand why.

Either something messed into the groups I was member of and PulseAudio got screwed up because I lost permission to use ALSA, either some configuration files needed to be updated but APT decided to keep my current versions. Purging PulseAudio removed the configuration files and reinstalling reverted to sane defaults. At least, sound is working now.

Even better, the system seems more responsive now and didn’t freeze on startup. It really seems that PulseAudio and ALSA had trouble communicating, causing these hangups.

Why not a clean install?

Because my home directory lives on the same partition as my Ubuntu install. Any attempt to put my /home on a separate partition leads to insufficient disk space after a time. I have a 250Gb SSD shared with Windows so I cannot put 50Gb for my /home and 30Gb for just Ubuntu. One simple solution would be to move my /home to an hard drive. As long as Ubuntu is on a SSD, I’ll have a fine boot time. Or I would need a way to use SSD only as a cache and put everything on the hard drive. I could also come back to the dual SSD strategy: one SSD for Windows, one for Ubuntu. I’ll think about it, but at least I don’t have to do anything short term. Maybe I can wait and replace everything, and get a better case to fix the power button hickup at the same time…

If everything had failed, before the clean install, I could have tried to restore my Clonezilla image of my SSD, and that would have got me Ubuntu 18.04 back to normal. In case of failure, then I would have had to clean install 18.04, 18.10 or something else, or just give up on Linux for the moment. At least this is not necessary anymore.

Categories
Bug

Unsolvable and annoying VirtualBox slowdown

Since end of December 2016, I was using my personal computer to work from home. All my development environment was living in a VirtualBox CentOS virtual machine. That was working very nicely until March 13th. All of a sudden, the virtual machine became annoying slow. Starting the virtual machine took twice the time as before and everything was taking several seconds to start. It became impossible to start Thunderbird and connect to my corporate email through IMAP.

I tried to restart the virtual machine, reinstall VirtualBox, fiddle with the parameters of the virtual machine, nothing worked! The only workaround that helped a bit and at least allowed me to work was to increase the number of virtual CPUs from 2 to 4. However, even with that, when I was switching windows with Alt-Tab, it was taking several seconds for the window selector to pop up. Typing on the terminal was sluggish, sometimes as if I was connected through SSH on a slow server! But my machine was running locally! Running the unit tests of the main component I was developing used to take 4 minutes and a half on that machine. This bumped to more than 5 minutes, and even 6 minutes last week.

None of the suggestions on forums and nothing I thought about could solve this. I set the power plan to maximum performance so the CPU wasn’t throttled down. I fiddled with virtualization and paravirtualization options, disabled 3D acceleration, enabled it again, absolutely no change. Even the GRUB menu was taking a few seconds to display rather than appearing instantly as before. Maybe this is a CPU issue? I checked the CPU temperature using Speccy: everything seemed normal. VirtualBox was the mainly affected program, the rest was working fine.

This really seemed like a nasty Windows 10 update that screwed everything up. I could attempt a system restore or I could recover the last CloneZilla image I had, but if that was a Windows 10 update, the update would reinstall and thinks would screw up again. I would thus need to downgrade to Windows 7.

The issue persisted more than a month. Sometimes, I tried stuff without success: recreating the virtual machine, copying the VDI files somewhere and putting them back on my SSD (in case of filesystem corruption), checking my NVIDIA settings because the slowdown was mainly affected GUI rendering, etc.

The day before the first slowdown, two things happened: I ran CCleaner and Avast got updated. I tried to restore the registry backup CCleaner made before it cleaned registry: that failed once again. Windows couldn’t restore the registry backup. I tried to disable Avast temporarily: no effect. The hypothesis of a flaky Windows update was the main one, but no update were coming to fix it. Maybe they won’t fix it, because they offer Hyper-V, and they would love people switching from VirtualBox to Hyper-V. But that would involve reinstalling CentOS on a new virtual machine and recreating all my development environment. I could try VMWare Player instead, I think it can migrate a VirtualBox machine, but nothing told me it wouldn’t be affected as well by this issue.

Monday morning, April 17th, I tried all what I could to fix this. I was ready to attempt the system restore if needed. However, it was too late for a simple restore through Windows: I would need to use my CloneZilla image and take the risk Dropbox wouldn’t be smart enough to detect there is an old directory to update and not replace the contents in the cloud with the restored old directory. In case Dropbox screwed up, I backed up my Dropbox folder. I also backed up my virtual machines that I would need to restore after the Clonezilla recovery.

Before doing that, I tried to uninstall CCleaner: no success. I tried to remove NVIDIA Experience: no success. Two days ago, I removed the 3D vision drivers without effect. Then I removed Avast. At this point, I had little hope. I was ready to plug in the external hard drive with the Windows 10 image on it. But before, I rebooted, tried the virtual machine one more time and then, finally, it worked! It wasn’t slow anymore!

I pushed as far as starting my Ubuntu virtual machine and upgrading it from 16.10 to 17.04. This went well, without any issue. I’m writing this post from Ubuntu 17.04, running inside my VirtualBox environment which is finally fixed.

It is not the first time VirtualBox is hindered by an anti-virus program. It happened at work with Symantec Endpoint Connection. I had to downgrade to an old VirtualBox for a few releases until that finally got fixed. VirtualBox or Avast will have to fix something for this system to work again. For now, I am using the anti-virus built into Windows 10. I don’t know if I will retry Avast, reinstall AVG or finally give up and pay for Symantec’s Anti-virus or Kaspersky.

Categories
Bug Configuration

Spurious mail delivery errors

A few weeks ago, I started to receive email containing error messages about the delivery of some mails I didn’t send. The contents of such emails looked like spam, but why weren’t they detected by the anti-spam functionality of GMail? Maybe spammers found a new way to send their junk that circumvents current filters. But few weeks later, the annoyance persisted. I was receiving at least one of these emails per day, sometimes several per day. I started to suspect some people hacked into my GMail account and were using it to send spam, but I couldn’t find any trace of these in my “Sent” folder. Maybe they can circumvent it as well. Will I have to change my password just in case? And what would  tell me they wouldn’t hack again?

Friday, April 14th, I got fed up of this. First, do these come from the same sender or group of senders? If it does, I could block these addresses. Otherwise, there is a problem with GMail that would need to be solved eventually, otherwise I would have to switch from GMail to some other email service. Looking at the sender’s address, I found out the message was coming from something @ericbuist.com. Could it be because my mail account from my Web host was misconfigured?

I logged onto my HostPapa cPanel and reached the mail options. I found out that anything @ericbuist.com not corresponding to a valid email account is sent to a default email address. As a result, spammers in need of a fake origin email address can take anything @ericbuist.com in the hope this won’t correspond to a valid address. I thus reconfigured the default route to return an error email instead of redirecting the message. I also found out that besides redirecting traffic to my GMail account, the HostPapa mail service is keeping a copy of the messages. I thus had 250Mb of junk emails there that I deleted to free space. Although the disk space is unlimited on my HostPapa, if every customer abuses it by leaving junk on their account, HostPapa will have to impose quotas at some point.

I didn’t receive other emails about mail delivery failures after that. Unfortunately, this is not the only cause of such problems. Other people had issues with that because they forwarded all their GMail emails to a service sending SMS, and the service went down. They had to disable that forwarding from their GMail accounts. Things get worse when other email addresses are redirected to a central email account. All these can be the cause of spurious emails and thus need to be checked in case of issues.

Categories
Bug Configuration

Bumpy Ableton Live session

Yesterday, I tried upgrading to latest Ableton’s Live, the 9.7.1 version. Everything went well, but I got other issues, not related to Live, that made my work session quite bad and frustrating.

S/PDIF not working great

A month ago, I got a new audio interface: the Focusrite’s Scarlett 18i20. This amazing device provides eight analog audio inputs and 10 outputs. This is far from the advertised 18 inputs and 20 outputs, but these include S/PDIF and an add-on card that plugs into the optical ports of the interface. Anyway, 8 inputs is more than enough for my needs. I have difficulty playing one instrument reliably, so I won’t start playing multiple instruments at the same time, at least not now!

I didn’t have enough long audio jack cables to plug my Novation’s Ultranova (two channels), my Korg’s EMX (two channels) and my Nord’s Drum (1 one channel), so I decided to try hooking my Ultranova through S/PDIF instead. For this, I used a RCA cable I had got somewhere I don’t remember. I plugged the S/PDIF coaxial output of the synthesizer to the appropriate input of the audio interface, then fiddled with MixControl to figure out HOW to enable S/PDIF. Easy, I thought: just set up one entry in the Mix 1 to route S/PDIF L to left channel and S/PDIF R to right channel. The Mix 1 mix was already routed to the two monitor outputs of the interface. With that, I should have obtained sound from my Ultranova into my audio monitors. No, nothing! I verified that the S/PDIF output was enabled from my Ultranova: it was.

I tried, checked many times, searched on the Web, ok, set the sync source to S/PDIF instead of Internal, from MixControl. Did it, no result. I spent at least half an hour trying, checking, trying again, to find that the volume of my Ultranova was turned all the way to minimum. Turning up the volume solved it!

BUT I started to hear cracking sounds from time to time. This happens especially when playing long notes with pad-style sounds. That means S/PDIF doesn’t work well out of my Ultranova, in my audio interface, or that requires a special cable I don’t have. But then WHY is the S/PDIF the exact same shape as an RCA connector?

There is no solution for the moment, except using the analog jacks and not being able to plug my EMX, Ultranova and Drum at the same time.

Jumpy mouse

While trying to work with Ableton’s Live and the MixControl, I had to cope with too small fonts all the times. I ended up using Windows zoom (Windows key plus +). But regularly, the zoom was jumping all around. I figured out that this was the mouse pointer that was regularly moving around without obvious reason. Ah, this is why I am now literally constantly loosing the pointer, forced to bring it back at upper left corner of the screen almost each time I want to click on something! The pointer is really jumping around, I’m not getting crazy! This made working with the mouse a real pain, similar to what I experienced with the old Mac my brother’s wife gave me a year ago. I thought about running Live on that Mac, because many people pretend that Mac’s are more stable for music production, but the machine is way way way too slow for that, I just forgot and never tried!

I ended up trying with another mouse, that seemed to be a bit better, but I realized that the right button was completely non-working!!! Why the hell did I keep this stupid mouse then? I threw it in the thrash can and put back the first one. Then I figured out that putting the mouse on a piece of white paper helped, making it a lot less jumpy.

Windows update restarting computer while I’m using it

Windows 10 sometimes automatically restarts the computer to apply some updates. Up to now, this only happened while the machine was idle. Well yesterday, it happened right in my face, while I was working with Live! I got so pissed off by this that I tried to disable this really bad functionality. I fortunately figured out a way to disable these forced updates. It was relatively easy, although it caused me trouble because my Windows is in French and the procedure was in English. If this procedure doesn’t work and spurious reboots happen too often, this may force me to downgrade to Windows 8 or Windows 7, or switch to Mac and have constant trouble with too small fonts. This could be a dead end case leading me to stop using my computer, at least stop trying to make music with this.

Slower and slower machine

My main computer is on a desk while my music gears are on a table on the opposite wall. I tried to link them together using a long USB cable and a hub, but that failed with crashes from Ableton’s Live. However, my attempts were with the audio interface built into my Ultranova. Maybe I’ll have more luck with my Focusrite, if the cable and hub are stable enough. Why an hub? Well, this is to get a keyboard and mouse next to my music table. I will also transport video through an HDMI cable and get a screen nearby as well.

But for now, I ended up having to use my Lenovo’s IdeaPad Yoga 13 ultrabook for attempts at music production. This worked relatively well, but the machine is starting to be slow since I updated it to Windows 10. Searching on forums gives no result, except other people are experiencing performance problems, sometimes on Windows 10, sometimes on Windows 8.1. Starting Live is now taking almost 45 seconds on this machine. Fortunately, the program is responding correctly for now, until of course I add enough tracks and effects to my Live set to make it choke up like crazy. I guess this will happen if I go far enough in music production.

Difficulties with music production itself

Creating the track I had in mind caused me great trouble. While not super complex, it is not a trivial repeat drum beat. I managed to play it a couple of times, started the recording on my EMX and messed it up completely. I tried again, messed it up again. I cannot play it reliably unless I try 25 times and more. The workaround is to correct notes, but this is quite tedious on the EMX. Tired of this, I tried to record MIDI using my Ultranova as a source and Live as a sequencer. But even from Live, fixing the incorrect notes was a real pain. I experimented with the quantization which also didn’t work correctly.

There is no well-defined workflows and no comprehensive tutorials about music production. All I can find is case-specific pro tips, sometimes involving plugins I don’t want to install yet. I’m just overwhelmed with Live itself, having to constantly check and redo what I am doing, this is not a great time to complicate stuff with plugins.

Conclusion

Although I am having less and less fun with all this for the moment, I feel I can manage to get something good out of it. If I gave up because of difficulties, I would not have been able to get a Ph.D, to keep my job for more than seven years and to create a modded Minecraft map.

Categories
Bug

An intricate audio puzzle

Since I moved to Windows 8, I am having multiple and increasingly annoying issues with my audio setup for computer-assisted music creation. I am slowly reaching a dead end that will force me to give up on creating music. At best, I will only be able to play some beats for fun with hardware synthesizers: no way to record, mix, apply effects etc., no way to ever come up with a full end-to-end song, just repeated audio patterns.

Initial setup

Here is my configuration:

  • Intel Core i7
  • 16Gb of RAM
  • 240Gb SSD
  • 1Tb HDD
  • M-Audio’s Fast Track Pro
  • (About to become infamous) Ableton’s Live 9.1
  • Infamous Windows 8.1

The M-Audio interface is accepting input from two TRS jacks. I was usually plugging the output of my Korg’s EMX+Kaoss Pad during session recording. The interface also has S/PDIF input that I was using to feed in audio from my Novation’s UltraNova hardware synthesizer. This trick allowed me to record four separate tracks with that otherwise stereo-only interface. Sound is output to two KRK audio monitors.

During a few months, this configuration worked correctly. However, from October 2014, things started to misbehave in multiple random ways. I initially thought this was because of my UltraNova, but problems persist even if I uninstall UltraNova’s drivers and unplug it from USB.

Here are the issues I am facing with the setup:

  • Some recording sessions go well, but after I close Ableton’s Live, turn off my devices and shut down my computer, Windows is stuck in a loop, incapable of completely turning off the machine. Screen remains black, computer fan continues spinning. Problem can happen if I leave the audio interface plugged and turned on, or if I turn it off.
  • Sometimes, sound has an incorrect pitch. There seems to be a mismatch between audio frequencies: Ableton’s Live sends 44.1kHz while audio interface plays at 48kHz. There is no solution, except repeatedly unplugging and plugging the audio interface from a USB port, try in another port, until it works.
  • Sometimes the M-Audio driver goes corrupt and cannot play sound anymore except through ASIO using Live. When this happens, I have to completely uninstall and reinstall the driver.
  • A few weeks ago, my UltraNova started to sound awfully distorted when sound was going through S/PDIF. Worried, I tried hooking up the synthesizer to my home theater AV receiver, through S/PDIF of course, and sound was super clean. Problem is thus caused by the M-Audio interface.
  • The behavior is different but incorrect on Linux: S/PDIF sound goes out distorted unless I lower my UltraNova’s volume to more than the half. Under Windows+Live, distortion happens at that volume level as well. As a result, it seems that some software component is involved in S/PDIF handling, and that component now misbehaves differently on Windows and Linux.

After I started getting distortion through S/PDIF, I felt it was too much for me and tried to change my configuration.

Second setup: another audio interface

My UltraNova offers an onboard audio interface. In order to isolate the M-Audio interface from the overwhelmingly complex equation, I decided to give it a shot. That resulted in the second setup described below.

  • Audio input is now plugged to my UltraNova’s jacks. I ended up plugging a mixer in order to get signals from my EMX+Kaoss Pad as well as a Nord Drum module I recently acquired.
  • My UltraNova sends audio to my computer through USB. As a result, sound from the synthesizer is transferred digitally while sound from my EMX and Nord is sampled by the UltraNova’s onboard chip.

I don’t like this configuration very much because my mixer has an annoying tendency to clip and I end up with a stereo mix of my recording, making it impossible to separate the tracks. I will thus be unable to experiment with mixing using this setup, unless I manually sample each track separately.

But this is not the main issue. Here are the problems:

  • I was able to record a couple of sessions of improvisation using the above configuration. However, a week ago, recorded sound started to be distorted. Even Live’s test tone, played through ASIO, then passing into UltraNova, started to sound distorted!
  • Not only playback is distorted but also recordings are not clean anymore, dirty of distortion. A promising session that could almost have been used as is for a sound track in a future Minecraft video got screwed up by this and is good for throwaway!
  • I got fed up and tried to disable ASIO, using DirectSound instead. That worked, no distortion, but more latency. I can even hear the latency when playing through Live with a MIDI keyboard.
  • Thinking it could be UltraNova’s ASIO driver, I tested with ASIO4ALL instead and got similar issues.
  • Yesterday, Live suddenly died and could not start anymore. For no obvious reasons, I had to reinstall it completely. After that, sound with ASIO worked without distortion, but playback and recording were frequently cutting.

Contradictions, no gos, no solutions!

  • My best friend thinks the issue is caused by Windows 8 and that I should downgrade to Windows 7. However, my machine takes forever, at least one hour, to install Windows, excluding drivers and applications, installation destroys Linux boot loader and it takes me at least half an hour to find out how to restore it because there is no builtin ways in Ubuntu to do so simply. Activation of Windows 7 is likely to fail because I upgraded my license to Windows 8, and I will be left with no solution if that happens, other than trying one crack after the other or installing some piece of crap that would make sure my system’s date is set to something that won’t go past the activation grace period. I just cannot accept to have all messed up file date/times because of a single piece of software.
  • I thought about purchasing a new computer, that would have only Windows 7 and be dedicated to music creation. However, there is no Windows 7 machine anymore; every new system comes with Windows 8. I could try, maybe I would be lucky and the new Windows 8 configuration would work better, but that is a hit or miss without any way to increase chances of success! Probably my best guess would be a custom-made machine with a Windows 7 license for it, if this can at all still be purchased.
  • Some forum posts pretend that Ableton’s Live will work fine on Windows 8 and 8.1, others not, others pretend it worked on Windows 8 and not well on Windows 8.1. Downgrading to Windows 8 is just a non-sense for me, better downgrade to Windows 7 instead. I’m likely to experiment the same activation issues and will loose the same amount of time reinstalling everything and repairing my Ubuntu configuration.
  • Some forum posts suggest that it will never work well on a PC and that I should try with a Mac. Well, I tried on a Mac, and that was the most awful, most frustrating experience through my whole life. The machine was awfully slow and keyboard was not responding in a deterministic way. From this experience, I had to conclude with concerns that the Mac will work well only when used with the mouse, keyboard is just for typing text in fields and emails!!! My visual impairment makes using the mouse a pain for me. Does that mean I am excluded from computer-assisted music creation, unless somebody helps me out all the time? I am slowly but surely reaching that very frustrating conclusion.

For now it somewhat works with non-ASIO configuration and stereo recording, but I know a lot more can be done. Does the problem come from Live, Windows, my motherboard’s USB ports? I just cannot figure out.

I started to investigate about ASIO itself, to try understand how and why it could misbehave. I may have to dig straight into ASIO SDK for that. I investigated about using ASIO with Audacity, because that could help me test without Ableton’s Live and eliminate it from the equation. Even that promises to be major hassle, forcing me to install Visual Studio, SDKs and compile the whole planet to get ASIO support into Audacity!

Each new issue is decreasing my motivation to persist. I have less and less fun playing with that music creation setup and I am often thinking about putting it aside. I’m not to the point of selling the components yet, but if I cannot find out something to unblock the computer part of it, I may come to that. I will at least wait for Windows 10 to come out and give it a shot before selling anything.

I hope some people will read that post and start thinking about concrete solutions, not just “it works for me, so no problem”, “buy a Mac”, “reinstall”, “try on your friend’s computer”. Maybe a Linux-based music creation platform could help, maybe a Linux distribution dedicated to music and providing just the needed components, nothing that can interfere, or something that would make Mac platform faster and more usable.

I am a software developer so I could help in developing the platform, but because Live is closed source, I cannot take what exists and improve over it, I would have to start from something inferior such as Ardour or Audacity, and rebuild/reinvent/rewrite on top of it. This would be a very time consuming experience, and without Ableton’s expertise, I would certainly do it wrong. If all Mac OS X was open source, if it could be compiled from streamlined autoconf-based build processes rather than XCode where small-sized fonts are king, I would be able to open the hood, examine the code and improve the GUI. No, I would have to start from scratch, using just the BSD kernel, and reinvent the WHOLE GUI. Without Apple’s experience, I would certainly do it wrong and messy!

Categories
Bug

Is Linux overusing hard drives?

Monday, May 18 2015, when I tried to synchronize files between my main computer and my HTPC, I got error messages from Unison telling it was failing for some files. Tired of repeated unexpected failures, I tried copying the files manually. Manual copy was failing too. I quickly noticed that the whole partition I was copying the files to turned into read only. I tried to enable read/write by remounting the partition, that worked, but it reset back to read only after a few minutes. Some Google search later, I was thinking my hard drive was failing. I tried running a self-test using sudo smartctl -t short /dev/sdb and after a few minutes, sudo smartctl -a /dev/sdb. The drive was now failing the short auto-test. Damn, again? Yep, another dead hard drive!

Short term fix

First step was to stop using that beast, so move away all I can before it goes even worse. A failing drive can always become totally unusable, corrupting ALL data on it.

Transfer of my music files from that failing drive (the 1.5Tb one) to another one (the 3Tb one) started to output multiple I/O errors. No, it will fail like this for half of the files and take forever! I aborted that and instead copied my music files from my main PC. At least I would get all the files, not half of them. Fortunately, I was able to transfer most of my ripped DVD and blu-ray disks from my 1.5Tb to my 3Tb drive. I finished the transfer the next day; it ran in the background while I was working from home. I could then unmount the 1.5Tb drive, putting it out of the way for Ubuntu.

I got a new 3Tb drive Friday, May 22 2015 and installed it the morning after. This time, I was able to pick the right drive to remove, because the 1.5Tb was colored green while the 3Tb was red. The new 3Tb is also a red WD model. Unfortunately, things are never smooth when modifying that machine: I had to remove my SSD drive+bracket assembly in order to unscrew the 1.5Tb hard drive. However, after that, I was able to install the new 3Tb drive, put back the SSD assembly and reconnect everything. This went surprising well as opposed to last time where I had trouble finding a way to connect power on the four drives.

The new 3Tb drive seemed to work correctly. I was able to partition it as GPT, create a 3Tb Ext4 partition and mount that partition. I then started a long self-test to make sure it will not fail at me just a couple of days after. The self-test completed during the evening and showed no error.

Tentative prevention

Problem solved? Partly. The thing is that machine runs 24/7 to power a Minecraft server. This makes both hard drives spin non-stop. I would like Ubuntu to stop the drives when they are unused. I moved the Minecraft files to my SSD and will use the hard drives only for media and backup.

No, Ubuntu never ever spins down any hard drive! I tried to set this up with sudo hdparm -S 241 /dev/sdb, no result. Only thing that worked is manually spin down the drive with sudo hdparm -Y /dev/sdb (or /dev/sdg for the other drive). I recently found that gnome-disks has an option to set drive spindown timeout. The spindown setting from gnome-disks was honored once on my main computer, but I need to check if it’s reliable or not.

What if it happens again and again?

Otherwise, it seems I need Windows just to get my hard drives automatically spin down when unused, which is quite a shame! I don’t want to format this HTPC as a Windows machine, because the Minecraft server won’t run smoothly on Windows. I will be stuck with an always-open command prompt window with the server running, unless I search forever to figure out a way to run this as a system service, assuming it is possible.

A colleague at my workplace suggested the use of a Ubuntu virtual machine, but my HTPC doesn’t have enough memory to reliably run a VM and I cannot bump it up more than 4Gb because of motherboard limitation. Well, I could try to stick it 4 4Gb DDR2 modules and see, but I’m not sure the board would accept this at all, even though that could fit physically! If that fails, I would be stuck with useless DDR2 while newer systems use DDR3. What a pain!

I also investigated the possibility of using a RAID to improve reliability of storage. If I put a third 3Tb hard drive, I could configure a RAID5 array of total 6Tb, and even increase to 9Tb with a fourth 3Tb hard drive! The RAID5 splits and mirrors the data in such a way that two drives are involved when accessing files, increasing performance. It also makes sure that if one drive fails, ALL the data can be recovered and the array can be rebuilt by simply removing the failed drive and adding a replacement drive.

I was tempted by this, but that would have forced me to purchase two hard drives and another PSU to have more than four SATA power connectors. I wasn’t sure I wanted to spend more than 300$ just to get this up and running. Moreover, creating the RAID array would have forced me to move all the files away from my current 3Tb drive to combine it with the two new drives, unless I jumped directly to the 4-drive array.

I will instead wait for that machine to die and next system could be a smaller SSD-only HTPC combined with a NAS offering easier drive installation and replacement. I could purchase a dedicated NAS, or build myself a generic computer configured as a NAS. Fortunately, Ubuntu has facilities to configure software RAID, I checked that recently. I’m not sure about the fake RAID using the motherboard, that may or may not work, that may or may not be better than software RAID.