Categories
Bug

Cascade of problems

It is unbelievable how things can go bad starting from a small number of problems. This afternoon, I was overwhelmed by several hurdles, but there were only two main root causes: keyboard instability and network bandwith issue.

Everything started with the delivery of my new AZIO keyboard and Razer Taipan mouse. Well, these are low-risk plug-in devices that won’t disturb my work too much, so let’s plug them in and see how that goes. The mouse worked fine. It seems sturdy and scroll wheel is working well, far better than this Microsoft Comfort mouse I used  a while ago. Pointer moves smoothly and there are several extra buttons that could be useful. Sensitivity can be adjusted with the touch of a button, so it can be decreased for office work for the pointer to move at reasonable speed and increased in games like Fract where I had to move the mouse five meters away to manipulate some controls!

Keyboard, on the other hand, didn’t work too well. Keys are large and the LED backlight is very nice, but the keyboard has a tendency to skip keys when I type. It skips randomly, especially the e and the s. This is a real pain when trying to do anything other than looking at emails or analyzing code or data. I tried to give it some time, but the problem persisted, up to the point I got fed up and put back my old keyboard.

Some time after I put back the old keyboard, my NX connection to my company’s remote server dropped. I had to reconnect and then got the DPI bug again. The remote desktop was running in a low resolution as if DPI scaling wasn’t disabled for the NX client anymore. I tried to restart the client, checked the DPI setting, everything was fine. I had to reboot the whole system to get my resolution back at 1680×1050 in the remote desktop.

That worked for some time, then things became laggy. Ok, we need a plan B: a VirtualBox guest running Linux and accessing the files remotely using sshfs. I already had a VM on my home PC, so I wanted to copy it on my company’s ultrabook as a first step. That was intended to run in the background and not disturb anything, but file transfer became unstable and started to slow down and stop completely. I had to initiate file transfer from my home PC: it wouldn’t work the other way round, again because of Windows. File transfer from Ubuntu failed, because the VirtualBox image was on my Windows partition and Ubuntu refused to mount it because now, Windows 8 doesn’t unmount partitions correctly, so NTFS-3G will eventuall have to adapt and implement very patchy and ugly workaround against this!

That forced me to switch back and forth between the two PCs and I was having trouble finding the buttons to switch the KVM and the HDMI switch. I don’t have a large enough desk to put two displays, two keyboards and mice so I am stuck with that stupid KVM and HDMI switch.

Things came to a total dead end when the network connection of the ultrabook stopped completely. Windows was unable to interact with my router and thus connect to the Internet. All I could do is turn wi-fi back on. I turned it off this morning, because Windows was stubbornly trying to use wi-fi instead of wired Ethernet! It took a while for wi-fi to come up, it didn’t connect automatically to my router, it took a while to connect, and connection was limited.

If I remember well, I could connect back to my NX server, but everything hung up and I had to terminate the NX client. Nothing would work: no ALT-F4, no right-click+Close program, I had to use the task manager. Then any attempt to connect back to the server returned to hung up NX session. I would have had to find back a long and complicated command on company’s wiki to reset the X server. No way! Tired of all this, I rebooted the whole server instead.

I ended up copying my files locally to not use the remote server at all and transfer the VirtualBox image using an external hard drive. That counter-productive end of afternoon totally drained me out and I was quite exasperated and overwheled after that. I have been fighting for weeks against the NX Client and the grid infrastructure I was connecting to, without nothing other than patchy workarounds that sometimes apply, sometimes fail. I felt I reached the dead end at this point. I needed a solution.

But that was working fine during the morning. Why, all of a sudden, things went south? All started from network issues: the ultrabook preferred wi-fi over wired Ethernet, the network connection to my NX server dropped all of a sudden, file transfer was unstable, Windows couldn’t access the network anymore, etc. So let’s act on network first, before fixing NX client again! Maybe that USB network interface is flawky and I would have to try with a new one.

But first, let’s remove it from that USB hub and plug it in directly into the ultrabook. That hub worked for a while, when I was using network just for sparse file transfers, but higher bandwith is needed for a full remote desktop connection. It is still good for keyboard and mouse, and necessary since that ultrabook has just two USB ports!

So I tried this, and that seemed to help! Connection to wired Ethernet happened almost instantly, Windows didn’t fall back to wi-fi as this morning, and I worked for half an hour, remotely connected, without any issue. As a final test, I transferred an Ubuntu ISO over the network and that worked without a flaw.

That hub is capable of transferring only a theoretic 480Mbts/s. It is used to carry over information from small devices like keyboards, mice, occasional data transfer from  USB stick or external hard drive, but how about something requesting 100Mbits/s constantly? That may well overload the poor little hub.

If that still bugs, I will give a shot to the Gigabit Ethernet adapter I have in the office. If that one fails as well, I will probably have to give up on this utrabook and start carrying over the heavier laptop from office to home.

Categories
Bug

Why do I suddenly need to use source to call ANY Bash script?

This week, I ran into a somewhat weird and annoying Bash issue that took a couple of minutes to solve. It was a very simple problem, but it caused quite a bit of headaches. A colleague wrote a script that was setting up some configuration variables. The script, named config.sh, was intended to be called using source config.sh. The source command tells Bash to run the script in the current interpreter rather than spawning a new process and run in it. Any variable in such a sourced script gets set up in the current Bash process, so they can be used after the script finished.

The script was containing variables of the form

TOOLS_DIR=<some path>
TSVTOOLS_DIR=<some path>
...

I tried to refer to these variables in one of my script, by using $TOOLS_DIR, for example. However, each time I was calling the script, Bash was acting the same was as if the variable wasn’t defined! The variable was accessible from the Bash process that invoked the config.sh script. Why?

There were two workarounds:

  1. Call my script using source.
  2. Modify my script to call source config.sh in it. I didn’t like this, because this was adding an extra step to all my scripts and running config.sh was taking several seconds.

My colleague and I looked at this to no avail. Then I found the culprit.

The config.sh script was declaring variables local to the Bash process. For the variables to be transferred to forked processes like invoked scripts, they needed to be exported as environment variabies! So the solution was as simple as modifying config.sh to prefix every variable declaration with export! For example,

export TOOLS_DIR=<some path>
export TSVTOOLS_DIR=<some path>
...

After this very simple change, I was able to use the variables in my script, without sourcing the scripts or invoking config.sh locally.

Categories
Bug Technical analysis

Groovy + Maven + Eclipse = headache

Java is a general-purpose programming language that matured over more than ten years. It provides a solid platform on which many third party libraries (of various quality and complexity of course) were developed. Maven is one of the several ways large Java projects can be described formally and built automatically. Maven has the ability to manage dependencies a lot better than the traditional way of bundling everything together in a large archive and aims at simplifying and unifying build process, although advanced configuration quickly drifts into XML nightmares. Then comes Eclipse, an integrated development environment. Although not perfect, very far from it, Eclipse has been a time saver for me, especially when comes time to search into large code bases and refactor code.  Eclipse is designed for Java, and it has a plugin to integrate well with Maven, called M2Eclipse. We can safely state that Java, Maven and Eclipse play well together.

Then comes Groovy, a language built on top of Java. Source code in the Groovy language is compiled into byte-code like Java is, and the byte-code can run under the same virtual machine as regular Java programs, with the exception they need a set of runtime classes and the generated byte-code has some more indirections as compared to one generated by a traditional Java compiler. As a Java extension, we would expect Groovy to play well with Maven and Eclipse. Well in practice, I found out not to be exactly the case.

I experienced what follows with Eclipse Kepler, Groovy 2.2 and Maven 3. Things may have been better with older versions or with newer ones, that will have to be seen.

Groovy and Eclipse, almost well but…

First time you will try to write a Groovy program in Eclipse, you will notice that there is absolutely no IDE support for that language. You won’t be able to use any code assist and Eclipse will not allow to compile or run Groovy code for you. You will need to install an extension to get Groovy support. This is the Groovy Eclipse plugin. The plugin works relatively well, but it has a couple of annoying drawbacks.

First, code completion works in random erratic ways. I sometimes get tired and turn it off. For example, I had a variable of type String. I knew it was a String and the IDE had the ways to know, because I declared the type of the variable in my code. In Groovy, you can use variables without declaring the type. However, when I was trying to get proposed completions for to, I was getting toUpperCase() but not toLowerCase(). This was completely arbitrary.

When running a Groovy script, the arguments in the launch configuration get prepopulated with a list of standard stuff that you must not delete. If you want to pass your own arguments to your script, you have to append them at the end of what the Groovy extension inserted in the Arguments box and you need to be careful not to delete the predefined stuff if you completely replace your custom arguments.

Debugging Groovy code in Eclipse is like playing Russian roulette. Sometimes you can print the contents of a variable, sometimes you cannot; you don’t know when it will fail and why.  Sometimes you can expand an object and see its fields, sometimes the + icon is not there and you cannot expand, again for no obvious reasons. Execution may step into closures or may not, you don’t know, at least I didn’t. You can work around by putting breakpoints in the closures, but when you go out the closure, you end up in strange places of the execution within internals of Groovy. Conditional breakpoints never worked, at all, so I had to constantly pollute my code with insane if (some condition) println(“Bla”) and be careful to remove all the junk after I’m done debugging.

Error messages are sometimes cryptic. If you are unlucky enough, you can even manage to get an Internal error from the Groovy Eclipse compiler! I was getting that in one of my classes and had to disable static type checking for that class to get rid of it.

On Monday, August 4th 2014, things went completely south after I upgraded my build to Groovy 2.3. Everything was working fine with Maven on the command line. Eclipse was compiling the code fine. I set up the project to use Groovy 2.3 and there was no issue. However, when running the project, I was getting the following runtime error.

Conflicting module versions. Module [groovy-all is loaded in version 2.2.1 and you are trying to load version 2.3.6

I looked at my POM file, analyzed Maven dependencies with both mvn dependency:tree and Eclipse, found no Groovy artifact except the 2.3.6 one, verified my PATH to make sure only Groovy 2.3 was on it, checked Eclipse preferences many many times, restarted Eclipse several times, to no avail. There seems to be something in the Groovy Eclipse plugin hard-coded for Groovy 2.2, even if the compiler is set to 2.3!

Any search on Google is yielding results about Grails and Spring, as if nobody is using Groovy alone anymore, only with other frameworks. Nobody else seems to be having the issue.

Maven + Groovy = fire hazard!

Maven relies on plugins to perform its tasks, so the ability to build something with Maven depends on the quality of the plugins. There is unfortunately no official well known, well tested and stable plugin to build Groovy stuff using Maven. The page Choosing your build tool gives a good idea of what is currently available.

First I read about GMaven, but I quickly learned it was not maintained anymore, so I didn’t try to use it. Then I read that the Groovy Eclipse Compiler was the recommended one. I was a bit reluctant, thinking this was a piece of hackery that would pull out a bunch of dependencies from Eclipse, resulting to an heavy weight solution. But this was in fact well isolated and just the compiler, no need to pull the whole Eclipse core!

Maven Eclipse compiler worked well a couple of months for me. However, yesterday, things went south all of a sudden. First, there were compilation errors in my project that would not show up into Eclipse but appeared when compiling with Maven. These were error messages related to the static type checking. After fixing these, compilation went well, but all of a sudden, at runtime, I was getting a ClassNotFondError about ShortTypeHandling. I read that this class was introduced by Groovy 2.3 while my project was using Groovy 2.2. Digging further, it seemed that the Groovy Eclipse Compiler was pulling Groovy 2.3, compiling code against it but the code was executed with Groovy 2.2. This should in principle not cause any problem, but it seems that in Groovy, byte-code is not fully compatible between versions!

I tried updating my dependency to the Groovy Eclipse Compiler in the hope that would fix the issue. However, that changed my ShortTypeHandling exception for stack overflows. It happened that the clone() method of one of my class was calling super.clone(), which is perfectly normal. But Groovy was making something nasty that was causing super.clone() to recursively call clone() of my subclass! This resulted to an infinite loop causing the stack overflow.

I found this issue to be even more intricate after I tried to compile my code on JDK8 and found it out to be working correctly. In other words, the JDK was affecting how Groovy Eclipse Compiler was building the byte-code!!! In JDK7, something would fluke the byte-code, causing the stack overflow errors, while in JDK8, everything would go fine!

I then tried updating the compiler once more, to the latest and greatest. Things compiled, but I was back at square one with the ShortTypeHandling exception! So no matter what I was trying, Maven was unable to build the project anymore.

I was about to give up on Maven and use a batch file to call Groovy directly, but that would have been a lot of fiddling with the class path. I was not happy at all about this solution.

Then I found out about the GMavenPlus plugin. I tried it and it worked like a charm! The plugin makes use of the Groovy artifact defined in the project’s dependencies rather than hard-coding a compiler for its own version of Groovy. It uses the official Groovy compiler API rather than its own, so things get compiled the same way as when using the Groovy Ant task or the standalone groovyc compiler. GMavenPlus saved my day yesterday, freeing me from a lot of hassle.

Is it worth it?

I’m not sure at all. I got several problems with Groovy that would deserve a separate post. The integration difficulties with Maven and Eclipse make me believe it is better just to use Java directly. JDK8 introduced lambda expressions that fulfill a part of what Groovy is trying to implement in its own special way. For projects that really need a true scripting language, there are already several of them, like Python, which is built from the basis for scripting.