Java Gotchas

I was writing what should have been (and, in the end, was) a very simple Java program yesterday. It wound up taking considerably longer than it should have, due to my tripping over two "gotchas", which I will document here for my own benefit (the next time I trip over them).

Issue 1: Importing from the default package

The program I was writing was actually a homework assignment for a MOOC. It involved a single Java class, and I was writing it in the NetBeans IDE (which will prove germane). I had earlier taken another MOOC (Algorithms, Part I, taught by Robert Sedgewick and Kevin Wayne of my alma mater, Princeton University -- highly recommended!) and downloaded a pair of Java libraries associated with their book (Algorithms, 4th ed.). The libraries are licensed under the GPLv3, and I wanted to use a convenience class from one in the new program. So I added it to the library list in my NetBeans project and serenely tried to import the class I wanted. And failed. Miserably.

After various unproductive dinking around (and swearing), I realized that the problem was that my new Java class was in a named package but the class I was importing was in the library's default package. Using named packages is the norm in Java -- NetBeans will issue a warning if you try to park a class in the default package -- but for reasons related to grading in their MOOC, Sedgewick and Wayne had put all their classes in the default package (and required classes written by students for homework to be in the default package). It turns out that Java will not let a class in any other package import from the default package. Classes in the default package can import from each other and from classes in named packages. I guess this is a security matter. Sedgewick and Wayne provide alternative versions of their libraries using named packages, and point out this very issue in the "Q & A" section of their download page, which I had read and immediately forgotten. Hence this note to myself (since I will inevitably forget again, in some other context).

Issue 2: "Could not find or load main class ..."

Once I resolved the aforementioned problem, I thought I was good to go -- no compilation errors -- and so I tried to run my program (emphasis on the word "tried"). NetBeans announce that it could not find or load my main class. The message is a trifle unhelpful, as it does not specify whether the issue is that the main class could not be found or that it could not be loaded.

Since I selected the main class from NetBeans's run configuration menu, it seemed apparent to me that the class could be found, and for the life of me I had no idea why it could not be loaded. So I was off on a Google search that found a fair number of (somewhat dated) pages with people asking for help and specifying the same error message. Unfortunately, none of the specifics were germane to me, and I did not find any responses that helped.

Eventually I correctly guessed this issue, and I'll record it here in case anyone else is searching for that error message. The problem in my case was the path to the NetBeans project. Recall that the program I was writing was for another MOOC. I had named the parent folder with the full name of the MOOC, including spaces and a colon (which I escaped). My operating system (Linux Mint) has no problem with the presence of the colon, but apparently NetBeans (or Ant, which NetBeans uses to process build scripts) did. As a veteran of DOS and survivor of early versions of Windows, I cut my teeth on 8.3 file names (and directory names), so I suppose I should know better than to get creative with folder names. Renaming the parent folder without any punctuation marks solved that problem.

Not an issue: Regex

This last item is not a problem, it's just a neat web site I found (in an answer on Quora) whose link I wish to preserve. Online regex tester and debugger provides a web application that lets you test regular expressions to see if they do what you think/want them to do. Very useful -- which I'd seen it sooner!

NetBeans 8 Update Bugs

An update to the NetBeans IDE (8.0 patch 2) this morning seems to have introduced some problems that I was fortunately able to work around. I'll document them here in case anyone else runs into them.

For some context, when I opened NetBeans, it correctly showed three projects in the project navigator, the main project on which I am currently working (call it C) and two related library projects (A and B). My task for the morning was to make a minor tweak to B. Happily (for my sanity), I use Git for version control and keep everything backed up to a remote repository (Bitbucket).

The IDE informed me that there were three updates available, which I downloaded and installed, forcing a restart of the IDE. Unfortunately, I took no notes on which updates they were, so I can't point to a particular culprit. Following the restart, I made B my main project, made the necessary code changes, tested them, committed them and pushed the commit up to the repository. I'll note here that I changed the method I used to access the repository from HTTPS to SSH, which may be significant for the second and third bugs. Then I did a clean and build to get a new jar file. So far, so good.

Bug #1: Can't generate Javadoc

The first problem came when I went to regenerate the Javadoc files for B. The Generate Javadoc menu option was disabled (grayed out), and nothing I could do enabled it. I wasted a fair bit of time reading some very old bug reports that, as it turns out, were not germane, and generally futzing around before I finally did what I should have tried immediately: I shut down and reopened the IDE. Just like that, Generate Javadoc was enabled again, and I was able to generate the updated Javadoc files. I have no idea why the additional restart was required.

Bug #2: Project won't open

Okay, whatever, time to make C my main project again. Small problem: after the latest restart, C is no longer listed in the project navigator (although several of its source files, which had been open the entire time, are again open). Fine, I'll open it from the recent projects list. Oops, it's not listed there either (?!). So I navigate to it with File > Open Project, try to open it and discover the following stuffed into the project name field:

Error in project.xml: The content of elements must consist of well-formed character data or markup.

Looking at project.xml in a text editor, I find what appears to be some Git-related comments stuffed into it (as if Git was trying to reconcile two conflicting versions of the file). Note that (a) this corruption must have occurred some time after I first opened the IDE, since C was listed in the project navigator then (and still was after the first post-update restart), and (b) up to that point I had not touched C, and certainly had not done any commits etc. relating to C.

Anyway, the fix was to open a terminal window in the project main folder and run git reset --hard to revert to the most recent version in the remote repository ... or so I thought.

Bug #3: Projects merged

After the reset, I was able to open project C (without having to restart the IDE) and make it my main project. I was surprised to discover that, on top of C's own source files, C now contained all the packages and files from B (?!). Again reverting to a terminal (after closing the IDE), I ran git remote -v and found that, on top of listing the various known URLs for pushing and fetching C, it listed the SSH URL for project B as being one of C's URLs. How it got there I have no idea. When I added the SSH URL for B, both B and C were open, but B was the main project, and I added the URL while pushing B, not C.

Lacking an elegant way to fix this, I opened Git's configuration file (.git/config) in a text editor, found the line for B's repository and deleted it, saved that, deleted all the source files for C from C's project folder (while holding my breath), and ran git reset --hard again. That got C back to a state where it contained only its own files.

So everything is back to normal, I've only wasted one entire morning (sheesh!), and I've had the wisdom of remote backups reinforced.

NetBeans and the UTF-8 Nag

I had a mildly annoying experience with the NetBeans IDE recently, and I'm making a note to myself here what to do if it happens again.

I was debugging a Java program that I wrote using an open-source library, and it became necessary for me to set a breakpoint in the library code. This was a first time experience for me: I use open-source libraries extensively, but I virtually never open the source code, and in fact I usually don't bother to include the path to the source code when I define the library in NetBeans. I've found that nothing spoils the taste of the sausage quite like seeing how it was made. :-)

At any rate, when I opened the source file to insert the breakpoint, NetBeans warned me that the source file "cannot be safely opened with encoding UTF-8" and asked if I wanted to open it anyway. I said yes, it opened, I set the breakpoint, found the bug (in my code), closed the source file and that was that. Or so I thought. There's a discussion of this warning on the NetBeans support forum, including various possible remedies (including converting the source library to UTF-8). That wasn't necessary or worth the effort in my case, since I was opening one file, one time only.

The first occurrence of the UTF-8 warning wasn't bothersome. What was annoying was that every time thereafter that I opened the project in NetBeans, I got the warning. Whether I answered the "really do it?" question yes or no did not matter; none of the source files opened, and the IDE thereafter gave me no grief ... until the next time I went into that project.

It turns out that I had left the breakpoint in the source file set. Even though the source file was closed, and I was not attempting to reopen it, the presence of the breakpoint was apparently causing NetBeans to look at the file, see the non-UTF-8 encoding, and nag me. Once I figured that out, the solution was simple: remove the breakpoint. Note to self: be extra careful to remove breakpoints from foreign source code in the future.

CP Optimizer, Java and NetBeans

After years of coding CPLEX applications in Java, I've just started working with CP Optimizer (the IBM/ILOG constraint programming solver) ... and it did not take me long to run into problems.

As with CPLEX, you access CP Optimizer from Java through the Concert API. As always, I am using the NetBeans IDE to do the coding. With CPLEX, this required two steps: add a library to the project, containing the cplex.jar file (for compilation); and add an option of the form -Djava.library.path=<path to CPLEX>/cplex/bin/x86-64_linux to Project Properties > Run > VM Options (for execution). Your path will differ depending on which version of CPLEX you have, where it is installed, and what operating system you are using.

So I naively expected to have to do the same two things to use CP Optimizer, albeit with different paths and files. I was half right (which, like being half alive, is not all that useful).

The good news was that for compilation, all that was required was adding a library to the project, this time pointing to ILOG.CP.jar. The bad news was that putting the corresponding path to the binary libraries in java.library.path most emphatically did not work. I got a ton of linking errors.

I won't bore you with the details of the errors, nor my fruitless pounding on Google to find me an answer. The CPLEX Optimization Studio distribution comes with Java examples and a make file for them. I ran the make file (outside NetBeans), and everything worked. That gave me an idea of what I needed to do. Apparently, the path to the binaries cannot go in java.library.path. I'm not sure why, but I think it has something to do with the way the binaries, which are C/C++ code wrapped for use by Java, are linked. Instead, on my Linux system they have to be specified in the LD_LIBRARY_PATH environment variable.

Thus, per this earlier post, I hacked the <NetBeans directory>/etc/netbeans.conf file and added the line

export LD_LIBRARY_PATH=<path to CPLEX Studio 12.6>/cplex/bin/x86-64_linux:<path to CPLEX Studio 12.6>/cpoptimizer/bin/x86-64_linux

Note that if you have other goodies already on your library path that your NetBeans programs might need, you'll need to append them to the path. Also, I'm not sure what the equivalent of LD_LIBRARY_PATH is in the Windows world, let alone on a Mac ... but I suspect that something similar will be required.

I also added the same LD_LIBRARY_PATH export to ~/.bashrc so that, should I run one of my programs from a terminal, the correct paths will be in place.

A couple of warnings:

• Obviously this hack will need to be updated whenever you upgrade to a newer version of CPLEX Optimization Studio.
• This will complicate the testing of different versions of CPLEX or CP Optimizer in the same program (other than by editing netbeans.conf for each version). With CPLEX and my previous configuration, I could just switch the CPLEX library version and alter the java.library.path string in the NetBeans project properties for that project. So much for the good old days.

A NetBeans Bug

I ran into a momentarily scary bug in the NetBeans 7.2 IDE as I was working on a Java program. I'm posting details here in case anyone else is doing what I did -- desperately googling for a fix.

I had just successfully run my code from the IDE and committed the changes (using Git, although I doubt that's relevant). Somewhere between doing the commit and possibly starting to edit a file -- I forget exactly where -- the IDE gave me a low memory warning. So I exited and restarted the IDE, thinking that might free up some memory. When I went to run my code (unmodified from the previous successful run), the program would not start. Instead, I was informed in the terminal window that a ClassNotFoundException had occurred. The named class was definitely there, at least in source form. I tried a "clean and build" to recompile the code, but the exception repeated. Restarting the IDE (again) did not help. Rebooting the PC did not help.

So I went off to Google to hunt for the reason for the exception, and found a variety of bug reports (from 2011 onward) that might or might not be pertinent. Fortunately, one of them contained a suggestion to turn off the Compile on Save feature. You can access this, which is on by default (at least for me), by right-clicking the project in the Projects window and clicking Properties > Build > Compiling. I turned it off, and sure enough the program would again run from the IDE. So I tried a "clean and build", verified the program ran, switched Compile on Save back on, and the bug returned.

Compile on Save does what is sounds like: it compiles source files when you save them. This saves time when you run or debug the program, so I was not entirely thrilled at having to turn it off. Before submitting a bug report, I did a quick search of the NetBeans issue tracker, and found only one bug report (again, from 2011) that mentioned both Compile on Save and ClassNotFoundException. The person who filed that report resolved his problem by deleting the NetBeans cache (which NetBeans rebuilds when next run). So I tracked down the cache (on my Linux Mint PC, it's in the folder ~/.cache/netbeans/7.2), deleted it, restarted NetBeans, turned Compile on Save back on, and happily discovered that the program once again runs.

Producing UML Diagrams in Netbeans 7

I need to revise some Java code I wrote several years ago, and I thought it might be helpful to have some visual aids as a reminder of how it works. Sadly, when you get older, memory is the second thing to go. (The first thing is ... er ... um ...) I use the Netbeans 7.2 IDE, and I knew that Netbeans had a plugin to produce UML diagrams for Java code, so I figured it was finally time to bite the bullet and learn UML.

Good news first: The book "UML 2.0 in a Nutshell" is available as a free e-book. The authors assume that you know the key concepts of object-oriented programming (abstraction, classes, fields, methods ...) but not much else. I've had good luck with O'Reilly's "nutshell" series before, and this book is proving no exception. (Caveat: I'm only in chapter 2. There's still time for my head to explode.) [Update: I just found out, via the comments, that the copy I downloaded is probably pirated. So I need to reevaluate and either buy the book or switch to a truly free resource. Mea culpa: I should have checked the source more carefully. Usually I can spot pirate sites.] There is also a rather nice online resource by Scott W. Ambler [that really is free].

Now the bad news (you knew this was coming, right?): Apparently the UML plugin for Netbeans was discontinued when Netbeans moved up to version 7. What to do, what to do?

Back to good news: I found this blog post by Matthew Johnson (grad student at UNC-Charlotte), in which he identifies an alternative solution that works in Netbeans 7.x. It uses the yWorks UML Doclet, which is available as in both a free "community edition" and a paid "professional edition". (I'm using the free edition, at least until I figure out whether I'll do enough UML diagrams to warrant investigating the paid version.) The yWorks doclet embeds UML diagrams in documentation generated by Javadoc for existing Java code (when you generate the javadocs), but does not work in the other direction (producing code stubs from UML diagrams). That's fine with me; I never put enough forethought into a program to produce any sort of diagram in advance of the onset of coding.

Installing the doclet is trivial; just download it, unpack the zip archive, and park the directory somewhere you can find. Using it proved a bit problematic at first, but a combination of a tip by commenter Kai Zander on Johnson's blog and a little trial-and-error by me got it working. Here are the key steps (once the software is installed). For any Netbeans Java project:

1. Per Kai's tip,  go to the resources folder of the UML Doclet installation directory, edit ydoc.cfg, find the tag <group name="tiling"> and change the enabled property from true to false. Otherwise, you tend to get UML diagrams with an empty swath down the middle.
2. Clean and build the project. (The user guide page for UML Doclet says that you need to add paths to any external libraries used by the project to the docletpath string, but I found that if you build the project first -- which copies any external libraries into the dist/lib folder -- UML Doclet will find them without needing to add them to docletpath.)
3. Right-click the project name, click Properties, then drill down to Build>Documenting. Add the following to the Additional Javadocs Options entry: -docletpath "<ypath>/lib/ydoc.jar" -doclet ydoc.doclets.YStandard -resourcepath "<ypath>/resources" -umlautogen, where <ypath> is the absolute path to the UML Doclet installation folder. The last switch (-umlautogen) tells UML Doclet to generate overview, package and class diagrams for all documented files. There are alternative switches to generate some but not all three types of diagrams, and there are switches that will restrict diagrams to documented files containing an @y.uml tag. If you omit all the -uml... switches, or use -umlgen but don't have @y.uml tags in any of your files, you won't get any UML diagrams because the doclet will think it has no work to do.
4. Now right-click the project, click Generate Javadoc, and discover that your javadocs have embedded diagrams.

Here are samples of package and class diagrams (from my command line utility):

Package Diagram

Class Diagram

Updating Java on Mint (and Ubuntu)

For various reasons, including compatibility issues with some third-party programs and a need to generate cross-platform code on occasions, I routinely use Sun (now Oracle) Java on Linux systems, rather than GCJ (which is frequently included in Linux distributions). Following a suggestion in a recent email from the folks at Mozilla, I checked my Firefox plugins and discovered a security warning for the Java 6 plugin. I'm not sure if I could upgrade just the plugin, using the Java 7 browser plugin while still using the Java 6 JRE (runtime environment) and the Java 6 JDK (development kit); so I decided to upgrade everything to Java 7. Small problem: the Ubuntu repositories still contain version 6 and not version 7.

A quick search turned up a blog post at Web Upd8 which explains why the official repositories lag behind:

Oracle JDK7 itself is not hosted in the PPA because that's not allowed by the new Java license (which is also the reason why it has been removed from the official Ubuntu repositories); the package in the PPA automatically downloads (and installs) Oracle Java JDK 7 from its official website and installs it on your computer, just like the flashplugin-installer package does.

The PPA (personal package archive) repository to which they refer contains a package that automatically downloads and installs the current version of the Oracle Java 7 JDK. (Note that there is no comparable package to install just the JRE; you have to install the full JDK, which contains the JRE as a subset. That's fine with me, since I need the JDK for program development.)

The instructions are straightforward and worked fine for me. Once I had confirmed that Java 7 was installed and was the default choice, there was one more step. I use the Netbeans IDE for writing Java code. I could have installed the latest version with Java 7 bundled, but that's a hefty download, and my current version of Netbeans is recent enough. In Netbeans, I clicked Help > About and discovered that Netbeans was still launching with Java 6. The solution is to edit the configuration file <netbeans>/etc/netbeans.conf (where <netbeans> is the top directory of the Netbeans installation) and change the setting for netbeans_jdkhome to point to the Java 7 directory. When I restarted Netbeans, it launched with Java 7 and also changed the default Java platform for compiled code to Java 7.

Big thanks to the PPA maintainers for simplifying the process.

Dopey Defaults II

Like many programs, NetBeans IDE automatically checks for updates.  Like many programs built to be extensible, NetBeans updates components (or plugins) individually.  So it came as no shock when I cranked up NetBeans 6.9 on my home PC this morning and it announced that there were 23 or so updates available.  I told it to download and install them.  Normally I get out of the way while NB does this, but I was in a bit of a hurry today, so I told it to download/install in the background while I worked on a small Java app ... which may have led to my downfall.

After it was done updating itself, NB called for me to restart it.  I held off until I'd finished and saved some changes, then gave it the go-ahead.  When it tried to restart, I was present with a very length list of error messages, all of which seemed to have something to do with either missing plugins or unsatisfied dependencies.  I clicked past that, and NB opened up -- sort of.  I didn't get the usual start page.  I did get the file I'd been editing (a Java main class) and had left open, but the corresponding project did not open.  So I tried to open it the usual way, but NB would not open it -- or any other Java project -- and did not seem to recognize them as projects at all.

Grump, grump: I bit the bullet, downloaded the Java SE version (all I need, and the smallest at about 54MB), uninstalled and reinstalled, but keeping my old settings (which, again, may turn out to have been shooting myself in the foot).  The installer ran successfully and when I started it I got no error messages ... but also no projects.  Once again, the file I'd been editing was open but I could not open any projects.  This time, though, I noticed that when I highlighted a project directory there was a curious message in the open-project dialog that said something about a possibly uninstalled plugin (without specifying which).  So I checked in Tools > Plugins > Installed and, lo and behold, 23 plugins (the correct number) were installed but most were disabled -- including all the Java development plugins.

So there's the dopey part: either the borked upgrade of the original copy of NB 6.9 changed a setting somewhere telling NB that none of the Java development stuff was active, or the 6.9.1 installer, designed for Java development, installed with the Java stuff inactive (which it's never done before).  Dopey default?  Or bad handling of upgrade errors?  At least the fix is easy, once you recognize the problem.

Setting Environment Variables for Netbeans

I use both NetBeans and Eclipse as IDEs for developing Java code.  In some cases (notably when the code employs the CPLEX optimization library), I need to set an environment variable at run time (in the case of CPLEX, to point to the license file).  Eclipse provides an easy way to do this as part of the run time configuration for the project, but for whatever reason NetBeans lacks this feature (a singularly dopey omission IMHO).

After considerable time spent with Google, it appears that if you need to set environment variables on a project-level basis, the best option in NetBeans involves hacking the build.xml file in ways that I cannot begin to fathom.  Fortunately, if the environment variable works globally (across projects), there's a simpler answer.  In the NetBeans installation tree there's an etc/ folder containing a configuration file (netbeans.conf).  Just add the line export key=value and the problem is solved.