Java APIs and Ivy: Click-Through Agreements Suck

We've been using Ivy for project dependency management at OpenGamma since day-one. It was one of the earliest decisions we made, and I'm quite happy that we made it. Furthermore, the work that the Ivyroundup guys have done to package up Ivy-specific files has been fantastic.

That being said, there's one big, honkin problem with it: Sun and their click-through agreements to download almost any of the "interesting" APIs that you need for enterprise software development. These packages (like the JMS or JavaMail APIs) are necessary for pretty much anything that you might want to do on a real system, and you can't legally put them up for download. And unfortunately, there are a lot of them.

The workaround, of course, is that someone in your organization downloads them all, clicks-through on the agreement that is completely and utterly unenforceable, and then sticks them on a private Ivy repository and you configure your ivysettings.xml file to refer to that location, but that's not particularly scalable. And for open source projects, it just doesn't work at all.

Yeah, we've got this Ivy thing so that it automatically identifies all the dependencies and downloads them so you don't have to do anything but have Ant installed. Except for the Java stuff, which you have to manually download and put into these particular locations, and that's a 20 minute task minimum.

Kinda takes the shine off the whole experience, doesn't it?

Now given that everybody seems to know that you can handle licensing restrictions in different ways (by a disclaimer in each file and a copy of the license included), why does Sun still live in the past? Why do they require a clickthrough for something that is essentially just an explicit disclaimer of warranties and liability? Why can I download Eclipse or NetBeans without a clickthrough but not the JMS 1.1 API interface files?

Man, I hope the EU finally lets Oracle buy Sun and put this type of stupidity out of its misery.

The Ivy Tutorial Strikes Back

We're continuing on with the Jim Moores Guest Post series, as Ivy proceeds to destroy his source code directory.

After my previous problem with the build.xml file supplied in the Ivy tutorial I hit another issue that I thought I'd share. The default ant build file given in the Ivy tutorial generates an example source file within the ant file and then builds it. I, and I presume many others, based their new ivy-aware projects on that file. It has the default targets for downloading and installing ivy. Yeah. Problem is that one of the targets looks like this:

Can you see where this is going? Note where it says to delete the entire source directory. Yeah. That just happened to me.

I lost about half a week's work. It was my own fault. Because I'm a Perforce refugee and we've decided to try using git, I was more hesitant than usual about checking non-working code into my own code branch just because i wasn't so familiar with the git style of doing things. So while I was trying to get the ivy problems ironed out I ran ant clean which promptly deleted all my source code. All of it. Well, except the unit tests.

Please, let this be a lesson that I've learned so that you don't have to. While I accept that this was my fault, I can't help thinking that as a rule no build file should ever delete the entire contents from the src directory, particularly not one shipped as part of a tutorial that one would attempt on existing code.

Ant, Ivy 2.0.0, and java.net.UnknownHostException

This is a Very Special Guest Post by Jim Moores, another one of the OpenGamma team. We thought this would be useful to the interwebs, and until we have the OpenGamma blog ready to go, we're all ranting as one.

Ivy is a great way of avoiding shipping all the dependent libraries with your code and making upgrading dependencies smoother and easier, but it can still be a bit rough around the edges. For those of you new to Ivy, it's a dependency manager that is built on top of Ant that goes off and downloads and installs all your dependent libraries given a pretty simple list in an xml file. It hit version 2.0 a few months back and started to be really useful with more and more projects switching over to it. Of course Maven has done many of the same things for much longer, but Ivy doesn't require the whole head-warping change of viewpoint that Maven does. To be fair to Maven, I've only used it in passing when building the MyFaces JSF source code and while I did get it working, and was impressed, it took a bit of fiddling. My general understanding of Maven is that it's more declarative than 'normal' build systems - you tell it what you need and you have to rely on it figuring the whole thing out and it requires you let go of the idea of targets and explicit flow control (apologies to Maven fans if any of this is inaccurate). Ivy requires less of a leap of faith because it fits in with the Ant style that is more familiar to many but still brings the advantages of automatic dependency tracking. It can even install itself.

When I first tried Ivy three or four months ago I quickly found that some of the packages I wanted weren't available in the default ibiblio maven repository, in my case it was BerkeleyDB. I soon came across the excellent Ivy RoundUp repository. Ivy RoundUp doesn't actually host the packages themselves, it's a meta-repository that hosts information on how to download extract and repackage artifacts on demand. Unfortunately I couldn't just switch over to RoundUp because I needed some packages that were only on ibibio, so I had to delve deeper into the ivysettings.xml file to set up what is called a chain resolver. Chain resolvers do just what they sound like they do - they try one repository and then fall back to another if the package can't be found in the first. Here is my settings file:

Unfortunately this didn't work at all. It turned out that the example ivy build.xml in the tutorial contains the line:

  <property name="ivy.install.version" value="2.0.0-beta1">
</property>

And that 2.0.0-beta1 has a bug in it that means that chain resolvers don't work. Once I updated the ivy.install.version property to the latest version (2.0.0 at the time) it worked fine. This is still a problem with the tutorials, so be aware.

Since then I've needed to start using Hibernate in my application so I thought adding it would be easy as it's one of the most widely used Java packages. Err, no. After finding Hibernate in RoundUp easily enough (the latest ibibio version is really old), my build hung for some time, apparently unpacking a file, and then went totally exception-tastic on me.

[ivy:cachepath]
[ivy:cachepath] download.1.N65558:
[ivy:cachepath]
[ivy:cachepath]       [get] Getting: file://C:/DOCUME~1/Jim/LOCALS~1/Temp//jta-1_1-classes.zip
[ivy:cachepath]
[ivy:cachepath]       [get] To: C:\Documents and Settings\Jim\.ivy2\packager\cache\jta-1_1-classes.zip
[ivy:cachepath]
[ivy:cachepath]       [get] Error getting file://C:/DOCUME~1/Jim/LOCALS~1/Temp//jta-1_1-classes.zip to C:\Documents and Settings\Jim\.ivy2\packager\cache\jta-1_1-classes.zip
[ivy:cachepath]
[ivy:cachepath] C:\Documents and Settings\Jim\.ivy2\packager\build\javax.transaction\jta\1.1\build.xml:53: The following error occurred while executing this line:
[ivy:cachepath] C:\Documents and Settings\Jim\.ivy2\packager\build\javax.transaction\jta\1.1\packager-output.xml:28: java.net.UnknownHostException: C
[ivy:cachepath]         at org.apache.tools.ant.ProjectHelper.addLocationToBuildException(ProjectHelper.java:541)
[ivy:cachepath]         at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:418)
[ivy:cachepath]         at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
[ivy:cachepath]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ivy:cachepath]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[ivy:cachepath]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

and another ten pages of related stuff which I'll spare you. There was clearly some problem in downloading the JTA 1.1 classes, which are a dependency of Hibernate. I tried upgrading to the latest version of ivy - didn't work. I then assumed it was an ivy package file and went digging around. When I looked to see the temporary file it was trying to open it wasn't actually there. Hmm. Just before that it implies that it's downloading the file in question. Could it be deleting it? I eventually decided it must be a problem with the file in the repository, so I went looking for help there. That's when I came across this thread on the roundup mailing list archives, and then onto the Roundup discussion of manually downloaded software.

It seems some of the software DOES NOT get downloaded automatically by ivy. Unfortunately, instead of giving you a nice error message telling you where to put it, you get a java.net.UnknownHostException. It turns out you need to download the artifacts manually from the Sun website and put them in the temporary directory ivy uses to store package downloads.

Once it did this it all works magically. Apparently the reason for this is that the download requires ticking a license agreement box which can't be done automatically. I hope this knowledge helps someone save some time. I think in the long run it'll probably mean me having a private repository for those artifacts, but for now I'll leave that for another day.