updating git svn local copies

Post-release in maven, I have a lot of svn modules to update:

Settig up Git mirrors of SVN

At work, I've been using git-svn for quite a while. I like the workflow options, and better merging capabilities. As a casualty from the recent laptop hard drive failure, all of my git repositories had gone, checked out from our main SVN server. I'm the only one using git where I work, but I love the workflow that it gives me and going back to SVN is a no-no. That had previously been created by doing a

git svn clone -s svn://svn.example.com/module

That took ages (3 days for all of the stuff I need to work on) and was quite slow when doing commits. My backups are mildly corrupt too, so I've started over, and set it up properly this time. Thanks to the guide here.

On the server, I created a directory to hold the git mirrors, and a text file containing the SVN modules that I wanted initially. Then a simple bash script to loop through the file and create a mirror of each SVN module:

for f in `cat svn-modules.txt` ; do svn2git.sh $f ; done



Then just make the repositories available:

git-daemon --export-all --base-path=/opt/git --verbose

and create a cron job to refresh the git mirrors periodically.

git --bare svn fetch --all

There are other ways, but that's the quick-n-dirty approach. Then a similar script on the client, which used the same list of modules that I wanted to check out.



Benefits of this approach:

• Much faster to set up - it took just over a couple of hours this time.


• Available to other people to try out - not just me.


• Provides a migration path off SVN as we eventually migrate off SVN (my long-term aim, muhahaha)

Separation of concerns

The shower control described here resonated with me. Having a separate control for temperature and volume appeals to me as a software developer. But most sinks don't work this way. Instead, you have a tap for hot water and one for cold. I've seen a sink which did have controls as per the shower. I liked it; most other people complained and eventually it was replaced with a more typical Western arrangement. I wonder why the idea doesn't transport well from shower to sink, or if there are places where that is the norm?

JRuby on Rails Tomcat logging

We have a mixed dev team - Windows and Linux currently, although I'm considering a Mac.

In your environment:

os = java.lang.System.get_property 'os.name'

config.logger = Logger.new('/var/log/my-company/my-app/rails.log', 5, 104857) if os.downcase =~ /linux/

Maven Woes 2

More time spent fighting with maven. Projects which used hibernate starting failing yesterday. We use XDoclet to generate the hibernate mapping files.

Unable to find resource 'xdoclet-plugins:xdoclet-plugin-qtags:jar:1.0.4-SNAPSHOT' in repository
central (http://repo1.maven.org/maven2)
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.

Missing:
----------
1) xdoclet-plugins:xdoclet-plugin-qtags:jar:1.0.4-SNAPSHOT
Try downloading the file manually from the project website.

  Then, install it using the command: 
      mvn install:install-file -DgroupId=xdoclet-plugins -DartifactId=xdoclet-plugin-qtags -Dversion=1.0.4-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file

  Alternatively, if you host your own repository you can deploy the file there: 
      mvn deploy:deploy-file -DgroupId=xdoclet-plugins -DartifactId=xdoclet-plugin-qtags -Dversion=1.0.4-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

For whatever reason, maven wouldn't use the SNAPSHOT version that I had locally. I couldn't find a 1.0.4 released version on t'Internet, so I just deployed the snapshot that I had in my local repository as the 1.0.4 released version. Naughty me. Had to do the same for the xdoclet-plugin-hibernate plugin as well.

UPDATE: And today it's working with the SNAPSHOT versions again. Still just doing mvn clean install, on Ubuntu, Windows Vista and within Hudson on Red Hat. Silently breaks, and then fixes itself. WTF is that all about?

Maven Woes 1

My mental model of Maven is that there is a small kernel and lots of plugins which provide functionality. On a daily basis, maven will try to update plugins that it uses; e.g. for dependency resolution. You can configure maven to not upgrade certain core plugins, but people don't tend to do this. Perhaps they should...

At the beginning of August this year, I started getting this:

 Unable to find resource 'bouncycastle:bctsp-jdk14:jar:138' in repository
central (http://repo1.maven.org/maven2)
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.

Missing:
----------
1) bouncycastle:bctsp-jdk14:jar:138

  Try downloading the file manually from the project website.

  Then, install it using the command:
      mvn install:install-file -DgroupId=bouncycastle -DartifactId=bctsp-jdk14 -
Dversion=138 -Dpackaging=jar -Dfile=/path/to/file

  Alternatively, if you host your own repository you can deploy the file there:

      mvn deploy:deploy-file -DgroupId=bouncycastle -DartifactId=bctsp-jdk14 -Dv
ersion=138 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

This broke not only all of our trunk builds, but also previously released items. Bouncy Castle jars presumably aren't in the main repositories due to crypto-export issues for some countries. We don't need to ship them. The cause of the problem? We had a dependency on jasperreports. This has an open-ended dependency:

<dependency>
  <groupId>com.lowagie</groupId>
  <artifactId>itext</artifactId>
  <version>[1.02b,)</version>
  <scope>compile</scope>
</dependency>

Version 2.1.7 of com.lowagie itext (released a couple of months ago) introduced a dependency on bouncycastle. Before that time, maven had been resolving the com.lowagie itext version to use version 1.3.1. Presumably a plugin was updated to fix a known bug in open-ended dependencies like the one in jasperreports and it exposed us to this problem. We were OK until the bug was fixed! Our current solution is to explicitly define the com.lowagie itext version as 2.1.5, which doesn't have the bouncycastle dependency. The closer dependency wins over the transitive dependency, yada...

Technology predictions revisited

An email that I sent internally about some of our RSS and Atom feed processing, on 2008-03-04:

"The push aspect is a little annoying. We can do polling quite well now, but push via SFTP is messy. One day people will wake up and do this over Jabber, using something based on Atom, but until then, I guess we need to have a scheduled task that polls the SFTP directory and copies content about. Plus ça change, plus c’est la meme chose..."

That would be PubSubHubbub then.

Ubuntu Firefox 3.5 update

I've switched to 3.5 as my main Firefox (using Google Chrome / Chromium Web Browser as well). Just needed to update the symlink in /usr/bin/firefox so that applications using the system default browser were opening the right version of Firefox.

$ sudo rm /usr/bin/firefox
$ sudo ln -s /usr/bin/firefox-3.5 /usr/bin/firefox

Scripting JMX via JRuby

I'm gradually growing some operations scripts which use JMX to alter application behaviour according to system load / required maintenance, etc. JRuby is a nice easy way to create these scripts.

host = 'some-ip'
port = 'port-number'
serviceUrl = javax.management.remote.JMXServiceURL.new("service:jmx:rmi:///jndi/rmi://#{host}:#{port}/jmxrmi")
connector = javax.management.remote.JMXConnectorFactory.connect(serviceUrl)
remote = connector.getMBeanServerConnection()
remoteRuntime = java.lang.management.ManagementFactory.newPlatformMXBeanProxy(remote, 
                     java.lang.management.ManagementFactory::RUNTIME_MXBEAN_NAME, 
                     java.lang.management.RuntimeMXBean.java_class)
p remoteRuntime.getName()
connector.close()

Seriously - another standard plugin for viewing video on the web?

I was trying to view some content via Gilad Bracha but it wasn't easy enough to do and I can't be bothered with it.

FAIL!

Indoctrination

So it begins. Took the boys climbing today, of sorts. I thought Callum was ready for it; Connor less so, but organisational issues being how they are, I had the big two while Al was off with the little one.

The venue was Brant Fell. I'd gone there the previous day and done pretty much everything apart from the traverse and a fingery eliminate at about Font 7b (similar to Perfect Day direct at Gardoms, but smaller holds. Failed with a bad split tip, same as the last time I tried Perfect Day! I think that's related to Callum's steroid cream thinning my finger-tip skin). Callum was pretty happy, romped up 3 short things. Connor, not so happy. He tied on, but then didn't like it so bailed. He then tried to solo stuff instead. That boy!

Java doesNotUnderstand-like behaviour in Eclipse

Kent Beck tweeted this recently. I've been doing this for years, but I guess it's not as widely used as I assumed.

Window | Preferences | Java | Code Style | Code Templates | Code | Method Body


// ${todo} Auto-generated method stub
throw new UnsupportedOperationException("Not implemented");


Then add a breakpoint to your Debugger Breakpoint view:

New Java Exception for UnsupportedOperationException that hasn't been caught.

IDEA supports something similar, since I had it set up then as well, but I've not used IDEA for a couple of years.

I tend not to use debuggers; I prefer tests, but sometimes a debugger's the thing.

Java REPL

Obviously, most good dynamic languages for the JVM have this, but still, it's sweet.

$ jirb 
irb(main):001:0> require 'lib/org.restlet.jar'
=> true
irb(main):002:0>  http = Java::OrgRestletData::Protocol::HTTP
=> #<Java::OrgRestletData::Protocol:0x59cbda @java_object=#>
irb(main):003:0> client = Java::OrgRestlet::Client.new http 
=> #<Java::OrgRestlet::Client:0x800aa1 @java_object=#>
irb(main):004:0> r = client.get 'http://www.apache.org/'
01-Feb-2009 22:54:08 org.restlet.engine.http.StreamClientHelper start
INFO: Starting the HTTP client
=> #<Java::OrgRestletData::Response:0x12a416a @java_object=#>

Body Swerve

Cameron making us laugh again. I picked him up from nursery the other day and got home. Al was in the kitchen with open arms looking for a hug. He started off towards her, then dropped his shoulder, sold a beautiful dummy and went straight for the drawer with the pans to get some out and start banging! Not bad considering he's only been walking properly for less than a month.

JAXP pipelines using SAX

XProc looks handy, but is not in a usable state yet. So I had to roll my own pipeline as a one-off recently, and seemed to struggle more than I expected. The requirement was fairly simple

1. Ingest some HTML and convert to well-formed XML for processing;


2. filter that XML to remove unwanted content;


3. convert the XML to a different format.

Step 2 was a new bit - I already had well-tested code for the other two parts. So I wanted to re-use that as much as possible. JAXP pipelines using SAX looked to be (and is!) very nice for this, but examples seemed a bit thin on the ground. I've put a version of it here, in the hope that others may find it useful.

/* Create an InputSource for the pipeline input document. */
InputSource in = new InputSource(new ByteArrayInputStream(StringUtils.getBytes(text, "utf-8")));

/* Step 1. TagSoup parsing to get well-formed XML */
XMLReader reader = new Parser();

try {
    SAXTransformerFactory stf = (SAXTransformerFactory) SAXTransformerFactory.newInstance();

    StringWriter sb = new StringWriter();

    OutputFormat outputFormat = new OutputFormat();
    outputFormat.setOmitXMLDeclaration(true);

    XMLSerializer serializer = new org.apache.xml.serialize.XMLSerializer(outputFormat);
    serializer.setOutputCharStream(sb);

    /* Step 2. Remove unwanted markup from the well-formed XML. */
    InputStream stripContent = getResourceAsStream("strip-content.xslt");
    XMLFilter removeUnwanted = stf.newXMLFilter(new StreamSource(stripContent));

    /* Step 3. Convert to preferred markup format. */
    InputStream xsltResourceInputStream = getResourceAsStream("xhtml2dial.xslt");
    XMLFilter xhtml2dial = stf.newXMLFilter(new StreamSource(xsltResourceInputStream));

    removeUnwanted.setParent(reader);
    xhtml2dial.setParent(removeUnwanted);
    xhtml2dial.setContentHandler(serializer.asContentHandler());

    reader.parse(in);

    return sb.toString();
} catch (TransformerException e) {
    throw new ConversionException(e.getMessage(), e);
} catch (IOException e) {
    throw new ConversionException(e.getMessage(), e);
} catch (SAXException e) {
    throw new ConversionException(e.getMessage(), e);
}

More learning

I like Mercurial. I'm trying to use it more, to see how it can improve my workflow (and trying to resist the complexity of git for the moment). I'm also trying to do a lot with Scheme, thus scratching my itch of learning about language design generally and a Lisp in a more in-depth fashion.

UPDATE: I've actually restricted access to the bitbucket repository at one of the author's request. Leave a comment here if you want access and I'll see about allowing access on a per-person basis. Allowing access will at the very least depend on good verification of the requestor and posting to the group to allow any instructors to veto access.

Socially responsible internet citizenship

I may not yet be knowledgeable enough to contribute editorially, but I can contribute to Wikipedia financially. Just donated.



Update: Not sure how cool a URI this is - not sure they've read Bill de hÓra's post on paging.

mysociety.org - How great!

I've previously used WriteToThem.com to communicate my concerns on various government / EU policies. I didn't realise that the people behind that site also had a host of other initiatives. mysociety.org is a fabulous concept that shows what can be done by moderately organised people with a little bit of technology.

Enforced SVN upgrade

I migrated from Eclipse Europa (3.3) to Eclipse Ganymede (3.4), re-installed the necessary plugins and noticed an issue with using svn on the CLI.


$ svn up
svn: This client is too old to work with working copy '.'; please get a newer Subversion client

$ svn --version
svn, version 1.4.6 (r28521)
compiled Mar 11 2008, 08:26:35

Copyright (C) 2000-2007 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).

The following repository access (RA) modules are available:

* ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol.
- handles 'http' scheme
- handles 'https' scheme
* ra_svn : Module for accessing a repository using the svn network protocol.
- handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
- handles 'file' scheme


Apparently, Subclipse had upgraded my working copies and my CLI client was too old to cope with this. The options appeared to be either upgrade to Ibex, build svn locally or use hardy-backports. Backports seemed like the best option and has a nice option to restrict what I want to install.

$ cat /etc/apt/preference
Package: *
Pin: release a=hardy-backports
Pin-Priority: 400

$ sudo aptitude install subversion=1.5.1dfsg1-1ubuntu2~hardy2
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initialising package states... Done
Building tag database... Done
The following packages are BROKEN:
subversion
1 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1274kB of archives. After unpacking 713kB will be used.
The following packages have unmet dependencies:
subversion: Depends: libsvn1 (= 1.5.1dfsg1-1ubuntu2~hardy2) but 1.4.6dfsg1-2ubuntu1 is installed.
Resolving dependencies...
The following actions will resolve these dependencies:

Upgrade the following packages:
libsvn1 [1.4.6dfsg1-2ubuntu1 (hardy, now) -> 1.5.1dfsg1-1ubuntu2~hardy2 (hardy-backports)]

Score is 20

Accept this solution? [Y/n/q/?] Y
The following packages will be upgraded:
libsvn1 subversion
2 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1995kB of archives. After unpacking 971kB will be used.
Do you want to continue? [Y/n/?] Y
Writing extended state information... Done
Get:1 http://gb.archive.ubuntu.com hardy-backports/main subversion 1.5.1dfsg1-1ubuntu2~hardy2 [1274kB]
Get:2 http://gb.archive.ubuntu.com hardy-backports/main libsvn1 1.5.1dfsg1-1ubuntu2~hardy2 [721kB]
Fetched 1995kB in 27s (72.2kB/s)
(Reading database ... 169200 files and directories currently installed.)
Preparing to replace subversion 1.4.6dfsg1-2ubuntu1 (using .../subversion_1.5.1dfsg1-1ubuntu2~hardy2_i386.deb) ...
Unpacking replacement subversion ...
Preparing to replace libsvn1 1.4.6dfsg1-2ubuntu1 (using .../libsvn1_1.5.1dfsg1-1ubuntu2~hardy2_i386.deb) ...
Unpacking replacement libsvn1 ...
Setting up libsvn1 (1.5.1dfsg1-1ubuntu2~hardy2) ...

Setting up subversion (1.5.1dfsg1-1ubuntu2~hardy2) ...
Installing new version of config file /etc/bash_completion.d/subversion ...
Installing new version of config file /etc/subversion/config ...
Installing new version of config file /etc/subversion/servers ...
Processing triggers for libc6 ...
ldconfig deferred processing now taking place
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initialising package states... Done
Building tag database... Done


Subversion upgrade - DONE!

Ubuntu Hardy issues

Just a place holder for stuff as I hit it.

1. Hibernate / Resume has stopped working. Damn, now it takes me 10 minutes to start up and shutdown at the end of the day. That's time my employer is losing. With Gutsy, Sleep was great. Just shut the lid and leg it out at the end of the day to beat the traffic and pick up the kids. Now I have to stop writing code early. Tracking this, but I'm don't have an idea of when I might get the old functionality back.
2. Thunderbird with Lightning 0.8 stopped understanding iCal, VCALENDAR and all remote and local Calendar data disappeared. I eventually narrowed it down to this. With libstdc++5 installed locally, a reinstall of the lightning 0.8 plugin brought all my calendars back. Might be moving more of them into the cloud from now on though.
3. IO seems a slower, or the disk is getting spun quite a lot more. That's just a subjective impression though, and I don't have any Bonnie(++) numbers for my laptop before the upgrade.
4. Network Manager seems to have a hard time with roaming now. At work, I'm on a LAN all the time. At home, I'm on my LAN or WiFi. Switching at home seems a lot more troublesome that it used to be, and /etc/init.d/dbus restart as my usual brute force approach seems to stop the NM applet from getting put back into the System Tray. No bug report for that yet, since that sounds a little woolly!