ScaleCamp - LittleBigPlanet

Alex and James from mediamolecule.com. Fascinating perspective of embedded developers coming to server-programming and refusing to accept commonly held views on best practices for doing so. This was the surprise hit of the conference for me; I just elected to go since there wasn't anything else in that slot that I was really passionate about. I'd been talking to them both in the queue for tea earlier and made a poorly judged joke about Map-Reduce (we pretty much had this conversation). The session they ran was an awesome talk about scaling server-side within the games sector - Little Big Planet is theirs.

Written their own C-based key-data structure store, of which we're spoilt for choice just now. Alex commented that he's looked at Redis and it has some nice stuff, but when they came to need it, there wasn't anything that met their needs, and experience with running the recommended Java stack had left them with the impression that they should stick to what they know. What they know is writing very tight code in constrained environments, so applying that mind-set to server-side development seemed to have yielded some very pleasing numbers. Other parts are in Ruby (presumably 1.9, since they're using Fibers?). I didn't get around to asking James how well that works or which implementation they're using. Very happy with that programming model though - James is or was a Java guy - funny how nice Ruby feels coming from there!

ScaleCamp - Varnish

Artur Bergman talking about Varnish; this followed on from the Squid talk and most of the same crowd hung around for this one.

purges gone from multicast to Rabbit MQ (damn can't remember if I got that right or what it means!)
2 8-core servers in London data centre 350MB/s with 5000 requests/sec. Intel X25 SSDs have changed a certain class of application. If disk is the new tape, then it's probably still acceptable to go to disk if you're running those babies. See also Last.FMs experience with them.

Attempt cache hits in all data centres (UK -> US) before going to the app. Much better performance.

CDN gets broken with query string parameters - common misconfiguration which can be defended against.

Varnish protects against thundering herd. Interesting - need to read more about that to better understand it.

ScaleCamp - scaling with Squid

Summarising a recent Thoughtworks experience with this, from Chris Read and Sam Newman.

This was for a high volume retailer. A Proxy / caching solution was supposed to be provided and TW would do the app. At t-4, it turned out that TW would also have to provide the proxy / cache, so this was a hasty investigation into Squid.

1 hardware LB
2 Squid boxes - 8 core machine

1 carp process
lots of child processes

going to 16 app servers.

16,000 requests per second = 5% of traffic

TTL for items ranged from 5 minutes to 1 hour for stable furniture.

whole site does 250 million requests per day

peak 24,000 requests per second

importance of good HTTP Caching directives. Discussion of making the application (and by implication, the application developers) aware of considering ETag / Expires / Vary for all parts of the application, versus just adding it via apache ReWrite or similar. Most people in the room (including me, very strongly; RFC2616 is my favourite RFC) were in the former camp.

heap LFUDA was a good change to make.

Tried Varnish, but couldn't get good numbers out of it, in the timescales available. Artur opined that Varnish should provide better numbers than Squid; he's arguably conflicted, but seemed pretty convincing! Another factor in that was very likely that they were running RHEL old shit, and Varnish works best with a shiny new kernel (cite?)

Update: they also presented a version of this talk at DevOps 2010.

ScaleCamp - State of the Nation for Monitoring

Where are we now and what's broken with it?

• Cacti


• Collectd


• Munin


• Ganglia


• Hyperic


• Zabbix


• Zenoss

People are almost getting to the point of needing more powerful machines to do the monitoring than the app servers! Maybe something's broken somewhere...

RRDTool - overall, the consensus seemed to be that this was a little dated.

1. Does lots of writes due to the way it stores data


2. throws away data by the way it aggregates - to see fine-grained data of last years sales, you need to keep a backup of the files / graphs, rather than being able to query it.


3. can't be cleansed of bad data, or it's a bitch of a job to do so.

Alternative options

• hbase


• reconnoiter


• Tokyo Tyrant / Cabinet


• timesplicedb looks to be an interesting attempt to provide a replacement. More language bindings needed, don't be shy!

A good start to the conference for me and it gave me a flavour of the depth and breadth of discussions available.

ScaleCamp UK 2009

On Friday I was fortunate enough to attend the inaugural ScaleCamp UK event, organised by Michael Brunton-Spall at the Guardian. This was a great conference. It was a BarCamp-style approach (not that I've been to BarCamp yet!) with the schedule evolving over conversations and planned on a board in the morning. Some of the sessions I took notes at; others were standing room only, so I'll try to remember what was talked about. Obviously, this is a personal perspective focused on my interests; others should be blogging about Javascript and the like.

I met lots of very passionate, smart people doing cool stuff. That bodes well for the economy; if you want to do interesting work, then hooking up with any of the people that attended there wouldn't be a bad place to start.

iMac - Guide for Linux Users

Got a MacBook Pro recently at work, for doing more iPhone stuff. I've long admired Macs as hardware, but haven't ever owned one due to an irrational distrust of Steve Jobs. Oh well, lots of friends recommend them and have told me it's the best computing experience going. I'm expecting a learning curve, but here goes:

1. Go through basic setup for my user account. No friction so far apart from the keyboard. I know which keys to check when I'm installing a Linux, so I do the same. SHIFT+2 and SHIFT+' give me @". WTF? Need to remap certain stuff; that's not the British English layout I'm used to; none of my other computers are Macs and I have 10 years of muscle memory when it comes to typing. I'll come back to fixing that. First off, Apple | System Preferences | Keyboard | Modifier Keys and sort out Caps Lock and CTRL, for good emacs usage.


2. Where's a bloody terminal? Spend 5 minutes learning the nuances of the trackpad (I'm used to a nipple) and then drag one out of Applications | Utilities onto the Dock.


3. Good, ssh is available. Copy my SSH keys and config off the Dell laptop running Ubuntu 9.10. Test all ssh stuff and grin like a maniac.


4. Right, best install any system updates before I start configuring the arse off it. For a Unix, Mac OS X seems to need a lot of restarts for simple stuff like iTunes updates.


5. WTF!? Nothing like aptitude? That seems like a glaring omission. What are my options? Googling seems to point to Macports, Fink and Homebrew as the available options.


6. IRC client - download Colloquy and start talking to real people about their experiences.


7. After that very small skewed sample, decide to go with Macports for now with an intention to properly evaluate Homebrew Real Soon Now.


8. Install git with git-svn support. $ sudo port install git-core +svn gitX


9. Checkout my github stuff. Git is missing something - completion!


10. $ sudo port install bash_completion
    $ curl -o git "http://repo.or.cz/w/git.git/blob_plain/HEAD:/contrib/completion/git-completion.bash"
    $ sudo mv git /opt/local/etc/bash_completion.d/


11. Download the behemoth that is Xcode from the Apple Developer site and start checking out Objective-C

That'll do as a minimally usable system for now. Hardware-wise, it's a delight. Being able to watch all of InfoQ content without the teething problems that I always seem to have on Ubuntu is just a major relief - I've got a lot of stuff in delicious tagged from there that I've never managed to watch, so I can start getting through that backlog as well.

Maturity

Got home from Font around midnight on Saturday night (Halloween) to see a single carved pumpkin at home with a light inside. In the morning it transpired that someone that evening had taken Connor's pumpkin and smashed it just around the corner. I was taken with his reaction - very laissez-faire and musing on how he had the enjoyment from making the pumpkin. He was just a bit disappointed that I didn't get to see it. He's almost human sometimes!

Font 2009 summary

This was a bonus trip. I had a bad back for a month beforehand thanks to a Connor Kung-Fu Panda drop on me when stretching after a run. So no climbing and whacked out on painkillers - not ideal before a bouldering trip.

Traditional Friday night / Saturday morning driving ferry and drive only to find wet Font greeting us.

Sunday

Apremont

Monday

Cusiniere

Tuesday

Rocher Canon - running

Wednesday

Haute Pleines / Isatis

Thursday

Cuvier - rest day

Friday

95.2

Saturday

Gorge aux Chats

Joined the inveterate tickers club - that and I can't really remember what I've done in Font apart from Carnage, l'Abbatoir and a few others.

Great trip; too hot for getting on some stuff, but rather that that raining all week. Lucky to get stuff done too, with my lack of preparation. Not done 7b in the forest for a few years!

Google Gears for Firefox 3.5 on Ubuntu Karmic 9.10

Built my own - seems to work fine so far.

jabley@miq-jabley:~/work/gears-read-only$ svn info
Path: .
URL: http://gears.googlecode.com/svn/trunk
Repository Root: http://gears.googlecode.com/svn
Repository UUID: fe895e04-df30-0410-9975-d76d301b4276
Revision: 3410
Node Kind: directory
Schedule: normal
Last Changed Author: gears.daemon
Last Changed Rev: 3410
Last Changed Date: 2009-11-10 01:49:08 +0000 (Tue, 10 Nov 2009)

Made some changes:



make mode=OPT

and then install the resulting xpi.

updating git svn local copies

Post-release in maven, I have a lot of svn modules to update:

Settig up Git mirrors of SVN

At work, I've been using git-svn for quite a while. I like the workflow options, and better merging capabilities. As a casualty from the recent laptop hard drive failure, all of my git repositories had gone, checked out from our main SVN server. I'm the only one using git where I work, but I love the workflow that it gives me and going back to SVN is a no-no. That had previously been created by doing a

git svn clone -s svn://svn.example.com/module

That took ages (3 days for all of the stuff I need to work on) and was quite slow when doing commits. My backups are mildly corrupt too, so I've started over, and set it up properly this time. Thanks to the guide here.

On the server, I created a directory to hold the git mirrors, and a text file containing the SVN modules that I wanted initially. Then a simple bash script to loop through the file and create a mirror of each SVN module:

for f in `cat svn-modules.txt` ; do svn2git.sh $f ; done



Then just make the repositories available:

git-daemon --export-all --base-path=/opt/git --verbose

and create a cron job to refresh the git mirrors periodically.

git --bare svn fetch --all

There are other ways, but that's the quick-n-dirty approach. Then a similar script on the client, which used the same list of modules that I wanted to check out.



Benefits of this approach:

• Much faster to set up - it took just over a couple of hours this time.


• Available to other people to try out - not just me.


• Provides a migration path off SVN as we eventually migrate off SVN (my long-term aim, muhahaha)

Separation of concerns

The shower control described here resonated with me. Having a separate control for temperature and volume appeals to me as a software developer. But most sinks don't work this way. Instead, you have a tap for hot water and one for cold. I've seen a sink which did have controls as per the shower. I liked it; most other people complained and eventually it was replaced with a more typical Western arrangement. I wonder why the idea doesn't transport well from shower to sink, or if there are places where that is the norm?

JRuby on Rails Tomcat logging

We have a mixed dev team - Windows and Linux currently, although I'm considering a Mac.

In your environment:

os = java.lang.System.get_property 'os.name'

config.logger = Logger.new('/var/log/my-company/my-app/rails.log', 5, 104857) if os.downcase =~ /linux/

Maven Woes 2

More time spent fighting with maven. Projects which used hibernate starting failing yesterday. We use XDoclet to generate the hibernate mapping files.

Unable to find resource 'xdoclet-plugins:xdoclet-plugin-qtags:jar:1.0.4-SNAPSHOT' in repository
central (http://repo1.maven.org/maven2)
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.

Missing:
----------
1) xdoclet-plugins:xdoclet-plugin-qtags:jar:1.0.4-SNAPSHOT
Try downloading the file manually from the project website.

  Then, install it using the command: 
      mvn install:install-file -DgroupId=xdoclet-plugins -DartifactId=xdoclet-plugin-qtags -Dversion=1.0.4-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file

  Alternatively, if you host your own repository you can deploy the file there: 
      mvn deploy:deploy-file -DgroupId=xdoclet-plugins -DartifactId=xdoclet-plugin-qtags -Dversion=1.0.4-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

For whatever reason, maven wouldn't use the SNAPSHOT version that I had locally. I couldn't find a 1.0.4 released version on t'Internet, so I just deployed the snapshot that I had in my local repository as the 1.0.4 released version. Naughty me. Had to do the same for the xdoclet-plugin-hibernate plugin as well.

UPDATE: And today it's working with the SNAPSHOT versions again. Still just doing mvn clean install, on Ubuntu, Windows Vista and within Hudson on Red Hat. Silently breaks, and then fixes itself. WTF is that all about?

Maven Woes 1

My mental model of Maven is that there is a small kernel and lots of plugins which provide functionality. On a daily basis, maven will try to update plugins that it uses; e.g. for dependency resolution. You can configure maven to not upgrade certain core plugins, but people don't tend to do this. Perhaps they should...

At the beginning of August this year, I started getting this:

 Unable to find resource 'bouncycastle:bctsp-jdk14:jar:138' in repository
central (http://repo1.maven.org/maven2)
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.

Missing:
----------
1) bouncycastle:bctsp-jdk14:jar:138

  Try downloading the file manually from the project website.

  Then, install it using the command:
      mvn install:install-file -DgroupId=bouncycastle -DartifactId=bctsp-jdk14 -
Dversion=138 -Dpackaging=jar -Dfile=/path/to/file

  Alternatively, if you host your own repository you can deploy the file there:

      mvn deploy:deploy-file -DgroupId=bouncycastle -DartifactId=bctsp-jdk14 -Dv
ersion=138 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

This broke not only all of our trunk builds, but also previously released items. Bouncy Castle jars presumably aren't in the main repositories due to crypto-export issues for some countries. We don't need to ship them. The cause of the problem? We had a dependency on jasperreports. This has an open-ended dependency:

<dependency>
  <groupId>com.lowagie</groupId>
  <artifactId>itext</artifactId>
  <version>[1.02b,)</version>
  <scope>compile</scope>
</dependency>

Version 2.1.7 of com.lowagie itext (released a couple of months ago) introduced a dependency on bouncycastle. Before that time, maven had been resolving the com.lowagie itext version to use version 1.3.1. Presumably a plugin was updated to fix a known bug in open-ended dependencies like the one in jasperreports and it exposed us to this problem. We were OK until the bug was fixed! Our current solution is to explicitly define the com.lowagie itext version as 2.1.5, which doesn't have the bouncycastle dependency. The closer dependency wins over the transitive dependency, yada...

Technology predictions revisited

An email that I sent internally about some of our RSS and Atom feed processing, on 2008-03-04:

"The push aspect is a little annoying. We can do polling quite well now, but push via SFTP is messy. One day people will wake up and do this over Jabber, using something based on Atom, but until then, I guess we need to have a scheduled task that polls the SFTP directory and copies content about. Plus ça change, plus c’est la meme chose..."

That would be PubSubHubbub then.

Ubuntu Firefox 3.5 update

I've switched to 3.5 as my main Firefox (using Google Chrome / Chromium Web Browser as well). Just needed to update the symlink in /usr/bin/firefox so that applications using the system default browser were opening the right version of Firefox.

$ sudo rm /usr/bin/firefox
$ sudo ln -s /usr/bin/firefox-3.5 /usr/bin/firefox

Scripting JMX via JRuby

I'm gradually growing some operations scripts which use JMX to alter application behaviour according to system load / required maintenance, etc. JRuby is a nice easy way to create these scripts.

host = 'some-ip'
port = 'port-number'
serviceUrl = javax.management.remote.JMXServiceURL.new("service:jmx:rmi:///jndi/rmi://#{host}:#{port}/jmxrmi")
connector = javax.management.remote.JMXConnectorFactory.connect(serviceUrl)
remote = connector.getMBeanServerConnection()
remoteRuntime = java.lang.management.ManagementFactory.newPlatformMXBeanProxy(remote, 
                     java.lang.management.ManagementFactory::RUNTIME_MXBEAN_NAME, 
                     java.lang.management.RuntimeMXBean.java_class)
p remoteRuntime.getName()
connector.close()

Seriously - another standard plugin for viewing video on the web?

I was trying to view some content via Gilad Bracha but it wasn't easy enough to do and I can't be bothered with it.

FAIL!

Indoctrination

So it begins. Took the boys climbing today, of sorts. I thought Callum was ready for it; Connor less so, but organisational issues being how they are, I had the big two while Al was off with the little one.

The venue was Brant Fell. I'd gone there the previous day and done pretty much everything apart from the traverse and a fingery eliminate at about Font 7b (similar to Perfect Day direct at Gardoms, but smaller holds. Failed with a bad split tip, same as the last time I tried Perfect Day! I think that's related to Callum's steroid cream thinning my finger-tip skin). Callum was pretty happy, romped up 3 short things. Connor, not so happy. He tied on, but then didn't like it so bailed. He then tried to solo stuff instead. That boy!

Java doesNotUnderstand-like behaviour in Eclipse

Kent Beck tweeted this recently. I've been doing this for years, but I guess it's not as widely used as I assumed.

Window | Preferences | Java | Code Style | Code Templates | Code | Method Body


// ${todo} Auto-generated method stub
throw new UnsupportedOperationException("Not implemented");


Then add a breakpoint to your Debugger Breakpoint view:

New Java Exception for UnsupportedOperationException that hasn't been caught.

IDEA supports something similar, since I had it set up then as well, but I've not used IDEA for a couple of years.

I tend not to use debuggers; I prefer tests, but sometimes a debugger's the thing.

Java REPL

Obviously, most good dynamic languages for the JVM have this, but still, it's sweet.

$ jirb 
irb(main):001:0> require 'lib/org.restlet.jar'
=> true
irb(main):002:0>  http = Java::OrgRestletData::Protocol::HTTP
=> #<Java::OrgRestletData::Protocol:0x59cbda @java_object=#>
irb(main):003:0> client = Java::OrgRestlet::Client.new http 
=> #<Java::OrgRestlet::Client:0x800aa1 @java_object=#>
irb(main):004:0> r = client.get 'http://www.apache.org/'
01-Feb-2009 22:54:08 org.restlet.engine.http.StreamClientHelper start
INFO: Starting the HTTP client
=> #<Java::OrgRestletData::Response:0x12a416a @java_object=#>

Body Swerve

Cameron making us laugh again. I picked him up from nursery the other day and got home. Al was in the kitchen with open arms looking for a hug. He started off towards her, then dropped his shoulder, sold a beautiful dummy and went straight for the drawer with the pans to get some out and start banging! Not bad considering he's only been walking properly for less than a month.

JAXP pipelines using SAX

XProc looks handy, but is not in a usable state yet. So I had to roll my own pipeline as a one-off recently, and seemed to struggle more than I expected. The requirement was fairly simple

1. Ingest some HTML and convert to well-formed XML for processing;


2. filter that XML to remove unwanted content;


3. convert the XML to a different format.

Step 2 was a new bit - I already had well-tested code for the other two parts. So I wanted to re-use that as much as possible. JAXP pipelines using SAX looked to be (and is!) very nice for this, but examples seemed a bit thin on the ground. I've put a version of it here, in the hope that others may find it useful.

/* Create an InputSource for the pipeline input document. */
InputSource in = new InputSource(new ByteArrayInputStream(StringUtils.getBytes(text, "utf-8")));

/* Step 1. TagSoup parsing to get well-formed XML */
XMLReader reader = new Parser();

try {
    SAXTransformerFactory stf = (SAXTransformerFactory) SAXTransformerFactory.newInstance();

    StringWriter sb = new StringWriter();

    OutputFormat outputFormat = new OutputFormat();
    outputFormat.setOmitXMLDeclaration(true);

    XMLSerializer serializer = new org.apache.xml.serialize.XMLSerializer(outputFormat);
    serializer.setOutputCharStream(sb);

    /* Step 2. Remove unwanted markup from the well-formed XML. */
    InputStream stripContent = getResourceAsStream("strip-content.xslt");
    XMLFilter removeUnwanted = stf.newXMLFilter(new StreamSource(stripContent));

    /* Step 3. Convert to preferred markup format. */
    InputStream xsltResourceInputStream = getResourceAsStream("xhtml2dial.xslt");
    XMLFilter xhtml2dial = stf.newXMLFilter(new StreamSource(xsltResourceInputStream));

    removeUnwanted.setParent(reader);
    xhtml2dial.setParent(removeUnwanted);
    xhtml2dial.setContentHandler(serializer.asContentHandler());

    reader.parse(in);

    return sb.toString();
} catch (TransformerException e) {
    throw new ConversionException(e.getMessage(), e);
} catch (IOException e) {
    throw new ConversionException(e.getMessage(), e);
} catch (SAXException e) {
    throw new ConversionException(e.getMessage(), e);
}

More learning

I like Mercurial. I'm trying to use it more, to see how it can improve my workflow (and trying to resist the complexity of git for the moment). I'm also trying to do a lot with Scheme, thus scratching my itch of learning about language design generally and a Lisp in a more in-depth fashion.

UPDATE: I've actually restricted access to the bitbucket repository at one of the author's request. Leave a comment here if you want access and I'll see about allowing access on a per-person basis. Allowing access will at the very least depend on good verification of the requestor and posting to the group to allow any instructors to veto access.

Socially responsible internet citizenship

I may not yet be knowledgeable enough to contribute editorially, but I can contribute to Wikipedia financially. Just donated.



Update: Not sure how cool a URI this is - not sure they've read Bill de hÓra's post on paging.

mysociety.org - How great!

I've previously used WriteToThem.com to communicate my concerns on various government / EU policies. I didn't realise that the people behind that site also had a host of other initiatives. mysociety.org is a fabulous concept that shows what can be done by moderately organised people with a little bit of technology.

Enforced SVN upgrade

I migrated from Eclipse Europa (3.3) to Eclipse Ganymede (3.4), re-installed the necessary plugins and noticed an issue with using svn on the CLI.


$ svn up
svn: This client is too old to work with working copy '.'; please get a newer Subversion client

$ svn --version
svn, version 1.4.6 (r28521)
compiled Mar 11 2008, 08:26:35

Copyright (C) 2000-2007 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).

The following repository access (RA) modules are available:

* ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol.
- handles 'http' scheme
- handles 'https' scheme
* ra_svn : Module for accessing a repository using the svn network protocol.
- handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
- handles 'file' scheme


Apparently, Subclipse had upgraded my working copies and my CLI client was too old to cope with this. The options appeared to be either upgrade to Ibex, build svn locally or use hardy-backports. Backports seemed like the best option and has a nice option to restrict what I want to install.

$ cat /etc/apt/preference
Package: *
Pin: release a=hardy-backports
Pin-Priority: 400

$ sudo aptitude install subversion=1.5.1dfsg1-1ubuntu2~hardy2
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initialising package states... Done
Building tag database... Done
The following packages are BROKEN:
subversion
1 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1274kB of archives. After unpacking 713kB will be used.
The following packages have unmet dependencies:
subversion: Depends: libsvn1 (= 1.5.1dfsg1-1ubuntu2~hardy2) but 1.4.6dfsg1-2ubuntu1 is installed.
Resolving dependencies...
The following actions will resolve these dependencies:

Upgrade the following packages:
libsvn1 [1.4.6dfsg1-2ubuntu1 (hardy, now) -> 1.5.1dfsg1-1ubuntu2~hardy2 (hardy-backports)]

Score is 20

Accept this solution? [Y/n/q/?] Y
The following packages will be upgraded:
libsvn1 subversion
2 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1995kB of archives. After unpacking 971kB will be used.
Do you want to continue? [Y/n/?] Y
Writing extended state information... Done
Get:1 http://gb.archive.ubuntu.com hardy-backports/main subversion 1.5.1dfsg1-1ubuntu2~hardy2 [1274kB]
Get:2 http://gb.archive.ubuntu.com hardy-backports/main libsvn1 1.5.1dfsg1-1ubuntu2~hardy2 [721kB]
Fetched 1995kB in 27s (72.2kB/s)
(Reading database ... 169200 files and directories currently installed.)
Preparing to replace subversion 1.4.6dfsg1-2ubuntu1 (using .../subversion_1.5.1dfsg1-1ubuntu2~hardy2_i386.deb) ...
Unpacking replacement subversion ...
Preparing to replace libsvn1 1.4.6dfsg1-2ubuntu1 (using .../libsvn1_1.5.1dfsg1-1ubuntu2~hardy2_i386.deb) ...
Unpacking replacement libsvn1 ...
Setting up libsvn1 (1.5.1dfsg1-1ubuntu2~hardy2) ...

Setting up subversion (1.5.1dfsg1-1ubuntu2~hardy2) ...
Installing new version of config file /etc/bash_completion.d/subversion ...
Installing new version of config file /etc/subversion/config ...
Installing new version of config file /etc/subversion/servers ...
Processing triggers for libc6 ...
ldconfig deferred processing now taking place
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initialising package states... Done
Building tag database... Done


Subversion upgrade - DONE!

Ubuntu Hardy issues

Just a place holder for stuff as I hit it.

1. Hibernate / Resume has stopped working. Damn, now it takes me 10 minutes to start up and shutdown at the end of the day. That's time my employer is losing. With Gutsy, Sleep was great. Just shut the lid and leg it out at the end of the day to beat the traffic and pick up the kids. Now I have to stop writing code early. Tracking this, but I'm don't have an idea of when I might get the old functionality back.
2. Thunderbird with Lightning 0.8 stopped understanding iCal, VCALENDAR and all remote and local Calendar data disappeared. I eventually narrowed it down to this. With libstdc++5 installed locally, a reinstall of the lightning 0.8 plugin brought all my calendars back. Might be moving more of them into the cloud from now on though.
3. IO seems a slower, or the disk is getting spun quite a lot more. That's just a subjective impression though, and I don't have any Bonnie(++) numbers for my laptop before the upgrade.
4. Network Manager seems to have a hard time with roaming now. At work, I'm on a LAN all the time. At home, I'm on my LAN or WiFi. Switching at home seems a lot more troublesome that it used to be, and /etc/init.d/dbus restart as my usual brute force approach seems to stop the NM applet from getting put back into the System Tray. No bug report for that yet, since that sounds a little woolly!

Java Triple DES example

I had to do this recently, and the examples that I found through Google seemed a little lacking. So a pointer to others:







UPDATE

Changed to use a gist on github. That will probably dilute the search ranking of this post (which is reasonably popular) but gives me nicer formatting!

Environmental influences

Thinking back to leaving Yorkshire also popped into my head how we learn things. I left Yorkshire just before I was 10. When I left, I said "Ow" when I fell over. Within a couple of months of living in Warwickshire, I said "Aie-Yah". WTF? Call it mirroring, blending, whatever. I had changed my behaviour from what was previously defined to suit my new environment.

We've recently had cause to consider how the kids are being raised, after being exposed to some new ideas and rejecting them. That's not what I want to directly talk about now though. It's more how things are learned. We think Connor has a very high tolerance to pain. Obviously, this isn't a boundary that most people want to find the limits of, so I say that based on watching him when he was smaller, and how he'd happily fall over when learning to walk, and had no fear. When I say fall, face-plant would be more accurate. He never seemed to learn to put his hands out, because nothing hurt him, therefore he never used to cry. But he's learned. Callum is much more vocal when he's upset, even if it's just a flesh wound. I think Connor has learned to cry from Callum even though he's not hurt, or in pain to the level where it causes him discomfort. But if he falls over, he's learned that he should cry. Interesting.

Crystal tones

Al's been pointing out a few occasions when I lapse into Northern; e.g. "Look at them cars, boys!". Not the greatest grammar in the world, and the kids have started to pick up on it. It reminded me of my own childhood, when my folks used to find mine and my brother's habit of dropping our H's quite painful. I often wonder if a contributing factor to leaving Yorkshire was to correct our speaking before it became an insurmountable problem. "'Ow do Fred, what's tha 'aving fer t' tea?". Whereas personally I find it problematic that our kids are currently growing up saying 'glarse' rather than glas. Oh well, plus ça change, plus c'est la même chose.

Broken ETags in the Wild

I've had a good teacher. Just noting some of the things I've come across.

Default Apache httpd configuration in a cluster with multiple filesystems

By default, Apache httpd generates an ETag of the format inode-filesize-timestamp. A symptom of this sort of physical architecture is seeing ETags for the same resource and representation that have the same last part; e.g. "518854-3504d-ce290380" and "c8578-3504d-ce290380", and probably the same Last-Modified value too. This can be fixed by changing the Apache configuration or ignoring the ETag and just using Last-Modified. I would normally recommend retaining the ETag and just using the MTime and Size parts to calculate the ETag value.

EScenic CMS

I'm not sure whether this is an application developer issue, or a problem with the EScenic server itself, but the ETag values that I've seen from this server aren't quoted.

Etag: 20080523124147BST-39-6

The ETag value should be a quoted string

Django on Google App Engine 1.0

The version of Django 0.96.1 that shipped with Google App Engine had a similar problem to the EScenic Server, so I'd recommend bundling Django with your app until Google update the bundled version. I went to fix this in Django trunk, only to find someone beat me to it!

Cameron

Something happened earlier this year and I'm only now comfortable enough to able to talk about it. It happened on 22nd June. At the time, it was terrifying, but it has a good ending.

Al had gone away for the weekend and I was at home playing single parent. Sunday morning, not quite woken from my reverie by Cameron crying softly, I got up, changed his nappy then put him on the floor in his room to play. Connor started calling to get me to come help him on the toilet. I left Cameron's room, pulled the stairgate shut and went to the bathroom to help Connor, rubbing the sleep from my eyes. About 10 seconds later I heard the worst sound that I've ever heard. A soft thudding as the boy went down the stairs, followed by immediate screaming. Fuck. I raced down to find a scene from a Tarantino film. I don't know whether the detachment comes from my (long expired) medical training, or that's just what people naturally do in these situations. I noted the pools of blood on the Mediterranean-tiled floor. Then the bump on his forehead where the blood was coming from, as I cradled him. "That's a strange looking bump, more of an indentation. Oh, that's his skull." Reaching for a muslin to try to apply pressure to the wound. It was too wide really anyway to allow effective compression, so I tried to pack the wound while talking to the emergency services operator. It was very hard to hear them with my own state and Cameron either in pain or understandably upset. Got the other two to dress themselves while the ambulance came and I got some clothes on and mopped up the blood. A big part of me even at that point was worrying Al'll kill me if she sees this, and I didn't want his brothers to see all the blood.

Nee-nah

The ambulance came and we all piled in. Callum and Connor were thrilled to be in the van with the sirens going off. I was probably white as and trying to speak calmly to Cameron as they strapped me on the bed and we moved swiftly to Frimley Park. The ambulance crew were great and we got there, Cameron having thrown up on the journey and coming close to closing his eyes a few times.

A&E was very quiet. Lots of fabulous nursing staff were there, and able to take some of the load in terms of getting out toys for the other two boys and Cameron showed an interest as well. Then I could phone Al at her hotel on the hen-weekend and relay the bad news. She took it as well as could be expected and we had lots of crying phone calls as she was driven from Bath to get to the hospital. Due to the lack of urgency in how we were being treated, I was getting more confident at this point that it wasn't life-threatening. By this time, it was the boy's nap time and the nursing staff were happy enough that he could go to sleep. Al arrived maybe 15 minutes after he went to sleep and came into the ward to see him sleeping with his eyes open, head lolling back in my arms and a cut wider maybe a centimetre wide and 6 centimetres long on his forehead, with the bone showing through. It looked ghastly and we were both crying again. We had some fraught communication with the Reg / SHO about where the best place for treatment was. Doubtless that wasn't helped by our frame of mind, and the language issue with the guy. But we were immensely happy with the level of treatment and service that the emergency services provide. You only really miss something when you don't have it anymore; I don't ever want to experience that loss with our primary medical care.

Support

Due to the nature of the head injury, Cameron was kept in overnight for observation. Our friends Alan and Alison provided invaluable help in taking the two older boys with them to a rowing regatta where they got spoiled on BBQ and ice-creams until I went to pick them up later. As an indicator of where my head was, I picked them up and promptly drove the wrong way around a roundabout. Fortunately the guy coming the other way was going fairly slowly, then my brain kicked in and I recognised where I was. Al phoned later with the results of the CT scan (she's very persuasive!).

Cameron and Al were transferred to St George's Tooting to go to the Maxillo Facial surgical unit, where Dr Singh did an amazing job of closing him up. We got to take him home the same day.

Retro

We examined the stairgate. It is a retractable one, which clips onto the opposing wall at the top and the bottom. The top clip was on a block screwed into the wall. I had placed the block 4 years earlier, which had then been removed and replaced by the painter, and endured some abuse from the other two swinging on it. We saw that it was fairly easy to just clip the gate on the top alone, without clipping the attachment on the skirting board as well. When set like this, effectively it acted like a catflap and Cameron would have had hardly any resistance to crawling against it. Not ensuring that this was properly attached was completely my fault. We were so lucky with how this turned out; from the blood pattern it looked like he'd surfed on his stomach down the first 5 steps and hit the skirting board with his head. From there, he'd rolled down the rest of the stairs to the tiled floor below, but somehow not done further damage to himself when he got to the tiles. Then the first order was shooting some video so that both sets of grandparents could see the extent of the damage and be assuaged that they didn't need to travel down to see him straight away.

So lucky, and so blessed.

Facebook Mega-Data

Just noticed that Facebook open-sourced a version of their BigTable data store and it's hosted on Google code, written in Java. I'll omit the obvious snickers about Facebook cloning BigTable and then hosting on Google Code.

FogCreek Copilot - impressed

I was hoping that now I don't have a Windows machine any more that I could wriggle out of IT support duties for the family. Well, Copilot works under Wine on Ubuntu 7.10. Arse. I was wondering about talking my folks through installing a VNC server on their machine, but just tried the Copilot quick free demo, and it all worked. And it's free on the weekend. So I guess that's my IT support position.

How to dismantle an Atomic Bomb (otherwise known as de-stressing by migrating from Vista to Ubuntu)

Finally it's happened! I've been able to get off Windows Vista and install Gutsy Gibbon on my work laptop. Herein follows a few notes on things that I noticed, which may help others take the leap. This will probably get updated as I notice things that I've missed, or that went well.

1. Cygwin. I can't say enough good things about that project. It's been great to have on Windows and ultimately played a very important part in getting stuff off my laptop before wiping the hard-drive. I've found the people involved very helpful and responsive. My initial attempts to rsync data to a safe location failed. If you're using Windows, go fuck yourself, jwz, 2007. Contacting the mailing list led me to try the snapshot versions, report issues and see them fixed very quickly. Thanks Corinna and others.


2. Firefox. My profile was split over ${HOME}/AppData/Local/Mozilla/Firefox and ${HOME}/AppData/Roaming/Mozilla/Firefox. I consolidated the two by copying to ~/.mozilla/firefox. The profiles.ini file needed editing, since it has an incorrect relative path to the profile folder. I also needed to rename extensions.rdf, so that a new one is built on startup. This preserved all of my extensions (apart from Google Toolbar; there is a Linux-specific version?) and associated data such as ModifyHeaders values, UserAgentSwitcher values, etc. I have needed to manually edit some preferences, such as Download location, and then restart Firefox for it to take effect, but otherwise it all seems to be working fine. Similarly for Thunderbird, although I should have exported my Lightning Calendars first - I seem to have lost them, but that's no biggie. I still have all of the emails (MBOX!) (although my tags seem to have disappeared. That's a bit of a pisser). Maybe I should have migrated that to GMail, rather than copying mboxes around, but it's worked.


3. SSH. I made sure that I copied my keys over and that they all worked. I seem to have missed the full known_hosts file; I have a copy, but some entries are missing, which is slightly annoying. Also, I forgot my System32/drives/etc/hosts changes, so I'll need to recreate the local aliases that I have for some servers. Sure I can remember that, so not too painful an omission.


4. sudo aptitude install tofrodos

Ubuntu just flies, versus the same hardware running Vista. That's a shocker, obviously ;-).

Update:

Eclipse was using a lot of file handles (I just install the world in terms of the number of plugins I have), which required an addition to /etc/security/limits.conf.

# raise limits due to Eclipse complaining about too many open files
* soft nofile 5120
* hard nofile 5120


Shell Meme for migrating off Vista

Backing up required files to my home machine first, prior to wiping Vista off the laptop this evening...


jabley@python:~$ history | awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}' | sort -rn | head
89 ll
70 sudo
61 cd
41 exit
38 aptitude
18 svn
15 rsync
10 mkdir
10 cp
6 which


Mouse in your hair is nothing

Just finished Russell Brand's Booky Wook, which is what the title references. Interesting in terms of the honest coverage, but it did feel a little rushed (zeitgeist is everything, darling). At one point he talks about when a mouse lived in his hair (I think it was referenced in the Jonathan Ross interview as well). We can top that. Callum has a nit comb from school; apparently part of the national curriculum to inform hygiene issues. Connor was using it on Al the other night and kept lifting her hair.

Al

Connor, what are you doing?

Connor

Looking for bees, Mummy.

Al

Not nits, Connor?

Connor

No Mummy, BEES!

Minor casualties from the disk crash

There was some loss - mainly source code for personal hacking. My Jython UnicodeData implementation was lost, although I did have a copy of that in my GMail Sent folder which has been contributed to the committers in the hope that someone with more time and enthusiasm than me finds it useful. Apparently they have. The other thing lost was my WideFinder sketching. I surprisingly did have an early warning that the disk was close to failure, when my implementation (which was clocking ten seconds or something) suddenly started taking twenty minutes to complete. Obviously I didn't pick up on this warning; instead blaming it on some horrible mistake in my code. Some learnings from my WideFinder experience.

• NIO can be tricky, and should probably have an easier API in comparison to classical IO. It was nice to look at this anyway, and I'll probably have a read of the Jython and JRuby IO stuff to see how others do it, and also the Apache NIO stuff.
• Java string matching is slow using regexp. I'd not got as far as implementing Boyer-Moore or similar, but I expected to get a big hike out of my 10 second time when switching to that.
• CAS is good, versus synchronized blocks. Brian Goetz has talked about the advantages of non-blocking forms elsewhere.
• Java is quite restrictive. I must have been doing too much Common Lisp, Erlang and Scheme recently!

MTBF

(File under this will happen to you)

Former colleagues may recall how smug I once had occasion to be, when I migrated my old home machine from dual boot with Linux and W2K to solely Ubuntu. My preparation for that was very informed by this, but it was still slightly nerve-wracking to delete everything and start from scratch. After that, my backups got a little slack. A single 40GB IBM Deskstar with content being intermittently copied over to my external USB drive, with very infrequent backups to DVD. Then the IBM drive crashed. Arse. Oh well, still got the USB drive. Then the USB drive crashed two days later. Oh shit, that's five years of my life in pictures and film gone. The first photos of all the boys...
I was feeling physically sick. A few days later and I have a better appreciation for what I've lost. Personal stuff - nothing. The USB drive started working again after leaving it on for half an hour. Everything was taken off that and put on my work laptop, and I still had photos and video on the cameraflash disk and digital tape respectively. Thank fuck for that.

Rebuilt machine - two Maxtor 360GB PATA drives (hey, my motherboard is seven years old!). My motherboard aslo has RAID 1 capabilities, but didn't appear to work too well when I installed Gutsy. It might be possible to do it, but I didn't have the patience and was worried about what happens the next time I upgrade the kernel. So I went for the software RAID option instead. I'm now doing the backups more frequently along with using S3 for anything that would give me serious pain to lose.

So just adding to the general noise around this issue. Look after your data, people!

Clever!!!

Cooking pancakes with the boys today, and Connor was getting a bit impatient waiting for a pancake to cool.

Connor

Can I have it?

Me

No Connor, it's too hot.

Cue the boy going into the cooking draw to get the oven gloves to pick up his pancake. Clever!

Apple envy

Does Apple have some quality issues? I'm in the market for a new laptop, and was considering a Mac, having heard good things from some. But recently, there seems to be a spate of problems. Think I'll be staying with Ubuntu on something for now, thanks all the same.

Scaling

Now here's a subject I'd love to read a book about...

What if pedantic civil servants wrote software?

Well, it looks like Bernard Woolley has started to do just that!

WideFinder - late entrant

Snagged me a log file and starting cranking out some code tonight. I had some free time to think about this a couple of weeks ago, and got some sketches down, but I've only recently got the data to start seeing how the code can fly. So, I'm starting out on my work Dell laptop, Dual Core Pentium 2GHz with 2GB RAM and a really shitty disk, judging by the slowness and noises it makes (or is that just Vista?) [stay on topic! - Ed]. The Ruby version runs in just over a minute, once all of the caches are warmed up. My initial naive Java version runs in 14 seconds (I haven't figured out yet how to run it using time as per *nix environments - Cygwin says it can't find the time command when I pipe zcat output into it).

Now to start implementing my ideas. I have what I think is the shared update of the accumulator as well as I'm going to get it. I'm hypothesising that most of the updates are uncontended and so don't require the full weight of Java's locking capabilities. Now I just need to parallelize the I/O and determine the most efficient matching algorithm, which seems to be Boyer-Moore from reading the Wide-Finder series. That particular algorithm seems to pop up fairly regularly in searching. Might be interesting to see what else is available in that field, but it should be in a library, surely?

Development updates

Just logging the early signs. Connor was walking to nursery with Al the other day, and kicked a cat. There is a history there, the cat had previously scratched him, but even so! The other sign to be aware of is from when the boys recently watched the middle Star Wars trilogy. Afterwards, Callum wanted to role play and be Luke. Connor wanted to be Darth Vader. Enough said.

Oakley For Ever

I love Oakley _[1], and not just for the quality of the purchased product.

• From Lance Armstrong's book, he recounts telling his Oakley sponsor that he's been diagnosed with cancer, but he's not got health insurance. The guy from Oakley tells Armstrong not to worry and he sorts it out by telling the insurance company that Armstrong had better damn well be covered, otherwise Oakley will stop doing their company heath insurance scheme through that insurance company. I'm happy spending money with a company that employs that sort of human being.
• When I recently lost one of the rubber noseclips from one of my pairs of Oakleys, I phoned up customer service and they posted some out the same day, no charge. That's just great service, and ensures that I'll continue to buy Oakleys and will probably be getting them from my three sons over time as well. Why aren't more companies that switched on about having a great long-term relationship with customers?

[1] Ok, the website does appear to suck somewhat!

Albarracin trip day six - return to Sol

Went back to Sol, to go to the other end of it. Really struggled warming up; my arms would not work, and I have very mushy thin tips on all of my fingers and thumbs. Scott did a nice roof, first or second go. I really struggled to link it until I just got really angry, and just pissed up it. Why didn't I do that the first time and save all of the skin and energy, fool? Found it hard to get pysched today, tiredness and missing the family is starting to tell. Next we tried a really cool roof traverse into a slopey top-out. Great moves, but a bit hard for me in that state. Around 7a again, but I split a tip on about fourth go and that was me done. I'd got overlapping halves on it apart from topping out, which I'd split my tip on. Scottie got overlapping halves and topped out, but then seemed to lose power, the sun came around a bit more or something and he didn't look as smooth on it. I then tucked into the baguettes while he thrashed himself on a couple of other things.

While we were in the area, we went over to Masia. That looks like it's got a few nice things to do as well, so something else to go back for. Final night, we went back to El Molina del Gato, which serves nice beer, has strange music and all of the climbing topos, including French ones so we could work out where we'd gone wrong in looking for Tierra Media. And then another disappointing meal in a place off the main square. Not sure about Spanish food; it must be better than that. Maybe we couldn't speak the language that well, but people couldn't recommend stuff either, and we were getting cold peppers out of a tin served with bad fries and indeterminate meat. Not inspiring. I was already dreaming of Thai green chicken curry.

Albarracin trip day five - return to Arrastradero

Had some unfinished business with a sit down start to an arete here, so wanted to go back. Mint conditions, quite cold and hard to warm up. Skin still felt trashed. Met up with the two guys from Bristol that we'd met at the car park earlier in the week. I think it was their first time to the area, so they were running around dead keen. Managed to do a few nice things; there was a place round the back quite similar to l'Elephant with sloping flared cracks. There was a really good-looking 7a rib (8a sit start) which I didn't try due to trying to preserve skin, and the original walls that we'd seen on our first visit to this area that we still hadn't tackled. Oh well...

Albarracin trip day four - rest day

Day four for the first rest day? Well, not really. The first day wasn't much; an hour or so of tinkering. So we've pretty much done the 2 days on, 1 day off thing. It seems to have worked out well, as this was a planned rest day, and it started raining last night.

So a very late start today, after a lot of vino tinto last night. We decided to go exploring and work out where best to go on our last two days. First off, up towards Techo. This was the first area that we found using the topo and didn't get lost! This looks like quite a brutal hardcore place. OK, so it was a rest day and we're feeling (or I am anyway) quite shredded, but most stuff here looks like you need to be doing 7b to get much out of coming here. There were some inspiring roof lines that looked around 8a; I'll save them until I've lost 10 kg and got strong again.

We carried on up the hill towards Madriles and Pyschokiller; more big stepped roofs again. Although apparently Pyschokiller might not allow climbing there. Some of the areas are restricted, but not at the time we were here.

Then we went looking for the visitor's centre that was listed on the national park signs. Unfortunately, our written Spanish was enough to understand that it was only open at weekends and holiday periods. But the drive to get there was quite something. The park seems to be raised about the plateau, and we had some fantastic views of the flood plain and the sandstone towers elsewhere. It looked like going for a walk in that area would be worthwhile on the rest day as well, if we were there for a longer period.

To round off the day, we tried to find Tierra Media, but our Spanish wasn't up to the task. We did find some striking red trees though; not sure if they were seasonally that colour, or that was their natural plumage, so to speak.

Albarracin trip day three - Sol

Well, a bit more about where we're staying. We're at Camping Albarracin. It's a somewhat strange place in terms of requiring an international bank transfer to pay the deposit, but they accept Visa when I came to pay the balance on arrival. We're in a bungalow advertised as being suitable for four, and it would do that, but it feels very bijou with just the two of us. Not sure how Andy, Emma et al will find it next week!

Went to Sol today, and again had slight problems finding our way there. We followed our Spanish topo, but went a bit far before breaking up the hill parallel to the road. Sol was good, lots of stuff to warm-up on. My skin is feeling it already though. Trying a big roof today; I couldn't quite use Scott's sequence due to not having sixty foot arms, but need to work on press moves a bit more. I couldn't quite press out over the lip enough.

Albarracin trip day two - Arrastradero

Had the most awful food at Hotel Albarracin last night. It was steak, but well-done at one end and completely raw (proper raw, not just the raw that I've had in certain French establishments. So, late start before getting up to go climbing. Lovely day again, got a bit lost going to where we intended. We went up to the main car park, and then along the road a bit before striking left into the forest, past a collection of cave paintings. This was thanks to our reading of the topo, and we had inadvertently gone to the wrong side of the hill. We noticed some enticing looking walls on the left (minimum 7a?, so a bit hard to warm up on) and eventually got to where we wanted to be. What we should have done is walked up past the swing park, until you see the climbing area on your right. But our Spanish was non-existent. Another good day, good rock and lovely area. I'm not getting any power onto the rock really, been pulling on blobs too much and got no topping out skills for these slopey topouts, so I'm struggling on everything except the crimpy overhanding walls. Scott is climbing really well, but then he climbs on rock lots! Jealous, moi!?

Albarracin trip day one

Flew into Madrid, managed to get all of the bouldering kit including my unfeasibly large extending stick on the plane and then drove the Albarracin. No map in the hire care (thanks Europcar!) but a combination of the AA and Google Maps directions got us there without any mishaps. We got there at about six in the evening and Scott was mad keen. I was more up for a beer, but we went for a drive, found some rocks and had a quick play. The rock and national park is really cool, and we were climbing as the moon came up. Beautiful. Sadly Al's got the camera this week, so I hope that my phone will suffice. Think we're going to have a good week.

How to remove a key when a 2 year old has snapped the key off in the lock

Google to the rescue again. In case this helps anyone else avoid locksmith callout charges on a Saturday, I started off here and here, and was sufficiently enlightened as to solve it myself. Try checking that the key is in the correct orientation to be removed, twisting using a screwdriver where necessary. Then you should hopefully be able to use another key to push it out from the other side, and use tweezers to coax it out the rest of the way. The suggested superglue method wasn't necessary, but is a nice trick.

Java Mock Objects tip

Discovered while using JMock, but I would imagine it's also good for EasyMock, RMock, ...


checking(new Expectations() {
    {
        one(httpServletRequest).getParameter("c");
        will(returnValue("-2"));

        one(httpServletRequest).getParameterNames();

        // StringTokenizer implements Enumeration. A bit cheeky!
        will(returnValue(new StringTokenizer("c")));
    }
});

Surgery

How I love the smell of burning vas in the evening! Recovery tips:

• treat it as a soft tissue injury and ICE it!
• Arnica; there's going to be a lot of bruising, so any old snake oil is worth a try.

Novarra makes a play for End Game

So there's been a bit of a kerfuffle over here. What's happened is that Vodafone UK have started using Novarra's content adaptation technology to transcode the internet for mobile phones. Why has this caused such irate responses in some quarters? Well, a history lesson seems to be in order...

The mobile internet started out with devices unable to render nearly all of the internet due to hardware constraints. WML was introduced as a very limited markup which phones were able to use to display content. Early WML sites were, needless to say, fairly limited in functionality, and tended to be exposed as distinct resources; e.g. Apache user-agent detection would be used to send you to http://wap.example.com/ if you accessed http://www.example.com/ with your phone. Moore's law still holds in this arena; devices got a bit more capable and iMode (CHTML) was introduced in Japan. This was initially something like HTML 2.0 / 3.2 without <table/>s, but with the better networks over there, this was somewhat more successful in the marketplace. The W3C stuck in their thumb and pulled out XHTML Basic, and then there was the related XHTML Mobile Profile.

So who is so offended by what Vodafone UK and Novarra have done? Well, mainly the mobile internet community. The solutions that the community provide have evolved over time from hand-coded WML sites, iMode sites, XHTML Basic sites, sites where the view renders the model to the appropriate markup and DIAL processors / Drutt / Volantis's offerings in this market. There is an entire industry sector dedicated to providing solutions in this area and nearly all of those solutions rely on the HTTP header User-Agent being present and being a reliable indicator of the requesting client. Novarra / Vodafone UK have introduced a fairly disruptive change which could be viewed as an attempt to change the rules of the game. They're making some pretty provacative statements along the way.

2.3.1.7 A content transformation server can do a better job of following
mobile best practices
The "Mobile Web Best Practices 1.0" W3C Proposed Recommendation [1]
contains many recommendations for authoring content that is intended for
viewing on a mobile device. A well-designed content transformation
server can do a better job of following the mobile best practices than a
human author, especially when taking into account the capabilities of
the many different mobile devices. The result will be a more
consistent, uniform experience.

I call BS. The incumbent mobile content industry is feeling the pain, but this could just be a game-changing move like Google upp-ing the storage limits on webmail. Time will tell whether the market (mobile phone customers) feels that the Vodafone UK solution is good enough, or whether a more open market will be preferred. I'm all for the Ubiquitous Web when it's good for the customer. So here's a little gedank-experiment. What happens to the mobile content industry if all carriers start using a content-adaptation proxy? How else is your company adding value? Evolving markets are hardly a new phenomenon, although maybe the rate of change is a little faster in these modern times...

Connor responding to discipline

We were out in a restaurant for Al's birthday meal and Connor was playing with the door where we were sitting. We warned him not to do that or the waitress would come over and tell him off. Five minutes later, the waitress came over and the boy's first reaction?

"I'm not touching it!"

Key log

Not blogged for a while. Usual issues:

1. Been busy at work.


2. Been on holiday and had a lot of catching up (reading blogs!)


3. Been ill

Well, I can fix the second one at least - the first thing isn't really something to complain about, and there's not a lot I can do about the third. Unsubscribed from Planet Intertwingly (sob). That should free up some more time to finish off Jython UnicodeData, read stuff and do more hacking.

Happy Birthday Babe

8 years ago I met my wife. She still amazes me every day and I can't believe how lucky I am.

Rugby World Cup - France vs Argentina

I don't think much of the English chances, but bring it on!

Update 1 9 - 17. The French seem to be choking. Lots of basic errors and not converting their chances. The Pumas are really going for it. We could be on for a massive upset and blow to the hosts (and favourites in some eyes) chances... (Loving Will Greenwood as a pundit as well).

All consuming

That's a major one of my frustrations with my current feed reader of choice dealt with then. Now, about those duplicate entries...

Amazon losing capability in core function?

I was watching Steve Yegge's talk from OSCON about branding, and one of the things he touched on was Amazon's brand is books, and they want to get out of that since Jeff Bezos thinks that certain consumables will become totally digitized. My current thinking is that they are focusing on other things anyway (hosting, anyone?) and maybe this is taking away from their original core competency. Recent orders of mine have taken a lot longer than they used to. Previously, I'd order stuff on a Monday and it would arrive on Wednesday or Thursday the same week, with Super Saver Delivery. Now it takes a lot longer. So I'm still waiting for the Erlang, ANTLR, Haskell and REST books to arrive that I'd hoped to read on holiday.

Hello Cameron

There we go, another smasher has arrived. I think there were various events and conversations that I wanted to record today, but it's late and I'm knackered. Blog more with photos tomorrow or the weekend.

Jython UnicodeData hacking in the CDS

Got numeric and decimal working today in between contractions in the Central Delivery Suite at Frimley Park Hospital today. Nearly got categories working as well, except I'm not clear how CPython has implemented unicodedata.category for undefined codepoints.

e.g. Python 2.5 uses Unicode 4.2 for the Unicode database. The integer codepoint 13313(decimal) / 3401 (hex) is not defined within Unicode 4.1.

3400;;Lo;0;L;;;;;N;;;;;
4DB5;;Lo;0;L;;;;;N;;;;;

It isn't defined in Unicode 5.0, which is what I've been using to do the Jython implementation.

3400;;Lo;0;L;;;;;N;;;;;
4DB5;;Lo;0;L;;;;;N;;;;;

So how does CPython define unicodedata.category(unichr(13313)) to be 'Lo'? And it doesn't seem to be just 'Lo' in all cases of undefined items. I'm speculating that it might be falling back to the preceding valid codepoint category. Think I need to post to a CPython list to confirm.

Google Reader Bug report - use Atom <id/> elements

I am directly subscribed to Sam Ruby's feed. I recently added Planet Intertwingly as well, which contains Sam's blog. Both feeds are served as application/atom+xml. In Google Reader, duplicate items show up (for Sam, Steve Loughran, and others that overlap from my other subscriptions. I don't want to remove individual subscriptions in case Sam removes them from Planet Intertwingly.

From my readings of the spec a while ago, that was an explicit rationale for having id's associated with each entry. As you would expect, the id's are the same. From Planet Intertwingly:

  <entry>
    <id>tag:intertwingly.net,2004:2619</id>
    <link href="http://intertwingly.net/blog/2007/07/31/Agile-Financial-Publishing" rel="alternate" type="text/html"/>
    <link href="http://intertwingly.net/blog/2619.atom" rel="replies" type="text/html"/>
    <title>Agile Financial Publishing</title>
    ...
  </entry>

From Intertwingly:

  <entry>
    <id>tag:intertwingly.net,2004:2619</id>
    <link href="2007/07/31/Agile-Financial-Publishing"/>
    <link rel="replies" href="2619.atom" thr:count="1" thr:updated="2007-07-31T12:15:28-04:00"/>
    <title>Agile Financial Publishing</title>
    ...
  </entry>

Those atom:id element IRIs appear to be the same to me...

Objective review of why Vista pisses me off (or Why isn't Vista more like Ubuntu?

So I'm being completely upfront in the title as to where I am on this one, but I think that it's worth giving some airtime to a few of these. Readers might care to note that they should route around the "grumpy Yorkshireman that doesn't like change" for some of these. You never know, I might get some comments explaining why I'm a bozo when it comes to Windows.

Avalon and the new Windows Aero UI.

I gave it two weeks and then I'd had enough. Does it really make me more productive having all that shit, or is it just effects for the sake of it? I have the same opinion on Compiz - I haven't yet seen a compelling reason for it to exist, beyond being secretly sponsored by Nvidia / ATI to make people get shiny new graphics cards. Let the conspiracy theorists chew on that one. And despite paying Dell more for a fancy graphics card in this laptop,



apparently it's not that good.

Blue screen of Death

No really, I had one in my first week, and I've had one since then. Lovely. Occasionally (once a quarter?) when running Ubuntu Dapper, I've had X lock up on me and be completely unresponsive, to the extent that I couldn't even switch to a virtual terminal or ssh into the box and do something to it. The first time this happened, I went climbing at lunchtime, came back and it was still borked, so just a hard reboot to fix that. A reboot once a quarter on a development machine doesn't strike me as too bad. Vista is managing once a week at the moment, and in neither case am I writing C or any system level code. It's all Java / Python / Ruby and that sort of level.

Black screen of death

That was a Vista new feature for me; I've never had that on previous versions of Windows. This is progress people!

Still no decent shell

I'm using Cygwin, but it doesn't seem to let me tail files and press Return to get some space in between lines. Minor, but annoying.

Continual swapping

Previously I was on a 2GB RAM Dell workstation, and now I have a 2GB RAM Dell laptop. The laptop is always swapping. What's changed? Well, I'm now on Vista rather than Ubuntu and I'm not running Oracle XE anymore, but otherwise the services running are much the same. IDEA / Eclipse, Tomcat, MySQL, Firefox, intermittent email client and a text editor. Don't know why Vista is always swapping (TaskMngr thinks it has 700MB free) but it's bloody annoying.

Broken file permissions

Doing a release today, the VPN crashed (don't know if that was Vista, BT, the Windows 2003 Server or something else. That appears to have left me with the following undeletable file.



So Vista has let me create a file that I don't have the rights to delete. That's smashing!

Random security policies

Or that's what I'm guessing is causing this anyway. If so, then a learning mode like AppArmor has would be nice. See


and



No, when I ask for a large amount of heap to be allocated for my Java process, I don't really know what I'm doing, so please stop me. Thank you Vista, you're my hero.

Update: LazyWeb to the rescue, at least about the disk thrashing issue.

The Hustler

There was a bag of sweets up on the kitchen top. Connor spotted these and was after them, but I told him "Not for breakfast". After I'd gone to work, he asked Al the same and got the same response. His reaction?

Connor:

OK Mummy, me just hold it, OK?

Al:

OK Connor.

(Al continues ironing. Some time passes)

Connor: (Coming back into the kitchen)

Look Mummy, me just hold it, OK?

Al:

Well done Connor.

(More time passes)

Connor: (Comes back into the kitchen again)

Look Mummy, me still hold it

Al:

Good boy, Connor!

After an hour, the ironing is done and Al's about to phone me to say how well-disciplined the boy's been, just holding the sweets. So she takes the ironing upstairs and comes back down to see Connor doing his Muttley laugh and just shy of shovelling all of the sweets into his mouth! Gamed by a 2 year old.

Good ETag support requires thinking about it up-front

I posted a comment on this but I thought it worthwhile going a little deeper.

Blogger doesn't support Trackback so I'll just post and link.

My point was not to argue about how little code is required to implement sending an ETag and checking an ETag based on the MD5 hash of your content (that's pretty much a library issue which should level out to be equal over time) but to go a little deeper into ETags.

I've been reading Sam Ruby long enough to have had the benefit of ETags drummed into me. The posts that Bill links to are focused on the network savings aspect of conditional GET. But you can also save server processing power, if you put a little more thought into your application model.

So we come back to the requirements for Java frameworks to support ETags such that it is possible to avoid doing the bulk of the server side processing. Caveat this could well be premature optimization, and is merely me thinking out loud. Struts is the one I'm most familiar with and I think with the struts-chain RequestProcessor, this approach could be used, but anything that works as a chain would do for this (so pure Filters would also work).

public void doFilter(ServletRequest request, ServletResponse response,
        FilterChain filterChain) throws IOException, ServletException {

    HttpServletRequest httpRequest = (HttpServletRequest) request;
    
    /*
     * Do something that works out what is required to render a response 
     * for this request and generate an ETag based on that. So here we 
     * have moved away from the approach of generating ETags from MD5 
     * hashes of the response body.
     */
    ETag currentResourceETag = calculateETag(httpRequest);
    ETag incomingETag = extractETag(httpRequest);

    if (currentResourceETag.equals(incomingETag)) {
        response.sendStatus(HttpServletResponse.SC_NOT_MODIFIED);
    } else {
        filterChain.doFilter(request, response);
        response.addHeader("Etag", currentResourceETag.stringValue());
    }
    
}

You would need to be able to obtain the items responsible for determining the ETag value reasonably early in the request processing, before any really expensive operations. Not sure what implications that has for the layers in your application, or if you were using strict MVC how disruptive / worthwhile it would be to try this approach...

Jython - UnicodeData mirrored is complete!

from test_support import verify, verbose
import sha

encoding = 'utf-8'

def test_mirrored():
    
    h = sha.sha()
    
    for i in range(65536):
        c = unichr(i)
        h.update(str(unicodedata.mirrored(c)))
        print  "%i : %i%c" % (i, unicodedata.mirrored(c), unichr(10)),
        
    # Value returned by Python 2.5, which uses Unicode 4.2
    #verify('91cd30c6c81911835dbcbed083f99fc9fc073e4a' ==  h.hexdigest(), 
    #       h.hexdigest())
    
    # Value returned by current Jython implementation, which uses Unicode 5.0
    verify('595795a212ca0ac629d6b2dfb09c703a472adb03' ==  h.hexdigest(), 
           h.hexdigest())

# Add next test!
    
if __name__ == '__main__':
    import unicodedata
    test_mirrored()

OK, it's only for the BMP, but it's a good start. Supporting supplementary characters (in Java terminology) or the other sixteeen planes would need a more fundamental change to PyUnicode, methinks. Now I need to start adding the other unicodedata methods which should be fairly straightforward. Then I'll have a working implementation to post to the dev list. Maybe end of this month, unless Baby comes and I lose my late night hacking time?

Jython UnicodeData mirroring

for i in range(65536):
        c = unichr(i)
        print  "%i : %i%c" % (i, unicodedata.mirrored(c), unichr(10)) ,

jabley@miq-jabley ~/work/eclipse/workspaces/personal/jython-trunk/jython
$ diff jython-mirrored.txt python-mirrored.txt
10177c10177
< 10176 : 0
---
> 10176 : 1
10180,10183c10180,10183
< 10179 : 0
< 10180 : 0
< 10181 : 0
< 10182 : 0
---
> 10179 : 1
> 10180 : 1
> 10181 : 1
> 10182 : 1
11779,11782c11779,11782
< 11778 : 0
< 11779 : 0
< 11780 : 0
< 11781 : 0
---
> 11778 : 1
> 11779 : 1
> 11780 : 1
> 11781 : 1
11786,11787c11786,11787
< 11785 : 0
< 11786 : 0
---
> 11785 : 1
> 11786 : 1
11789,11790c11789,11790
< 11788 : 0
< 11789 : 0
---
> 11788 : 1
> 11789 : 1
11805,11806c11805,11806
< 11804 : 0
< 11805 : 0
---
> 11804 : 1
> 11805 : 1

I was hoping to use java.lang.Character.isMirrored(char), but the above is the result of diffing the output for jython and python running my test and diff-ing the output. Looking in more detail, Java 1.4 supports UCD 3.2, then Java 5 and Java 6 both only have support for UCD 4.0.

jabley@miq-jabley ~/work/eclipse/workspaces/personal/jython-trunk/jython
$ python
Python 2.5.1 (r251:54863, May 18 2007, 16:56:43)
[GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import unicodedata
>>> unicodedata.unidata_version
'4.1.0'

And I'm getting the sneaking feeling that I've done something like that before, but it's been so long since I did any development in this area that I've forgotten it!

The Graduate

The Bash had his nursery graduation a couple of weeks ago (must put the photos up on Flickr) and then his last day in nursery was yesterday. He didn't seem too bothered; more excited about going on holiday and spending the summer with Al. But that 4 years has gone by fast!

Our little philosopher

Me

Callum, we don't need to take your toys into Legoland since there's enough things in there to keep you busy. How does playing with your toys make you feel versus going on a ride?

Callum

Happy.

Me

So what's a ride like then...like playing with toys.

Callum

So Daddy, when you're on the ride, it's like the ride is playing with you. You're the toy!

Me (slightly shocked)

Yes, that's right Callum. Good analogy.

Me

So Callum, your homework is to write an essay on the inside of a brick.

Callum

Silly Daddy, that's just pretend!

He's sharp, that one.

The Negotiator

A breakfast discussion with Connor:

Me

What would you like, Golden Nuggets?

Connor

Schweetie.

Me

No Connor, no sweetie for breakfast. Crispies?

Connor

OK Daddy.

Connor

Wee schweetie (holding his thumb and middle finger to indicate the size).

Linux goodness

I'm posting this using the Ubuntu Feisty Live Disk running on my work laptop. I was able to connect to my wireless connection and it Just Worked, WEP using a passphrase as well, rather than requiring the large hexadecimal keys. So colour me happy with that (for anyone that's interested, this is on a Dell Latitude D820, more details available on request) Think I'll be grabbing an XP install disk from our MSDN Universal subscription or whatever it's called these days and taking a serious look at Xen, along with possibly trying to pick Steve Loughran's brain about virtualisation on Linux.

Career Development

Steve Yegge's recent post about the second most important course in CS made me smile. I discovered Eric Raymond's How To Become A Hacker fairly early on in my career, and to a Mathematics graduate with a job doing Visual Basic 6.0, that was eye-opening stuff. But you always have to remember that all authors have an agenda (including this one, obviously - Ed) and it has been commented by people other than myself that esr's HOWTO could be alternatively titled 'How To Be Like Eric Raymond'. Well, so be it. It's always the journey that's the interesting part. So can we consider Steve's post to be an equivalent 'How To Be Like Steve Yegge'? Doesn't matter. Again, it's the journey that has value.

As an aside, I had an email from Amazon today.

Your order #xxx-xxxxxxx-xxxxxxx (received 27-March-2007)
-------------------------------------------------------------------------
Ordered  Title                          Price  Dispatched  Subtotal  
---------------------------------------------------------------------
Amazon.co.uk items (Sold by Amazon EU S.a.r.L.):

   1     Compilers: Principles, Tec...   £47.49      1   £47.49

Shipped via Home Delivery Network Limited (estimated arrival date: 
28-June-2007).

Note the order date as well. Three months to get that book. Ouch! But at least it's given me time to read these two, and I can interpret Steve's post (and Joe's comment) as a good barometer of where I'm heading.

Seasonal Suffering

I took the boys to the park yesterday and Callum was very chatty. He noticed that I was quite snottery* and talked about how I had B-fever, rather than hayfever. Clever!

* I must have been married to a Scot for too long to corrupt the English language so.

Jython Update - actually doing some work on UnicodeData

So I had put this on hold to finish reading Josh Bloch and Neal Gafter's Java Puzzlers. Great book; highlighted some new things for me, which is all I ask for in any book. Partly that is since I'm still not doing Java 5 apart from at home for various minor things, but it was good.

So now back to Jython, finally. Well, reasons for my procrastination first:

1. New Job - busy as hell.


2. New job comes with a new laptop, which I was hoping would be a big step up from my nearly six year old self-built machine. The new laptop has reasonably impressive hardware specification, but it's running Vista. What an absolute pile of shit. I honestly don't know how developers are productive using that OS. I've given it nearly two months, just to be sure that it's not the fact that I've been off Windows for three years that is causing me all of the problems, but really. It's got to the point where I'm looking seriously at Xen on Ubuntu for the odd application that I do need to run Windows for. The other alternative would be to install XP, put up with the half life cost and initial downtime of getting the laptop set up for development all over again. I'll enumerate my grievances in a separate post. I don't think XP would get in my way as much as Vista (I did knock up the xmlunit XMLSchema validation patch on my wife's XP machine over Christmas and it wasn't that painful), but I'm not a fan of Windows after using GNU/Linux exclusively for three years.

So I've been doing a little work on UnicodeData again. Since I've not touched it for so long, I wanted to get some code up and running to start seeing how many tests were failing. Until Jython goes to Java 5 and above, I can't use java.lang.Character to do parts of it, or I could do a piecemeal approach of use java.lang.Character for the BMP, and then implement a new part for supplementary characters, or try to provide behaviour based on the running JVM. All a bit more work than I wanted to do, laziness and hubris being key. Instead, go for the brute force approach of the simplest thing that will possibly work. So I wrote a Python script (what else - it's a nice way of bootstrapping this problem) to parse the UnicodeData.txt file and generate some Java classes. The initial approach was to partition the UnicodeData.txt into a class for each plane in Unicode. Anyone that knows Unicode and the assigned codepoints will know that the BMP will take up most of this, but I was interested in getting something working, and then maybe refine it once the tests are passing. Well, my first cut was to have a simple interface:


interface UnicodePlane {

/**
* Return a UnicodeCodepoint for the specified codepoint.
*
* @param codepoint the Unicode codepoint
*
* @return a UnicodeCodepoint, or null if there is no match
*/
UnicodeCodepoint getCodepoint(int codepoint);

}


I would have a class that implements this interface for each plane and a static initializer within each class that fills a Map of UnicodeCodepoint classes keyed by Integer codepoint.

Eclipse gives me this error:

The code for the static initializer is exceeding the 65535 bytes limit 

Whereas ANT gave me this variation:

    [javac] Compiling 2 source files to c:\Users\jabley\work\eclipse\workspaces\personal\jython-trunk\jython\build
    [javac] c:\Users\jabley\work\eclipse\workspaces\personal\jython-trunk\jython\UnicodeData\generated-src\org\python\modules\unicodedata\UnicodeCharacterDataBasicMultilingualPlane
.java:11: code too large
    [javac]     private static final Map CODEPOINTS = new HashMap();
    [javac]                              ^
    [javac] 1 error

BUILD FAILED
c:\Users\jabley\work\eclipse\workspaces\personal\jython-trunk\jython\build.xml:456: Compile failed; see the compiler error output for details.

So I need to think a bit harder about the data structures. Turning to Bentley's Programming Pearls, the sparseness of certain items stands out, like the mirrored property.

I'll have a think. At least with the Python script that I have to cut up the UnicodeData.txt file, it's very easy to add another list comprehension to it, to see how many items in the file exhibit a certain property. The other way I'm considering is to just generate a properties file and lazily populate a Map as required. That's probably what I'll try next, rather than thinking too hard about how to compress a 1038607 bytes data file into something more reasonable.

Sleekit plumbing

Feels like Thames Water have reduced the water pressure in an effort to conserve reserves over the summer. Either that, or reserves are already low enough to cause a pressure drop. Either way, it means that the toilet can take more flushes to clear, and at nine litres a flush, I'm not sure how much water that conserves.

Happy Birthday!

Had a party for both the boys at the weekend. Knackering!