svn merging on OSX

I don't always use svn as a version control system with which I'll need to merge branches, but when I do, I use fmdiff.

$ brew install fmdiff

One minor annoyance - fmmerge (used for interactive conflict resolution) doesn't work. The number of arguments passed to the script has changed since it was first written. I patched it locally, but it still didn't work. FileMerge was launched, I could edit files, etc; but it kept saying that the merge needed resolving. Instead, I just postpone all merge conflicts during the merge, and then use fmresolve and svn resolve to resolve any individual merge conflicts.

[1] I like to branch by feature typically, but occasionally, branch by VCS is used.

Minifying Javascript at runtime

Steve Souders pointed at this today; I've done something similar in the past, but I struggled somewhat with the documentation. Hopefully this might be useful to others.



As part of a product that serves as a rendering runtime for mobile, this allows authors to create Javascript, and the runtime can optimise and cache on the fly. We like it!

Installing Graphite on OSX (Snow Leopard)

This wasn't entirely straightforward, so in the hope that it's useful for others:

python on Snow Leopard doesn't seem to come with the development headers, so we need to address that, since pycairo needs them.

$ brew install python

• edit the PATH to have /usr/local/share/python at the start
• open a new shell to recognise the new PATH
• install cairo as per http://stackoverflow.com/questions/6886578/how-to-install-pycairo-1-10-on-mac-osx-with-default-python
• I do --use-gcc since it doesn't work with LLVM / LLVM-based GCC.

$ brew install cairo --use-gcc
$ wget http://cairographics.org/releases/py2cairo-1.10.0.tar.bz2
$ tar xjf py2cairo-1.10.0.tar.bz2
$ pushd py2cairo-1.10.0
$ emacs wscript
$ export CC=/usr/bin/gcc
$ export PKG_CONFIG_PATH=/usr/local/Cellar/cairo/1.10.2/lib/pkgconfig/
$ python waf configure
$ python waf build
$ python waf install 

Now install python dependencies

$ pip install django
$ pip install django-tagging
$ pip install twisted
$ pushd path/to/graphite/
$ pushd whisper
$ python setup.py install
$ popd
$ pushd carbon
$ python setup.py install
$ popd 
$ python check-dependencies.py
$ python setup.py install
$ pushd /opt/graphite/webapp
$ export PYTHON_PATH=${PYTHON_PATH}:/opt/graphite/webapp
$ pushd graphite
$ python manage.py syncdb 

Grabbed this file and put it in /opt/graphite/bin. That means I don't need to setup apache httpd locally.

$ wget https://raw.github.com/tmm1/graphite/d0f76a659f4f2dea67f19902002710f601f534aa/bin/run-graphite-devel-server.py
$ python /opt/graphite/bin/carbon-cache.py start
$ python /opt/graphite/bin/run-graphite-devel-server.py /opt/graphite 

Browse to http://localhost:8080/ and I have a graphite webapp running

$ python path/to/graphite/examples/example-client.py 

I now have a script putting data into graphite. Might want to tweak local_settings.py (make it Europe/London, for example), and conf/carbon.conf to have reasonable retention periods / file sizes for the whisper data files.

Related links:

• http://stackoverflow.com/questions/7138076/installing-graphite-on-os-x-lion-how-to-configure-apache2
• http://stackoverflow.com/questions/1811783/how-do-you-install-pycairo-on-mac-os-x
• http://stackoverflow.com/questions/6886578/how-to-install-pycairo-1-10-on-mac-osx-with-default-python
  

Content Negotiation on Mobile considered harmful

This is kind of a follow up to my previous post on using Amazon CloudFront.

This post is to cover conneg, or Content Negotiation.

TL;DR - use HTTP as designed and follow the rules.

Alice tries to access http://example.com/ on her iPhone. She gets back some HTML which references some images.




The markup returned to Alice contains references to 3 images. I'll just look at the first one.



Alice's iPhone made a request for /images/1 and got back a 320x80px PNG image, since we have a clever server-side component which knows about different user-agents and tries to serve the most suitable version of an image for each client.

Along comes Bob. Bob is using a Google Nexus One. He similarly requests our home page and gets back a link to /images/1. When the Nexus One requests that resource though, it gets back a 420x120px PNG, again thanks to our fancy server-side detection.

What does this do to our caching? Well, it stuffs it up completely.

1. We're using a canonical URI for the image - /images/1.
2. We're serving different representations of the image from the same URL.
3. We cannot easily specify good HTTP expiration directives.

Going into point 3 in more detail, we cannot use the Vary header in our response to try to let proxy caches more efficiently. Vary can only specify a request header. This means that something like the User-Agent doesn't work:

1. There are many thousand User-Agent strings in existence.
2. How different are these 2 anyway?
   • Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0_1 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko)
   • Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0_1 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Mobile/7A400

Let's be conservative and say that there are 5000 different User-Agents hitting our site. That means that we could be telling proxy servers to cache 5000 copies of /images/1, rather than the perhaps 7 different sizes that our application might produce.

So alternatives to conneg?

• Use a distinct URI for every bag of bytes that the application can serve. This means that your server-side markup generation needs to be a little smarter, so that you render markup containing /images/1/320x80 and /images/1/480x120 for example (see the point in my CloudFront post about not wanting to use query string parameters for this information).
  
• It's still possible to support the old canonical URLs, either by continuing to perform conneg, or redirecting appropriately.
  
• Rev your URIs so that you can happily set Far Future Expires directives on these resources; i.e. if the image changes, give it a new URI.

I'm still musing on what this means for progressive enhancement. Andrea's post looks like a step in the right direction though.

Objective-C concurrency issues

Disclaimer - I've not shipped Java Swing / SWT apps. I'm a server guy where markup is the UI. Consequently, I don't have in-depth knowledge of Java to compare against. I'm aware of SwingUtilities.invokeLater(Runnable) but otherwise just assume I'm clueless about Swing.

First rule of GUI programming - don't block the main thread.

Second rule of GUI programming - don't block the main thread, etc.

Quantifying this, you have a device running at a refresh rate of 60Hz. So you if you do anything in the main thread, you need it to complete in under 16ms, or your UI will not be smooth and responsive.

In Java, I would normally look at Executors, Callables, Runnables and related APIs to do things off the main thread. In Objective-C, we have NSOperationQueue and NSOperation. Learn, use and love them. In particular, don't do what I did and start porting java.util.concurrent classes to Objective-C. I wrote a CountdownLatch, which was very nice and taught me about various low-level concurrency primitives. Unfortunately it was completely the wrong solution for the language. What I should have done was to use [NSOperation addDependency:] to chain tasks together.

Objective-C tooling

Java development tools are top of the pile out of anything I've used. The IDEs are massively powerful (they have to be, with the warts on the language). I'm also an emacs user day to day, and pragmatically use vim as well. But Eclipse / IDEA / Netbeans are pretty amazing tools for Java The Language development.

Respectively for Objective-C development, Xcode isn't.

If Apple Ts&Cs allow, IntelliJ could probably make some impressive inroads into that market.

clang is a good (and getting better all the time) addition. The debugger needs some love; I don't find gdb as powerful as Java debuggers.

In Java-land, one can use maven, ant, ivy, Make, etc to build a project. For iOS development, the IDE rules a lot from the off. There is a command-line tool which can potentially be driven by Jenkins or Thoughtworks Go. That would be my preferred option going forward; in my view, building in an IDE is not a repeatable build process.

Creating a Custom Origin Server for Amazon CloudFront

At the time of writing, tool support is limited to the REST API? The intention with this piece of work was to take content being served by our origin server; e.g. http://example.com/images/foo.png; and serve it via Amazon CloudFront on http://cdn.example.com/images/foo.png.

1. Download cfcurl.pl.
2. Get any dependencies from CPAN (the cfcurl.pl script tells you how to do that in case you aren't sure).
3. Create $HOME/.aws-secrets and chmod 600.
   

   $ cat /Users/jabley/.aws-secrets
   %awsSecretAccessKeys = (
    # my personal account
    'james-personal' => {
        id => 'foo',
        key => 'bar',
    },
   
    # my corporate account
    'james-work' => {
        id => 'AWS-ID',
        key => 'AWS-Secret-Key',
    },
   );
   

4. Create a file with the request data
   

   $ cat create-distribution.xml
   <?xml version="1.0" encoding="UTF-8"?>
   <DistributionConfig xmlns="http://cloudfront.amazonaws.com/doc/2010-11-01/">
   <CustomOrigin>
      <DNSName>example.com</DNSName>
      <OriginProtocolPolicy>http-only</OriginProtocolPolicy>
   </CustomOrigin>
   <CallerReference>20110106103700</CallerReference>
   <CNAME>cdn.example.com</CNAME>
   <Comment>example.com CloudFront CDN</Comment>
   <Enabled>true</Enabled>
   <Logging>
      <Bucket>accesslogs-example.com.s3.amazonaws.com</Bucket>
      <Prefix>cdn.example.com/</Prefix>
   </Logging>
   </DistributionConfig>
   

5. POST the file
   

      perl cfcurl.pl --keyname james-work -- -X POST -H "Content-Type: text/xml;charset=utf-8" --upload-file \
       create-distribution.xml https://cloudfront.amazonaws.com/2010-11-01/distribution
   

6. Poll to see when it has finished creating the distribution:
   

      perl cfcurl.pl --keyname james-work -- https://cloudfront.amazonaws.com/2008-06-30/distribution
   

7. Configure DNS so that cdn.example.com is a CNAME for the DomainName value of your newly created CloudFront Distribution.

You should now be able to request a resource using the new CDN name:

$ curl -v -s "http://cdn.example.com/images/foo.png" -o /dev/null
* About to connect() to cdn.example.com port 80 (#0)
*   Trying 192.168.1.1... connected
* Connected to cdn.example.com (192.168.1.1) port 80 (#0)
> GET /images/foo.png HTTP/1.1
> Host: cdn.example.com
> Accept: */*
> User-Agent: curl
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 06 Jan 2011 18:51:35 GMT
< Server: Apache/2.2.3 (Red Hat)
< Content-Length: 1233
< Cache-Control: max-age=86400
< Content-Type: image/png
< Age: 13311
< X-Cache: Hit from cloudfront
< X-Amz-Cf-Id: b769b423c54e2ffb0a6fb60369e2e0f7b103251ef3e2c549084fb4abe4ef9a236052f8eec40b3a14,80416de274eb8ee87c21ee41c863f9f6f9ef1c251823d9c12c46ab13dc33759dcbb04175b7d4a5a7
< Via: 1.0 83eb7919a5076e946a3a2d59d7f4415b.cloudfront.net:11180 (CloudFront), 1.0 26fb80d2abd86d7f52358cd1c2efd787.cloudfront.net:11180 (CloudFront)
< Connection: close
< 
{ [data not shown]
* Closing connection #0 

Note that Amazon CloudFront doesn't support query strings on resources, so you might need to use some Apache mod_rewrite stuff.