Tuesday, September 27, 2011

Installing Graphite on OSX (Snow Leopard)

This wasn't entirely straightforward, so in the hope that it's useful for others:

python on Snow Leopard doesn't seem to come with the development headers, so we need to address that, since pycairo needs them.
$ brew install python
$ brew install cairo --use-gcc
$ wget http://cairographics.org/releases/py2cairo-1.10.0.tar.bz2
$ tar xjf py2cairo-1.10.0.tar.bz2
$ pushd py2cairo-1.10.0
$ emacs wscript
$ export CC=/usr/bin/gcc
$ export PKG_CONFIG_PATH=/usr/local/Cellar/cairo/1.10.2/lib/pkgconfig/
$ python waf configure
$ python waf build
$ python waf install

Now install python dependencies

$ pip install django
$ pip install django-tagging
$ pip install twisted
$ pushd path/to/graphite/
$ pushd whisper
$ python setup.py install
$ popd
$ pushd carbon
$ python setup.py install
$ popd
$ python check-dependencies.py
$ python setup.py install
$ pushd /opt/graphite/webapp
$ export PYTHON_PATH=${PYTHON_PATH}:/opt/graphite/webapp
$ pushd graphite
$ python manage.py syncdb

Grabbed this file and put it in /opt/graphite/bin. That means I don't need to setup apache httpd locally.

$ wget https://raw.github.com/tmm1/graphite/d0f76a659f4f2dea67f19902002710f601f534aa/bin/run-graphite-devel-server.py
$ python /opt/graphite/bin/carbon-cache.py start
$ python /opt/graphite/bin/run-graphite-devel-server.py /opt/graphite

Browse to http://localhost:8080/ and I have a graphite webapp running

$ python path/to/graphite/examples/example-client.py 

I now have a script putting data into graphite. Might want to tweak local_settings.py (make it Europe/London, for example), and conf/carbon.conf to have reasonable retention periods / file sizes for the whisper data files.

Related links:

Tuesday, May 31, 2011

Content Negotiation on Mobile considered harmful

This is kind of a follow up to my previous post on using Amazon CloudFront.

This post is to cover conneg, or Content Negotiation.

TL;DR - use HTTP as designed and follow the rules.

Alice tries to access http://example.com/ on her iPhone. She gets back some HTML which references some images.




The markup returned to Alice contains references to 3 images. I'll just look at the first one.



Alice's iPhone made a request for /images/1 and got back a 320x80px PNG image, since we have a clever server-side component which knows about different user-agents and tries to serve the most suitable version of an image for each client.

Along comes Bob. Bob is using a Google Nexus One. He similarly requests our home page and gets back a link to /images/1. When the Nexus One requests that resource though, it gets back a 420x120px PNG, again thanks to our fancy server-side detection.

What does this do to our caching? Well, it stuffs it up completely.
  1. We're using a canonical URI for the image - /images/1.
  2. We're serving different representations of the image from the same URL.
  3. We cannot easily specify good HTTP expiration directives.
Going into point 3 in more detail, we cannot use the Vary header in our response to try to let proxy caches more efficiently. Vary can only specify a request header. This means that something like the User-Agent doesn't work:
  1. There are many thousand User-Agent strings in existence.
  2. How different are these 2 anyway?
    • Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0_1 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko)
    • Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0_1 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Mobile/7A400
Let's be conservative and say that there are 5000 different User-Agents hitting our site. That means that we could be telling proxy servers to cache 5000 copies of /images/1, rather than the perhaps 7 different sizes that our application might produce.

So alternatives to conneg?
  • Use a distinct URI for every bag of bytes that the application can serve. This means that your server-side markup generation needs to be a little smarter, so that you render markup containing /images/1/320x80 and /images/1/480x120 for example (see the point in my CloudFront post about not wanting to use query string parameters for this information).
  • It's still possible to support the old canonical URLs, either by continuing to perform conneg, or redirecting appropriately.
  • Rev your URIs so that you can happily set Far Future Expires directives on these resources; i.e. if the image changes, give it a new URI.
I'm still musing on what this means for progressive enhancement. Andrea's post looks like a step in the right direction though.

Tuesday, February 08, 2011

Objective-C concurrency issues

Disclaimer - I've not shipped Java Swing / SWT apps. I'm a server guy where markup is the UI. Consequently, I don't have in-depth knowledge of Java to compare against. I'm aware of SwingUtilities.invokeLater(Runnable) but otherwise just assume I'm clueless about Swing.

First rule of GUI programming - don't block the main thread.
Second rule of GUI programming - don't block the main thread, etc.

Quantifying this, you have a device running at a refresh rate of 60Hz. So you if you do anything in the main thread, you need it to complete in under 16ms, or your UI will not be smooth and responsive.

In Java, I would normally look at Executors, Callables, Runnables and related APIs to do things off the main thread. In Objective-C, we have NSOperationQueue and NSOperation. Learn, use and love them. In particular, don't do what I did and start porting java.util.concurrent classes to Objective-C. I wrote a CountdownLatch, which was very nice and taught me about various low-level concurrency primitives. Unfortunately it was completely the wrong solution for the language. What I should have done was to use [NSOperation addDependency:] to chain tasks together.

Objective-C tooling

Java development tools are top of the pile out of anything I've used. The IDEs are massively powerful (they have to be, with the warts on the language). I'm also an emacs user day to day, and pragmatically use vim as well. But Eclipse / IDEA / Netbeans are pretty amazing tools for Java The Language development.

Respectively for Objective-C development, Xcode isn't.

If Apple Ts&Cs allow, IntelliJ could probably make some impressive inroads into that market.

clang is a good (and getting better all the time) addition. The debugger needs some love; I don't find gdb as powerful as Java debuggers.

In Java-land, one can use maven, ant, ivy, Make, etc to build a project. For iOS development, the IDE rules a lot from the off. There is a command-line tool which can potentially be driven by Jenkins or Thoughtworks Go. That would be my preferred option going forward; in my view, building in an IDE is not a repeatable build process.

Friday, January 07, 2011

Creating a Custom Origin Server for Amazon CloudFront

At the time of writing, tool support is limited to the REST API? The intention with this piece of work was to take content being served by our origin server; e.g. http://example.com/images/foo.png; and serve it via Amazon CloudFront on http://cdn.example.com/images/foo.png.

  1. Download cfcurl.pl.
  2. Get any dependencies from CPAN (the cfcurl.pl script tells you how to do that in case you aren't sure).
  3. Create $HOME/.aws-secrets and chmod 600.
    $ cat /Users/jabley/.aws-secrets
    %awsSecretAccessKeys = (
    # my personal account
    'james-personal' => {
    id => 'foo',
    key => 'bar',
    },

    # my corporate account
    'james-work' => {
    id => 'AWS-ID',
    key => 'AWS-Secret-Key',
    },
    );
  4. Create a file with the request data
    $ cat create-distribution.xml
    <?xml version="1.0" encoding="UTF-8"?>
    <DistributionConfig xmlns="http://cloudfront.amazonaws.com/doc/2010-11-01/">
    <CustomOrigin>
    <DNSName>example.com</DNSName>
    <OriginProtocolPolicy>http-only</OriginProtocolPolicy>
    </CustomOrigin>
    <CallerReference>20110106103700</CallerReference>
    <CNAME>cdn.example.com</CNAME>
    <Comment>example.com CloudFront CDN</Comment>
    <Enabled>true</Enabled>
    <Logging>
    <Bucket>accesslogs-example.com.s3.amazonaws.com</Bucket>
    <Prefix>cdn.example.com/</Prefix>
    </Logging>
    </DistributionConfig>
  5. POST the file
       perl cfcurl.pl --keyname james-work -- -X POST -H "Content-Type: text/xml;charset=utf-8" --upload-file \
    create-distribution.xml https://cloudfront.amazonaws.com/2010-11-01/distribution
  6. Poll to see when it has finished creating the distribution:
       perl cfcurl.pl --keyname james-work -- https://cloudfront.amazonaws.com/2008-06-30/distribution
  7. Configure DNS so that cdn.example.com is a CNAME for the DomainName value of your newly created CloudFront Distribution.

You should now be able to request a resource using the new CDN name:

$ curl -v -s "http://cdn.example.com/images/foo.png" -o /dev/null
* About to connect() to cdn.example.com port 80 (#0)
* Trying 192.168.1.1... connected
* Connected to cdn.example.com (192.168.1.1) port 80 (#0)
> GET /images/foo.png HTTP/1.1
> Host: cdn.example.com
> Accept: */*
> User-Agent: curl
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 06 Jan 2011 18:51:35 GMT
< Server: Apache/2.2.3 (Red Hat)
< Content-Length: 1233
< Cache-Control: max-age=86400
< Content-Type: image/png
< Age: 13311
< X-Cache: Hit from cloudfront
< X-Amz-Cf-Id: b769b423c54e2ffb0a6fb60369e2e0f7b103251ef3e2c549084fb4abe4ef9a236052f8eec40b3a14,80416de274eb8ee87c21ee41c863f9f6f9ef1c251823d9c12c46ab13dc33759dcbb04175b7d4a5a7
< Via: 1.0 83eb7919a5076e946a3a2d59d7f4415b.cloudfront.net:11180 (CloudFront), 1.0 26fb80d2abd86d7f52358cd1c2efd787.cloudfront.net:11180 (CloudFront)
< Connection: close
<
{ [data not shown]
* Closing connection #0

Note that Amazon CloudFront doesn't support query strings on resources, so you might need to use some Apache mod_rewrite stuff.

Wednesday, August 04, 2010

Apple AppStore feedback

This morning the AppStore on my iPod Touch pointed out that the Twitter app had an update. I installed it and launched the app to see what was new. Crash! After 5 crashes in a row without a successful launch, it was obvious that a bad build had snuck into the App Store.

Incident Identifier: 61EE3335-1AD0-4099-8EC6-FAB4B6160A43

CrashReporter Key: 001e29bd4aa2ef81d42701ce5325da94b364e27b

Process: Twitter [8470]

Path: /var/mobile/Applications/8E13E345-CDD3-4CA4-899D-8E38BA6661C5/Twitter.app/Twitter

Identifier: Twitter

Version: ??? (???)

Code Type: ARM (Native)

Parent Process: launchd [1]


Date/Time: 2010-08-04 07:58:35.994 +0100

OS Version: iPhone OS 3.1.3 (7E18)

Report Version: 104


Exception Type: EXC_BREAKPOINT (SIGTRAP)

Exception Codes: 0x00000001, 0xe7ffdefe

Crashed Thread: 0


Dyld Error Message:

Symbol not found: __NSConcreteGlobalBlock

Referenced from: /var/mobile/Applications/8E13E345-CDD3-4CA4-899D-8E38BA6661C5/Twitter.app/Twitter

Expected in: /usr/lib/libSystem.B.dylib

Dyld Version: 149


Binary Images:

0x1000 - 0x14ffff +Twitter armv6 <43ca857e309a61ba8c5da3ab83e42218> /var/mobile/Applications/8E13E345-CDD3-4CA4-899D-8E38BA6661C5/Twitter.app/Twitter

As an iPhone app developer, I think I know what this problem is. We saw this problem in one of our apps. IIRC, the new, preferred llvm compiler has a bug with the new blocks language construct, and gcc doesn't, and the bug only shows up at runtime, in certain environments. To fix it, Twitter are going to have to recompile and use gcc rather than llvm, and then wait for the wheels at Apple to turn.

Other people have talked about the frustration of not being able to iterate at web speed or do continuous deployment, but that's part of the ecosystem that you operate in with Apple.

Testing, either by the Twitter team, or by Apple when they review the app prior to approving it, should have caught this issue. But these things happen.

We had a similar thing happen with an update to one of our apps recently. An update went live and thanks to the apparent difficulty in doing your own testing of the binary that gets sent to Apple, an issue only became apparent when the new version was available through iTunes. To me, this is where the ecosystem is broken. If I have a webapp and I deploy an update, then find an issue (via my cluster-immune system - one day!), I roll it back.

iTunesConnect has no rollback, even though it seems like a highly desirable feature. I know in our case, we would have liked the option to rollback to the last known good version and then wait for Apple to review an update, rather than having the world upgrade to a version that we didn't want them to be running. I imagine Twitter would appreciate a similar feature right about now.

Thursday, July 08, 2010

Capturing Mobile Network Traffic On OS X

Recently had to audit an app to ensure that it wasn't leaking any unwanted details over the network. This was an iPhone app, but the same process can be used for Android, etc.
  1. Ensure Macbook Pro is plugged into Ethernet.
  2. Open System Preferences
  3. Internet & Wireless | Sharing (in Snow Leopard).
  4. Click Internet Sharing.
  5. From Ethernet
  6. On Airport
  7. Close System Preferences
  8. Click Airport
  9. Select Create Network...
  10. On the phone, open the WiFi controls and connect to the network that you've just created.
  11. Run Wireshark.
  12. Start capturing traffic on the wireless card.
  13. Check stuff is using SSL that should be, etc.

Thursday, May 20, 2010

mod_python in apache on OS X with Homebrew

Recently had to install mod_python to test something for a customer. It needed some nudging, so including it here. Snow Leopard, Homebrew and default httpd.

Wednesday, January 13, 2010

Objective-C - the language

First off, I read the Objective-C Primer and Objective-C Programming Language guides. I collect languages, so there was some underlying familiarity there. Ruby, Smalltalk and C obviously shone through for me. Second off, I re-read Smalltalk Best Practice Patterns. I first read that book maybe 6 years ago and it had a massive impact on my Java style. Objective-C is the most Smalltalk-like language that the 'masses' will actually use professionally. Sadly, it's not enough Smalltalk for me, and the C abstractions leak quite a bit.

Java Developers Guide to Objective-C on the iPhone

This will be a place-holder page containing links to the other entries that I create in this series. I've got a lot of commercial experience with Java, some Python and Ruby. This has all been server-side; I've not really touched GUIs (apart from GWT, HTML and Javascript) for a while, so this series will necessarily reflect that. Hopefully it will prove useful to others.

Topics that I hope to cover:

Tuesday, December 29, 2009

Objective-C for Java Developers

I'm coming from an Eclipse on Ubuntu background, but this is equally applicable for IDEA on Windows. What are the equivalents for iPhone development?

Java iPhone Notes
JUnit (unit testing framework) ? It is possible to use TDD for Swing apps, although I've been predominantly a server-side guy with client stuff happening in the browser for quite a while now. Cucumber with iPhone looks worth exploring...
Hudson (continuous integration tool) ? On my first iPhone app, it rapidly became apparent how easy it was for people to do bad merges and delete classes from the Xcode project file / strings from the UTF-16 l14n Localizable.strings file. You can argue that people should take more care; yeah, that'll fix it. git bisect is great, but a tool that builds on each commit is better.

Saturday, December 26, 2009

ScaleCamp - Queue PUBSUB

From some reason I went to this thinking PubSubHubbub, but it was more a refresher about making an app asynchronous, why you'd want to do that and how implementation complexity goes up as you go after certain desirable behaviours.

ScaleCamp - How do you scale Activity Feeds?

Popular session this - standing room only, so no notes from me.

The short answer is Redis, courtesy of Simon Willison. Since the consensus was that Redis would do the trick, we then touched on Simon's other new favourite technology - node.js.

Digression: Alex from mediamolecule made a comment about 100MB of data in a key-data structure store should only require 100MB of memory to store in such a server app (plus a little extra for housekeeping, but it shouldn't be a 1:10 ratio or similar. I didn't take that as a direct criticism of Redis but more of a reminder about choosing good data structures and the importance of CompSci (says this mathematician). I mention that, since it pricked me to investigate a suspected bad data structure in one of our apps, and coupled with the Eclipse Memory Analyser recommended by the Guardian guys, I found something that was using far too much of the heap for our Tomcat nodes; and had a change rolled out within 4 days of attending this conference. That reduced the memory footprint for that data structure from 250MB to 16MB. Ouch, shocker, but great to have found, prioritised and fixed.

Monday, December 07, 2009

ScaleCamp - Scaling Java and Oracle for the Guardian

Guardian.co.uk

Graham and various other people from the development and operations team pitching in.

3 years ago - published static files with apache SSI to fill-in gaps. Moved to a fully dynamic system. Now, they're somewhere in between.

Stack -
  • apache

  • resin

  • spring / hibernate / velocity

  • Oracle DB backend (not recommended!)


Measured the application - 1300 requests to DB just to render homepage.

Added ehcache to hibernate as 2nd level cache and added a warmup script before putting into load balancer

30m unique users per month
270m pages per month
250 requests/second at lunchtime
1500 requests/second peak.

GC tools



Google weakref cache (part of Google Collections)

Eclipse memory analyser - what's using all my memory?

Cacti for monitoring - DB usage was killing it.

8 app servers in each co-lo (London and Manchester).

400MB used by cache - churn meant was pretty ineffective.

Tried or considered ehcache distribution and jboss cache distribution.

Rejected since cache eviction via replication would have thrashed it.

memcached



massive improvement in response times, but DB load still high.

went to caching every query for 5 minutes. DB load vanished and is flat even as more app servers come on-line.

servlet filter writing to memcached made it stink fast.

took a days worth of logs and Hadoop to see how long the cache should be. 1 minute was the sweet spot.

Emergency switch to serve a static copy of the site, minus personalization features.

Daemon or script scrapes the sit; they can handle 700req/s/node when the site's operating in this mode.

new content published in this mode has a copy pressed so it can be served from disk - publish is slower than with the other system but updates still possible

Highly recommend that this sort of emergency degrade read-only mode should be built-in from the off - they've used this approach with the MPs Expenses apps built to crowd-source investigations.

ScaleCamp - Scaling Java with Shared Nothing

Thoughtworks guys again.

Basic Servlet overview - in-memory sessions don't scale, duh!

Preferred options - state goes into cookies and serialized. Security, legal aspects? Pretty well common to most frameworks these days.

Page composition in the server with proxy server holding StringTemplate objects. Interesting idea - seen variants of this in other places. I'm curious as to whether doing this could mean having a poor man's macro system for Java, since XSLTs can be written to create XSLTs; maybe Velocity templates could similarly generate Velocity templates or StringTemplate -> StringTemplate?

Again, application developers need to have a good idea of caching directives for this to work. One objection I had with this approach is that you potentially increase your hardware requirement and can open the app up to liveness failures here. Request A comes in and is serviced by Thread 1. As part of that, it makes a request to the proxy server for a template. At the proxy server, a cache miss means that another request needs to be made to the app server. Then Request A is tying up 2 app server threads. What about applications which parallelize the requests? They might use more than 2 app server request-handling threads at a time, etc.

Thoughtworks seem to do fun, interesting work.

ScaleCamp - LittleBigPlanet

Alex and James from mediamolecule.com. Fascinating perspective of embedded developers coming to server-programming and refusing to accept commonly held views on best practices for doing so. This was the surprise hit of the conference for me; I just elected to go since there wasn't anything else in that slot that I was really passionate about. I'd been talking to them both in the queue for tea earlier and made a poorly judged joke about Map-Reduce (we pretty much had this conversation). The session they ran was an awesome talk about scaling server-side within the games sector - Little Big Planet is theirs.

Written their own C-based key-data structure store, of which we're spoilt for choice just now. Alex commented that he's looked at Redis and it has some nice stuff, but when they came to need it, there wasn't anything that met their needs, and experience with running the recommended Java stack had left them with the impression that they should stick to what they know. What they know is writing very tight code in constrained environments, so applying that mind-set to server-side development seemed to have yielded some very pleasing numbers. Other parts are in Ruby (presumably 1.9, since they're using Fibers?). I didn't get around to asking James how well that works or which implementation they're using. Very happy with that programming model though - James is or was a Java guy - funny how nice Ruby feels coming from there!

ScaleCamp - Varnish

Artur Bergman talking about Varnish; this followed on from the Squid talk and most of the same crowd hung around for this one.

purges gone from multicast to Rabbit MQ (damn can't remember if I got that right or what it means!)
2 8-core servers in London data centre 350MB/s with 5000 requests/sec. Intel X25 SSDs have changed a certain class of application. If disk is the new tape, then it's probably still acceptable to go to disk if you're running those babies. See also Last.FMs experience with them.

Attempt cache hits in all data centres (UK -> US) before going to the app. Much better performance.

CDN gets broken with query string parameters - common misconfiguration which can be defended against.

Varnish protects against thundering herd. Interesting - need to read more about that to better understand it.

ScaleCamp - scaling with Squid

Summarising a recent Thoughtworks experience with this, from Chris Read and Sam Newman.

This was for a high volume retailer. A Proxy / caching solution was supposed to be provided and TW would do the app. At t-4, it turned out that TW would also have to provide the proxy / cache, so this was a hasty investigation into Squid.

1 hardware LB
2 Squid boxes - 8 core machine

1 carp process
lots of child processes

going to 16 app servers.

16,000 requests per second = 5% of traffic

TTL for items ranged from 5 minutes to 1 hour for stable furniture.

whole site does 250 million requests per day

peak 24,000 requests per second

importance of good HTTP Caching directives. Discussion of making the application (and by implication, the application developers) aware of considering ETag / Expires / Vary for all parts of the application, versus just adding it via apache ReWrite or similar. Most people in the room (including me, very strongly; RFC2616 is my favourite RFC) were in the former camp.

heap LFUDA was a good change to make.

Tried Varnish, but couldn't get good numbers out of it, in the timescales available. Artur opined that Varnish should provide better numbers than Squid; he's arguably conflicted, but seemed pretty convincing! Another factor in that was very likely that they were running RHEL old shit, and Varnish works best with a shiny new kernel (cite?)

Update: they also presented a version of this talk at DevOps 2010.

ScaleCamp - State of the Nation for Monitoring

Where are we now and what's broken with it?


People are almost getting to the point of needing more powerful machines to do the monitoring than the app servers! Maybe something's broken somewhere...


RRDTool - overall, the consensus seemed to be that this was a little dated.



  1. Does lots of writes due to the way it stores data

  2. throws away data by the way it aggregates - to see fine-grained data of last years sales, you need to keep a backup of the files / graphs, rather than being able to query it.

  3. can't be cleansed of bad data, or it's a bitch of a job to do so.


Alternative options



  • hbase

  • reconnoiter

  • Tokyo Tyrant / Cabinet

  • timesplicedb looks to be an interesting attempt to provide a replacement. More language bindings needed, don't be shy!

A good start to the conference for me and it gave me a flavour of the depth and breadth of discussions available.

ScaleCamp UK 2009

On Friday I was fortunate enough to attend the inaugural ScaleCamp UK event, organised by Michael Brunton-Spall at the Guardian. This was a great conference. It was a BarCamp-style approach (not that I've been to BarCamp yet!) with the schedule evolving over conversations and planned on a board in the morning. Some of the sessions I took notes at; others were standing room only, so I'll try to remember what was talked about. Obviously, this is a personal perspective focused on my interests; others should be blogging about Javascript and the like.

I met lots of very passionate, smart people doing cool stuff. That bodes well for the economy; if you want to do interesting work, then hooking up with any of the people that attended there wouldn't be a bad place to start.