Content Negotiation on Mobile considered harmful

This is kind of a follow up to my previous post on using Amazon CloudFront.

This post is to cover conneg, or Content Negotiation.

TL;DR - use HTTP as designed and follow the rules.

Alice tries to access http://example.com/ on her iPhone. She gets back some HTML which references some images.




The markup returned to Alice contains references to 3 images. I'll just look at the first one.



Alice's iPhone made a request for /images/1 and got back a 320x80px PNG image, since we have a clever server-side component which knows about different user-agents and tries to serve the most suitable version of an image for each client.

Along comes Bob. Bob is using a Google Nexus One. He similarly requests our home page and gets back a link to /images/1. When the Nexus One requests that resource though, it gets back a 420x120px PNG, again thanks to our fancy server-side detection.

What does this do to our caching? Well, it stuffs it up completely.

1. We're using a canonical URI for the image - /images/1.
2. We're serving different representations of the image from the same URL.
3. We cannot easily specify good HTTP expiration directives.

Going into point 3 in more detail, we cannot use the Vary header in our response to try to let proxy caches more efficiently. Vary can only specify a request header. This means that something like the User-Agent doesn't work:

1. There are many thousand User-Agent strings in existence.
2. How different are these 2 anyway?
   • Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0_1 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko)
   • Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0_1 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Mobile/7A400

Let's be conservative and say that there are 5000 different User-Agents hitting our site. That means that we could be telling proxy servers to cache 5000 copies of /images/1, rather than the perhaps 7 different sizes that our application might produce.

So alternatives to conneg?

• Use a distinct URI for every bag of bytes that the application can serve. This means that your server-side markup generation needs to be a little smarter, so that you render markup containing /images/1/320x80 and /images/1/480x120 for example (see the point in my CloudFront post about not wanting to use query string parameters for this information).
  
• It's still possible to support the old canonical URLs, either by continuing to perform conneg, or redirecting appropriately.
  
• Rev your URIs so that you can happily set Far Future Expires directives on these resources; i.e. if the image changes, give it a new URI.

I'm still musing on what this means for progressive enhancement. Andrea's post looks like a step in the right direction though.

Novarra makes a play for End Game

So there's been a bit of a kerfuffle over here. What's happened is that Vodafone UK have started using Novarra's content adaptation technology to transcode the internet for mobile phones. Why has this caused such irate responses in some quarters? Well, a history lesson seems to be in order...

The mobile internet started out with devices unable to render nearly all of the internet due to hardware constraints. WML was introduced as a very limited markup which phones were able to use to display content. Early WML sites were, needless to say, fairly limited in functionality, and tended to be exposed as distinct resources; e.g. Apache user-agent detection would be used to send you to http://wap.example.com/ if you accessed http://www.example.com/ with your phone. Moore's law still holds in this arena; devices got a bit more capable and iMode (CHTML) was introduced in Japan. This was initially something like HTML 2.0 / 3.2 without <table/>s, but with the better networks over there, this was somewhat more successful in the marketplace. The W3C stuck in their thumb and pulled out XHTML Basic, and then there was the related XHTML Mobile Profile.

So who is so offended by what Vodafone UK and Novarra have done? Well, mainly the mobile internet community. The solutions that the community provide have evolved over time from hand-coded WML sites, iMode sites, XHTML Basic sites, sites where the view renders the model to the appropriate markup and DIAL processors / Drutt / Volantis's offerings in this market. There is an entire industry sector dedicated to providing solutions in this area and nearly all of those solutions rely on the HTTP header User-Agent being present and being a reliable indicator of the requesting client. Novarra / Vodafone UK have introduced a fairly disruptive change which could be viewed as an attempt to change the rules of the game. They're making some pretty provacative statements along the way.

2.3.1.7 A content transformation server can do a better job of following
mobile best practices
The "Mobile Web Best Practices 1.0" W3C Proposed Recommendation [1]
contains many recommendations for authoring content that is intended for
viewing on a mobile device. A well-designed content transformation
server can do a better job of following the mobile best practices than a
human author, especially when taking into account the capabilities of
the many different mobile devices. The result will be a more
consistent, uniform experience.

I call BS. The incumbent mobile content industry is feeling the pain, but this could just be a game-changing move like Google upp-ing the storage limits on webmail. Time will tell whether the market (mobile phone customers) feels that the Vodafone UK solution is good enough, or whether a more open market will be preferred. I'm all for the Ubiquitous Web when it's good for the customer. So here's a little gedank-experiment. What happens to the mobile content industry if all carriers start using a content-adaptation proxy? How else is your company adding value? Evolving markets are hardly a new phenomenon, although maybe the rate of change is a little faster in these modern times...