tag:blogger.com,1999:blog-134732432024-02-21T14:11:57.422+00:00James' PadMusings on my family, work and things I find interesting. Mainly, this was / is intended to record things for my kids so that they can get an insight into how I feel about them, plus I haven't done any writing for a long time, so would like to get some skill in that sphere again.James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.comBlogger168125tag:blogger.com,1999:blog-13473243.post-69116363004011432682014-11-21T16:23:00.001+00:002014-11-21T20:58:06.935+00:00VelocityConf EU 2014 – Day 1TLDR; Velocity is a great conference for web and operations people, and why didn't you go already?<br />
<h3>
Is TLS Fast Yet?</h3>
<div>
This was a talk by <a href="https://twitter.com/igrigorik">Ilya Grigorik</a> full of practical, actionable things that you can do to serve your site over TLS, and make it fast.<br />
<br /></div>
<div>
</div>
<div>
<a href="https://flic.kr/p/q6qf3z">My notes on Ilya's talk</a>.</div>
<h3>
Monitoring: The Math Behind Bad Behavior</h3>
<div>
<a href="https://twitter.com/postwait">Theo Schlossnagle</a> gave an excellent talk (which didn't involve much maths) about the problems that <a href="https://www.circonus.com/">Circonus</a> see with handling massive amounts of data, and reliably detecting anomalies. I found this quite hard to take notes, and it wasn't as practical in my context as the first talk, but still really interesting.</div>
<div>
<br /></div>
<div>
<a href="https://flic.kr/p/q6yfZ5">My notes on Theo's talk</a>.</div>
<h3>
Design Reviews for Operations</h3>
<div>
<a href="https://twitter.com/lnxchk">Mandi Walls</a> of Chef showed us how operations should be involved early on. She did a great job of emphasising the importance of having the right people having the right conversations at the right time.</div>
<div>
<br /></div>
<div>
I felt a little over-qualified for this talk, given that I've worked with <a href="https://twitter.com/garethr">Gareth Rushgrove</a> for most of the last 3 years, and helped write some of the <a href="https://www.gov.uk/service-manual/operations/web-operations-stories">user stories for operations</a> that GDS published on GOV.UK. Not everyone has had that privilege though!</div>
<div>
<br /></div>
<div>
<a href="https://flic.kr/p/q4jUJ7">My notes on Mandi's talk</a>.</div>
<h3>
What Ops Can Learn From Design</h3>
<div>
<a href="https://twitter.com/robtreat2">Rob Treat</a> of <a href="http://www.omniti.com/">Omniti</a> brought together <a href="https://en.wikipedia.org/wiki/The_Design_of_Everyday_Things">The Design of Everyday Things</a> and <a href="http://www.catb.org/esr/writings/taoup/">The Art of UNIX Programming</a> to show how designing with empathy to create intuitive interfaces can be easy to overlook, but can have a massive impact on people using your stuff.</div>
<div>
<br /></div>
<div>
<a href="https://flic.kr/p/q6qhwn">My notes on Rob's talk</a>.</div>
<h3>
Statistical Learning-based Automatic Anomaly Detection @Twitter</h3>
<div>
Anomaly Detection seemed to be quite popular this year (see Theo's talk and Baron's proposed talk). Here, <a href="https://twitter.com/arun_kejariwal">Arun Kejariwal</a> talked about the state of the art, how it didn't quite fit for Twitter's usage, and what they did about it. The tools and code should be open-sourced in a few weeks, so people can plug it into their own problems.</div>
<div>
<br /></div>
<div>
<a href="https://flic.kr/p/q4jVLs">My notes on Arun's talk</a>.</div>
<h3>
Your Place or Mine: A Discussion of Where to Host Your Site</h3>
<div>
This was an emergency panel convened since the originally planned speaker had something come up. Nice end to the day, talking about cloud and similar issues. <a href="https://twitter.com/bruntonspall">Michael</a> did a nice job of not answering someone that seemed to be either aggrieved, or trolling quite hard. He's a proper civil servant.</div>
<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-90465005564777944442014-04-13T12:26:00.000+01:002014-04-13T12:26:05.704+01:00Things GDS doesn't tell you<span style="font-family: Arial; font-size: 13px;">Working at the <a href="https://gds.blog.gov.uk/about/">Government Digital Service (GDS)</a> changes a person. It’s not a thing we highlight; just acknowledge internally in furtive conversations. This post should expose a few truths about that.</span><br />
<br />
<h3>
<span style="font-family: Arial; font-size: 13px;">Pedantry</span></h3>
<span style="font-family: Arial; font-size: 13px;"><a href="https://www.gov.uk/design-principles/style-guide">Words are really important</a>. A common effect of working at GDS is that large parts of the internet become unusable for you, since the writing is so poor. Caring about serial commas is the norm.</span><div>
<span style="font-family: Arial; font-size: x-small;"><br /></span>
<h3>
<span style="font-family: Arial; font-size: 13px;">New ideas</span></h3>
<span style="font-family: Arial; font-size: 13px;"><a href="https://gds.blog.gov.uk/2014/03/14/what-we-mean-when-we-talk-about-content-design/">Content design</a>. <a href="https://www.gov.uk/service-manual/user-centered-design/user-research">User research</a>. These are all things that were new to me, and it turns out they have a <a href="http://www.bbc.co.uk/news/entertainment-arts-22164715">massive impact</a> in creating<a href="https://gds.blog.gov.uk/2013/07/02/pencils/"> award-winning web sites</a>. Another portion of the internet becomes blacklisted since it fails to meet your minimum standards for user experience.</span></div>
<div>
<span style="font-family: Arial; font-size: x-small;"><br /></span>
<h3>
<span style="font-family: Arial; font-size: 13px;">Intolerance</span></h3>
<span style="font-family: Arial; font-size: 13px;">Working with amazing people every day has a horrible effect on an individual. Working with less talented people becomes very unattractive. This is a deliberate retention policy strategy, and seems to work very well.</span></div>
<div>
<span style="font-family: Arial; font-size: x-small;"><br /></span>
<h3>
<span style="font-family: Arial; font-size: 13px;">Elitism</span></h3>
<span style="font-family: Arial; font-size: 13px;">Presenting well is a learned skill. Once you’ve learned it <a href="http://russelldavies.typepad.com/home/">from one of the best there is</a>, you start to notice things. Bad things. Powerpoint things. You cannot unsee these things.</span></div>
<div>
<span style="font-family: Arial; font-size: 13px;"></span><br />
<span style="font-family: Arial; font-size: 13px;">If these side-effects repulse you, make sure you don’t <a href="https://gds.blog.gov.uk/jobs/">apply to work here</a>.</span><br /></div>
<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-26858864565747643122012-09-27T09:27:00.002+01:002012-09-27T09:27:36.003+01:00How to check the encryption used in a zip fileSadly, I've not found a nice CLI way of doing this, but I recently had to validate that a 3rd party was transferring files to us in an approved way (AES 256-bit) and this is what I did:<br />
<ol>
<li>Open the zip file in emacs.</li>
<li>Use fundamental-mode to stop showing a listing of the zip contents. (<span style="font-family: Courier New, Courier, monospace;">M-x fundamental-mode</span>)</li>
<li>Use hexl-mode to get a binary view of the file. (<span style="font-family: Courier New, Courier, monospace;">M-x hexl-mode</span>)</li>
<li>Search for the string "0199 0700" to find the <a href="http://www.winzip.com/win/en/aes_info.htm#extra-data">AES Extra header field</a>. (<span style="font-family: Courier New, Courier, monospace;">C-S 0199 0700</span>)</li>
<li>Check that 2 bytes after the 0700 (skip the 2 vendor bytes; 0200 below) is 4145 (the characters AE) followed by 01, 02 or 03 representing the AES encryption strength. In our case, we wanted 03, or AES-256.</li>
</ol>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsn3qkSAgWBdTMq5nU6fTxMyYmM-Bfrxkx92JoGgt-Znss1UUY-pO5skRL1yvPz6q-JqbiRkzFTAkTsisXExZgP9PxBTWHnu66y7ZHy0Nmj6jrAtSdZzmLIAFEnb1XMqXI3Evm/s1600/Screen+Shot+2012-09-27+at+09.24.36.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="12" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsn3qkSAgWBdTMq5nU6fTxMyYmM-Bfrxkx92JoGgt-Znss1UUY-pO5skRL1yvPz6q-JqbiRkzFTAkTsisXExZgP9PxBTWHnu66y7ZHy0Nmj6jrAtSdZzmLIAFEnb1XMqXI3Evm/s320/Screen+Shot+2012-09-27+at+09.24.36.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com1tag:blogger.com,1999:blog-13473243.post-35884678968577689592012-09-13T13:21:00.001+01:002013-01-21T09:24:45.442+00:00How to convert an Oracle .dmp into a more portable formatOne of the things that I've come across has been legacy applications which use Oracle; where we don't have access to the database, but do get provided with Oracle .dmp files. These aren't very helpful when all you have (or are used to!) is MySQL and PostgreSQL.<br />
One approach that I've had success with is to <a href="http://www.oracle.com/technetwork/community/developer-vm/index.html">download a VirtualBox image of Oracle</a>, and then play with the data in there. I chose the <a href="http://www.oracle.com/technetwork/database/enterprise-edition/databaseappdev-vm-161299.html">Database App Development VM</a> since I wasn't sure what parts of Oracle I'd need, having strenuously, pretty successfully, avoided Oracle for most of my time in the industry.<br />
I then imported this into VirtualBox (running on OSX Lion) and configured networking:
<br />
<ol>
<li>One adapter running NAT so that I can browse the internet in the guest OS. This let me download the .dmp file to the guest OS filesystem.</li>
<li>Set up Port Forwarding for the NAT interface so that I can ssh to port 2022 on the host which will go to port 22 on the guest, thus allowing ssh access.</li>
<li>Optionally, I set up the VM to have another adapter (Host-only) so that I can set up NFS shares to mount part of the host filesystem under the guest.</li>
</ol>
<br />
Next, I needed to get the .dmp data onto the guest OS (and later get the transformed data off the guest). <code>ssh-copy-id</code> is good for this, to put an SSH public key into the <code>authorized_keys</code> for the oracle user on the guest. You can also get data into and out of the guest using <code>python -m SimpleHTTPServer</code> ran in the appropriate directory, which let me browse the host filesystem or guest filesystem as needed. <code>ifconfig</code> in the host or guest lets me know which IP address to use.<br />
Now, I needed to create a tablespace and user to allow me to import the data. I advise doing this, since (for me at least!) importing the data is an iterative process, and creating a separate tablespace (with separate data files) is a good practice since it avoids bloating the system tablespace and means that disk space can be reclaimed. Pretty much the only Oracle knowledge I have! Before you create the tablespace, it's a good idea to check the size of your .dmp and available space on the filesystem. I had a 1.4GB .dmp which didn't fit into the space left on the fs and I burnt a bit of time figuring out Oracle error messages for the failed import before I worked out the filesystem wasn't big enough. In this case, I created a symlink in <code>$ORACLE_HOME/dbs/</code> which pointed to a large enough partition and set the owner / permissions as required. Creating the tablespace was just a case of running:
<br />
<pre>$ sqlplus / as sysdba
...
SQL> CREATE BIGFILE TABLESPACE mytablespace DATAFILE 'mytablespace/f1.dat' SIZE 20M AUTOEXTEND ON;
Tablespace created.
SQL> CREATE USER myuser IDENTIFIED BY password DEFAULT TABLESPACE mytablespace;
User created.
SQL> GRANT CREATE SESSION,CREATE SYNONYM,CONNECT,RESOURCE,CREATE VIEW,IMP_FULL_DATABASE to myuser;
Grant succeeded.
SQL> exit
</pre>
We should now be in a position to try to import the data.<br />
<pre>$ time imp myuser/password file=path/to/data.dmp full=yes</pre>
If this fails since the user that it was exported as is not the same as the user you created, then stop the import and clear out the user and tablespace.<br />
<pre>$ sqlplus / as sysdba
SQL> DROP USER myuser CASCADE;
User dropped.
SQL> DROP TABLESPACE mytablespace INCLUDING CONTENTS AND DATAFILES;
</pre>
Then re-create the tablespace and the new user and try the import again.<br />
Once the import has succeeded, you want to get the data out of the database into a less proprietary format. One way is to use SQL Developer (a GUI tool included in the VM image).<br />
Open SQL Developer and define a new database connection:<br />
<dl>
<dt>Connection name</dt>
<dd>myuser</dd>
<dt>User name</dt>
<dd>myuser</dd>
<dt>Password</dt>
<dd>password</dd>
<dt>Save Password?</dt>
<dd>Checked</dd>
</dl>
SID is orcl rather than xe, in the Developer Days VM that I used.<br />
Test the connection. It should work. Then open the connection and examine the tables.<br />
<ol>
<li>In the menu, click Tools | Database Export</li>
<li>Want to export the data only, into CSV.</li>
<li>Choose the connection, choose the tables, choose the destination file.</li>
</ol>
For large databases, this can take a while to process (2.5 hours for my case). It may be faster to write your own export routine using Perl, PL/SQL or similar. In the end, that's what I did, so that I could script the entire process like so:<br />
<script src="https://gist.github.com/3713603.js?file=convert2csv.sh"></script>
<script src="https://gist.github.com/3713603.js?file=create-db.sql"></script>
<script src="https://gist.github.com/3713603.js?file=dump2csv.sql"></script>
<script src="https://gist.github.com/3713603.js?file=export.sql"></script>
<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com3tag:blogger.com,1999:blog-13473243.post-66370366933286604722012-05-08T06:35:00.000+01:002012-07-10T06:38:50.345+01:00Joining GDSBasically echoing what <a href="http://www.morethanseven.net/2011/08/19/On-her-majestys-digital-service.html">others</a> <a href="http://blog.geekmanager.co.uk/2012/04/16/joining-the-revolution/">have</a> said. I've held off working in London forever, not wanting to spend a large portion of my day on a train. But then <a href="http://digital.cabinetoffice.gov.uk/about/">something like this</a> comes along, with an opportunity to transform how Government delivers services (<a href="http://radar.oreilly.com/2009/01/work-on-stuff-that-matters-fir.html">work on stuff that matters</a>), and working with a ridiculously talented set of people. Chances like that don't come along often. It's going to be an exciting ride, and I'm grateful to my wife and kids for letting me get on.<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-38135350214648725502012-03-03T21:39:00.005+00:002012-03-03T23:11:56.223+00:00My reaction to Raganwald's "How to do what you love"You should <a href="http://leanpub.com/dowhatyoulove">buy this book</a>. I know (all?) the content is available online already and if you've been reading <a href="http://weblog.raganwald.com/">raganwald's</a> <a href="https://github.com/raganwald/homoiconic/blob/master/homoiconic.markdown">output over the years</a>, you might have already read the articles collated in this slim volume. I still suggest you should buy the book; my only nitpick was that (at the time I purchased it) the maximum payable price seemed lower than what I would have paid.<br /><br />I guess for me there were 2 reasons to buy it. One is partly a reflection on my evolving personal philosophy, that people who create great stuff should be somehow rewarded, so that they can carry on creating great stuff. In Renaissance times, this would be patronage. These days, tip jars or similar can be simple, low-friction ways of allowing a much larger potential audience to support an artist. Also, I prefer to buy stuff that is free, because I am fortunate enough to be in a position to do that, and to try to ensure that the supply of free stuff doesn't dry up.<br /><br />The second reason is that I am grateful to Reg for providing me with so many hours of stimulating thought.<br /><br />I don't believe I had previously read all of the compositions, and 3 things struck me upon reading this book. <br /><br />First, I don't have a publicly viewable portfolio demonstrating that I am in any way a competent professional. There are the <a href="https://rails.lighthouseapp.com/projects/8994/tickets/6069">odd</a> <a href="https://github.com/jruby/jruby/commit/200b4e0">normal</a> <a href="https://github.com/apache/httpclient/commit/d384ec4e3a5daf624a64760955c72251140efc4e">bunch</a> of <a href="https://github.com/jabley/homebrew/commit/7aa41cd1ad2a929be2d175cbf051ca6f497f5a59">patches</a> littered in various libraries that I use or have used, and one former employer <a href="http://gateway.volantis.com/docs/archives/vms/index.html">released a large chunk of their code as open source</a> (but with all identification / attribution removed) but there is nothing meaty that is mine (<a href="https://masterbranch.com/jabley">apparently</a>, apart from <a href="http://sourceforge.net/projects/vbunitfree/">vbunitfree</a>, which is very dead). I have in the past railed against walled gardens in terms of mobile carriers and their view of the web; in this case I have been working with other walled gardens, in terms of writing code that is proprietary, for corporate entities. <a href="https://github.com/jabley">My github account</a> needs some TLC to showcase my skills.<br /><br />Second, in recent years I have neglected communication and other soft skills, choosing instead to focus on technical skills for quite some time. That is a mistake. As I've got older, I've come to think that communication is more important; it's all about the conversations you have with people. Reg certainly seems to share that viewpoint. This blog was initially created since all of my blogging output was going onto an internal, employer-owned blog and I wanted to develop those skills further (and stop putting all of the good stuff in a walled garden!). I need to dedicate some time to this.<br /><br />Finally, NDAs are evil. In that instance, not only is your professional output (in terms of code at least) locked up in a walled garden so that no-one can view it, but neither can you even talk about it. I agonised for a long time about the last NDA that I signed. No more. If you need me to sign an NDA, I suggest that perhaps you need to examine why you are asking me to do that. Surely you should have confidence in your ability to execute on a plan, and the speed at which you will iterate?<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-76304184327810446962012-02-24T16:24:00.007+00:002012-02-24T16:52:14.072+00:00Merging Subversion trunk into a branch; how to deal with merge conflictsTLDR meh.<br /><br />I'd inherited a 4 month old branch which needed to be merged back into trunk at some point. As a first step, I wanted to merge the (hopefully smaller) changeset from trunk back into the branch. I tried git-svn. It didn't work for me. This has not been a pretty task.<br /><pre><br />$ pushd path/to/svn/repo<br />$ svn sw https://example.com/svn/project/branches/my-branch<br />$ svn up<br />$ svn merge https://example.com/svn/project/trunk --accept postpone<br />...<br />svn: One or more conflicts were produced while merging r3097:4432 into<br />'.' --<br />resolve all conflicts and rerun the merge to apply the remaining<br />unmerged revisions<br /></pre><br />At this point I have my working copy in a partially merged state with various file-level and tree/directory level conflicts. As an example of how a repository might get into this state, imagine this happening in the branch<br /><pre><br />$ svn mv dir1 dir2<br />$ mkdir dir1<br />...<br /># add files to dir1<br /># and commit a few times.<br /></pre><br />Meanwhile in trunk<br /><pre><br />...<br /># add and modify files in dir1<br /># commit a few times.<br /></pre><br />Since the changes hadn't been cherrypicked, you get tree conflicts. Let's take a look at those conflicts.<br /><pre><br />$ svn stat | grep 'C '<br /></pre><br />I had 46 issues listed for the merge up to this point. File level conflicts can be easily resolved using <a href="//eternusuk.blogspot.com/2011/12/svn-merging-on-osx.html">fmresolve which I've written about previously</a>.<br /><pre><br />$ fmresolve path/to/conflicted/file<br /></pre><br />and then<br /><pre><br />$ svn resolve --accept working<br /></pre><br />or<br /><pre><br />$ svn resolve --accept theirs-full<br /></pre><br />or<br /><pre><br />$ svn resolve --accept mine-full<br /></pre><br />Tree conflicts can only be resolved using the working copy, so I needed to checkout / copy the relevant file and edit until I was happy with each one, and then mark each conflict as resolved, accepting the working copy. 21 of these needed attention at this stage.<br />Then you can proceed with the merge.<br /><pre><br />$ svn merge https://example.com/svn/project/trunk --accept postpone<br /></pre><br />Repeat until done.<br />Hopefully merging the branch back into trunk will go a little easier.<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-24245314014575415922011-12-16T13:02:00.008+00:002011-12-16T14:20:12.339+00:00svn merging on OSXI don't always use <code>svn</code> as a version control system with which I'll need to merge branches, but when I do, I use <a href=""http://www.defraine.net/~brunod/fmdiff/">fmdiff</a>.<br /><br /><code>$ brew install fmdiff</code><br /><br />One minor annoyance - <code>fmmerge</code> (used for interactive conflict resolution) doesn't work. The number of arguments passed to the script has changed since it was first written. I patched it locally, but it still didn't work. FileMerge was launched, I could edit files, etc; but it kept saying that the merge needed resolving. Instead, I just postpone all merge conflicts during the merge, and then use <code>fmresolve</code> and <code>svn resolve</code> to resolve any individual merge conflicts.<br /><br />[1] I like to branch by feature typically, but occasionally, branch by VCS is used.<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-52196428429441921892011-10-04T18:31:00.003+01:002011-10-04T18:35:53.595+01:00Minifying Javascript at runtime<a href="http://www.stevesouders.com/">Steve Souders</a> <a href="http://twitter.com/souders/status/121257539936591874">pointed</a> at <a href="http://www.aliaspooryorik.com/blog/index.cfm/e/posts.details/post/340">this</a> today; I've done something similar in the past, but I struggled somewhat with the documentation. Hopefully this might be useful to others.<br /><br /><script src="https://gist.github.com/1262250.js?file=ClosureMinifier.java"></script><br /><br />As part of a product that serves as a rendering runtime for mobile, this allows authors to create Javascript, and the runtime can optimise and cache on the fly. We like it!<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-13270810962427777382011-09-27T11:20:00.002+01:002011-09-27T11:27:27.267+01:00Installing Graphite on OSX (Snow Leopard)This wasn't entirely straightforward, so in the hope that it's useful for others:<br /><br />python on Snow Leopard doesn't seem to come with the development headers, so we need to address that, since pycairo needs them.<br /><pre>$ brew install python</pre><ul><li>edit the PATH to have /usr/local/share/python at the start</li><li>open a new shell to recognise the new PATH</li><li>install cairo as per <a href="http://stackoverflow.com/questions/6886578/how-to-install-pycairo-1-10-on-mac-osx-with-default-python">http://stackoverflow.com/questions/6886578/how-to-install-pycairo-1-10-on-mac-osx-with-default-python</a></li><li>I do --use-gcc since it doesn't work with LLVM / LLVM-based GCC.</li></ul><div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ brew install cairo --use-gcc<br />$ wget http://cairographics.org/releases/py2cairo-1.10.0.tar.bz2<br />$ tar xjf py2cairo-1.10.0.tar.bz2<br />$ pushd py2cairo-1.10.0<br />$ emacs wscript<br />$ export CC=/usr/bin/gcc<br />$ export PKG_CONFIG_PATH=/usr/local/Cellar/cairo/1.10.2/lib/pkgconfig/<br />$ python waf configure<br />$ python waf build<br />$ python waf install </pre> </div></div> <p>Now install python dependencies</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ pip install django<br />$ pip install django-tagging<br />$ pip install twisted<br />$ pushd path/to/graphite/<br />$ pushd whisper<br />$ python setup.py install<br />$ popd<br />$ pushd carbon<br />$ python setup.py install<br />$ popd <br />$ python check-dependencies.py<br />$ python setup.py install<br />$ pushd /opt/graphite/webapp<br />$ export PYTHON_PATH=${PYTHON_PATH}:/opt/graphite/webapp<br />$ pushd graphite<br />$ python manage.py syncdb </pre> </div></div> <p>Grabbed this file and put it in /opt/graphite/bin. That means I don't need to setup apache httpd locally.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ wget https://raw.github.com/tmm1/graphite/d0f76a659f4f2dea67f19902002710f601f534aa/bin/run-graphite-devel-server.py<br />$ python /opt/graphite/bin/carbon-cache.py start<br />$ python /opt/graphite/bin/run-graphite-devel-server.py /opt/graphite </pre> </div></div> <p>Browse to <a href="http://localhost:8080/">http://localhost:8080/</a> and I have a graphite webapp running</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ python path/to/graphite/examples/example-client.py </pre> </div></div> <p>I now have a script putting data into graphite. Might want to tweak local_settings.py (make it Europe/London, for example), and conf/carbon.conf to have reasonable retention periods / file sizes for the whisper data files.</p><p>Related links:</p><ul><li><a href="http://stackoverflow.com/questions/7138076/installing-graphite-on-os-x-lion-how-to-configure-apache2">http://stackoverflow.com/questions/7138076/installing-graphite-on-os-x-lion-how-to-configure-apache2</a></li><li><a href="http://stackoverflow.com/questions/1811783/how-do-you-install-pycairo-on-mac-os-x">http://stackoverflow.com/questions/1811783/how-do-you-install-pycairo-on-mac-os-x</a></li><li><a href="http://stackoverflow.com/questions/6886578/how-to-install-pycairo-1-10-on-mac-osx-with-default-python">http://stackoverflow.com/questions/6886578/how-to-install-pycairo-1-10-on-mac-osx-with-default-python</a><br /></li></ul><div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com4tag:blogger.com,1999:blog-13473243.post-74295962233724066332011-05-31T23:17:00.000+01:002011-05-31T23:21:15.623+01:00Content Negotiation on Mobile considered harmfulThis is kind of a follow up to <a href="http://www.blogger.com/2011/01/creating-custom-origin-server-for.html">my previous post on using Amazon CloudFront</a>.<br /><br />This post is to cover conneg, or <a href="http://en.wikipedia.org/wiki/Content_negotiation">Content Negotiation</a>.<br /><br />TL;DR - use HTTP as designed and <a href="http://developer.yahoo.com/performance/rules.html">follow the rules</a>.<br /><br />Alice tries to access http://example.com/ on her iPhone. She gets back some HTML which references some images.<br /><br /><br /><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 222px; height: 211px;" src="http://www.websequencediagrams.com/cgi-bin/cdraw?lz=cGFydGljaXBhbnQgQWxpY2UgYXMgQQoACwxTZXJ2ZXIgYXMgUwoKQS0-UzogR0VUIC8KUy0-QTogaHRtbCByZXNwb25zZQoK&s=napkin" alt="" border="0" /><br /><br />The markup returned to Alice contains references to 3 images. I'll just look at the first one.<br /><br /><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 222px; height: 211px;" src="http://www.websequencediagrams.com/cgi-bin/cdraw?lz=cGFydGljaXBhbnQgQWxpY2UgYXMgQQoACwxTZXJ2ZXIgYXMgUwoKQS0-UzogR0VUIC9pbWFnZXMvMQpTLT5BOiAzMjB4ODAgUE5HIAAVBQoK&s=napkin" /><br /><br />Alice's iPhone made a request for /images/1 and got back a 320x80px PNG image, since we have a clever server-side component which knows about different user-agents and tries to serve the most suitable version of an image for each client.<br /><br />Along comes Bob. Bob is using a Google Nexus One. He similarly requests our home page and gets back a link to /images/1. When the Nexus One requests that resource though, it gets back a 420x120px PNG, again thanks to our fancy server-side detection.<br /><br />What does this do to our <a href="http://tomayko.com/writings/things-caches-do">caching</a>? Well, it stuffs it up completely.<br /><ol><li>We're using a canonical URI for the image - /images/1.</li><li>We're serving different representations of the image from the same URL.</li><li>We cannot easily specify <a href="http://developer.yahoo.com/performance/rules.html#expires">good HTTP</a> <a href="http://www.mnot.net/cache_docs/#CACHE-CONTROL">expiration directives</a>.</li></ol>Going into point 3 in more detail, we cannot use the Vary header in our response to try to let proxy caches more efficiently. Vary can only specify a request header. This means that something like the User-Agent doesn't work:<br /><ol><li>There are <a href="http://deviceatlas.com/devices">many thousand</a> User-Agent strings in existence.</li><li>How different are these 2 anyway?<ul><li>Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0_1 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko)</li><li>Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0_1 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Mobile/7A400</li></ul></li></ol>Let's be conservative and say that there are 5000 different User-Agents hitting our site. That means that we could be telling proxy servers to cache 5000 copies of /images/1, rather than the perhaps 7 different sizes that our application might produce.<br /><br />So alternatives to conneg?<br /><ul><li>Use a distinct URI for every bag of bytes that the application can serve. This means that your server-side markup generation needs to be a little smarter, so that you render markup containing /images/1/320x80 and /images/1/480x120 for example (see the point in my CloudFront post about not wanting to use query string parameters for this information).<br /></li><li>It's still possible to support the old canonical URLs, either by continuing to perform conneg, or redirecting appropriately.<br /></li><li>Rev your URIs so that you can happily set Far Future Expires directives on these resources; i.e. if the image changes, give it a new URI.</li></ul>I'm still musing on what this means for progressive enhancement. <a href="http://blog.trasatti.it/2011/05/responsive-images-and-tinysrc.html">Andrea's post</a> looks like a step in the right direction though.<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-90599010399714289842011-02-08T20:32:00.002+00:002011-02-08T20:54:49.618+00:00Objective-C concurrency issuesDisclaimer - I've not shipped Java Swing / SWT apps. I'm a server guy where markup is the UI. Consequently, I don't have in-depth knowledge of Java to compare against. I'm aware of <a href="http://download.oracle.com/javase/6/docs/api/javax/swing/SwingUtilities.html#invokeLater(java.lang.Runnable)">SwingUtilities.invokeLater(Runnable)</a> but otherwise just assume I'm clueless about Swing.<div><br /></div><div>First rule of GUI programming - don't block the main thread.</div><div>Second rule of GUI programming - don't block the main thread, etc.</div><div><br /></div><div>Quantifying this, you have a device running at a refresh rate of 60Hz. So you if you do anything in the main thread, you need it to complete in under 16ms, or your UI will not be smooth and responsive.</div><div><br /></div><div>In Java, I would normally look at <a href="http://download.oracle.com/javase/6/docs/api/java/util/concurrent/Executor.html">Executor</a>s, <a href="http://download.oracle.com/javase/6/docs/api/java/util/concurrent/Callable.html">Callable</a>s, <a href="http://download.oracle.com/javase/6/docs/api/java/lang/Runnable.html">Runnable</a>s and related APIs to do things off the main thread. In Objective-C, we have <a href="http://www.google.co.uk/search?ie=UTF-8&q=nsoperationqueue">NSOperationQueue</a> and <a href="http://www.google.co.uk/search?ie=UTF-8&q=nsoperation">NSOperation</a>. Learn, use and love them. In particular, don't do what I did and start porting java.util.concurrent classes to Objective-C. I wrote a CountdownLatch, which was very nice and taught me about various low-level concurrency primitives. Unfortunately it was completely the wrong solution for the language. What I should have done was to use <a href="http://www.google.co.uk/search?&ie=UTF-8&q=nsoperation+adddependency">[NSOperation addDependency:]</a> to chain tasks together.</div><div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-72974202757101579312011-02-08T20:19:00.003+00:002011-02-08T20:28:38.643+00:00Objective-C toolingJava development tools are top of the pile out of anything I've used. The IDEs are massively powerful (they have to be, with the warts on the language). I'm also an emacs user day to day, and pragmatically use vim as well. But Eclipse / IDEA / Netbeans are pretty amazing tools for Java The Language development.<div><br /></div><div>Respectively for Objective-C development, Xcode isn't.</div><div><br /></div><div>If Apple Ts&Cs allow, IntelliJ could probably make some impressive inroads into that market.</div><div><br /></div><div><a href="http://clang.llvm.org/">clang</a> is a good (and getting better all the time) addition. The debugger needs some love; I don't find gdb as powerful as Java debuggers.</div><div><br /></div><div>In Java-land, one can use maven, ant, ivy, Make, etc to build a project. For iOS development, the IDE rules a lot from the off. There is a command-line tool which can potentially be driven by Jenkins or Thoughtworks Go. That would be my preferred option going forward; in my view, building in an IDE is not a repeatable build process.</div><div><br /></div><div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-33842120370996538312011-01-07T10:18:00.008+00:002011-01-07T11:33:02.334+00:00Creating a Custom Origin Server for Amazon CloudFront<p>At the time of writing, tool support is limited to the <a href="http://docs.amazonwebservices.com/AmazonCloudFront/latest/APIReference/index.html?CreateDistribution.html">REST API</a>? The intention with this piece of work was to take content being served by our origin server; e.g. http://example.com/images/foo.png; and serve it via Amazon CloudFront on http://cdn.example.com/images/foo.png.<br /></p><ol><li>Download <a href="http://aws.amazon.com/code/1878">cfcurl.pl</a>.</li><li>Get any dependencies from CPAN (the cfcurl.pl script tells you how to do that in case you aren't sure).</li><li>Create $HOME/.aws-secrets and chmod 600.<br /><pre>$ cat /Users/jabley/.aws-secrets<br />%awsSecretAccessKeys = (<br /> # my personal account<br /> 'james-personal' => {<br /> id => 'foo',<br /> key => 'bar',<br /> },<br /><br /> # my corporate account<br /> 'james-work' => {<br /> id => 'AWS-ID',<br /> key => 'AWS-Secret-Key',<br /> },<br />);<br /></pre></li><li>Create a file with the request data<br /><pre>$ cat create-distribution.xml<br /><?xml version="1.0" encoding="UTF-8"?><br /><DistributionConfig xmlns="http://cloudfront.amazonaws.com/doc/2010-11-01/"><br /><CustomOrigin><br /> <DNSName>example.com</DNSName><br /> <OriginProtocolPolicy>http-only</OriginProtocolPolicy><br /></CustomOrigin><br /><CallerReference>20110106103700</CallerReference><br /><CNAME>cdn.example.com</CNAME><br /><Comment>example.com<comment> CloudFront CDN</Comment><br /><Enabled>true</Enabled><br /><Logging><br /> <Bucket>accesslogs-example.com.s3.amazonaws.com</Bucket><br /> <Prefix>cdn.example.com/</Prefix><br /></Logging><br /></DistributionConfig><br /></comment></pre></li><li>POST the file<br /><pre> perl cfcurl.pl --keyname james-work -- -X POST -H "Content-Type: text/xml;charset=utf-8" --upload-file \<br /> create-distribution.xml https://cloudfront.amazonaws.com/2010-11-01/distribution<br /></pre></li><li>Poll to see when it has finished creating the distribution:<br /><pre> perl cfcurl.pl --keyname james-work -- https://cloudfront.amazonaws.com/2008-06-30/distribution<br /></pre></li><li>Configure DNS so that cdn.example.com is a CNAME for the DomainName value of your newly created CloudFront Distribution.</li></ol><p>You should now be able to request a resource using the new CDN name:<br /></p><pre>$ curl -v -s "http://cdn.example.com/images/foo.png" -o /dev/null<br />* About to connect() to cdn.example.com port 80 (#0)<br />* Trying 192.168.1.1... connected<br />* Connected to cdn.example.com (192.168.1.1) port 80 (#0)<br />> GET /images/foo.png HTTP/1.1<br />> Host: cdn.example.com<br />> Accept: */*<br />> User-Agent: curl<br />><br />* HTTP 1.0, assume close after body<br />< HTTP/1.0 200 OK<br />< Date: Thu, 06 Jan 2011 18:51:35 GMT<br />< Server: Apache/2.2.3 (Red Hat)<br />< Content-Length: 1233<br />< Cache-Control: max-age=86400<br />< Content-Type: image/png<br />< Age: 13311<br />< X-Cache: Hit from cloudfront<br />< X-Amz-Cf-Id: b769b423c54e2ffb0a6fb60369e2e0f7b103251ef3e2c549084fb4abe4ef9a236052f8eec40b3a14,80416de274eb8ee87c21ee41c863f9f6f9ef1c251823d9c12c46ab13dc33759dcbb04175b7d4a5a7<br />< Via: 1.0 83eb7919a5076e946a3a2d59d7f4415b.cloudfront.net:11180 (CloudFront), 1.0 26fb80d2abd86d7f52358cd1c2efd787.cloudfront.net:11180 (CloudFront)<br />< Connection: close<br />< <br />{ [data not shown]<br />* Closing connection #0 </pre><p></p><p>Note that Amazon CloudFront doesn't support query strings on resources, so you might need to use some <a href="http://serverfault.com/questions/219418/rewriting-apache-urls-to-use-only-paths-and-set-response-headers">Apache mod_rewrite stuff</a>.</p><div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com2tag:blogger.com,1999:blog-13473243.post-26560328971807474262010-08-04T08:45:00.003+01:002010-08-04T09:00:58.355+01:00Apple AppStore feedbackThis morning the AppStore on my iPod Touch pointed out that the Twitter app had an update. I installed it and launched the app to see what was new. Crash! After 5 crashes in a row without a successful launch, it was obvious that a bad build had snuck into the App Store.<div><br /></div><div><code></code></div><code></code><div><code> <p class="p1">Incident Identifier: 61EE3335-1AD0-4099-8EC6-FAB4B6160A43</p> <p class="p1">CrashReporter Key: 001e29bd4aa2ef81d42701ce5325da94b364e27b</p> <p class="p1">Process: Twitter [8470]</p> <p class="p1">Path: /var/mobile/Applications/8E13E345-CDD3-4CA4-899D-8E38BA6661C5/Twitter.app/Twitter</p> <p class="p1">Identifier: Twitter</p> <p class="p1">Version: ??? (???)</p> <p class="p1">Code Type: ARM (Native)</p> <p class="p1">Parent Process: launchd [1]</p> <p class="p2"><br /></p> <p class="p1">Date/Time: 2010-08-04 07:58:35.994 +0100</p> <p class="p1">OS Version: iPhone OS 3.1.3 (7E18)</p> <p class="p1">Report Version: 104</p> <p class="p2"><br /></p> <p class="p1">Exception Type: EXC_BREAKPOINT (SIGTRAP)</p> <p class="p1">Exception Codes: 0x00000001, 0xe7ffdefe</p> <p class="p1">Crashed Thread: 0</p> <p class="p2"><br /></p> <p class="p1">Dyld Error Message:</p> <p class="p1"> Symbol not found: __NSConcreteGlobalBlock</p> <p class="p1"> Referenced from: /var/mobile/Applications/8E13E345-CDD3-4CA4-899D-8E38BA6661C5/Twitter.app/Twitter</p> <p class="p1"> Expected in: /usr/lib/libSystem.B.dylib</p> <p class="p1"> Dyld Version: 149</p> <p class="p2"><br /></p> <p class="p1">Binary Images:</p> <p class="p1"> 0x1000 - 0x14ffff +Twitter armv6 <43ca857e309a61ba8c5da3ab83e42218> /var/mobile/Applications/8E13E345-CDD3-4CA4-899D-8E38BA6661C5/Twitter.app/Twitter</p></code><p class="p1"></p><p class="p1">As an iPhone app developer, I think I know what this problem is. We saw this problem in one of our apps. IIRC, the new, preferred llvm compiler has a bug with the new blocks language construct, and gcc doesn't, and the bug only shows up at runtime, in certain environments. To fix it, Twitter are going to have to recompile and use gcc rather than llvm, and then wait for the wheels at Apple to turn.</p><p class="p1">Other people have talked about the frustration of not being able to iterate at web speed or do continuous deployment, but that's part of the ecosystem that you operate in with Apple. </p><p class="p1">Testing, either by the Twitter team, or by Apple when they review the app prior to approving it, should have caught this issue. But these things happen.</p><p class="p1">We had a similar thing happen with an update to one of our apps recently. An update went live and thanks to the apparent difficulty in doing your own testing of the binary that gets sent to Apple, an issue only became apparent when the new version was available through iTunes. To me, this is where the ecosystem is broken. If I have a webapp and I deploy an update, then find an issue (via my cluster-immune system - one day!), I roll it back.</p><p class="p1">iTunesConnect has no rollback, even though it seems like a highly desirable feature. I know in our case, we would have liked the option to rollback to the last known good version and then wait for Apple to review an update, rather than having the world upgrade to a version that we didn't want them to be running. I imagine Twitter would appreciate a similar feature right about now.</p></div><div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com4tag:blogger.com,1999:blog-13473243.post-2638907800334216172010-07-08T10:13:00.004+01:002010-07-08T10:21:06.135+01:00Capturing Mobile Network Traffic On OS XRecently had to audit an app to ensure that it wasn't leaking any unwanted details over the network. This was an iPhone app, but the same process can be used for Android, etc.<br /><ol><li>Ensure Macbook Pro is plugged into Ethernet.</li><li>Open System Preferences</li><li>Internet & Wireless | Sharing (in Snow Leopard).</li><li>Click Internet Sharing.</li><li>From Ethernet</li><li>On Airport</li><li>Close System Preferences</li><li>Click Airport</li><li>Select Create Network...</li><li>On the phone, open the WiFi controls and connect to the network that you've just created.</li><li>Run Wireshark.</li><li>Start capturing traffic on the wireless card.</li><li>Check stuff is using SSL that should be, etc.</li></ol><div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com1tag:blogger.com,1999:blog-13473243.post-78414435851887645682010-05-20T14:03:00.004+01:002010-05-20T14:11:55.415+01:00mod_python in apache on OS X with HomebrewRecently had to install mod_python to test something for a customer. It needed some nudging, so including it here. Snow Leopard, Homebrew and default httpd.<div><br /><div><div><script src="http://gist.github.com/407547.js?file=gistfile1.sh"></script></div></div></div><div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-90596799980216581782010-01-13T10:21:00.004+00:002011-02-08T20:19:51.451+00:00Objective-C - the languageFirst off, I read the <a href="http://developer.apple.com/iphone/library/referencelibrary/GettingStarted/Learning_Objective-C_A_Primer/index.html">Objective-C Primer</a> and <a href="http://developer.apple.com/iphone/library/documentation/Cocoa/Conceptual/ObjectiveC/Introduction/introObjectiveC.html">Objective-C Programming Language</a> guides. I collect languages, so there was some underlying familiarity there. Ruby, Smalltalk and C obviously shone through for me. Second off, I re-read Smalltalk Best Practice Patterns. I first read that book maybe 6 years ago and it had a massive impact on my Java style. Objective-C is the most Smalltalk-like language that the 'masses' will actually use professionally. Sadly, it's not enough Smalltalk for me, and the C abstractions leak quite a bit.<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-78448502228250198112010-01-13T10:17:00.005+00:002011-02-08T20:55:31.158+00:00Java Developers Guide to Objective-C on the iPhoneThis will be a place-holder page containing links to the other entries that I create in this series. I've got a lot of commercial experience with Java, some Python and Ruby. This has all been server-side; I've not really touched GUIs (apart from GWT, HTML and Javascript) for a while, so this series will necessarily reflect that. Hopefully it will prove useful to others.<br /><br />Topics that I hope to cover:<br /><ul><li><a href="http://eternusuk.blogspot.com/2009/12/objective-c-for-java-developers.html">Initial questions</a> coming from a mainly Java background.</li><li><a href="http://eternusuk.blogspot.com/2010/01/objective-c-language.html">Objective-C the language</a>, including comparisons with Java.</li><li>OO with Objective-C, covering how a typical Java app would use interfaces and how Objective-C might approach the problem.</li><li>Collections in Java and Objective-C.<br /></li><li><a href="http://eternusuk.blogspot.com/2011/02/objective-c-concurrency-issues.html">Concurrency utils in Java and Objective-C</a>.<br /></li><li><a href="http://eternusuk.blogspot.com/2011/02/objective-c-tooling.html">Tools</a>.</li></ul><div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com1tag:blogger.com,1999:blog-13473243.post-88626539657195706902009-12-29T23:20:00.006+00:002011-02-09T07:11:32.765+00:00Objective-C for Java Developers<div>I'm coming from an Eclipse on Ubuntu background, but this is equally applicable for IDEA on Windows. What are the equivalents for iPhone development?</div><br /><table border="1" cellpadding="2" cellspacing="2"><tbody> <tr> <th>Java</th> <th>iPhone</th> <th>Notes</th> </tr> <tr> <td>JUnit (unit testing framework)</td> <td style="text-align: center;">?</td> <td>It is possible to use TDD for Swing apps, although I've been predominantly a server-side guy with client stuff happening in the browser for quite a while now. <a href="http://www.google.com/search?&ie=UTF-8&q=cucumber+with+iphone">Cucumber with iPhone</a> looks worth exploring...</td> </tr> <tr> <td>Hudson (continuous integration tool)</td> <td style="text-align: center;">?</td> <td>On my first iPhone app, it rapidly became apparent how easy it was for people to do bad merges and delete classes from the Xcode project file / strings from the UTF-16 l14n Localizable.strings file. You can argue that people should take more care; yeah, that'll fix it. git bisect is great, but a tool that builds on each commit is better.</td> </tr></tbody></table><table></table><div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-71372714584189847842009-12-26T23:53:00.003+00:002009-12-26T23:54:57.791+00:00ScaleCamp - Queue PUBSUBFrom some reason I went to this thinking PubSubHubbub, but it was more a refresher about making an app asynchronous, why you'd want to do that and how implementation complexity goes up as you go after certain desirable behaviours.<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-73769647962507662742009-12-26T23:17:00.006+00:002009-12-27T12:23:39.284+00:00ScaleCamp - How do you scale Activity Feeds?Popular session this - standing room only, so no notes from me. <br /><br />The short answer is Redis, courtesy of <a href="http://simonwillison.net/">Simon Willison</a>. Since the consensus was that Redis would do the trick, we then touched on <a href="http://simonwillison.net/2009/Nov/23/node/">Simon's other new favourite technology - node.js</a>.<br /><br />Digression: Alex from <a href="http://www.mediamolecule.com/">mediamolecule</a> made a comment about 100MB of data in a key-data structure store should only require 100MB of memory to store in such a server app (plus a little extra for housekeeping, but it shouldn't be a 1:10 ratio or similar. I didn't take that as a direct criticism of Redis but more of a reminder about choosing good data structures and the importance of CompSci (says this mathematician). I mention that, since it pricked me to investigate a suspected bad data structure in one of our apps, and coupled with the <a href="http://www.eclipse.org/mat/">Eclipse Memory Analyser</a> recommended by the <a href="http://eternusuk.blogspot.com/2009/12/scalecamp-scaling-java-and-oracle-for.html">Guardian guys</a>, I found something that was using far too much of the heap for our Tomcat nodes; and had a change rolled out within 4 days of attending this conference. That reduced the memory footprint for that data structure from 250MB to 16MB. Ouch, shocker, but great to have found, prioritised and fixed.<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-48788449619276599132009-12-07T12:11:00.003+00:002009-12-26T23:16:09.823+00:00ScaleCamp - Scaling Java and Oracle for the GuardianGuardian.co.uk<br /><br />Graham and various other people from the development and operations team pitching in.<br /><br />3 years ago - published static files with apache SSI to fill-in gaps. Moved to a fully dynamic system. Now, they're somewhere in between.<br /><br />Stack - <br /><ul><li>apache</li><br /><li>resin</li><br /><li>spring / hibernate / velocity</li><br /><li>Oracle DB backend (not recommended!)</li></ul><br /><br />Measured the application - 1300 requests to DB just to render homepage.<br /><br />Added ehcache to hibernate as 2nd level cache and added a warmup script before putting into load balancer<br /><br />30m unique users per month<br />270m pages per month<br />250 requests/second at lunchtime<br />1500 requests/second peak.<br /><br /><h3>GC tools</h3><br /><br />Google weakref cache (part of <a href="http://code.google.com/p/google-collections/">Google Collections</a>)<br /><br /><a href="http://www.eclipse.org/mat/">Eclipse memory analyser</a> - what's using all my memory?<br /><br />Cacti for monitoring - DB usage was killing it.<br /><br />8 app servers in each co-lo (London and Manchester).<br /><br />400MB used by cache - churn meant was pretty ineffective.<br /><br />Tried or considered ehcache distribution and jboss cache distribution.<br /><br />Rejected since cache eviction via replication would have thrashed it.<br /><br /><h3>memcached</h3><br /><br />massive improvement in response times, but DB load still high.<br /><br />went to caching every query for 5 minutes. DB load vanished and is flat even as more app servers come on-line.<br /><br />servlet filter writing to memcached made it stink fast.<br /><br />took a days worth of logs and Hadoop to see how long the cache should be. 1 minute was the sweet spot.<br /><br />Emergency switch to serve a static copy of the site, minus personalization features.<br /><br />Daemon or script scrapes the sit; they can handle 700req/s/node when the site's operating in this mode.<br /><br />new content published in this mode has a copy pressed so it can be served from disk - publish is slower than with the other system but updates still possible<br /><br />Highly recommend that this sort of emergency degrade read-only mode should be built-in from the off - they've used this approach with the MPs Expenses apps built to crowd-source investigations.<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-51670664602837744122009-12-07T12:09:00.002+00:002009-12-26T23:02:26.783+00:00ScaleCamp - Scaling Java with Shared Nothing<a href="http://eternusuk.blogspot.com/2009/12/scalecamp-scaling-with-squid.html">Thoughtworks guys again</a>.<br /><br />Basic Servlet overview - in-memory sessions don't scale, duh!<br /><br />Preferred options - state goes into cookies and serialized. Security, legal aspects? Pretty well common to most frameworks these days.<br /><br />Page composition in the server with proxy server holding StringTemplate objects. Interesting idea - <a href="http://www.holovaty.com/writing/django-two-phased-rendering/">seen variants of this in other place</a>s. I'm curious as to whether doing this could mean having a poor man's macro system for Java, since XSLTs can be written to create XSLTs; maybe Velocity templates could similarly generate Velocity templates or StringTemplate -> StringTemplate?<br /><br />Again, application developers need to have a good idea of caching directives for this to work. One objection I had with this approach is that you potentially increase your hardware requirement and can open the app up to liveness failures here. Request A comes in and is serviced by Thread 1. As part of that, it makes a request to the proxy server for a template. At the proxy server, a cache miss means that another request needs to be made to the app server. Then Request A is tying up 2 app server threads. What about applications which parallelize the requests? They might use more than 2 app server request-handling threads at a time, etc.<br /><br />Thoughtworks seem to do fun, interesting work.<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0tag:blogger.com,1999:blog-13473243.post-80282417922978701182009-12-07T12:08:00.004+00:002009-12-26T22:41:39.526+00:00ScaleCamp - LittleBigPlanetAlex and James from mediamolecule.com. Fascinating perspective of embedded developers coming to server-programming and refusing to accept commonly held views on best practices for doing so. This was the surprise hit of the conference for me; I just elected to go since there wasn't anything else in that slot that I was really passionate about. I'd been talking to them both in the queue for tea earlier and made a poorly judged joke about Map-Reduce (<a href="http://browsertoolkit.com/fault-tolerance.png">we pretty much had this conversation</a>). The session they ran was an awesome talk about scaling server-side within the games sector - <a href="http://www.littlebigplanet.com/">Little Big Planet</a> is theirs.<br /><br />Written their own C-based key-data structure store, of which we're spoilt for choice just now. Alex commented that he's looked at Redis and it has some nice stuff, but when they came to need it, there wasn't anything that met their needs, and experience with running the recommended Java stack had left them with the impression that they should stick to what they know. What they know is writing very tight code in constrained environments, so applying that mind-set to server-side development seemed to have yielded some very pleasing numbers. Other parts are in Ruby (presumably 1.9, since they're using Fibers?). I didn't get around to asking James how well that works or which implementation they're using. Very happy with that programming model though - James is or was a Java guy - funny how nice Ruby feels coming from there!<div class="blogger-post-footer">All content Copyright 2005, 2006 James Abley</div>James Ableyhttp://www.blogger.com/profile/10389773375250732709noreply@blogger.com0