Objective-C concurrency issues

Disclaimer - I've not shipped Java Swing / SWT apps. I'm a server guy where markup is the UI. Consequently, I don't have in-depth knowledge of Java to compare against. I'm aware of SwingUtilities.invokeLater(Runnable) but otherwise just assume I'm clueless about Swing.

First rule of GUI programming - don't block the main thread.

Second rule of GUI programming - don't block the main thread, etc.

Quantifying this, you have a device running at a refresh rate of 60Hz. So you if you do anything in the main thread, you need it to complete in under 16ms, or your UI will not be smooth and responsive.

In Java, I would normally look at Executors, Callables, Runnables and related APIs to do things off the main thread. In Objective-C, we have NSOperationQueue and NSOperation. Learn, use and love them. In particular, don't do what I did and start porting java.util.concurrent classes to Objective-C. I wrote a CountdownLatch, which was very nice and taught me about various low-level concurrency primitives. Unfortunately it was completely the wrong solution for the language. What I should have done was to use [NSOperation addDependency:] to chain tasks together.

Objective-C tooling

Java development tools are top of the pile out of anything I've used. The IDEs are massively powerful (they have to be, with the warts on the language). I'm also an emacs user day to day, and pragmatically use vim as well. But Eclipse / IDEA / Netbeans are pretty amazing tools for Java The Language development.

Respectively for Objective-C development, Xcode isn't.

If Apple Ts&Cs allow, IntelliJ could probably make some impressive inroads into that market.

clang is a good (and getting better all the time) addition. The debugger needs some love; I don't find gdb as powerful as Java debuggers.

In Java-land, one can use maven, ant, ivy, Make, etc to build a project. For iOS development, the IDE rules a lot from the off. There is a command-line tool which can potentially be driven by Jenkins or Thoughtworks Go. That would be my preferred option going forward; in my view, building in an IDE is not a repeatable build process.

Objective-C - the language

First off, I read the Objective-C Primer and Objective-C Programming Language guides. I collect languages, so there was some underlying familiarity there. Ruby, Smalltalk and C obviously shone through for me. Second off, I re-read Smalltalk Best Practice Patterns. I first read that book maybe 6 years ago and it had a massive impact on my Java style. Objective-C is the most Smalltalk-like language that the 'masses' will actually use professionally. Sadly, it's not enough Smalltalk for me, and the C abstractions leak quite a bit.

Java Developers Guide to Objective-C on the iPhone

This will be a place-holder page containing links to the other entries that I create in this series. I've got a lot of commercial experience with Java, some Python and Ruby. This has all been server-side; I've not really touched GUIs (apart from GWT, HTML and Javascript) for a while, so this series will necessarily reflect that. Hopefully it will prove useful to others.

Topics that I hope to cover:

• Initial questions coming from a mainly Java background.
• Objective-C the language, including comparisons with Java.
• OO with Objective-C, covering how a typical Java app would use interfaces and how Objective-C might approach the problem.
• Collections in Java and Objective-C.
  
• Concurrency utils in Java and Objective-C.
  
• Tools.

Objective-C for Java Developers

I'm coming from an Eclipse on Ubuntu background, but this is equally applicable for IDEA on Windows. What are the equivalents for iPhone development?

Java	iPhone	Notes
JUnit (unit testing framework)	?	It is possible to use TDD for Swing apps, although I've been predominantly a server-side guy with client stuff happening in the browser for quite a while now. Cucumber with iPhone looks worth exploring...
Hudson (continuous integration tool)	?	On my first iPhone app, it rapidly became apparent how easy it was for people to do bad merges and delete classes from the Xcode project file / strings from the UTF-16 l14n Localizable.strings file. You can argue that people should take more care; yeah, that'll fix it. git bisect is great, but a tool that builds on each commit is better.

ScaleCamp - Scaling Java with Shared Nothing

Thoughtworks guys again.

Basic Servlet overview - in-memory sessions don't scale, duh!

Preferred options - state goes into cookies and serialized. Security, legal aspects? Pretty well common to most frameworks these days.

Page composition in the server with proxy server holding StringTemplate objects. Interesting idea - seen variants of this in other places. I'm curious as to whether doing this could mean having a poor man's macro system for Java, since XSLTs can be written to create XSLTs; maybe Velocity templates could similarly generate Velocity templates or StringTemplate -> StringTemplate?

Again, application developers need to have a good idea of caching directives for this to work. One objection I had with this approach is that you potentially increase your hardware requirement and can open the app up to liveness failures here. Request A comes in and is serviced by Thread 1. As part of that, it makes a request to the proxy server for a template. At the proxy server, a cache miss means that another request needs to be made to the app server. Then Request A is tying up 2 app server threads. What about applications which parallelize the requests? They might use more than 2 app server request-handling threads at a time, etc.

Thoughtworks seem to do fun, interesting work.

Java doesNotUnderstand-like behaviour in Eclipse

Kent Beck tweeted this recently. I've been doing this for years, but I guess it's not as widely used as I assumed.

Window | Preferences | Java | Code Style | Code Templates | Code | Method Body


// ${todo} Auto-generated method stub
throw new UnsupportedOperationException("Not implemented");


Then add a breakpoint to your Debugger Breakpoint view:

New Java Exception for UnsupportedOperationException that hasn't been caught.

IDEA supports something similar, since I had it set up then as well, but I've not used IDEA for a couple of years.

I tend not to use debuggers; I prefer tests, but sometimes a debugger's the thing.

Java REPL

Obviously, most good dynamic languages for the JVM have this, but still, it's sweet.

$ jirb 
irb(main):001:0> require 'lib/org.restlet.jar'
=> true
irb(main):002:0>  http = Java::OrgRestletData::Protocol::HTTP
=> #<Java::OrgRestletData::Protocol:0x59cbda @java_object=#>
irb(main):003:0> client = Java::OrgRestlet::Client.new http 
=> #<Java::OrgRestlet::Client:0x800aa1 @java_object=#>
irb(main):004:0> r = client.get 'http://www.apache.org/'
01-Feb-2009 22:54:08 org.restlet.engine.http.StreamClientHelper start
INFO: Starting the HTTP client
=> #<Java::OrgRestletData::Response:0x12a416a @java_object=#>

JAXP pipelines using SAX

XProc looks handy, but is not in a usable state yet. So I had to roll my own pipeline as a one-off recently, and seemed to struggle more than I expected. The requirement was fairly simple

1. Ingest some HTML and convert to well-formed XML for processing;


2. filter that XML to remove unwanted content;


3. convert the XML to a different format.

Step 2 was a new bit - I already had well-tested code for the other two parts. So I wanted to re-use that as much as possible. JAXP pipelines using SAX looked to be (and is!) very nice for this, but examples seemed a bit thin on the ground. I've put a version of it here, in the hope that others may find it useful.

/* Create an InputSource for the pipeline input document. */
InputSource in = new InputSource(new ByteArrayInputStream(StringUtils.getBytes(text, "utf-8")));

/* Step 1. TagSoup parsing to get well-formed XML */
XMLReader reader = new Parser();

try {
    SAXTransformerFactory stf = (SAXTransformerFactory) SAXTransformerFactory.newInstance();

    StringWriter sb = new StringWriter();

    OutputFormat outputFormat = new OutputFormat();
    outputFormat.setOmitXMLDeclaration(true);

    XMLSerializer serializer = new org.apache.xml.serialize.XMLSerializer(outputFormat);
    serializer.setOutputCharStream(sb);

    /* Step 2. Remove unwanted markup from the well-formed XML. */
    InputStream stripContent = getResourceAsStream("strip-content.xslt");
    XMLFilter removeUnwanted = stf.newXMLFilter(new StreamSource(stripContent));

    /* Step 3. Convert to preferred markup format. */
    InputStream xsltResourceInputStream = getResourceAsStream("xhtml2dial.xslt");
    XMLFilter xhtml2dial = stf.newXMLFilter(new StreamSource(xsltResourceInputStream));

    removeUnwanted.setParent(reader);
    xhtml2dial.setParent(removeUnwanted);
    xhtml2dial.setContentHandler(serializer.asContentHandler());

    reader.parse(in);

    return sb.toString();
} catch (TransformerException e) {
    throw new ConversionException(e.getMessage(), e);
} catch (IOException e) {
    throw new ConversionException(e.getMessage(), e);
} catch (SAXException e) {
    throw new ConversionException(e.getMessage(), e);
}

Java Triple DES example

I had to do this recently, and the examples that I found through Google seemed a little lacking. So a pointer to others:







UPDATE

Changed to use a gist on github. That will probably dilute the search ranking of this post (which is reasonably popular) but gives me nicer formatting!

Facebook Mega-Data

Just noticed that Facebook open-sourced a version of their BigTable data store and it's hosted on Google code, written in Java. I'll omit the obvious snickers about Facebook cloning BigTable and then hosting on Google Code.

WideFinder - late entrant

Snagged me a log file and starting cranking out some code tonight. I had some free time to think about this a couple of weeks ago, and got some sketches down, but I've only recently got the data to start seeing how the code can fly. So, I'm starting out on my work Dell laptop, Dual Core Pentium 2GHz with 2GB RAM and a really shitty disk, judging by the slowness and noises it makes (or is that just Vista?) [stay on topic! - Ed]. The Ruby version runs in just over a minute, once all of the caches are warmed up. My initial naive Java version runs in 14 seconds (I haven't figured out yet how to run it using time as per *nix environments - Cygwin says it can't find the time command when I pipe zcat output into it).

Now to start implementing my ideas. I have what I think is the shared update of the accumulator as well as I'm going to get it. I'm hypothesising that most of the updates are uncontended and so don't require the full weight of Java's locking capabilities. Now I just need to parallelize the I/O and determine the most efficient matching algorithm, which seems to be Boyer-Moore from reading the Wide-Finder series. That particular algorithm seems to pop up fairly regularly in searching. Might be interesting to see what else is available in that field, but it should be in a library, surely?

Java Mock Objects tip

Discovered while using JMock, but I would imagine it's also good for EasyMock, RMock, ...


checking(new Expectations() {
    {
        one(httpServletRequest).getParameter("c");
        will(returnValue("-2"));

        one(httpServletRequest).getParameterNames();

        // StringTokenizer implements Enumeration. A bit cheeky!
        will(returnValue(new StringTokenizer("c")));
    }
});

Jython UnicodeData hacking in the CDS

Got numeric and decimal working today in between contractions in the Central Delivery Suite at Frimley Park Hospital today. Nearly got categories working as well, except I'm not clear how CPython has implemented unicodedata.category for undefined codepoints.

e.g. Python 2.5 uses Unicode 4.2 for the Unicode database. The integer codepoint 13313(decimal) / 3401 (hex) is not defined within Unicode 4.1.

3400;;Lo;0;L;;;;;N;;;;;
4DB5;;Lo;0;L;;;;;N;;;;;

It isn't defined in Unicode 5.0, which is what I've been using to do the Jython implementation.

3400;;Lo;0;L;;;;;N;;;;;
4DB5;;Lo;0;L;;;;;N;;;;;

So how does CPython define unicodedata.category(unichr(13313)) to be 'Lo'? And it doesn't seem to be just 'Lo' in all cases of undefined items. I'm speculating that it might be falling back to the preceding valid codepoint category. Think I need to post to a CPython list to confirm.

Good ETag support requires thinking about it up-front

I posted a comment on this but I thought it worthwhile going a little deeper.

Blogger doesn't support Trackback so I'll just post and link.

My point was not to argue about how little code is required to implement sending an ETag and checking an ETag based on the MD5 hash of your content (that's pretty much a library issue which should level out to be equal over time) but to go a little deeper into ETags.

I've been reading Sam Ruby long enough to have had the benefit of ETags drummed into me. The posts that Bill links to are focused on the network savings aspect of conditional GET. But you can also save server processing power, if you put a little more thought into your application model.

So we come back to the requirements for Java frameworks to support ETags such that it is possible to avoid doing the bulk of the server side processing. Caveat this could well be premature optimization, and is merely me thinking out loud. Struts is the one I'm most familiar with and I think with the struts-chain RequestProcessor, this approach could be used, but anything that works as a chain would do for this (so pure Filters would also work).

public void doFilter(ServletRequest request, ServletResponse response,
        FilterChain filterChain) throws IOException, ServletException {

    HttpServletRequest httpRequest = (HttpServletRequest) request;
    
    /*
     * Do something that works out what is required to render a response 
     * for this request and generate an ETag based on that. So here we 
     * have moved away from the approach of generating ETags from MD5 
     * hashes of the response body.
     */
    ETag currentResourceETag = calculateETag(httpRequest);
    ETag incomingETag = extractETag(httpRequest);

    if (currentResourceETag.equals(incomingETag)) {
        response.sendStatus(HttpServletResponse.SC_NOT_MODIFIED);
    } else {
        filterChain.doFilter(request, response);
        response.addHeader("Etag", currentResourceETag.stringValue());
    }
    
}

You would need to be able to obtain the items responsible for determining the ETag value reasonably early in the request processing, before any really expensive operations. Not sure what implications that has for the layers in your application, or if you were using strict MVC how disruptive / worthwhile it would be to try this approach...

Jython - UnicodeData mirrored is complete!

from test_support import verify, verbose
import sha

encoding = 'utf-8'

def test_mirrored():
    
    h = sha.sha()
    
    for i in range(65536):
        c = unichr(i)
        h.update(str(unicodedata.mirrored(c)))
        print  "%i : %i%c" % (i, unicodedata.mirrored(c), unichr(10)),
        
    # Value returned by Python 2.5, which uses Unicode 4.2
    #verify('91cd30c6c81911835dbcbed083f99fc9fc073e4a' ==  h.hexdigest(), 
    #       h.hexdigest())
    
    # Value returned by current Jython implementation, which uses Unicode 5.0
    verify('595795a212ca0ac629d6b2dfb09c703a472adb03' ==  h.hexdigest(), 
           h.hexdigest())

# Add next test!
    
if __name__ == '__main__':
    import unicodedata
    test_mirrored()

OK, it's only for the BMP, but it's a good start. Supporting supplementary characters (in Java terminology) or the other sixteeen planes would need a more fundamental change to PyUnicode, methinks. Now I need to start adding the other unicodedata methods which should be fairly straightforward. Then I'll have a working implementation to post to the dev list. Maybe end of this month, unless Baby comes and I lose my late night hacking time?

Jython UnicodeData mirroring

for i in range(65536):
        c = unichr(i)
        print  "%i : %i%c" % (i, unicodedata.mirrored(c), unichr(10)) ,

jabley@miq-jabley ~/work/eclipse/workspaces/personal/jython-trunk/jython
$ diff jython-mirrored.txt python-mirrored.txt
10177c10177
< 10176 : 0
---
> 10176 : 1
10180,10183c10180,10183
< 10179 : 0
< 10180 : 0
< 10181 : 0
< 10182 : 0
---
> 10179 : 1
> 10180 : 1
> 10181 : 1
> 10182 : 1
11779,11782c11779,11782
< 11778 : 0
< 11779 : 0
< 11780 : 0
< 11781 : 0
---
> 11778 : 1
> 11779 : 1
> 11780 : 1
> 11781 : 1
11786,11787c11786,11787
< 11785 : 0
< 11786 : 0
---
> 11785 : 1
> 11786 : 1
11789,11790c11789,11790
< 11788 : 0
< 11789 : 0
---
> 11788 : 1
> 11789 : 1
11805,11806c11805,11806
< 11804 : 0
< 11805 : 0
---
> 11804 : 1
> 11805 : 1

I was hoping to use java.lang.Character.isMirrored(char), but the above is the result of diffing the output for jython and python running my test and diff-ing the output. Looking in more detail, Java 1.4 supports UCD 3.2, then Java 5 and Java 6 both only have support for UCD 4.0.

jabley@miq-jabley ~/work/eclipse/workspaces/personal/jython-trunk/jython
$ python
Python 2.5.1 (r251:54863, May 18 2007, 16:56:43)
[GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import unicodedata
>>> unicodedata.unidata_version
'4.1.0'

And I'm getting the sneaking feeling that I've done something like that before, but it's been so long since I did any development in this area that I've forgotten it!

Jython Update - actually doing some work on UnicodeData

So I had put this on hold to finish reading Josh Bloch and Neal Gafter's Java Puzzlers. Great book; highlighted some new things for me, which is all I ask for in any book. Partly that is since I'm still not doing Java 5 apart from at home for various minor things, but it was good.

So now back to Jython, finally. Well, reasons for my procrastination first:

1. New Job - busy as hell.


2. New job comes with a new laptop, which I was hoping would be a big step up from my nearly six year old self-built machine. The new laptop has reasonably impressive hardware specification, but it's running Vista. What an absolute pile of shit. I honestly don't know how developers are productive using that OS. I've given it nearly two months, just to be sure that it's not the fact that I've been off Windows for three years that is causing me all of the problems, but really. It's got to the point where I'm looking seriously at Xen on Ubuntu for the odd application that I do need to run Windows for. The other alternative would be to install XP, put up with the half life cost and initial downtime of getting the laptop set up for development all over again. I'll enumerate my grievances in a separate post. I don't think XP would get in my way as much as Vista (I did knock up the xmlunit XMLSchema validation patch on my wife's XP machine over Christmas and it wasn't that painful), but I'm not a fan of Windows after using GNU/Linux exclusively for three years.

So I've been doing a little work on UnicodeData again. Since I've not touched it for so long, I wanted to get some code up and running to start seeing how many tests were failing. Until Jython goes to Java 5 and above, I can't use java.lang.Character to do parts of it, or I could do a piecemeal approach of use java.lang.Character for the BMP, and then implement a new part for supplementary characters, or try to provide behaviour based on the running JVM. All a bit more work than I wanted to do, laziness and hubris being key. Instead, go for the brute force approach of the simplest thing that will possibly work. So I wrote a Python script (what else - it's a nice way of bootstrapping this problem) to parse the UnicodeData.txt file and generate some Java classes. The initial approach was to partition the UnicodeData.txt into a class for each plane in Unicode. Anyone that knows Unicode and the assigned codepoints will know that the BMP will take up most of this, but I was interested in getting something working, and then maybe refine it once the tests are passing. Well, my first cut was to have a simple interface:


interface UnicodePlane {

/**
* Return a UnicodeCodepoint for the specified codepoint.
*
* @param codepoint the Unicode codepoint
*
* @return a UnicodeCodepoint, or null if there is no match
*/
UnicodeCodepoint getCodepoint(int codepoint);

}


I would have a class that implements this interface for each plane and a static initializer within each class that fills a Map of UnicodeCodepoint classes keyed by Integer codepoint.

Eclipse gives me this error:

The code for the static initializer is exceeding the 65535 bytes limit 

Whereas ANT gave me this variation:

    [javac] Compiling 2 source files to c:\Users\jabley\work\eclipse\workspaces\personal\jython-trunk\jython\build
    [javac] c:\Users\jabley\work\eclipse\workspaces\personal\jython-trunk\jython\UnicodeData\generated-src\org\python\modules\unicodedata\UnicodeCharacterDataBasicMultilingualPlane
.java:11: code too large
    [javac]     private static final Map CODEPOINTS = new HashMap();
    [javac]                              ^
    [javac] 1 error

BUILD FAILED
c:\Users\jabley\work\eclipse\workspaces\personal\jython-trunk\jython\build.xml:456: Compile failed; see the compiler error output for details.

So I need to think a bit harder about the data structures. Turning to Bentley's Programming Pearls, the sparseness of certain items stands out, like the mirrored property.

I'll have a think. At least with the Python script that I have to cut up the UnicodeData.txt file, it's very easy to add another list comprehension to it, to see how many items in the file exhibit a certain property. The other way I'm considering is to just generate a properties file and lazily populate a Map as required. That's probably what I'll try next, rather than thinking too hard about how to compress a 1038607 bytes data file into something more reasonable.

Jython - contributing

Got the go-ahead from my employer late last week that it's OK for me to contribute to Jython. Cool, I can start properly doing the unicodedata implementation (and I don't have to ask Stefan to back out my xmlunit patch!). The only annoying thing is that I started doing the Josh Bloch / Neal Gafter Java Puzzlers book while I was waiting for the approval to come through, so I want to finish that rather than having too many things on the go at one time. So I'll probably be a week before ramping up on unicodedata again.

Jython stalled

I haven't made as much progress with this as I would like, since I'm having some troubles. Not ones of a technical nature, but instead ones of a legal nature. My employment contract is apparently fairly standard and says that my employer owns all of my thoughts (even the ones when I'm writing this!). As what I thought was a courtesy, I asked my manager whether it was OK to contribute to open source projects, so that there would be no shady areas about who owned what code. Well, I'm still waiting for an answer and a bit of paper from my employer. So until I get that, I'm holding off writing any real code.

What I do have is a bit of sketching around the general area. First off, I wasn't sure about some aspects of the CPython implementation, so I asked. It got bounced by the python-dev moderator with the advice that I should post on the python-list. Which I did, but as I expected, it was more of a question for the python-dev list and the only (private) response that I've had is from Martin V. Loewis, who did the last change to that part of CPython. Maybe people are busy with PyCon. I was pointed at the C implementation, which doesn't generate that part from the UnicodeData.txt file, but instead is a horrible case statement, which looks bad. I think I need to raise a bug.

The other thing I've done with it is some Learning Tests about java.lang.Character, to see what it offers me. Obviously, this is attractive since it's a core library, is well tested, debugged and used by millions of people and all the other reasons Josh Bloch enumerates in Effective Java. It seems to have a few little idiosyncracies, which I have captured in my tests. (Note to self: I haven't seen any JUnit (or TestNG - Cedric!) tests in the Jython source tree. Must ask the dev list about that.) Then maybe I should check whether JRuby needs this sort of thing, and make it re-usable, with a Jython wrapper for the API that it needs, etc.