Jython UnicodeData hacking in the CDS

Got numeric and decimal working today in between contractions in the Central Delivery Suite at Frimley Park Hospital today. Nearly got categories working as well, except I'm not clear how CPython has implemented unicodedata.category for undefined codepoints.

e.g. Python 2.5 uses Unicode 4.2 for the Unicode database. The integer codepoint 13313(decimal) / 3401 (hex) is not defined within Unicode 4.1.

3400;;Lo;0;L;;;;;N;;;;;
4DB5;;Lo;0;L;;;;;N;;;;;

It isn't defined in Unicode 5.0, which is what I've been using to do the Jython implementation.

3400;;Lo;0;L;;;;;N;;;;;
4DB5;;Lo;0;L;;;;;N;;;;;

So how does CPython define unicodedata.category(unichr(13313)) to be 'Lo'? And it doesn't seem to be just 'Lo' in all cases of undefined items. I'm speculating that it might be falling back to the preceding valid codepoint category. Think I need to post to a CPython list to confirm.

Jython - UnicodeData mirrored is complete!

from test_support import verify, verbose
import sha

encoding = 'utf-8'

def test_mirrored():
    
    h = sha.sha()
    
    for i in range(65536):
        c = unichr(i)
        h.update(str(unicodedata.mirrored(c)))
        print  "%i : %i%c" % (i, unicodedata.mirrored(c), unichr(10)),
        
    # Value returned by Python 2.5, which uses Unicode 4.2
    #verify('91cd30c6c81911835dbcbed083f99fc9fc073e4a' ==  h.hexdigest(), 
    #       h.hexdigest())
    
    # Value returned by current Jython implementation, which uses Unicode 5.0
    verify('595795a212ca0ac629d6b2dfb09c703a472adb03' ==  h.hexdigest(), 
           h.hexdigest())

# Add next test!
    
if __name__ == '__main__':
    import unicodedata
    test_mirrored()

OK, it's only for the BMP, but it's a good start. Supporting supplementary characters (in Java terminology) or the other sixteeen planes would need a more fundamental change to PyUnicode, methinks. Now I need to start adding the other unicodedata methods which should be fairly straightforward. Then I'll have a working implementation to post to the dev list. Maybe end of this month, unless Baby comes and I lose my late night hacking time?

Jython UnicodeData mirroring

for i in range(65536):
        c = unichr(i)
        print  "%i : %i%c" % (i, unicodedata.mirrored(c), unichr(10)) ,

jabley@miq-jabley ~/work/eclipse/workspaces/personal/jython-trunk/jython
$ diff jython-mirrored.txt python-mirrored.txt
10177c10177
< 10176 : 0
---
> 10176 : 1
10180,10183c10180,10183
< 10179 : 0
< 10180 : 0
< 10181 : 0
< 10182 : 0
---
> 10179 : 1
> 10180 : 1
> 10181 : 1
> 10182 : 1
11779,11782c11779,11782
< 11778 : 0
< 11779 : 0
< 11780 : 0
< 11781 : 0
---
> 11778 : 1
> 11779 : 1
> 11780 : 1
> 11781 : 1
11786,11787c11786,11787
< 11785 : 0
< 11786 : 0
---
> 11785 : 1
> 11786 : 1
11789,11790c11789,11790
< 11788 : 0
< 11789 : 0
---
> 11788 : 1
> 11789 : 1
11805,11806c11805,11806
< 11804 : 0
< 11805 : 0
---
> 11804 : 1
> 11805 : 1

I was hoping to use java.lang.Character.isMirrored(char), but the above is the result of diffing the output for jython and python running my test and diff-ing the output. Looking in more detail, Java 1.4 supports UCD 3.2, then Java 5 and Java 6 both only have support for UCD 4.0.

jabley@miq-jabley ~/work/eclipse/workspaces/personal/jython-trunk/jython
$ python
Python 2.5.1 (r251:54863, May 18 2007, 16:56:43)
[GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import unicodedata
>>> unicodedata.unidata_version
'4.1.0'

And I'm getting the sneaking feeling that I've done something like that before, but it's been so long since I did any development in this area that I've forgotten it!

Jython Update - actually doing some work on UnicodeData

So I had put this on hold to finish reading Josh Bloch and Neal Gafter's Java Puzzlers. Great book; highlighted some new things for me, which is all I ask for in any book. Partly that is since I'm still not doing Java 5 apart from at home for various minor things, but it was good.

So now back to Jython, finally. Well, reasons for my procrastination first:

1. New Job - busy as hell.


2. New job comes with a new laptop, which I was hoping would be a big step up from my nearly six year old self-built machine. The new laptop has reasonably impressive hardware specification, but it's running Vista. What an absolute pile of shit. I honestly don't know how developers are productive using that OS. I've given it nearly two months, just to be sure that it's not the fact that I've been off Windows for three years that is causing me all of the problems, but really. It's got to the point where I'm looking seriously at Xen on Ubuntu for the odd application that I do need to run Windows for. The other alternative would be to install XP, put up with the half life cost and initial downtime of getting the laptop set up for development all over again. I'll enumerate my grievances in a separate post. I don't think XP would get in my way as much as Vista (I did knock up the xmlunit XMLSchema validation patch on my wife's XP machine over Christmas and it wasn't that painful), but I'm not a fan of Windows after using GNU/Linux exclusively for three years.

So I've been doing a little work on UnicodeData again. Since I've not touched it for so long, I wanted to get some code up and running to start seeing how many tests were failing. Until Jython goes to Java 5 and above, I can't use java.lang.Character to do parts of it, or I could do a piecemeal approach of use java.lang.Character for the BMP, and then implement a new part for supplementary characters, or try to provide behaviour based on the running JVM. All a bit more work than I wanted to do, laziness and hubris being key. Instead, go for the brute force approach of the simplest thing that will possibly work. So I wrote a Python script (what else - it's a nice way of bootstrapping this problem) to parse the UnicodeData.txt file and generate some Java classes. The initial approach was to partition the UnicodeData.txt into a class for each plane in Unicode. Anyone that knows Unicode and the assigned codepoints will know that the BMP will take up most of this, but I was interested in getting something working, and then maybe refine it once the tests are passing. Well, my first cut was to have a simple interface:


interface UnicodePlane {

/**
* Return a UnicodeCodepoint for the specified codepoint.
*
* @param codepoint the Unicode codepoint
*
* @return a UnicodeCodepoint, or null if there is no match
*/
UnicodeCodepoint getCodepoint(int codepoint);

}


I would have a class that implements this interface for each plane and a static initializer within each class that fills a Map of UnicodeCodepoint classes keyed by Integer codepoint.

Eclipse gives me this error:

The code for the static initializer is exceeding the 65535 bytes limit 

Whereas ANT gave me this variation:

    [javac] Compiling 2 source files to c:\Users\jabley\work\eclipse\workspaces\personal\jython-trunk\jython\build
    [javac] c:\Users\jabley\work\eclipse\workspaces\personal\jython-trunk\jython\UnicodeData\generated-src\org\python\modules\unicodedata\UnicodeCharacterDataBasicMultilingualPlane
.java:11: code too large
    [javac]     private static final Map CODEPOINTS = new HashMap();
    [javac]                              ^
    [javac] 1 error

BUILD FAILED
c:\Users\jabley\work\eclipse\workspaces\personal\jython-trunk\jython\build.xml:456: Compile failed; see the compiler error output for details.

So I need to think a bit harder about the data structures. Turning to Bentley's Programming Pearls, the sparseness of certain items stands out, like the mirrored property.

I'll have a think. At least with the Python script that I have to cut up the UnicodeData.txt file, it's very easy to add another list comprehension to it, to see how many items in the file exhibit a certain property. The other way I'm considering is to just generate a properties file and lazily populate a Map as required. That's probably what I'll try next, rather than thinking too hard about how to compress a 1038607 bytes data file into something more reasonable.

Jython progress report

Not much really! I've moved jobs, so being busy on that meant my motivation to hack in the evening went a bit flat. But I'm settling in to the new thing now and have a shiny new dual core laptop for developing on. I set that up for Jython development last night, so I'm hopeful of picking it up. And for anyone that's interested, I've set up a pipe that will only have Jython-related posts in it, rather than all the family stuff that they might find slightly uninteresting if they don't know me!

Jython - contributing

Got the go-ahead from my employer late last week that it's OK for me to contribute to Jython. Cool, I can start properly doing the unicodedata implementation (and I don't have to ask Stefan to back out my xmlunit patch!). The only annoying thing is that I started doing the Josh Bloch / Neal Gafter Java Puzzlers book while I was waiting for the approval to come through, so I want to finish that rather than having too many things on the go at one time. So I'll probably be a week before ramping up on unicodedata again.

Jython stalled

I haven't made as much progress with this as I would like, since I'm having some troubles. Not ones of a technical nature, but instead ones of a legal nature. My employment contract is apparently fairly standard and says that my employer owns all of my thoughts (even the ones when I'm writing this!). As what I thought was a courtesy, I asked my manager whether it was OK to contribute to open source projects, so that there would be no shady areas about who owned what code. Well, I'm still waiting for an answer and a bit of paper from my employer. So until I get that, I'm holding off writing any real code.

What I do have is a bit of sketching around the general area. First off, I wasn't sure about some aspects of the CPython implementation, so I asked. It got bounced by the python-dev moderator with the advice that I should post on the python-list. Which I did, but as I expected, it was more of a question for the python-dev list and the only (private) response that I've had is from Martin V. Loewis, who did the last change to that part of CPython. Maybe people are busy with PyCon. I was pointed at the C implementation, which doesn't generate that part from the UnicodeData.txt file, but instead is a horrible case statement, which looks bad. I think I need to raise a bug.

The other thing I've done with it is some Learning Tests about java.lang.Character, to see what it offers me. Obviously, this is attractive since it's a core library, is well tested, debugged and used by millions of people and all the other reasons Josh Bloch enumerates in Effective Java. It seems to have a few little idiosyncracies, which I have captured in my tests. (Note to self: I haven't seen any JUnit (or TestNG - Cedric!) tests in the Jython source tree. Must ask the dev list about that.) Then maybe I should check whether JRuby needs this sort of thing, and make it re-usable, with a Jython wrapper for the API that it needs, etc.

Jython unicodedata - how complete is java.lang.Character anyway?

Aye, there's the rub! My initial reaction to java.lang.Character is the extensive use of char in the API. That obviously wouldn't cover all of Unicode. So as a best scenario, maybe the BMP plus a bit can be covered by Character, and then something extra would need to be implemented to support the rest.

A related issue is that all of the nice int overloaded versions of the methods are Java 5, and that's not what I'm targeting here. I'm hoping to get away with a target environment of Java 4, since haven't Sun end-of-lifed Java 3? Java 4 Character only has implemented Unicode 3.0 anyway, so there's a bit of a gap. Python 2.3 contains version 3.2.0 of UnicodeData. There's going to be a gap that I need to fill somewhere.

Thirdly, the API that Character offers seems to be rather different from what I and Python has interpreted under the Unicode specification.

>>> unicodedata.digit(u'\u2468')
9

but then in Java:

assertEquals(true, Character.isDigit('\u2468');

fails. Closer inspection of the isDigit API documentation shows that this is in fact a test for is DECIMAL DIGIT, so it equates to unicodedata.decimal rather than unicodedata.digit. Hopefully, Character will allow me to get a fair way into implementing and making some of the tests run, before I have to start thinking too hard about creating lookup tables and bit-masking the 11 most significant bits.

Jython unicodedata initial overview

So it looks like the problem breaks down into creating a suitable data structure from the contents of the UnicodeData.txt file. CPython uses a python script (what else?) to create a C header file with the contents of various data structures. So all I need to do is probably one of the following:

1. Use the same approach, generate a very similar structure and port the existing C code that accesses the data structures into Java. Not very appealing.


2. Do something similar, and create a class for each code point. Probably not very good from a resource perspective (OK, potentially premature optmisation since I haven't measured it, but the UnicodeData.txt file is 817k, so that's some data structure). That could be useful from a LearningTest perspective though; e.g. can I use java.lang.Character, or do I need something else entirely.


3. Use something in existing core Java libraries.


4. Use a third-party library.


5. Something else, that I haven't bothered to think about what it could be.

Just one problem. I don't fully understand what is required yet. From reading the UnicodeData commentary, that indicates to me the reasons why the below tests are fine.


2468;CIRCLED DIGIT NINE;No;0;EN; 0039;;9;9;N;;;;;

(UnicodeData.txt entry for code-point 0x2468)

verify(unicodedata.decimal(u'\u2468',None) is None)
verify(unicodedata.digit(u'\u2468') == 9)
verify(unicodedata.numeric(u'\u2468') == 9.0)

and those tests pass (for CPython - I haven't implemented the Jython version yet!). From the file entry and commentary, that code-point appears to have no decimal digit value, a digit value of 9 and a numeric value of 9. The tests confirm that. I don't understand why these don't also pass.

325F;CIRCLED NUMBER THIRTY FIVE;No;0;ON; 0033 0035;;;35;N;;;;;

(UnicodeData.txt entry for code-point 0x325F)

verify(unicodedata.decimal(u'\u325F',None) is None)
verify(unicodedata.digit(u'\u325F', None) is None)
verify(unicodedata.numeric(u'\u325F') == 35.0)


The last one fails with:
Traceback (most recent call last):
File "", line 1, in ?
ValueError: not a numeric character

Evidently I need to delve deeper into the spec, or start asking more knowledgeable people some questions.

Jython unicodedata proceedings

So a little background as to why I'm doing this. Well, I don't know Unicode as well as I'd like, and I know Python a lot better than I know Ruby, so no temptation to start hacking JRuby at this point (well, maybe just a little).

I've implemented the methods that were missing and now I'm getting failures in the test. For the first implementation, I grabbed the existing Python source. Shit, C programming rots your brain. I learned C at Uni and then again via K&R, but this is a little different. But it's enough to give me the method signatures for everything that I need to stub.

*sys-package-mgr*: processing modified jar, '/home/jabley/work/workspaces/main/jython/dist/jython.jar'
Testing Unicode Database...
Methods: 38ef24ef104d52e24f9b7c942676c6961f9233cc
Functions: 97f3b4a034c7d9a0d0c1f387e216d6b8bf309442
API:Traceback (innermost last):
File "dist/Lib/test/test_unicodedata.py", line 91, in ?
File "/home/jabley/work/workspaces/main/jython/dist/Lib/test/test_support.py", line 125, in verify
TestFailed: test failed


Bit tired tonight (Connor's been throwing up the last two days), so I won't dig much into this. It feels slightly weird to be running the tests as python tests against a Java implementation, but that's what you get for implementing a library like this. No Junit / TestNG in sight. I have a feeling that it's going to require a lot of reading, which is good in that I might learn something, but I also wanted to get back to Stefan with a DocBook example for an xmlunit proposal.

Jython 101

Inspired by Joe Gregario, I thought I'd take a gander at Jython.
The proposed JythonSprint seemed to suggest that unicodedata was required, so I thought I'd have a play and maybe even contribute something. We'll see.

So, grab the main trunk from subversion and off we go. TDD all the way, as much as possible. Are you sitting comfortably? Then I'll begin...

Step 1 - create an ant.properties file. This is in .cvsignore, so no worries about clashing with anyone else's settings. Straight off the developer guide.

build.compiler=modern
debug=on
optimize=off

Step 2 - build it using ANT.

Step 3 - make it accessible. I don't have jython on my machine already, so I take the dirty approach.

sudo vi /usr/bin/jython
#!/bin/sh

export JYTHON_HOME=/home/jabley/work/workspaces/main/jython
exec java -Dpython.home=${JYTHON_HOME}/dist/ -jar ${JYTHON_HOME}/dist/jython.jar $*


Slight tweak from the development guide, and yes, I'm using Eclipse.

sudo chmod 755 /usr/bin/jython

Now see what it looks like:

jython
*sys-package-mgr*: processing new jar, '/home/jabley/work/workspaces/main/jython/dist/jython.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-1.5.0-sun-1.5.0.06/jre/lib/rt.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-1.5.0-sun-1.5.0.06/jre/lib/jsse.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-1.5.0-sun-1.5.0.06/jre/lib/jce.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-1.5.0-sun-1.5.0.06/jre/lib/charsets.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-1.5.0-sun-1.5.0.06/jre/lib/ext/sunjce_provider.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-1.5.0-sun-1.5.0.06/jre/lib/ext/sunpkcs11.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-1.5.0-sun-1.5.0.06/jre/lib/ext/localedata.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-1.5.0-sun-1.5.0.06/jre/lib/ext/dnsns.jar'
Jython 2.2b1 on java1.5.0_06 (JIT: null)
Type "copyright", "credits" or "license" for more information.
>>>


Cool.

Next run the tests.

jython dist/Lib/test/test_unicodedata.py
Testing Unicode Database...
Methods: 38ef24ef104d52e24f9b7c942676c6961f9233cc
Traceback (innermost last):
File "dist/Lib/test/test_unicodedata.py", line 84, in ?
ImportError: no module named unicodedata


That's expected. So I added a class org.python.modules.unicodedata and added an entry in org.python.modules.Setup to have a unicodedata module. I'll probably go back to the class name and maybe create it's own package longer-term, but that's the simplest thing for now. Tests again.

jython dist/Lib/test/test_unicodedata.py
*sys-package-mgr*: processing modified jar, '/home/jabley/work/workspaces/main/jython/dist/jython.jar'
Testing Unicode Database...
Methods: 38ef24ef104d52e24f9b7c942676c6961f9233cc
Functions:Traceback (innermost last):
File "dist/Lib/test/test_unicodedata.py", line 86, in ?
File "dist/Lib/test/test_unicodedata.py", line 62, in test_unicodedata
AttributeError: class 'org.python.modules.unicodedata' has no attribute 'digit'

I seem to recall reading about some script that will generate stubs for these things - gexpose.py? I'll have a look, but otherwise it looks like my next step will be implementing stubs for the required methods and see where the tests fail next.