You are browsing the archive for Open Knowledge International.

Improvements to the Concordance

Open Knowledge International - January 3, 2007 in Technical

One of the main items scheduled for v0.4 of open shakespeare is improvements to the responsiveness of the concordance. Using the v0.3 codebase, using just the sonnets as test material, loading up the list of words for the concordance alone took around 24s on my laptop. This is because even with a single text there are already over 18,000 items in the concordance and we were having to read through all of these to generate the list of words. Some recent commits (e.g. r:72) have gone some way to improving this responsiveness (loading word list is now 3s now compared to 24s) but the result is not entirely satisfactory (printing full statistics is 13s compared to 40s previously). One obvious way to go futher is to use caching — either of individual web pages or of particular key parts such as all the distinct words occurring in the concordance (caching works because the concordance only changes when new texts are added which will usually only happen once — when the system is first initialised).

Relatedly and r:74 is a first step on filtering the concordance — in this case to exclude roman numerals and various non-words. Doing this made me think about whether the concordance should be storing actual words or just stems — for example, it does not seem to make much sense to have different entries for kill, kills, killed etc. Using a stemming algorithm such as the porter stemmer (which I notice has a nice python implementation directly available) we can easily stem each word as we go along. This would have several benefits one of the most prominent being a dramatic reduction in the basic dictionary size (i.e. the number of distinct words in the concordance).

Adding Web-Based Annotation Support

Open Knowledge International - December 18, 2006 in Technical

We intend to add annotation/commentarysupport to the open shakespeare web demo either in this release or next. As a first step we’ve been looking to see what (open-source) web-based annotation systems are already out there. Below is our list of what we’ve been able to find so far (if you know of more please post a comment). After examining several of these in some detail the one we’re going to try our properly is marginalia (if you’re interested our current efforts to do this including writing a python wsgi annotation service backend can be found here in the subversion repository).

  1. stet: javascript annotation system used for gpl v3 comments system

  2. commentary: javascript based wsgi middleware developed by ian bicking

    • Rather hacked together (apparently he coded it in a week). Had problems getting it working locally and no documentation to help in adaptation. Seems to be unmaintained (demo site is currently down) which is perhaps not surprising given how many other projects Ian has on the go.
    • One nice feature is that you don’t seem to have to mess with the underlying web pages you want to add comments to (this only works if you are sitting on top of another wsgi application)
  3. marginalia: javascript library and spec for adding web annotation to pages

  4. annotea: W3C project based on RDF

    • Been around a long time and now seems to be inactive
    • Server and client support rather lacking. No simple interface based on, e.g., javascript — you have to write a special client yourself — which is a major drawback
    • That said the protocol is well-documented and so writing a client (or a server) shouldn’t be that hard (other than having to mess around with rdf in javascript …)
    • The Schema seems reasonable
    • xpointer based which according to the marginalia site is a problem

Switch from Kid to Genshi for templating in the Web Interface

Open Knowledge International - November 4, 2006 in Uncategorized

Today we made the switch from kid to genshi as our templating toolkit in the web interface. Kid has served us well but there are some issues with debugging and including input that can’t be guaranteed to be well-formed. Genshi, as a direct derivative of Kid, delivers very similar syntax but is both simpler and a little more flexible to use.

Does an ‘open’ scan of a shakespeare folio exist?

Open Knowledge International - October 15, 2006 in Uncategorized

We’d really like to have some nice images of a shakespeare first folio (if possible from Hamlet) for use in the Open Shakespeare project. However all the scanned copies we’ve managed to find seem to be under full ‘all rights reserved’ copyright.

For example there’s an online version from the Schoenberg Schoenberg Center for Electronic Text and Image at the University of Pennsylvania. But checking the printable version one finds the following:

©2003 Schoenberg Center for Electronic Text and Image

University of Pennsylvania Library.

And this isn’t exceptional. There’s a list of available online folios on:

All of the copies listed are closed (copyrighted with no open license) — with most not allowing for any types of use without permission (the only exception being the State Library of New South Wales which allows for “educational, non-profit, purposes”).

It’s a rather unfortunate situation and it would be great to know if there is a scan of a shakespeare first folio out there which truly is open.

v0.3 of Open Shakespeare Released

Open Knowledge International - October 4, 2006 in News

A new version (0.3) of open shakespeare is out. Get it via the code page:


  1. Can now view mutiple texts side by side (ticket:15). See it in action at:

  2. Now include moby/bosak versions of shakespeare as well as gutenberg (ticket:10) (though more work remains to be done to process these versions to plaintext and html)

  3. Fix bug whereby we were missing some of the available gutenberg texts (ticket:18)

  4. Install the shakespeare python package (ticket:16)

  5. Move to py.test from unittest

  6. New project website at

Outstanding Issues

  1. Several of the source texts (all of them Gutenberg folios) seem to break the viewer due to kid (the templating system) complaining about about ‘not well-formed (invalid token) xml’. Any help in tracking this down would be greatly appreciated.

About Open Shakespeare

A full open set of Shakespeare’s works along with anciallary material, a variety of tools and a python API.

For more information see the about page:

Mailing list:

Open Shakespeare v0.2

Open Knowledge International - July 15, 2006 in News

With a little bit of free time over the last couple of weeks I’ve managed to do some more work on open shakespeare. The new version (v0.2dev) is up and running on the site:

NB: concordance only includes sonnets (this is not a necessary restriction but saved on concordance build time)

Many of the improvements in this iteration are internal and will make future work faster and easier. More details on the changes can be found below.

Any and all feedback most welcome and if anyone wanted to start hacking away with me that would be fantastic (there is now a trac installation to assist with this — details below).

Main improvements

  • move away from gutenberg-centric setup present in v0.1
    • will now be simple to add new material
  • using domain model and database backend
    • much more flexible concordance with faster creation
  • web interface improved
    • concordance now provides snippets and link through to sources

Trac Installation

There’s now a trac installation for project management:

For latest developments check out the timeline:

All the TODOs are now tickets. Active tickets:

A roadmap with links to current future tasks: