How the New York Public Library made ebooks open, and thus one trillion times better

Leonard Richardson isn't just the author of Constellation Games, one of the best debut novels I ever read and certainly one of the best books I read in 2013; he's also an extremely talented free/open source server-software developer who has been working for the New York Public Library on a software project that liberates every part of the electronic book lending system from any kind of proprietary lock-in, and, in the process, made reading library ebooks one trillion times better.

Richardson explained his project in exciting detail at Restfest 2015 in Greenville, SC in a talk called "The Enterprise Media Distribution Platform At The End Of This Book," and has posted his talk slides along with notes to his site. I've been discussing Leonard's ideas in light of the proposal for an open library ebook platform that I made in Locus magazine this past spring. Both of us see a nonprofit, mission-oriented infrastructure for ebooks as critical to serving patrons best while protecting their privacy.

Richardson's talk makes the point that in the age of the web, we had hypertext that acted like, well, hypertext. Lots of people contributed to the web in lots of ways, and all those ways joined up, more or less painlessly. In the age of the app, that is virtually unheard of, and when it does occur — as when Netflix and Twitter opened up APIs that turned into flourishing hothouses of third-party innovation — it gets shut down without warning and with extreme prejudice.

Richardson's system actually works: they're using it in NYPL and many affiliated libraries. It makes reading ebooks from the library one trillion times better, and it lets anyone improve it, at anywhere in the stack — it lets commercial suppliers play, too, but prevents them from locking libraries, publishers or readers in. It is a model of how mission-driven public agencies and nonprofits can be truly game-changing in online ecosystems that have been dominated by a single, monolithic corporation.

I'm going to start you off slow. Remember that there are three main vendors in the library ebook space. We did deals with two of them. Now that we've got the middleware in place, we can do a deal with the third vendor. We can license books from a third source without having to tell our patrons to install app #4 on their phones.

Okay, that's nothing to do with OPDS. Any kind of middleware would allow that sort of integration.

But then we decide we also want to offer Project Gutenberg books to our patrons. Unfortunately Project Gutenberg does not have an API. They have this ugly system where you have to use rsync to mirror the ebooks and then pull the metadata from a big RDF document.

So I write a simple content server, which rsyncs the ebooks and pulls the metadata and then offers a collection that is the equal, in quantity if not in quality, of the commercial collections. But instead of making up a custom API for my collection to talk to the middleware, the way the commercial vendors did, I use the API I already have—OPDS.

So now I'm using OPDS for machine-to-machine integration, not just to talk to the patrons. I can use this protocol whenever I am talking about books or collections of books.

Now other sources of free ebooks want to get in on the action. unglue.it is mostly an aggregator for Creative Commons books and other open-access books that aren't a hundred years old. Standard Ebooks is a little org that makes really nice editions of public domain ebooks, because Project Gutenberg ebooks have really horrible formatting.

So I told those people: you generate OPDS feeds, and I'll slurp them up into my content server, they'll show up in our collection and patrons will be able to download them. And that's what they did. I haven't set up my part of it yet, the part that slurps, because I haven't had time, but it's going to work.

At that point the OPDS protocol is doing machine-to-machine integration across organizational boundaries. It's hypermedia API heaven!