Skip to main content

Clarity, ClearSilver integration for Zope 3

I've just checked in a new package into the Zope 3 base subversion repository called Clarity. What it does is integrate ClearSilver templating into Zope 3 (trunk, though I expect Zope X3.0 or even Five support should be easy enough). It's all still rough, but initial tests show ClearSilver templates can be quite a bit faster than ZPT, and they have other possible benefits. In my simplistic experiments I got transaction rates about 2 to 5 times higher than you can reach with ZPT, testing this with the 'siege' utility.

Here's an extract from the README that describes the motivation for this experiment.

Why?

Zope 3 already has page template and even DTML, why would I want to use this? Some possible benefits ClearSilver brings are:

  • Absolute separation of logic from template. You cannot do complicated logic in a ClearSilver template, you can only extract data. You cannot call into the Zope 3 API at all.
  • Increased debuggability. Calling APIs is impossible from the template, which means that most bugs the API should appear in Python code already, where the programmer can worry about them, not the person who is tweaking the templates. While you are building a template you will never be bothered by, say, ComponentLookupErrors.
  • Increased testability. Since you know what HDF structure you expect to get in advance, you can test this using a normal unit test from Python code. Your template can stay simple.
  • Performance. Admittedly simplistic testing with 'siege' shows that for larger templates, ClearSilver templates integrated into Zope run several times faster than the equivalent ZPT page.

However, since no significant codebases exist that use this facility, these benefits are somewhat speculative.

More info and download

More about ClearSilver here:

http://www.clearsilver.net

A comparison with other templating system and a motivation for its design, which they call dataset driven templates, can be found here.

Among other systems which use ClearSilver templates is Trac, the Python-based software project management tool:

http://projects.edgewall.com/trac/

You can check out Clarity from the Zope 3 base, here:

http://codespeak.net/svn/z3/clarity/trunk/

lxml 0.5.1 released

I've just released version 0.5.1 of lxml, the Pythonic binding to the libxml2 and libxslt libraries. This because I got feedback to 0.5 which pointed to a critical bug in the way unicode was handled. This kind of feedback is why I should've released lxml long ago!

lxml 0.5.1, and more information, can be found here:

http://codespeak.net/lxml

lxml released at last

I've finally found the time to release lxml. So here then is lxml, release 0.5!

The very brief lxml FAQ

Q: What is lxml?

A: It's a Pythonic wrapping for libxml2 and libxslt

Q: What does that do?

A: It does XML, parsing it, writing it, manipulating it, searching in it (XPath), transforming it (XSLT), and validating it (Relax NG), plus more.

Q: But doesn't libxml2 already have a Python wrapper?

A: Yeah, but it's not pythonic at all. It's huge and difficult to comprehend, it's low level, you can get segfaults and you have to do your own memory management.

Q: Why isn't this in the Python package index?

A: I forgot my password, and resetting my password doesn't appear to work correctly. The error already turned out to be reported in the PyPI issue tracker on sourceforge.

What Zope can learn from Ruby on Rails

Ian Bicking just posted an insightful analysis of what makes Ruby on Rails work. What struck me most was the following:

Zope is also unique among web frameworks, and maybe even among Python projects, in that it's something where people choose Zope first, then Python comes along. With most other projects a developer chooses Python then finds a project in Python. As a result, Zope users are selected not for an aspect of the implementation -- Python -- but because they are specifically attracted to Zope and all its metaphors and design decisions.

This is what Ruby on Rails has. People are selecting Rails, and it just happens to be written in Ruby. This seems weird, because it's a developer framework and intimately tied to the language, but it's clearly what's happening. So as a result, the Rails community isn't going to fork as developers get frustrated with the specific metaphors it chooses. They are choosing the framework, not the implementation. In contrast Python web programmers have been choosing implementations, then seeing if they can accept the framework (and forking when they can't).

Zope has indeed mostly been picked historically for the platform and what is offers, instead of for the language it is implemented in. I think in many cases the decision to go with Zope was not a very technical decision, but more one based on community, visibility, and user-level features.

Zope has a user interface, for instance, and while it looks somewhat old fashioned now, this has been a feature that helped tremendously to sell it to non Python-programmer decision makers. Zope being backed by a company has also been important. Zope being extensible with all kinds of components (Products) that offer to add new functionality in a 'building block' way is also something that looks attractive. The ease of creating dynamic web pages using the web user interface also helped first impressions a lot; the amount of stuff you could do with a few simple (then) DTML pages impressed people a lot (and then the bad part of the Zope learning curve kicks in, but that's after the decision to adopt Zope has been made). That Zope is implemented in Python doesn't play a role for many decision makers at all.

Something that Ian doesn't say explicitly but that I think is very important in adoption patterns by the wider community is one of presentation. Ruby-on-Rails presents itself well. We (the Zope community) can learn from this. Just compare their website:

http://www.rubyonrails.com/

to the Zope one, for instance:

http://www.zope.org/

Zope's presentation is dull in comparison. Almost no images, and no hype.

Zope 3's web presence is even worse:

http://www.zope.org/DevHome/Wikis/DevSite/Projects/ComponentArchitecture/FrontPage

while zope.org is dull, Zope 3's presentation to the wider community positively sucks. Nobody is going to convinced to use Zope 3 by looking at that set of web pages. I know that wiki is not targeted to the wider community at all, but it's the only thing there is, besides the Zope 3 base, which does look somewhat more appealing:

http://codespeak.net/z3

Plone, in constrast, does this a lot better:

http://plone.org/

I do not think it is a coincidence that Plone does presentation well and that Plone took off as it did.

Zope is currently in a phase of transition from Zope 2 to Zope 3. It's fairly well understood in the community that Zope 3 has desirable technical merits; we do not need to convince Zope users of this. We do need to convince other Python programmers of this, but it's clear from Ian's analysis that this is an uphill battle.

If we want Zope 3 to be popular, what we need to do is learn the lesson of Plone, learn the lesson of Ruby on Rails, and do presentation well, and attract new people, from outside the immediate Python community, to Zope. It's likely we need other people for this than the Zope 3 core hackers. I'm not ideal either (just look at the 'design' of this blog), though I can help writing text. How to find the people who are good at this? I don't know, but if we run into some let's give them all the support they need.

Technical excellence and quality are important, but for this to be useful, we also need a community, and to gain a community, marketing can help a lot. So, let's think a bit about buzz, hype, and pretty pictures.

How to delay a Zope release

Jim Fulton, today, on the Zope-3 dev list:

Now that that the decision has been made to include Zope 3 in Zope 2.8, I'd really prefer that Zope 2.8 use X3.1 code, not X3.0 code. In general, having code shared by Zope 2 and Zope 3 will complicate deprecation, probably increasing the length of time we must keep backward-compatibility code. I'd like to try to keep the Zope 3 code used in Zope 2 and the Zope 3 code used in Zope 3 in sync as much as possible.

We've made more than just a decision. This week, about 5 to 10 of the best Zope developers in the world have been gathered here in Paris. Many of us have worked all day on Zope 2.8, and integrating Zope X3 into it. This is a quite significant expenditure of resources.

In this week, we've heard that:

  • really we should be learning zpkgtools, a system only used for Zope 3, so we can integrate Zope X3 into Zope 2.8
  • really there needs to be a ZODB 3.4 release. The ZODB 3.3 that Zope 2.8 is built for and revolved around is out of date.
  • and now, really there needs to be a new Zope X3.1 release before we can do a Zope 2.8 release. Note that Stephan Richter's best guess is that this is done in may or june or so.

Implications:

  • the Five hackers would have to adjust Five so it works with Zope X3.1. More dependencies on more people who may not have a lot of time.
  • the work we've done this week integrating Zope X3.0 into Zope 2.8 is basically thrown away. We can start over with Zope X3.1.
  • who knows what changes will sneak into Zope 2.8 and Zope X3.1 before june that will further delay matters.

There are always good technical reasons to delay a release. And the longer you delay it, the more good technical reasons come up.

If it is impossible to make a Zope release by throwing this amount of brain power at it, and this amount of time (and money; we're in business here), I give up.

Update: Jim again:

My intent is that X3.1 would be released at about the same time, as 2.8, so the X3.1 version of the software will be released. Of course, if it was in 2.8, it would be released. :)

...

OK, so maybe 2.8 should not include Zope 3 at all and aim for 2.9 in the same timeframe as X3.1. I doubt this makes much difference because 2.8 is not yet ready to go out.

Update: A clarification by Brian Lloyd cleared up the air a lot. We're going to stick with Zope X3.0 for Zope 2.8, which will make life a lot easier.

Fixing the Zope release process

In this article, I will identify problems with the Zope release strategy, attribute blame, propose solutions, and offer some hope for the future.

While this article contains criticism, I have great faith in the Zope developers, and I hope the criticsm will be considered constructive by those it targets. I also realize I'm part of this process myself -- this article is one way in which I try to participate.

Zope feature releases

Let's look at the release dates of Zope feature releases.

Zope 2.6.0 was released in october, 2002. Zope 2.7 was released in late may, 2004. It is now march, 2005, and there is no Zope 2.8 release yet.

On the Zope 3 front, Zope 3 developed started in 2001. The first release of Zope 3, Zope X3 3.0.0 appeared in november of 2004.

Note bugfix releases of Zope do appear on a regular basis, most recently thanks to the wonderful efforts of Andreas Jung and others. We can however safely say we do not see a very regular cycle of Zope feature releases.

Zope release announcements

Let's now look at a brief history of Zope release announcements. Some of these were made in an informal mailing list setting, others during more formal settings such as a conference.

In november 2003, we got the following signal from Jim Fulton:

We want Zope 2.8 to happen as soon as possible, so we can start working on Zope 2.9. Zope 2.9 will be the release that incorprates major parts of Zope 3.

In december 2003, we heard more (mail to zope3-dev by Jim Fulton, about a sprint the following january):

Zope 2.9 will begin the transition to Zope 3 by making some features of Zope 3 available in Zope 2. This sprint will map out and begin to integrate these features.

On the Zope Development Roadmap, a presentation at PyCon 2004, in march 2004, the first release of Zope X3 was said be july of 2004 (with a question mark):

Final in July (?)

And Zope 2.8 is announced to be in quarter 3, 2004:

Q3 2004 (After X3.0)

For Zope X3.1, the release after Zope X3.0, we see the following on a wiki page:

Depending on community distributions, we are thinking about releasing X3.1 in December, 2004. This release will include several cleanups to the framework and feature the new Pluggable Authentication Service (PAS) and Workflow packages.

('distributions' is a typo for 'contributions')

In november 2004, we heard the following in the Zope 3 newsletter, from Jim Fulton:

After our initial release of Zope X3.0, we are beginning to contemplate an X3.1 release, possibly as early as January 2005.

Then, in early march 2005, we see the following post from Stephan Richter about the Zope X3.1 release:

Theoretically, this could be next week, but my experience from the X3.0 release tells me it will be more of the time scale of 2-3 months.

I wish I could give you a much narrower date range, but the community is currently too small to make sound predictions.

which puts back Zope X3.1 to may or july, 2005.

Something is wrong

The core software, Zope, that the Zope community relies on, is not evolving very quickly. In itself, too quick evolution of a mature system like Zope 2 is probably not very desirable. For Zope 3, at this stage in its evolution, a faster release cycle is desirable. In the case of both however, we need more regularity.

Why? Because irregularity and unpredictability discourage people from contributing to the Zope platform. Imagine you want to contribute a feature to Zope. You need this feature for a customer project to be released within a few months (which is the typical horizon for many projects in the Zope world). You think this feature is general enough to benefit many Zope developers, so you want to spread the benefit and the burden of maintaining it by contributing it to the Zope software itself.

But then you realize that adding this to Zope will not show up in any stable release for perhaps a year. Perhaps you can ask the lead developers of the project, but since they're wrong so often, you can't really believe them.

So what do you do? You decide not to contribute to Zope, but maintain your contribution yourself. Perhaps as a Zope product, or perhaps, worse, as a monkey patch. The community does not gain the fix, and you don't gain the input of others.

(ironically in the mail referenced I make a release prediction about what was to become Five which was way off, and I also claim I don't care about the Zope release cycle being slow -- I've since changed my mind)

Who to blame?

The easy thing is to blame the Zope core developers. But they're the ones actually doing the work, and doing their best, not paid by us, so that wouldn't be fair at all. When they make a release estimate, they're not lying; they make their honest best guess. So, Jim Fulton and all the others, thank you.

Who then? What do the core developers themselves say? Lack of resources is the most frequent explanation I see from them. If they had more resources, releases might come faster.

Since this is an open source project, a large resource is community contributions. And these are lacking. So the community is to blame itself; why whine about the lack of Zope releases? They should contribute!

We can't just turn around and blame the community however. For one, the community has in fact contributed significantly to development of Zope, for instance by hosting a whole series of Zope 3 sprints. There should be more community activity in the form of feature additions and bug fixes, but we've just given an important reason of why we're not seeing more of this -- lack of regular releases!

We're in a chicken and egg situation. Regular releases drive contributions, and contributions drive regular releases. We need to get out of this trap.

Wait a moment -- do more contribution in fact drive regular releases? No, not necessarily. The process needs to be in place to channel contributions the right way. Unstabilizing contributions shouldn't come just before a release, for instance, as in that case, the contribution will delay the release.

So let's not blame anyone; let's blame the process. Let's now look at how to fix it.

How to handle lack of resources

Lack of resources delay a release, but we also need to face it: there is always a lack of developer resources. Good developers experienced with a project are always scarce. Doing regular and predictable releases will hopefully attract more resources, but they'll never be sufficient, as human ambition always outstrips any resources

If you want to do a release, you need to manage these scarce resources and prioritize correctly, so that the release will happen. Without this management, developers will only tend develop more cool features, and never release anything.

Luckily, Zope isn't the first project with this problem. We have examples to study. One project that faced a very similar problem and solved it is the Gnome project. The problem and the solution were presented in this post in mid 2002, by Havoc Pennington. This quote applies to Zope as well as it does to Gnome:

There are two goals of a release strategy: to create stable releases, and to generate a lot of excitement and productive work that moves the software forward.

and this one does too:

If we get "stuck" on the stable branch and don't make the jump to unstable, then pressure builds to add features to stable, stable destabilizes, unstable stagnates and stops being usable because only a few people are using it, and it's all downhill from there. If we get stuck on the unstable branch, then we never provide anything to the majority of end users - we become useful to hackers/testers only.

Gnome has been on a time-based released plan since then, shipping new feature releases of the Gnome platform to end users regularly.

How to prioritize?

So, how does one prioritize matters to have releases happen on time? This is all well-known, though it can't hurt quickly repeating some concepts.

  • announce a feature freeze and stick to it.
  • compromise, defer issues aggressively so that they do not all have to be done for this release. During the development process, more issues than can be solved at the time will always occur. Someone needs to defer a lot them, however painful it might be. A good issue tracking system is very helpful here to keep track of what needs to be done.
  • avoid bottlenecks. Avoid one person holding up a release because only they can fix a bug or do something essential in the process. It is impossible to always avoid this, but try to minimize this.
  • Try to use standard procedures for releases as much as possible, so that it is easier for people to help out. If they need to jump through special hoops just for your project, they're more likely to give up, even if they're better hoops. The project innovates enough already in its core focus. YAGNI applies here.
  • Do not depend on superheroic effort to make a release happen. Only very few people are actual superheros.

A special recommendation for Zope

One thing that repeatedly seems to have gone wrong with Zope in the past is setting a release date and sticking to it. I hope I have pointed out that this important. In my opinion, it should at times be more important than features, quality, or cleanups. By that I mean is that on a regular basis, the highest priority should be given to making a release happen on time, and everything else should be compromised to a certain extent to make this happen.

When you announce a release date, stick to it. Not announcing a release date or announcing it and then missing it by months are both bad options, as I hope to have explained convincingly in this article. Saying "When it's ready, it's ready", will inevitably delay releases almost indefinitely.

Hope for the future

Thanks to the efforts here at the Paris Zope 3/Five sprint, we may have given the final push to make the Zope 2.8 release possible. We hope to distribute Five along with it, which should finally help realize what was planned originally for Zope 2.9 in 2004.

For Zope 3, Jim has stated repeatedly that he intends to move to a time based release schedule. This is great! Jim, consider this article as a firm reminder. :)

I hope that our efforts here at the sprint will be followed up by a commitment of the various parties involved, such as the Silva, CPS and Plone developers, as well as Zope corporation to continue working together and to continue contributing on a regular basis. It's in all our best interest.

Five 0.3 released!

I'm happy to announce the release of Five 0.3 today, the Zope 3 in Zope 2 integration component. This release has great new features, including edit forms and add forms based on Zope 3 schemas. See the Five website for more information, and in particular the CHANGES.

Thanks to all the Five developers who made this release possible!

Zope 3 will support XML

Paul Everitt asks Should Zope 3 support XML?. The answer is of course, yes, it should, and, what's more, yes it will, and Infrae will be one party working on it. In fact, we've already worked on it.

Paul is right in that many Zope 3 developers, and many Python developers in general, don't give XML a high priority. I wrote about some of the possible attitudes surrounding this before.

I am involved in Zope 3 development, and I care deeply about XML support, and have been thinking it over for, well, years now. In Silva, we have been adding more XML support with every release, and are going to release more soon, so we now have quite an extensive background in these matters too. We'll be looking into porting this functionality onto the Zope 3 platform.

Infrae and Nuxeo recently announced that we will be working together on Zope 3 development. XML support will certainly be a part of this cooperation.

What shape this XML support should take in Zope 3 needs to be debated. It might be in the core, it might make more sense as an extension. It might be a combination of both. One way or another, Infrae be helping to develop it, and our aim will be to make the functionality attractive enough so that anyone who wants to do serious application development with Zope 3 will be tempted into using it.

A CMS as a pile of semi-structured data

Paul Everitt and I have long been communicating about the role of XML in the CMS world. Recently, he posted the following blog entry and asked for my opinions per email. I started writing a mail back to him, but then I realized I have a blog now too...

Firstly, I want the UdellCMS too. :) Infrae has been trying to build one for a while, staggering along the way.

Now as to some comments to Paul's vision. Within Infrae I have the reputation of being a devil's advocates, and I've done it to Paul before, so here goes with my comments.

Paul mentions, referring to the demo he links to, that:

it isn't a "programming language" (though it sure looks like one)

This one is never going to fly. It's a programming language all right, though a highly declarative one. It's going to take someone with programming skills to write these. The advantage, similar to SQL, is reporting: you do not need to know a lot of APIs in order to extract data out of the system.

Now to what Paul and I both agree is the more interesting bit:

What's more interesting, IMO, is the effect this has on CMS design. It totally changes your approach to navigation. Instead of thinking very hard about folder structures or topic structures, you just throw everything into a big pile and let stored semi-structured and full-text queries create smaller piles. This allows numerous approaches to site navigation.

I think that this approach opens more possibilities than some traditional CMS approaches have been taken, because of its non-API oriented query/reporting nature. This also ties in to various content repository projects, such as Infrae's Railroad and content repository APIs like JSR-170.

But, I also think that you cannot just give up (up-front) thinking about things like folder structures or topic structures, or structured document content, for a number of reasons.

While the files in the referenced demo are perhaps in one big pile, these files contain a lot of structure that can be exploited. It's just not represented as folders and topics. Someone has to put this structure in there.

How will you get (non-computer-savvy) people to produce such structured content? I.e. how do you get enough consistency to actually be able to do smart queries like the ones demonstrated? Jon Udell alone can be trusted to produce semantic XHTML, but a whole organization? With Silva, we took a lot of care so that we get some semi-structured content out of it all (Silva XML).

In addition, organizations use structuring techniques like folders and (mandatory) metadata to have something to hold on to and some coherence in the produced content. You can use folders for authorization, for instance. You also need some uniformity in the topics used by people, and often organizations want to mandate this.

So, some structuring facility seems unavoidable in a CMS. Wikipedia has shown that even minimal facilities can lead to grand things, though, but wikis aren't right for all use cases.

Where the real merit lies is in giving up some APIs and going more to declarative data. In one way that's just moving the problem. But in another way, what you can gain is that you can move some CMS-style tasks away from a potentially limiting set of APIs to a more powerful query & reporting model. In order words, you manufactor serendipity; you make it easier to reuse content in new ways. It all being done in a standardized way (the host of XML standards) is cool of course.

We've been doing things like this at Infrae and with Silva for quite a while now, and we're trying to open it up more. It's not easy and the benefits are sometimes hard to see, but we're trying. But we can extract PDFs and Word documents from Silva, and with custom apps on top of Silva we can expose and relate data in a lot of interesting ways. When you start turning it into a real applicat And we'd like to do more in the future, as we're opening up Silva to accept more kinds of XML content.

Criteria for evaluating specifications

As Andrew Tannenbaum said, "The nice thing about standards is that there are so many to choose from." Apparently he followed this up by: "And if you really don't like all the standards you just have to wait another year until the one arises you are looking for."

So how does one choose? Why adopt one specification and not another? There are many possible reasons, and this article is an attempt to catalog some of them.

In the course of my work at Infrae, I run into many specifications that we may decide to implement or use, so having a common vocabulary and set of questions about this is important to us.

Note that I use the word 'specification', not 'standard' in this article, as there are far more specifications than true industry standards. Many so-called "standards" are only aiming to become so, but naturally many fail in this process due to a variety of reasons.

What is specified?

Specifications fall into a number of categories. A non-exhaustive list:

  • protocol (TCP/IP, HTTP, FTP, XML-RPC, SOAP, WebDAV)
  • document format ('pure' XHTML, Docbook, TEI, many custom XML document formats)
  • visual design ('pragmatic' XHTML, CSS, SVG, XSL-FO)
  • metadata (RDF, Topic Maps, RSS, Dublin Core)
  • query/programming language (SQL, XPath, XSLT, XQuery, C, C++, Java, C#, etc)
  • API (DOM, SAX, COM specification for MS Office interoperability, UNO specification for OOo API)
  • structural, foundational (XML, RDF)
  • prescriptive/descriptive (DTD, XML Schema, Relax NG, RDF Schema)

Questions to ask when encountering a new specification

Industry adoption

How well is the specification adopted within the industry? I.e., is this specification industry-standard or seems to be on the way to be so? There are a lot of specifications that are really not standards. Industry adoption has two sides to it:

  • Systems that implement the specification.
  • Systems that actually make us of the specification implementation to accomplish specific use cases.

If there are few of the former, there won't be many of the latter. An example is SVG in browsers. Even though it is expected that the uptake of SVG technology will be big once it is established in popular browsers, this uptake is not taking place right now as the browsers don't support it very well. In other areas, like the Linux desktop, SVG is seeing some use now.

Even though systems that conform to the specification may be implementable fairly easily, it may be that few systems exist that actually use this for useful purposes. RDF has a problem like this; even though it is relatively easy to produce some RDF, and tools exist that can then take this RDF and do some analysis, there are actually few systems that use this RDF to fulfill practical use cases. RSS is an exception, but only a partial one, as only few RSS streams are actually fully RDF compliant.

Implementation quality

How well is the specification implemented within the industry? If a specification is well implemented, this helps interoperability between software. CSS is an example of a specification that is implemented commonly by browsers, but not all to the same level of quality. The practical result of this is that the same CSS needs to be tested on all browsers, and pragmatic subsets and tricks are developed in order to work around limitations in the implementation on some platforms.

The DOM specification has been implemented quite a few times in Python, but very little Python software actually works across Python DOM implementations, because the implementations differ a lot in quality, supported feature set, and there is a lot of inconsistency in the translation to Python of the various DOM APIs. In contract, because of more rigidly defined and checked interfaces in Java, this language gains more interoperability between its various DOM implementations. Another example is Javascript-based browser DOMs. While many inconsistencies exist, the browser DOM still allows cross-browser applications to be developed.

XSLT is an example of a specification that seems to be well implemented across many platforms, probably due to the clarity of the specification, the limited scope, and extensive conformance test suites.

Market demand

How much does the market (or a particular market, or a particular client) want to adopt the specification as a standard?

Even in cases where industry adoption is still limited few implementations exist, significant forces in the market may still very well want implementations. They may want this for rational reasons, or due to industry hype, or a combination of both. One good reason for adoption of a specification is its popularity, so industry hype may in fact be a worthwile reason to adopt a specification.

Hype may exist in limited markets. Most people for instance haven't heard of SCORM, but it does have some buzz in the education market.

Buzz

How much buzz is there in the industry and market? Are vendors promising support? Are there high visibility projects? Glowing magazine articles?

Suitability

Does the specification fit the actual problem domain? XSLT can be used well for templating and transformations, but it's not very well suited for writing whole applications.

Even if a specification is not optimally suited to a problem domain, other factors may still cause it to be adopted -- increased interoperability for instance, or reduced learning time.

Implementability

How easy is the specification to implement? There are a whole set of reasons why a specification would be easier or harder to implement.

  • How clear is the specification? Is the specification very readable by a software developer?
  • How complete is the specification? Are there areas essential for interoperability which are glossed over?
  • How big is the specification? If all other factors stay the same, the larger specification is harder to implement.
  • How conceptually pure is the specification? Does the specification specify a hodge-podge of features or is everything reducible to basic principles? The more pure a specification is, the easier it tends to be to implement.
  • Do test suites exist to verify conformance? How extensive are these test suites? If these are lacking, successful implementation becomes a lot harder.
  • Do proof of concept implementations to server as developer examples? Often an implementation in software actually clarifies many of the details that may otherwise be left undescribed.
  • Does the specification have dependencies on other ones? A specification may have a requirement on one or more other specifications that are not commonly implemented.
  • Is the specification tied to a particular operating environment or programming language? If there are many requirements before a specification can be implemented, this may limit the implementability.
  • On the other hand, a specification that tries to be very general may turns out to be harder to implement as well, as it may be more complicated as a result.

Interoperability

Will implementing a specification in fact increase interoperability?

There are a number of categories of interoperability:

  • Interoperability between client and server, swap out server. Here the server implements the specification (and the client makes use of it). As an example, HTTP makes it easy to swap out one webserver for other one, while the same client (a browser) is used for both.
  • Interoperability between client and server, swap out client. Here the client implements the specification, and the server produces something that fits the specification. HTML and CSS make it easier to swap out the client (browsers such as Mozilla, IE, Konqueror, Opera) as each implement HTML and CSS, even though the server may always be the same.
  • Interoperability between "peers". XML makes it easier for one system to read basic structured information coming from another, as they can make use of well established XML processing tools.

The more difficult it is to implement a specification (correctly or completely), the fewer the interoperability benefits.

Learnability

Will the implementation specification likely make it easier for the developer to learn the system (or learn how to interoperate with the system)?

  • If the specification is popular, it's more likely there's plenty of reading material about it that the developers heard about. There's also a bigger chance the developer already heard about it, or used it in the context of another system. XML is the traditional example of how this works. XSLT is another good example, though only in the last few years, now that it has been commonly implemented and much documentation exists.
  • Implementability and learnability are related. If the specification is for some reason difficult to understand (difficcult to read, hugely complicated, badly written), it will be more difficult for a developer to understand it in other to use it.

If only everybody

How much of a network effect is needed to make implementing or using a specification useful? If first everybody needs to implement the specification to actually make its use worthwhile, a chicken and egg problem exists.

An example of this is the semantic web. If only everybody had their web systems produce large amounts of semantic information in the form of RDF, it'd be very useful in many ways and useful tools would be created which process this information. Unfortunately nobody is doing this as there are no tools to process the information.

Reasons for usefulness

Specifications can be useful for a number of reasons:

  • Interoperability. This is the obvious one. Your system can use another system that implements the specification, or vice versa. Implementability is the main factor that determines whether this can be accomplished, and popularity determines whether this is worthwhile (see also tool leverage).
  • Attracting developers. We expect many developers to be already be familiar with the specification, or alternatively, there are copious resources for developers to familiarize themselves with a specification. Implementing the specification makes our system easier to learn and develop for. Learnability and popularity are important factors here.
  • Reuse of design. Can we take useful ideas from a specification? Even if an interoperable specification is not developed, we may still be helped as we may not have to reinvent the wheel. Learnability is the most important factor here.
  • Tool leverage. By implementing the specification, other existing toolsets and libraries can be used to extend the feature set of the software. Important factors here are popularity; lots of other systems that build on top of this specification will exist that now become useful.
  • Marketing. Implementing the specification will provide us with buzzword compliant "checkbox features" for marketing. This will be helped by popularity (at least perceived popularity). The specification itself might be pure nonsense, or our implementation may be bad, it can still work, at least for a while until everybody finds out.