Fixing the Zope release process

Zope feature releases

Let's look at the release dates of Zope feature releases.

Zope 2.6.0 was released in october, 2002. Zope 2.7 was released in late may, 2004. It is now march, 2005, and there is no Zope 2.8 release yet.

On the Zope 3 front, Zope 3 developed started in 2001. The first release of Zope 3, Zope X3 3.0.0 appeared in november of 2004.

Note bugfix releases of Zope do appear on a regular basis, most recently thanks to the wonderful efforts of Andreas Jung and others. We can however safely say we do not see a very regular cycle of Zope feature releases.

Zope release announcements

Let's now look at a brief history of Zope release announcements. Some of these were made in an informal mailing list setting, others during more formal settings such as a conference.

In november 2003, we got the following signal from Jim Fulton:

We want Zope 2.8 to happen as soon as possible, so we can start working on Zope 2.9. Zope 2.9 will be the release that incorprates major parts of Zope 3.

In december 2003, we heard more (mail to zope3-dev by Jim Fulton, about a sprint the following january):

Zope 2.9 will begin the transition to Zope 3 by making some features of Zope 3 available in Zope 2. This sprint will map out and begin to integrate these features.

On the Zope Development Roadmap, a presentation at PyCon 2004, in march 2004, the first release of Zope X3 was said be july of 2004 (with a question mark):

Final in July (?)

And Zope 2.8 is announced to be in quarter 3, 2004:

Q3 2004 (After X3.0)

For Zope X3.1, the release after Zope X3.0, we see the following on a wiki page:

Depending on community distributions, we are thinking about releasing X3.1 in December, 2004. This release will include several cleanups to the framework and feature the new Pluggable Authentication Service (PAS) and Workflow packages.

('distributions' is a typo for 'contributions')

In november 2004, we heard the following in the Zope 3 newsletter, from Jim Fulton:

After our initial release of Zope X3.0, we are beginning to contemplate an X3.1 release, possibly as early as January 2005.

Then, in early march 2005, we see the following post from Stephan Richter about the Zope X3.1 release:

Theoretically, this could be next week, but my experience from the X3.0 release tells me it will be more of the time scale of 2-3 months.

I wish I could give you a much narrower date range, but the community is currently too small to make sound predictions.

which puts back Zope X3.1 to may or july, 2005.

Something is wrong

The core software, Zope, that the Zope community relies on, is not evolving very quickly. In itself, too quick evolution of a mature system like Zope 2 is probably not very desirable. For Zope 3, at this stage in its evolution, a faster release cycle is desirable. In the case of both however, we need more regularity.

Why? Because irregularity and unpredictability discourage people from contributing to the Zope platform. Imagine you want to contribute a feature to Zope. You need this feature for a customer project to be released within a few months (which is the typical horizon for many projects in the Zope world). You think this feature is general enough to benefit many Zope developers, so you want to spread the benefit and the burden of maintaining it by contributing it to the Zope software itself.

But then you realize that adding this to Zope will not show up in any stable release for perhaps a year. Perhaps you can ask the lead developers of the project, but since they're wrong so often, you can't really believe them.

So what do you do? You decide not to contribute to Zope, but maintain your contribution yourself. Perhaps as a Zope product, or perhaps, worse, as a monkey patch. The community does not gain the fix, and you don't gain the input of others.

(ironically in the mail referenced I make a release prediction about what was to become Five which was way off, and I also claim I don't care about the Zope release cycle being slow -- I've since changed my mind)

Who to blame?

The easy thing is to blame the Zope core developers. But they're the ones actually doing the work, and doing their best, not paid by us, so that wouldn't be fair at all. When they make a release estimate, they're not lying; they make their honest best guess. So, Jim Fulton and all the others, thank you.

Who then? What do the core developers themselves say? Lack of resources is the most frequent explanation I see from them. If they had more resources, releases might come faster.

Since this is an open source project, a large resource is community contributions. And these are lacking. So the community is to blame itself; why whine about the lack of Zope releases? They should contribute!

We can't just turn around and blame the community however. For one, the community has in fact contributed significantly to development of Zope, for instance by hosting a whole series of Zope 3 sprints. There should be more community activity in the form of feature additions and bug fixes, but we've just given an important reason of why we're not seeing more of this -- lack of regular releases!

We're in a chicken and egg situation. Regular releases drive contributions, and contributions drive regular releases. We need to get out of this trap.

Wait a moment -- do more contribution in fact drive regular releases? No, not necessarily. The process needs to be in place to channel contributions the right way. Unstabilizing contributions shouldn't come just before a release, for instance, as in that case, the contribution will delay the release.

So let's not blame anyone; let's blame the process. Let's now look at how to fix it.

How to handle lack of resources

Lack of resources delay a release, but we also need to face it: there is always a lack of developer resources. Good developers experienced with a project are always scarce. Doing regular and predictable releases will hopefully attract more resources, but they'll never be sufficient, as human ambition always outstrips any resources

If you want to do a release, you need to manage these scarce resources and prioritize correctly, so that the release will happen. Without this management, developers will only tend develop more cool features, and never release anything.

Luckily, Zope isn't the first project with this problem. We have examples to study. One project that faced a very similar problem and solved it is the Gnome project. The problem and the solution were presented in this post in mid 2002, by Havoc Pennington. This quote applies to Zope as well as it does to Gnome:

There are two goals of a release strategy: to create stable releases, and to generate a lot of excitement and productive work that moves the software forward.

and this one does too:

If we get "stuck" on the stable branch and don't make the jump to unstable, then pressure builds to add features to stable, stable destabilizes, unstable stagnates and stops being usable because only a few people are using it, and it's all downhill from there. If we get stuck on the unstable branch, then we never provide anything to the majority of end users - we become useful to hackers/testers only.

Gnome has been on a time-based released plan since then, shipping new feature releases of the Gnome platform to end users regularly.

How to prioritize?

So, how does one prioritize matters to have releases happen on time? This is all well-known, though it can't hurt quickly repeating some concepts.

• announce a feature freeze and stick to it.

• compromise, defer issues aggressively so that they do not all have to be done for this release. During the development process, more issues than can be solved at the time will always occur. Someone needs to defer a lot them, however painful it might be. A good issue tracking system is very helpful here to keep track of what needs to be done.

• avoid bottlenecks. Avoid one person holding up a release because only they can fix a bug or do something essential in the process. It is impossible to always avoid this, but try to minimize this.

• Try to use standard procedures for releases as much as possible, so that it is easier for people to help out. If they need to jump through special hoops just for your project, they're more likely to give up, even if they're better hoops. The project innovates enough already in its core focus. YAGNI applies here.

• Do not depend on superheroic effort to make a release happen. Only very few people are actual superheros.

A special recommendation for Zope

One thing that repeatedly seems to have gone wrong with Zope in the past is setting a release date and sticking to it. I hope I have pointed out that this important. In my opinion, it should at times be more important than features, quality, or cleanups. By that I mean is that on a regular basis, the highest priority should be given to making a release happen on time, and everything else should be compromised to a certain extent to make this happen.

When you announce a release date, stick to it. Not announcing a release date or announcing it and then missing it by months are both bad options, as I hope to have explained convincingly in this article. Saying "When it's ready, it's ready", will inevitably delay releases almost indefinitely.

Hope for the future

Thanks to the efforts here at the Paris Zope 3/Five sprint, we may have given the final push to make the Zope 2.8 release possible. We hope to distribute Five along with it, which should finally help realize what was planned originally for Zope 2.9 in 2004.

For Zope 3, Jim has stated repeatedly that he intends to move to a time based release schedule. This is great! Jim, consider this article as a firm reminder. :)

I hope that our efforts here at the sprint will be followed up by a commitment of the various parties involved, such as the Silva, CPS and Plone developers, as well as Zope corporation to continue working together and to continue contributing on a regular basis. It's in all our best interest.