Skip to main content

Zope Foundation in the Google Summer of Code!

I'm happy to announce that the Zope Foundation was accepted by Google as mentoring organization in the Google Summer of Code! This means we're looking for students and more mentors who would like to take on projects, as well as good projects. We have a wiki page about it with project suggestions and prospective mentors; if you're interested in this project, please check it out.

If you're an interested student or mentor, please also join our mailing list and discuss your ideas here.

Grok Sprint Zwei: the Ascent of Man

Yesterday I returned from "Grok Sprint Zwei", the second grok sprint, hosted by Philipp von Weitershausen in Dresden, Germany (and partially at Gocept for the warming up). Grok is a project to make Zope 3 safe, easy and fun for cavemen and other hominids like ourselves. Zope 3 of course is the powerful and flexible framework for the construction of web applications. See here for my initial introduction of the Zope Grok project.

To summarize, Grok is an advanced alternate way to construct Zope 3 based applications that makes it possible to use the powerful Zope 3 component architecture (the glue that binds Zope 3 together) without having to write lots of configuration code. Instead, we apply "convention over configuration" to Zope 3. We aim for two things:

  • developers can write simple Python code to make Zope 3 applications
  • developers can still fully exploit the power of Zope 3 (object database, pluggability, authentication, virtual hosting support, etc, etc)

Besides a project, Grok is also a caveman. Here he is again, relaxing after his work at the sprint:

http://faassen.n--tree.net/grok_relax.png

We had five people participating in the sprint. Everybody present at the first sprint was back: Philipp von Weitershausen, Christian Theune, Wolfgang Schnerring and myself. New at this sprint was Jan-Wijbrand Kolman. Jan-Wijbrand and I go back a while by now; I've had the pleasure to work with him for some years. More recently, after the first Grok sprint, we have been doing regular work on Grok and Grok applications. I was particularly happy he could make it to the second sprint, as he has lots of experience with usability, and Grok is a (developer) usability project.

The second Grok sprint was a success: we set ourselves goals and we accomplished them. We also played lots of Guitar Hero in the evening. :) The major new things in Grok after this sprint are:

  • security declarations for views. Views are public by default but you can now restrict access using permissions. This allows Grok to make use of Zope 3's advanced authentication/authorization support.

    In Grok, we deliberately turn off security for anything but views (only security where your app faces the web), as we noticed that the standard pervasive Zope 3 security proyxing model can hinder rapid application development.

  • local utility support. Local utilities can be used to do context-specific configuration and storage (such as the storage of an index). They are equivalent to CMF's tools. Utilities are looked up by interface which makes it easy to plug in new ones implementing the same interface. We now have an easy way to use local utilities.

    Concretely, this means Grok applications can now easily use powerful Zope 3 components such as the Zope 3 catalog for indexing of objects.

We think Grok now has enough features to start developing real world applications. We could of course only go so quickly as we're exposing existing Zope 3 features.

Since we're now ready with features, what remains to be done? One thing we are going to continue to do is build real-world applications with Grok. This will help us polish Grok further.

One of the main areas that is still lacking is documentation that tells you how to use Grok, so we will be working on this now.

Another aspect we want to work on is a focused admin UI for Grok applications. This is explicitly much more narrowly targeted than the Zope Management Interface. The admin ui is aimed at developers and admins to install and inspect Grok-based applications.

We hope to do an initial release of Grok within a few weeks. We are also planning another sprint in april, where we plan to make Grok ready to exit the cave and step into the wide world, club in hand and smile on his face - ready for the ascent.

Grok, CherryPy, WSGI and Zope 3

Yesterday night I experimented with getting Zope 3/Grok to work CherryPy, through WSGI. This led to all kinds of interesting adventures and opportunities. Follow the path I took:

This started as I had been doing some work with CherryPy recently. I like what I see of CherryPy so far; a nice, Pythonic and powerful web server.

Currently Zope 3 uses Twisted as the default web server implementation. Using Twisted has lots of advantages. It's a high-quality framework with a lot of features. It's also maintained by people other than the Zope developers, which is good. Since Zope was one of the first in the Python web space, the Zope developers still maintain a version of Medusa for Zope 2 called ZServer. It made sense to do that at the time, but it doesn't now.

So if Twisted is so nice, why look at CherryPy at all? Because it's nice and because it's fun to try would be reasons that are good enough, but there are some other reasons that make CherryPy interesting for Zope too:

  • CherryPy supports threaded webservers in a non-begrudging manner. Since Zope uses a threaded model for web serving, that might be a nice match. Twisted supports threads just fine too, but when you do, one always gets this idea that thousands of Twisted developers screaming at you just outside the range of your hearing.
  • CherryPy is committed to supporting WSGI, the framework for Python web interoperability. Zope has committed to using WSGI as well.The Twisted community sometimes seems to be a bit more half-hearted about supporting this.
  • CherryPy installs cleanly as a Python egg. This means I can use setuptools and buildout easily. When I asked a few months ago I got the impression the Twisted developers don't like eggs very much and certainly have no plans to support them any time soon. Since Zope is moving towards using eggs, CherryPy is interesting.

Then I got slightly side-tracked. I figured that the easiest way to integrate Zope 3 with CherryPy was to use CherryPy's WSGI server and use Zope 3 as a WSGI app, which Zope 3 has support for. This gave me a nice opportunity to play with WSGI for a bit. Now CherryPy's WSGI server is not actually CherryPy's server, so there goes my goal of integrating the whole of CherryPy... I heard a few good things about CherryPy's WSGI server too though, and it was easier to do, so I tried that.

This became a nice opportunity to play with some other things. One of the reasons I wanted to try running Zope 3 and Grok with CherryPy is that it would allow me to experiment with what the minimal Zope 3 configuration of Zope 3 would need to be. Grok comes in nicely here: aiming for Grok support made that minimal configuration quite well-defined. I didn't need to aim to get Zope 3's own management UI up, just to get the basic Grok functionality to work.

One of my aims is to make Zope 3's startup procedure a lot faster when you're doing Grok development. Another aim is to be prepared for the time when Zope 3 comes as a whole bunch of eggs: we can just use the eggs that Grok needs and not anything else. Combined with buildout this makes for an easy and powerful development and deployment system.

So those are my aims and motivations - how far did I get? Quite far: I got a basic Grok view working with CherryPy's WSGI server when I stopped. Not needing the whole of Zope 3 to load up, the server also starts a bit faster, though not as fast as I'd like yet. Here it is:

http://svn.zope.org/megrok.cherry

There are some interesting parts. This is the site.zcml that includes the smallest minimum of ZCML I've managed to get to yet:

http://svn.zope.org/megrok.cherry/trunk/site.zcml?view=markup

This is the WSGI-based server that hooks Zope 3 as a WSGI app into CherryPy's WSGI server:

http://svn.zope.org/megrok.cherry/trunk/src/megrok/cherry/server.py?view=markup

Not a lot of code, thanks to WSGI!

This shows that Grok works at least minimally; we're registering a view for Zope 3 containers (such as the root folder):

http://svn.zope.org/megrok.cherry/trunk/src/megrok/cherry/demo.py?view=markup

A lot more work needs to be done if this wants to become anywhere near production ready. This code right now is just an experiment. I just thought I'd get my thoughts out there, and see who's interested.

Since it's all a buildout, you can install it safely and if you're on a sufficiently Unixy machine, make it work:

$ svn co svn://svn.zope.org/repos/main/megrok.cherry megrok.cherry
$ cd megrok.cherry
$ python2.4 bootstrap.py
$ bin/buildout
$ bin/startserver

This starts a server running on port 8080. It doesn't do much yet, but what it does is allowing you to access:

http://localhost:8080/test

This is rendered by the demo.py referenced before.

Let me know what you think!

Grok: or what I did on my holiday

I will use this blog entry to talk talk a bit about my holiday in Germany a few weeks ago. I went to a mini sprint at Gocept, in Halle. I had a great time! (yes, I am a geek) This sprint showed that good sprints aren't necessarily the ones with many people participating; we just had 4 sprinters (and less much of the time), and this was one of the most productive sprints I've been at for years. I left the sprint energized and excited. Thanks to Gocept for organizing this, it was awesome!

So what did we do on this sprint? You might have heard the news already here and there: we worked on Grok.

This is Grok:

http://faassen.n--tree.net/grok.jpg

Grok is a caveman.

Grok talks like this: ME GROK SAY HI!

We all have a bit of the caveman or woman in us, although any resemblance of Grok to me is solely in your imagination and a pure coincidence and has absolutely nothing to do with the fact that the artist who drew Grok is my wife. Cavemen, like us programmers, want powerful and flexible tools. Cavemen are great at tools after all; they practically invented tools. But cavemen, and we, also want our tools to be simple and effective.

Simple and effective like clubs: a club is powerful, flexible (you can bash in anything, mash potatoes too) and also simple and effective. Zope 3 is powerful and flexible all right, but we need it to be simpler, thus making it even more effective. This brings me to the Grok motto: Grok -- now even cavemen can use Zope 3.

Besides a caveman, Grok is also a piece of software, and a project. The project that started at this year's pre-EuroPython sprint with some of my musings about improving Zope 3's development experience. I wrote some of them these ideas down on a hot day at CERN in Geneva when I found out I had been the first sprinter to arrive. These musings turned into a discussion at the sprint. The discussion with the people there (among others Jim Fulton, Steve Alexander, Philipp von Weitershausen and Uwe Oestermeier) helped my thinking quite a bit.

The basic idea behind Grok is to provide an alternate way to configure Zope 3 applications: it doesn't use ZCML. Instead, configuration can be triggered from Python. The idea was to apply a few design guidelines like Don't Repeat Yourself (DRY) and Convention over Configuration, concepts popularized by Ruby on Rails.

For months I didn't have the opportunity to do much with these ideas, until I spoke with Christian Theune of Gocept at the DZUG meeting in september. He recently had been confronted with the complexities of Zope 3 in a new way: a very smart programmer, Wolfgang Schnerring, was doing an internship at Gocept. In the course of this, we was learning to develop with Zope 3 and got quite frustrated. We don't want to frustrate smart programmers if we can avoid it. Good programmers are in rare supply.

We started to talk about Grok, and Christian got quite interested, so we organized for me to visit Gocept and work with Christian and Wolfgang to try to make some of my vague plans for Grok a reality. Philipp von Weitershausen heard about these plans and we were happy to see him join us. This completed our core team. Grok is our joint effort.

The first day started with Philipp and me arguing quite aggressively on what Grok's design could be like, somewhat intimidating poor Wolfgang. He was happy to find out we had just been warming up for the real work, and by late afternoon all four of us were making great progress sketching out Grok's design.

After doing that design work we started implementing Grok the next day. We practiced quadruple-programming: one person typing, the rest of us giving the long-suffering typer (initially Philipp) our kind directions. The code in the editor was projected with a beamer. This worked a lot better than one would expect: we used the time effectively planning out our testing strategy (we practiced test-first programming), worrying about design and edge cases, etc. We seem to have hit a very effective combination of personalities in Christian, Philipp, Wolfgang and myself.

In fact it worked astoundingly well; for the first day of checkins Philipp might've looked from the outside like he was a "perfect programmer", checking in incremental improvements to Grok without any hestitation about the design, going straight for his targets, extensive testing, the works. Of course after a while we did hit some design issues we retreated from later, but overall we could move very fast. After the sprint I realized I'd actually not typed a letter of actual Grok code even though I knew the codebase very well as I'd seen all of it on the screen (since I've been back I've of course been checking in stuff by myself). The code was better than I could've written just by myself and I was entirely comfortable with it.

In the evenings we ate a number of delicious meals and played Guitar Hero on Christian's Playstation, where we discovered that I'm a guitar caveman and that Christian is a guitar god among otherwise quite respectable guitar heros.

So what does Grok-based Zope 3 code look like? I'm still working on a tutorial, but here is a trivial example of a complete Grok application:

import grok

class Document(grok.Model):
    def __init__(self):
        self.text = 'ME GROK SAY HI'

class Index(grok.View):
    def render(self):
        return "The text is: " + self.context.text

ME GROK NO NEED ZCML

Grok fits my brain and I hope it will fit yours. Grok is entirely compatible with Zope 3; all the power of Zope 3 is in there. We're not changing the way Zope 3 works; we're just making it easier to use. Grok combines the flexibility of a swiss army knife with the great usability of a club.

To see a bigger example of Grok-based code, please take a look at the Grok-based wiki implementation. As a small note of warning: the Grok wiki is not intended to be a pure example code-base; we're using it as a Grok test-bed application so it is subject to change.

You can get Grok and run Grok yourself (along with the wiki and some other test-bed apps). Thanks to the wonders of zc.buildout it's actually very easy:

$ svn co svn://svn.zope.org/repos/main/grok/trunk grok
$ cd grok
$ python2.4 bootstrap/bootstrap.py
$ bin/buildout

This will download and install Zope 3 (please wait a while) and will make an instance with Grok and GrokWiki in it. To try the GrokWiki, first start zope 3:

$ parts/instance/bin/zopectl fg

You can now access Zope 3 on port 8080; you can log in with the username grok and password grok. You can add 'grokwiki' in the ZMI here. After you add it, you get a rather empty page asking you to register things. Ignore that (it's a default Zope 3 thing), and just go click 'preview' or go to the URL of the object you just created, for instance:

http://localhost:8080/grokwiki

Grok is a work in progress; it's going to change. It's not complete yet: we have a lot of things to do still. Don't build your applications on it yet unless you want to help us developing Grok further. We do think we have made great progress in a short time and I'm confident we'll make a lot more progress in the months to come. I personally think we're going to have quite an impact with Grok in the Zope world in 2007, and perhaps beyond.

Grok is team work. If you're interested in Grok and want to help us out developing it, we have a grok-dev mailing list you can join:

Update: updated the installation instructions slightly in response to feedback in comments.

A buildout for lxml

I've created what's called a "buildout" for lxml, and I figured I'd let everybody know how cool that is.

Buildout is a new system for constructing Python applications and its constituent parts, based on setuptools and eggs. Much more information can be found here:

http://cheeseshop.python.org/pypi/zc.buildout/

Now before you read all that (it's pretty technical), let me explain what problem the lxml buildout tries to solve.

Recently I tried using lxml's XMLSchema validator against an XML file and a schema. It gave a validation error, but to the best of my knowledge, the XML file should've been valid. Thinking it might perhaps be a bug that was fixed in more recent versions of libxml2, I set out to build and install libxml2 and libxslt locally, compiled lxml against it, and confirmed it: the document does validate against the schema with the most recent release of libxml2. This was indeed a bug in libxml2.

Now downloading, building and installing libxml2 and libxslt and linking lxml against it is quite a bit of work. Buildout to the rescue! Now it's just this:

$ svn co https://infrae.com/svn/buildout/lxml-recent/trunk lxml-recent
$ cd lxml-recent
$ python bootstrap/bootstrap.py
$ bin/buildout

and then wait. It'll download, configure and compile libxml2, libxslt, will downoad the lxml source and compile it against the libraries.

In the end you end up with a script that invokes the Python interpreter with the right paths set up so it can import lxml. Here's how to start it:

$ bin/lxmlpython

Though you can also manipulate the PYTHONPATH yourself in your scripts to put the lxml development egg on it, and then you can use the regular Python you used to run bootstrap.py with.

The only thing needed is a buildout.cfg - that's basically what you're checking out above, along with a standard bootstrap script and a required recipe (zc.recipe.cmmi) that hasn't been released as an egg to cheesehop yet.

Let's go through each section of buildout.cfg:

[buildout]
develop = zc.recipe.cmmi
parts = libxml2 libxslt lxml lxmlpython

Each buildout.cfg file has a top section called [buildout].

This one has a line develop. It says that I have one development egg, zc.recipe.cmmi. If the recipe were uploaded to the cheeseshop, this line can go away entirely, but right now it hasn't been released as an egg yet, so I need the local checkout of zc.recipe.cmmi to be available as an egg.

The next line, parts, lists what parts need to be built to finish the whole buildout. I want libxml2, libxslt, lxml and lxmlpython. The rest of the buildout.cfg file contains information on how to build the specified parts. Each part uses a recipe, such as zc.recipe.egg, zc.recipe.cmmi, etc.

The first part specifies how to build libxml2. We use the configure;make;make install recipe (zc.recipe.cmmi), which we just made available as a (development) egg. Here is the section that builds libxml2:

[libxml2]
recipe = zc.recipe.cmmi
url = ftp://xmlsoft.org/libxml2/libxml2-2.6.26.tar.gz
extra_options = --without-python

The tarball we want to download, compile and install is specified with the url option. This is enough for many tarballs: the tarball would be compiled and installed as a part in the parts directory under the buildout.

We want to do one extra thing and pass the special option to the configure script of libxml2: do not compile libxml2 with its default Python bindings. After all, lxml replaces those default binding bindings with something better!

The next section for [libxslt] is very similar. We do pass another extra option along to configure to make it link to the libxml2 part we just built:

[libxslt]
recipe = zc.recipe.cmmi
url = ftp://xmlsoft.org/libxml2/libxslt-1.1.17.tar.gz
extra_options =  --with-libxml-prefix=${buildout:directory}/parts/libxml2/
              --without-python

Now we want to build the lxml egg in the [lxml] section. We want to compile lxml 1.1.1 - the system will automatically download its sources from the cheeseshop. We want to pass a few extra options to the lxml build process (its setup script), to make sure it links against the libxml2 and libxslt parts we just built:

[lxml]
recipe = zc.recipe.egg:custom
eggs = lxml == 1.1.1
include-dirs = ${buildout:directory}/parts/libxml2/include:${buildout:directory}/parts/libxslt/include
rpath = ${buildout:directory}/parts/libxml2/lib:${buildout:directory}/parts/libxslt/lib

Finally, we want to build a Python interpreter with the lxml egg as an available import on the PYTHONPATH:

[lxmlpython]
recipe = zc.recipe.egg
interpreter = lxmlpython
eggs = lxml

That's all that's needed to make it download, compile and build lxml and its requirements.

Many buildout recipes exist: we already saw zc.recipe.egg (which we use here just to build an interpreter that has the egg on its path), and zc.recipe.egg:custom to build the lxml egg with some custom options, and zc.recipe.cmmi to do configure;make;make install.

At Infrae we also built a recipe to deploy OpenOffice in server mode, and allowing us to connect to it with PyUNO with the Python interpreter we want, for document conversion purposes.

Other recipes available can install Zope 3 and the ZODB.

This expanding pool of recipes will allow us to put together extremely complicated applications in a reproducable fashion. The idea of buildout is to have control and reproducability - to know exactly what is installed and not to have to install all Python libraries into a global site-packages.

As I said before, buildout uses setuptools and easy_install internally, so the whole egg story just works; it expands on that.

I think this way of building lxml has more benefits than just my single use case of trying it with the latest version of libxml2. It'll allow one to deploy lxml against a known-good version of libxml2 and libxslt, at least on a linux system. It should also help testing and developing lxml - it becomes a lot more easy to test and develop against different versions of libxml2.

I hope this short introduction makes people interested in buildout!

Zope 3.3 released!

Zope 3.3 is released!

Thanks to the hard work by many contributors, but especially by Jim Fulton and Philipp von Weitershausen to get the release out, we now have a fresh Zope 3.3 waiting for us.

Besides many bugfixes, one of the most important new feature additions is a much simpler API to register local components, such as utilities. The old APIs still work, but now issue deprecation warnings. I've adjusted some code to Zope 3.3 already and am quite happy with the work.

Onwards now to Zope 3.4! Next year we expect we'll complete the process of turning Zope 3 into a collection of eggs, making it a lot easier to reuse components of Zope 3 in other Python applications, and also changing the way we develop Zope 3 ourselves.

We'll be continuing to improve our release process. For one, I'm now the Zope 3.3 maintainer. I'll be pushing people to be sure to apply bugfixes on the Zope 3.3 branch, and making sure we have a Zope 3.3.1 release in the not too distant future.

The current release was a few months later than we initially desired. We also had some comments on improving our documention on what changed in a release. An excellent start of this is already available for Zope 3.3! We have some ideas for making it work better, and will be trying to implement some of those in the next release process.

Jean-Marc's implications

Jean-Marc Orliaguet is rewriting CPSSkins again, this time in Java, as Nuxeo is switching their CPS platform from Python and Zope to the Java language . The reasons for this switch are detailed in the FAQ. I'm not going to debate them here.

Jean-Marc, part of the community around Nuxeo making the switch, now posted some interesting criticisms of Zope and the Zope community. I am compelled to give a response from the perspective from the Zope community. If it's all a bit long and ranty, please forgive me.

I'm going to quote Jean-Marc copiously here. Let's start:

"You probably read the news the other day. There have been very few reactions so far apart from Phillip reporting the news and Paul making an allusion to it."

Jean-Marc, the news means Nuxeo's exiting our Zope community. I sincerely regret Nuxeo's leaving, as they contributed a lot. It's a major loss and we're mourning it. We're a diverse lot, so we don't have a single reaction. Possibly many of us feel that too much reaction would result in an argument about how wrong the other side is, as we're not joining you in the switch to Java for our own Zope-based platforms, and you're not going to be convinced to switch back either. So, there's not all that much to say overall, except to wish you luck.

I'm sure Jean-Marc's motivations in posting the criticisms are a genuine attempt to help us improve our Zope community. Thank you, Jean-Marc, that's valuable. I do however admit frankly that I'm a bit bothered by some of Jean-Marc's implications. I realize that it makes sense psychologically to defend the choice you've made, much like it makes sense for me to defend Zope and Python in the face of criticism.

Overall I think my answers give me the impression that Zope 3 isn't doing so badly, even from a Java perspective. :)

He starts by saying: "For Zope, what can be learned from the Java technologies?" and then lists some points. I'll repeat his points and give a response for each.

The Importance of the IDE

IDEs can indeed be very helpful. It's a general issue with Python that IDEs are not as powerful and well-developed as those in the Java world. Tools can compensate for the deficiencies in a language, and languages can compensate for the deficiencies of tools.

For a lot more about this, there is an insightful article about the psychology behind all this. I'm a "language maven" myself, but I recognize that other people are "tool mavens". More power to them, but I'm doing fine myself, thank you.

Again as a language maven, I don't think we as the Zope community should make an IDE a priority, as we only have so many resources to go with. Since this is an open source community anyone can step up and try to implement one, though.

The Importance of stable APIs

Jean-Marc says:

"The bottom-line is that there is only one chance for a developer to get the API right, because other developers expect it to be stable and easy to extend."

I can't get an API right the first time unless I have a lot of experience with the problem at hand, meaning having gotten the API wrong already a number of times. That appears to be true at first sight for Jean-Marc too; I'm sure it's easier to get the API right for CPSSkins the third time around.

Stable APIs are, of course, important. That's one reason why in the Zope community we've adopted explicit interface descriptions heavily, and test everything to heck just so we know we're not accidentally breaking something.

Concerning getting things right the first time in the Java community: the Java community understands refactoring well though. The IDEs have built-in support.

Jean-Marc:

"And no, the deprecation feature is not a tool available to developers for doing cosmetic refactoring or for renaming packages."

This point is not backed up by argumentation. I'm a big believer in iterative development as I know I'm not smart enough to get things right the first time. Instead of living with a mistake forever, I think it's often more effective to correct the mistake when possible, instead of living with it forever.

Instead of just breaking people's code when we change things, we make the commitment to support the old code for a significant period while issuing deprecation warnings.

It's somewhat ironic that Jean-Marc's point is stated in the context of the Nuxeo switch away from Python to Java - I imagine that will break a few APIs in CPS here and there as people move over to the new platform... Of course, Nuxeo uses the same reason I did just now: it's better to change from Python to Java now by API breakage than it is living with it forever.

The importance of standards

Yes, following established specifications is often a good idea, so I (as a member of the Zope community) have often argued it's useful to follow specifications. In fact, I wrote a whole article about how to evaluate them. Note that not all specifications are equally valuable. Sometimes they're inappropriate for the job. Sometimes they're too heavy and more light-weight processes are in order.

"Lots of packages are being added in the Zope3 repository without proper reviewing, implying: "we find this useful, hence someone else may find it useful too". This is relatively harmless for utility packages that have a limited scope, but for framework foundation modules this is a problem."

Jean-Marc, you're implying we're adding new framework foundation modules to Zope 3 without proper reviewing. I don't think that's happening. We've added non-framework packages to the Zope 3 software repository. That doesn't mean they're core and that doesn't mean you have to use them.

Before I start using a piece of open source software I evaluate how mature it is, whether it has developers backing it up, and so on. This is a fact of life in the open source community (or for any software developer, really). If I were just focused on standards I might use PyXML, as it supports DOM. I then realize PyXML is not very actively developed anymore, and that it tends to suck to work with the DOM API in Python, so I might switch to lxml or Amara.

We've had extensive discussions in the Zope 3 community on how to formalize the process of component maturation and mark packages as more or less stable and more or less supported more explicitly, and we'll continue taking steps in that direction.

It's important to realize that people learn, though. Software components that are perfectly good at some stage may still be replaced by better ones later. This process of evolution is healthy.

"But having 3 or 4 different specifications of a same basic feature because they were hurried in simply means a lack of concertation."

Sure. Are you implying we're hurrying in features a lot? Are you implying we regularly have 3 or 4 different specifications of same basic features in Zope 3? Are you implying that such a thing never happens with Java software?

Concerning package naming convention, I don't think we're doing too badly. Zope uses namespace packages extensively.

"Otherwise as a developer I may well start using a zope.abc package to learn 3 months later that the package was just the result of an experiment, and that it will be replaced by a "better" implementation."

I'm missing something here. Are you implying we just dump everything into the 'zope' namespace package? We clearly do not. Zope Corporation uses 'zc' for its package namespace. I've been using 'hurry'. We also got 'lovely' for Lovely Systems packages, and 'z3c' for packages managed by the communit, and so on.

'zope' is used for core packages, though admittedly we do have some non-Zope core packages sitting around - a situation we want to resolve. I'm not saying our situation is perfect, but I think we struck the balance between too cumbersome and too lightweight pretty well overall.

The Design Patterns

"There are no such things as adapters, factories, utilities, events in the Java language, these are just design patterns."

The implication here is that these things are all available in the Python language, but I think Jean-Marc here is comparing the Java language with the Zope 3 component architecture here. Perhaps it would've made more sense to compare Zope 3's component architecture with some component model in Java, such as J2EE or whatnot. Perhaps that makes no sense - I don't know much about J2EE.

Indeed:

  • adapters are not in the Python language. Adapters can be easily implemented as a design pattern. In Zope, there's no adapter base class either. What the Zope 3 component architecture adds is adapter registration and lookup, not adapters themselves. Those are just simple classes.
  • factories are a basic concept in the Python language. It's something that can be called and returns an object. It's easy to implement a factory in Python: you just define a class. It's also easy to switch from classes to custom functions when necessary, without changing the calling location. Java, using the 'new' keyword, does not have this benefit and the calling location will need to be changed if a factory function is used instead.
  • utilities are a Zope 3 component architecture concept, and not a Python language concept. You can see it as context-dependent module import by interface. Again the implementation of the utility is free - Zope 3 just offers a way to register them and look them up. It's a design pattern supported by the framework. We noticed it was useful as we've been doing a lot of this stuff in Zope 2 (CMF tools, for instance).
  • events: frameworks written in the Java language, such as user interface toolkits, use events. Zope 3 has an event system too. It helps you implement the design pattern where you want to decouple the caller from the callee and such.

"In other words, in some cases using the Adapter pattern may be a good idea while in other cases, it may be a bad idea. One cannot know the solution until the problem to be solved has been described."

Indeed, sometimes the adapter pattern is not a good idea, and sometimes it is. You seem to be implying that we think it's always a good idea and that we think we can know the solution to a problem before it is has been described.

Sometimes it's nice to have a registry of adapters so you can find the right one automatically. The latter is what Zope's component architecture offers and it makes it more easy to devise APIs with plugin points.

"My impression is that the 4 basic patterns promoted in the zope3 architecture as the "new religion" are predefined solutions to problems that are yet to be identified."

Let me hereby correct your impression, as it doesn't appear to be correct. Zope 3's component architecture provides useful facilities for looking up adapters and utilities, yes. It's not a religion and I personally encourage everybody to write plain Python where possible.

We've also been doing this Zope thing for some years and indeed have identified several problems in architecture we keep running into and that we'd like to have some assistance with. Hence the component architecture.

"Conclusion: the Component Architecture is not a universal solution for particular problems."

You seem to be implying we think it is. No, it's a bit more subtle than that, just like Design Patterns.

It's interesting to contrast, by the way, how first you recommend specifications as a way to get APIs right the first time around, and now you argue against solving solving problems that are yet to be identified.

The Component Architecture

Jean-Marc argues that hooking up small dedicated components is overhead. He then says that small dedicated components are the wrong appproach. I agree that hooking up small dedicated components is overhead, and reducing this overhead is something that I'm personally interested in tackling (sprint in about 3 weeks coming up). I disagree that this means small components are the wrong approach. I do agree that it is useful for us to study plugin-architectures in other systems.

I'll note that Java-based frameworks are not generally well known for having low developer overhead for hooking things up. One of the reasons of the success of Ruby on Rails is that it is a reaction to the overhead of getting things set up in Java. This therefore appears to be a criticism we possibly share with the Java world, not necessarily something to learn from the Java world.

"Conclusion: the Component Architecture does not replace high-level API design"

You seem to be implying that someone thinks it does. We don't. We think that it helps in API design, as it gives you tools enabling you to write pluggable APIs, such as interface-based lookup of adapters and utilities.

ZCML is not XML

The title of this point is rather odd. ZCML is of course valid XML. If you run it through an XML parser and you get an error, then that's a bug, please report it to us.

Anyway, the general point is that we should reduce the amount of XML for component configuration. I agree with this point.

I'll note again that Java-based frameworks are not generally well known for having low developer overhead for hooking things up. That doesn't make this criticism less valid, though.

The Presentation Layer

Jean-Marc knows a lot more about this than I do, and I'm sure there are useful things to learn from various Java-based frameworks here. Thanks for the pointers!

The "pythonic" trip

"What is "pythonic" what is not? but who cares basically as long as you have a good API and a great framework? Really, it is better to spend energy on more important things."

You seem to be implying we spend a lot of time arguing about some mystical quantity called Pythonic, being on some "trip" so to speak, but you know that's not what we do most of the time. Besides, for a system to be Pythonic means that system has a good API, following the conventions of the Python language and community. Here is what I had to say about this, when I stopped for a change and did discuss it.

Balkanization

"The balkanization of the Python community is a real issue I think."

Indeed. As you might know I've been arguing for the Zope community to engage positively other areas of the Python community (and beyond) for a long time. So, good criticism, and one we've been aware of for a while. Zope 3 is in part one reaction to the criticism from the Python community on Zope 2, for instance - we've been developing that since 2001.

I believe there's strength in our diversity, but I also believe there's a lot to benefit from working together. Recent developments help ease the sharing of components between communities through the egg distribution mechanism, so I'm very hopeful we're on the right track.

But this is perhaps not so good a criticism in the context of learning from Java. From a distance I have the feeling that the Java community hasn't entirely solved the issue of balkanization just yet either. There appear to be multiple competing web frameworks for instance, Eclipse and NetBeans for IDEs, multiple UI toolkits, and so on.

Self-criticism

I'm going to quote this section entirely, as this one really upset me on a personal level:

"Self-criticism is about being able to see what others did and admit when it is the case that they did some things better; especially taking a look at JEE, JBoss, Eclipse, or Ruby-On-Rails, Turbogears, Django, etc.

JBoss Seam for instance was created as a Java response to the Ruby-on-Rails' success.

What has the Zope Community's response been to RoR?

  • "that's just because they have a better web site"
  • "that's just hot air and marketing, all that we need is a better web site"

Maybe it is time for constructive self-criticism?"

Come on, Jean-Marc. You imply here, no, outright say, that we do not engage in constructive self-criticism, while nothing could be further from the truth. I, along with many others in the Zope community, fully admit other communities have done things better than Zope, including the Java community, and I personally have been working on these issues where I can.

While I dislike what you've been saying by implication so far, at least most of them are points that can be honestly debated, but this one I think is just completely unfair. Constructive self-criticism is taking place continuously in the Zope community. I know, as I've been engaged in it for a long time now, as you should well know.

You also mischaracterize our position concerning a web site. I do believe we need a better web site for Zope and better marketing towards developers in general. I think that's one lesson we can learn from Ruby on Rails and friends. Do you disagree?

You imply we believe that's the only thing other platforms have going for it. I feel offended by the implication that you think we do, and that we're so blind we don't even look. We do.

Jean-Marc, you appear to be saying a lot by implication. It's not my preferred style of criticism.

DZUG keynote

I was honoured to be asked to give a keynote talk at the recent DZUG conference in Sankt Augustin near Bonn, Germany. Even though DZUG is the German speaking Zope User Group, I was allowed to speak in English. :) My talk was titled "Zope in Perspective". Christian Scholz was kind enough to film it, also wrote a blog entry about , and put the video online! This way I could see all my mistakes, but overall it went well. If you're interested in the history of Zope, where we are now, and a chuckle or two, you might want to take a look at the video. Enjoy!

the Infrae OAI-PMH components

Since these components are little known I'd like to give a brief introduction of the great set of open source OAI-PMH components that Infrae has developed.

What is OAI-PMH? It's the Open Archives Initiative Protocol for Metadata Harvesting. This is an XML and HTTP based protocol for harvesting large amounts of metadata from so-called data providers. Data providers are sites on the web that are interested in exposing some metadata collection to the world. Other systems can then harvest this information and do something with it; analyze it, index it, present it, and so on.

While its potential uses are many, the main use of OAI-PMH is by many academic libraries in the world. Universities have large amounts of papers written all the time, and this is a way for the university library to publish information on the papers (the title, who wrote it, where to get it, and so on).

Infrae has been working for some years to build a great set of OAI-PMH tools. We have the following:

  • a Python library called pyoai, building on lxml. It implements the harvesting protocol, allowing easy access to OAI-PMH servers using a simple Python API. Moreover since version 2.0 it also implements the server protocol. If you implement the same Python API as you use for harvesting yourself, you can expose your own datasets as OAI-PMH easily without worrying about the details of the protocol. This library can be deployed by any Python application that needs OAI-PMH support.
  • A Zope 2 extension called OAICore. This takes care of OAI-PMH harvesting from Zope, using the pyoai library. The data is harvested into the ZODB, and indexed using the Zope catalog. There are also components for presenting this information in a web page. It's pluggable with new metadata definitions that control the way the metadata is indexed and displayed.
  • A Silva extension called SilvaOAI. This extends OAICore and offers various facilities to display harvested metadata in the Silva CMS, integrating tightly into the Silva user interface.

I can confidently say that if you need anything done with OAI-PMH, especially in a Python or Zope environment, this set of tools (together called the OAI Pack) is a good base to start from. And let Infrae know if you need help!

Ruby misconceptions about Python

I just ran into a rather misleading article claiming to compare Ruby and Python. Let's please be done with some of these misconceptions:

  • Python 1.5.2 was released in april 1999. Talking about supposed design flaws in a version released in 1999 is a bit misleading. We're seven years past april 1999. Maybe the referenced article was written in 2000, in which case it's somewhat more understandable. I do not believe it was, as it talks about Python's support for sets, which was added later.

  • the opinion sketched about booleans is an opinion. Python did add an explicit True and False quite recently, in Python 2.3 I believe (we're at 2.4 now), but 0 is still considered to be False (try bool(0)).

  • Yes, empty sets, like other empty sequences, are False in Python. Sets are built-in in Python 2.4, and were a library in Python 2.3, and empty sets are False in both. I happen to like empty sequences being False, but this particular opinion has the benefit in at least being original. :)

  • Let's please cut the crap about OO being a bolt-on in Python. This is FUD. It's probable that many people who repeat this meme are not even aware of enough deep OO issues to ever have to care even if it were true.

  • Python's OO implementation has indeed been improved in version 2.0 (released in 2001). That doesn't mean we haven't been happily using multiple inheritance in Python 1.5.2; the notion that it was critically flawed is rather exaggarated. I think this needs to be said to back up the OO bolted-on FUD. Like, "if OO in Python is bolted on, why has Python had multiple inheritance support since forever and Ruby doesn't?" Advocate's answer: "Well, that's because that multiple inheritance was critically flawed in versions as recent as 1.5.2!"

  • You can add and remove methods from classes in Python. This has been possible for years and years (including Python 1.5.2):

    class Foo(object):
       pass
    def mymethod(self):
       print "hello world"
    foo = Foo()
    Foo.mymethod = mymethod
    foo.mymethod()
    

    We were doing this with Zope in 1999. We think it tends to lead to maintainability nightmares and call this monkey-patching.

    And you can also add methods to individual objects in Python, though this is indeed slightly more tricky.

  • Classes are types in Python too, if you inherit from object. This has been the case since Python 2.0 (released, remember, in 2001?):

    class Foo(object):
        pass
    foo = Foo()
    assert type(foo) is Foo
    
  • Python programs also have a great capacity for "reflection". Saying that Ruby can do it in a discussion comparing Ruby and Python is a bit misleading as you imply you can't do it in Python.

  • CPython's garbage collection is based on reference counting, which leads to predictable run-time behavior. For circular references CPython has had an additional garbage collection algorithm for years (since, I believe, Python 2.0 again, released in 2001...), so circular references aren't a problem.

    Jython by the way uses Java's garbage collection system, and IronPython uses the CLR's. I believe the PyPy implementation of Python uses Boehm's, but they're still working on it and are making it pluggable, as this is PyPy after all. :)

  • del, by the way, just removes a reference and does not garbage collect right now, unless you only have a single reference to an object.

  • Taking care of reference counting when building C extensions is indeed difficult. Luckily Python has access to automatic binding generator tools like Swig which take care of it, or systems like Pyrex which let you write C extensions in a Python syntax. I wrote a full suite of advanced libxml2 bindings (lxml) for Python with Pyrex and I didn't need to worry about reference counting.

    I wonder how Ruby deals with the interaction between its garbage collection and C libraries that use malloc? I can imagine this might sometimes make things a bit more difficult.

  • the author of the article claimed they got more sophisticated as a programmer, and Python seemed less wonderful and Ruby seemed more attractive. That may be more a reflection of the personal evolution of the programmer than the capacity of the languages. I think this is so because the author was clearly not aware of many of the features that Python actually has when he was a predominantly programming in Python.

So, Ruby advocates, please consider stopping to spread the following misconceptions about Python:

  • Python's OO is bolted on. It's not bolted on. This misconception is a mismash between FUD and seriously outdated notions of Python's OO support.

    Python just is a multi-paradigm language with great OO support. You can do insanely advanced OO stuff in Python, including changing methods on classes at runtime, metaclasses, the works. And yes, you can do multiple inheritance in Python.

    I've never had to write a metaclass personally, by the way. Have you, Ruby programmer who claims Python's OO is bolted on?

  • Python has no garbage collection. It does. This is an outdated notion about CPython, never true for other implementations.

If you want to advocate Ruby over Python, advocate things like support for anonymous blocks. That's something neat that Python actually doesn't have.