Skip to main content

megrok.rdf finally checked in

Jasper Op de Coul and I worked on megrok.rdf during the post-europython Grok sprint a few weeks ago. We had a few logistical problems concerning access to svn.zope.org, but yesterday Jasper finally managed to check in megrok.rdf:

http://svn.zope.org/megrok.rdf/trunk/

megrok.rdf is far from finished, and should be classified as a prototype. It has me excited for two reasons:

  • if people were to hack on this I think Grok could quickly grow first-class RDF support without tremendous effort. What megrok.rdf can do for you is represent RDF subjects as models. These can then get views and adapters and such. RDF predicates can show up as object relations. These relations can be made to be traversable using grok.traversable().
  • the way megrok.rdf works is quite similar to megrok.rdb, Grok's SQLAlchemy integration. This is not a surprise, as megrok.rdf's design is heavily influenced by my experiences with megrok.rdb. It still helps validate to me that Grok's way of exposing models makes a lot sense and is flexible. Grok has ZODB-backed Models and Containers, but it also has RDB backed Models and Containers and now, at least experimentally, RDF backed versions of the same.

A lot of work remains to be done. One thing we should do is to actually start documentation/doctests to solidify the approach. The way an RDF database is accessed (and integrated with Zope's transaction machinery) also needs attention.

I hope we can get some help from people interested in RDF. If people are interested, please join us at grok-dev.

Post-EuroPython Grok sprint report

This is a report on the Grok sprint that followed EuroPython. Thanks to the EuroPython organizers for giving us the space!

At EuroPython I was thrilled to meet Laurence Rowe. Laurence is the author of zope.sqlalchemy, the low-level integration layer between Zope 3 (and therefore Grok) and SQLAlchemy's transaction machinery. Also thanks to a lot of communication with Laurence in the last few months, I was inspired to write z3c.saconfig on top of this, which integrates SQLAlchemy's configuration system with the Zope 3 component architecture in what I think is a nice way.

This in turn is all resulted in massive improvements in megrok.rdb recently. megrok.rdb is the SQLAlchemy integration layer for Grok. megrko.rdb tries to make integration with SQLAlchemy to feel "Grok-like" (grokonic?) and as smooth as Grok's integration with the ZODB. At the sprint, Laurence and I talked a lot and we sat together to improve megrok.rdb so it can support database generated primary integer keys as container keys in relation containers.

Besides this, I also worked a bit on megrok.rdb's support for converting SQLAlchemy database schemas to Zope 3 schemas, so that nice forms can be generated automatically. I added some doctests for this, but it still needs a lot of polishing.

Jasper Op de Coul and I worked on a package called megrok.rdf. Jasper is very familiar with RDF, so his experience there was invalabule. The megrok.rdf package uses the approach from megrok.rdb to expose RDF content as a first-class citizen in Grok. It was surprising and satisfying to see how well the approach fits RDF-based data as well. The megrok.rdf package will make it to svn.zope.org soon. If we can form a group of interested contributors around this package Grok can grow a powerful way to build web applications on top of an RDF triple store. If you're interested, please join us at grok-dev!

Kit Blake worked on an mockup of the "local utilities configuration" screen for the Grok UI. It's very common that applications expose things to be configured (think relational database connection settings, catalogs, email configuration, user authentication, etc). It would be nice if the Grok UI had a screen that allowed per-application configuration that applications can plug in to. Kit has a lot of experience with configuration issues in a Zope 2 context (Silva in particular), so he had some good ideas on how to go ahead.

Sylvain Viollon and Eric Casteleijn worked on applying Grok technology (in particular grokcore.component and Martian) to Silva in a Zope 2 context. They worked on a lot of Silva-specific grokkers to make it easier to write Silva content objects and configure Silva in general. It's exciting to see that the Zope 2 community is now starting to use Grok technology more and more. This thursday and friday Godefroid Chapelle is hosting a two-day sprint to work on Grok in a wider Zope 2 context.

I also had some discussions with other people who were trying to learn Grok technologies (grokcore.component and Martian), also to see how to apply it to their own Python projects. Ignas Mikalajunas suggested it would be good to see a PyGTK-based example that used Martian, which is an interesting idea I hope someone will explore. I also talked with Christian Scholz about integration of grokcore.component with a non-web application package that was already using zope.component for configuration.

Thanks to all the sprinters for the work and feedback! As a post-conference sprint it tends to be less focused than a dedicated Grok sprint, but I still feel we made a lot of progress.

It wasn't all work, we also had fun and games. It was a very warm couple of days in Vilnius, Lithuania. I enjoyed the various dinners I had with people. I also enjoyed playing Galcon with a bunch of the sprinters at the end of the day on saturday. ME GROK CONQUER GALAXY

Zope's entry into the commit count pissing match

Mark Ramm-Christensen of the TurboGears project starts what looks like it will be a great commit count pissing match . I know that wasn't his intent. He just wanted to show the TurboGears project is active. But nevertheless, this kind of thing tends to get rather silly rather quickly, so we certainly can't leave Zope out!

Amounts of commits on a project are only a very vague measurement of project activity. Some projects may have more frequent, smaller commits than others, depending on project culture. Or maybe people with more commits simply make more mistakes! It's also only a single measurement of project activity. Lines of code changed might make more sense, but it's harder to do. Determining what belongs to the project and what does not is very tricky though. There are also other measurements of community activity. Amount of documentation produced, say. Or package release frequency on PyPI.

Anyway, back to Zope. How does it fare in this? We're not dead yet either, after all. In the month of june alone, the svn.zope.org repository saw 663 commits. In the last 30 days or so, Mark's measurement, we got 796 commits (give or take 10; I'm not being strict with day boundaries here). Almost 800!

If we start pulling in commits from other projects related to Zope but not kept in svn.zope.org, we'd get a bigger number. We could go counting commits in the Plone collective, for instance. If we included commits of projects that we sometimes use with Zope, such as SQLAlchemy and Paste, we'd get even more commits. That'd be silly though, as we've already won!

Hah! :)

Grok 0.13 released!

Grok 0.13 was released yesterday. Grok, of course, is the powerful web application development framework. See the details in the release notes here:

http://grok.zope.org/project/releases/0.13/

The really involved details are in the changelog here:

http://pypi.python.org/pypi/grok

Grok 0.13 is the first release of grok which uses grokcore.component, a standalone library that only depends on zope.component and its dependencies. We have also released a new, more powerful version of martian.

Why is this significant? There are a number of reasons:

  • grokcore.component allows you to more easily use the famous Zope 3 component architecture in arbitrary Python applications.
  • grokcore.component can be used in arbitrary Zope 3 projects without pulling in the rest of Grok.
  • grokcore.component works out of the box in Zope 2.
  • grokcore.component can also be used in arbitrary Python projects.
  • Martian makes developing domain-specific configuration systems in Python a lot easier, and the new version of Martian makes it easier still, introducing a powerful way to introduce custom class-level directives. Martian has evolved to take away a lot of ad-hoc behavior away from Grok, making Grok's configuration rules more explicit, even though they continue to work the same as they did before.

I'm proud we're able to spin off cool, reusable technology like this with the Grok project. This is one of the best aspects of Zope 3 development, which Grok shares.

Of course all these reusable libraries don't take away from Grok's integrated feel. Grok is a megaframework, but it's also integrated. If you just want to use Grok to build your web applications, you don't need to worry about these underlying components unless you want move into quite advanced areas.

The 0.13 release is also the first release that includes a lot work done at the Grokkerdam sprint in early may. This includes some of the Martian work mentioned above, but also many other things. Thank you sprinters!

The work will now start on Grok 0.14. One major topic on the agenda for 0.14 is to finally get good out of the box WSGI integration for Grok. WSGI has been possible with Grok for a long time, but since it's not out of the box lots of people had to figure it out for themselves.

Another release that will happen soon is the new and improved grokproject, the tool that helps you install grok and create grok-based projects. We ran into a few minor snags at release time yesterday so we stuck with the older version, but those should be resolved soon enough.

Hopefully we'll also soon be able to publish our Sphinx-based documentation. We've been maintaining some larger, more "official" documentation in SVN for a long time, and while we've published it to the web through Plone the workflow is not optimal. We are planning to roll out per-release official documentation along with the releases in the future.

Interested in Grok? Do check out http://grok.zope.org - there is a lot of documentation that will help you get started, and we're always working on more.

A misconception about the ZODB

The ZODB is a powerful object database for Python objects. It's very mature - it's been around for more than a decade. It is transactional, has advanced features like clustering (ZEO), blob support, and yes, it can be used independently from Zope. Zope 2, Zope 3 and Grok all use the ZODB as its default data storage, and it's seen a lot of battle testing.

As a result of various discussions in the past, I realized that some smart, informed people, seem to think the ZODB doesn't do what it actually does. The ZODB is really an object database. It really does get references between objects right. It's not an object store where references have to be indirect (a string, for instance). Somehow this misconception about the ZODB is widespread.

What do I mean when I say the ZODB "gets references right"? Let me give you an example with a lot of a, b and c. If you have object a that points to c, and object b` that also points to c, updating c will really matter to both a and b. You will reach the updated version of c through the reference in both a and b.

That kind of example sounds rather abstract, so here is some code that demonstrates it:

from persistent import Persistent

class Source(Persistent):
  def __init__(self, ref):
      self.ref = ref

class Target(Persistent):
   def __init__(self, message):
       self.message = message

Let's use this code:

>>> c = Target("First message")
>>> c.message
'First message'
>>> a = Source(c)
>>> b = Source(c)
>>> c.message = "Second message"
>>> a.ref.message
'Second message'
>>> b.ref.message
'Second message'

So what's special here? There's nothing special! All this is the way you'd expect it from Python. The ZODB's mission is to take normal Python objects and persist them. This means that when you restart the application, all your objects and the reference between them will still be there. There are a few extra requirements to make sure objects get persisted which I'll go into below, but in essence, the above example is complete.

Quite a few smart people seem to be under the impression the ZODB does far less than this. They believe references like this won't work properly in the ZODB. They believe, perhaps, that c will be persisted twice, once for a, once for b. This may be the case if c doesn't inherit from the special Persistent superclass, but if it does, there really will only be that instance.

The ZODB offers transparent object persistence. It's almost exactly like a pool of normal Python objects. They can reference each other just fine. The only requirements I know of are:

  • if you don't inherit your class from Persistent, or use a python builtin (which doesn't inherit from Persistent), instances of that class can be serialized multiple times.
  • if you want to persist your object, your object needs to be connected to another persistent object (such as the database root dictionary).
  • since the ZODB is a transactional storage, you need to actually commit the transaction sometimes to make sure your changes are stored.
  • if you have a non-Persistent subobject (like a list) and you change it, you need to manually flag the persistence machinery on the object that its subobject changed, with _p_changed. This is only necessary if some of the objects are not sublclasses of Persistent. For common built-in collections in Python such as list and dictionary there are replacements (PersistentList, PersistentMapping), and more advanced building blocks for indexes (BTrees), that don't have this issue.

The misapprehension that the ZODB somehow does less than it really does seems to be an easy one for people to develop. One reason is because in real-world Zope or Grok-based applications hard references like this are relatively rare. The reason people don't use hard references like this all the time in an application is that sometimes you want back references, and sometimes you want looser coupling between objects. So that's when things are referenced by a string or using some other form of lookup.

It's no different in Python programs, though. For the same reason, you sometimes put Python objects in a dictionary and look them up with a key. The wide application of such soft references seems to give people the impression that normal Python references somehow don't work.

Let's look at a complete example now. The only thing you need installed to make this work is ZODB3, which you can retrieve from the Python package index here. It demonstrates some of the details, such as how to set up a database and how to use the root dictionary:

from ZODB import FileStorage, DB
from persistent import Persistent
import transaction

class Source(Persistent):
  def __init__(self, ref):
      self.ref = ref

class Target(Persistent):
   def __init__(self, message):
     self.message = message

def getroot():
    # open the database
    storage = FileStorage.FileStorage('/tmp/mystorage.fs')
    db = DB(storage)
    conn = db.open()
    dbroot = conn.root()
    return dbroot

def main():
    dbroot = getroot()

    if 'a' not in dbroot:
        print "Filling database"
        fill_database(dbroot)
    else:
        print "Reusing existing database"
        # reset to first message
        dbroot['c'].message = 'First message'

    a = dbroot['a']
    b = dbroot['b']
    c = dbroot['c']

    print "message through a:", a.ref.message
    print "message through b:", b.ref.message
    print "ref is the same:", a.ref is b.ref
    print "ref is indeed c:", a.ref is c
    print "changing message c to: Second message"
    c.message = 'Second message'
    print "message through a:", a.ref.message
    print "message through b:", b.ref.message

    # commit any changes to the database
    transaction.commit()

def fill_database(dbroot):
    dbroot['c'] = c = Target('First message')
    dbroot['a'] = a = Source(c)
    dbroot['b'] = b = Source(c)

if __name__ == '__main__':
    main()

How could we do something about such misapprehensions? It would be good if the ZODB had a single, up to date to date web site that people could go to learn more about it. The ZODB is one of the coolest, most powerful libraries in the Python world, but it's less well known than it should be. I believe a good ZODB site, with some examples like the one above, would also help grow the ZODB community. The ZODB community is currently in a healthy enough state, with new developments always in progress, but it's a shame more people aren't aware of it.

Unfortunately the ZODB developers themselves seem to be too busy to put up this web site. It wouldn't be much work as it's mostly a matter of collecting existing information and redacting it. So, I hope that they will actually do it soon, so that I have some good hyperlinks to put at the end of this article. This wiki page seems inadequate, but it's what Google thinks is the most relevant when I search for "ZODB".

The ZODB PDF file is very useful, though I wish I knew of a better way to link to it than to the Subversion repository.

A recent good introduction was created by Brandon Rhodes and Noah Gift for IBM developerworks.

Grok Status Update part 3

Lots more is going on in the Grok world, so the Grok Status Update has received a part 3. See also part 1 and part 2.

grok.OrderedContainer

Thanks to work at the Grokkerdam sprint, Grok 0.13 will contain an IOrderedContainer implementation besides the normal container: grok.OrderedContainer.

grok.Require

We realized during the Grokkerdam sprint that we should allow the user to pass the permission class directly to grok.require, instead of having to pass the permission name. Using permission names is still possible (and sometime necessary if the permission isn't defined by Grok itself), but in Grok 0.13 it will also be possible to pass the permission class directly:

class MyPermission(grok.Permission):
   grok.name("my.permission")

class MyView(grok.View):
   grok.require(MyPermission)

menus in Grok

Jan-Wijbrand and myself discussed what menus in Grok could look like a week or two ago. We found many parallels with viewlet managers (menu itself) and viewlets (menu entries), and we think we can build this on top of viewlets. We need to work out a more detailed design proposal and discuss it on grok-dev next.

grokcore.component release

At the post-PyCon sprint, Brandon Craig Rhodes and Robert Marianski worked on extracting generic bits from Grok into a separate package called grokcore.component. At the Grokkerdam sprint, Philipp von Weitershausen polished it off, merged the branch that makes the Grok trunk depend on it, and released it to PyPI.

grokcore.component is compatible with Zope 3 and also with Zope 2. It is more light-weight than Grok itself, so it's easier to integrate in existing projects. Various Zope 3 and Zope 2 developers have started using this package in their own projects.

five.grok

At the Paris Plone sprint, held in the week before the Grokkerdam sprint, a new package five.grok was created. This package aims to provide Grok functionality to Zope 2. It builds on grokcore.component, and extends its functionality with other things like view support. During the Grokkerdam sprint Eric Casteleijn continued work on five.grok, adding tests and updating it as grokcore.component itself was updated.

The task is now to get a dedicated set of volunteers who want to bring five.grok up to speed. The idea is that five.grok will as much as possible make available Grok's APIs in Zope 2. It won't engage in the invention of new APIs; that is for other packages which build on this.

directive refactoring

A large project was completed at the Grokkerdam sprint by Philipp von Weitershausen and Jan-Wijbrand Kolman. During the Snow sprint in january, I noticed quite a bit of regularity in the way directives were used in Grok. I figured there was a way to abstract this out and push it into our Martian configuration library, so on the train trip back from the Snow sprint, I started a project to implement this in Martian.

Philipp and Jan-Wijbrand extended this work at the Grokkerdam sprint, started using it in grokcore.component and Grok itself, cleaning up Grok's grokkers quite a bit.

At the end of the sprint we reviewed this work and started work on the next step of this pattern: declare exactly what directives a grokker uses in the Grokker. This cleans up the code even more, and should make it much easier for the Grok introspector to deduce useful information for the introspector UI.

Philipp continued this work after the sprint. This is now merged into the Grok trunk as well (and grokcore.component uses it too, already released). Since then, we noticed another pattern in the way classes were grokked that actually register the methods of these classes. Philipp set out to work on this too, and this resulted in a new part of Martian called the MethodGrokker, which codifies this pattern. This further cleans up Grok.

Since then, I've further extended Martian. It now has a nicer tutorial, and it uses the new directive infrastructure to define grokkers as well. This makes a grokker look like any grokked class - it derives from a base class and has a few directives.

All this work will land in Grok 0.13.

CRUD

Joachim Schmitz and myself worked on a demonstrator project to see how we could construct a powerful CRUD framework on top of Grok, automating the user interface. This is code that typically needs a lot of pluggability, and pluggability is where the Zope component architecture shines. Hopefully we'll see more progress on CRUD in Grok in the future.

EuroPython Grok sprint

We will be having another Grok sprint after EuroPython this year. We are looking for people to participate. If you want to join in, please add your name here:

http://wiki.zope.org/grok/EuroPython2008Sprint

Grok Status Update Part 2

I continue yesterday's status update with some more topics. Lots of stuff is going on in the Grok world!

grokproject improvements

Maurits van Rees, Tim Terlegård, Reinout van Rees and others at the Grokkerdam sprint worked on improving grokproject in many ways:

  • improve paster integration
  • less questions asked and better defaults
  • less different URLs are involved when installing grokproject the first time, as we now have the ability to bundle a lot of eggs in a big file, lowering the chance of the installation failing because some site we pull in an egg from happens to be down.

After the sprint, Maurits continued to work on z3c.recipe.eggbasket, which should make this easy to manage for developers and release managers. All this is coming to you in Grok 0.13, if not earlier in an interim grokproject release.

megrok.kss

Godefroid Chapelle, Jeroen Vloothuis, Jean-Paul Ladage and others worked on polishing off KSS support in Grok, in the form of the Grok extension megrok.kss. KSS is a powerful, declarative way of doing AJAX in the form of something very much like a CSS sheet.

KSS actions have now become views on views, which seems to be a good place, as KSS actions actually typically apply to the view itself (on the browser-side, so HTML). Documentation on KSS was also extended, and some discussions were started about the KSS APIs. Jeroen switched over the extension to use KSS's new base library, kss.base with Grok, so we're up to speed on the latest and greatest KSS.

the url method

Thanks to the work done by Peter Bengtsson and Jan-Jaap Driessen at the Grokkerdam sprint, the url method on views now has an extra optional parameter, data, which can be passed a dictionary with query parameters.

A test coverage analysis and a bug report showed there were still problems with the support of the url method in viewlets and viewlet managers. This led to some discussions as to how fix viewlets. Work is being done to fix this on a branch by Jan-Wijbrand Kolman.

megrok.responseheaders

Peter Bengtsson (and I know others worked with him, but I now forget who.. sorry!) worked at the Grokkerdam sprint to develop a package called megrok.responseheaders. This is a nice example of how Grok can be extended with new directives for existing views. It adds two new directives, http_content_type and http_cache_control. These can be added to view classes to control view content type and cache control information.

Right now it's only available in SVN, here, as it is based on some changes made in a newer version of our Martian configuration library. Once Grok 0.13 is released, it'll need a bit of polishing and can be released.

Grok Status Update Part 1

Time for a Grok status report. I thought the Grokkerdam sprint was very successful. It is now a few weeks behind us. A fun time was had by the participants and we got a lot of work done. What's more we've had followup on many of the projects started there - people have continued the work started at the sprint.

Meanwhile our release mill is churning along: last week Grok 0.12.1, a bugfix release, was released!

Here is an overview of some of the recent interesting developments in Grok-land. There is really too much to mention, so forgive me if I don't mention your favorite topic. This is part 1. I'll be posting another blog entry soon which continues the update.

megrok.rdb

At the sprint, we started the megrok.rdb project to integrate Grok into SQLAlchemy. We had a long discussion on how to best do this, and got a lot of valuable input from Christian Theune. After this Christian Theune, Christian Klinger, Jasper Spaans and myself worked on the implementation. We now have a sketchy implementation that does have many of the features we discussed, albeit in unpolished form:

  • we use SQLAlchemy's declarative extension for defining classes and tables.
  • we have a megrok.rdb.Model and a megrok.rdb.Container, analogous to the ZODB-backed Model and Container in Grok core.
  • we can traverse a SQLAlchemy generated ORM mapping, both objects and through relations (which act like containers).

The current example code using SQLAlchemy with Grok looks like this:

http://svn.zope.org/grokapps/rdbexample/trunk/src/rdbexample/app.py?&view=markup

We definitely will be cleaning this up further and will make many changes, but it already looks pretty clean.

After the sprint the work continued. One important development was the creation of the zope.sqlalchemy package by Laurence Rowe, which is intended to become the universal foundation to the other SQLAlchemy projects in the Zope world. It provides transaction integration between Zope and SQLAlchemy. I have also been extensive discussion with Laurence and others on how to best implement a multi-application multi-database setup with SQLAlchemy. Zope, and Grok, support the pattern of having multiple separate applications in a single server, and when you are using a database, multiple databases. We need to convince SQLAlchemy to do the right thing there while preserving ease of use.

grok.traversable

During the relational database integration discussion at the Sprint we discovered a common traversal pattern we decided to codify. Thanks to the work done by Reinout van Rees and Jasper Spaans we now have a grok.traversable() directive. This makes it trivial to make an attribute of an object traversable and thus part of URL space.

This directive will be available in the next feature release of Grok (Grok 0.13).

Grok on Zope 3.4 (KGS)

Grok had pinned down the versions of its dependencies to "whatever worked" back at the Neanderthal sprint. Since then, Stephan Richter developed and maintains KGS, a package index for Zope 3 packages. Versions of these packages are tested together.

At the sprint, Maurits van Rees and Tim Terlegård updated the versions.cfg list of Grok to the KGS-based versions. We continue to maintain our own versions list, but at least Grok depends on the same versions again as a typical Zope 3.4 install.

After the sprint, we've checked that Windows binary versions are available of all the eggs currently in KGS. Jim Fulton kindly compiled them for us and made them available on the Python Package Index, so Grok 0.13 will be windows compatible.

Sphinx for Grok

Uli Fouquet did a lot of work before the sprint in integrating the Sphinx documentation utility for Grok's documentation. Jan-Wijbrand Kolman and Uli spent a lot of time polishing this off and landing this support in the Grok trunk. We intend to start publishing some of the SVN managed documents for Grok with this. We will also continue to publish documents using our Plone website, but some documents are better suited to being maintained within SVN instead.

Now we only need to start adding this to our documentation workflow; we've started discussions on how to do this, so hopefully this will land our website soon.

Documentation

A lot of work was done on documentation during the sprint. The website was reviewed and suggestions for improvements were made. Kamon Ayeva and Godefroid Chapelle also offered to help maintain the Grok Plone site. Since the sprint we've set up a grok-web mailing list. This is a welcome release for our hard-pressed website hero, Kevin Teague. More documentation has been written since the sprint, and the documentation section of our website is starting to look quite nice.

In Memoriam Joachim Schmitz

This is a rather unusual blog entry. I sincerely hope I don't have to write blog entries like this very often. I usually write about technology, and the community surrounding it. This community is, of course, composed of people. Yesterday I heard that Joachim Schmitz, long-standing member of the Zope and Python communities, had died suddenly last weekend. He was a regular presence at Zope-related events, and I have met him often. I feel that his passing should not go without notice for people in our community who knew him and met him.

I last saw Joachim just a week before he died, at the Grokkerdam sprint here in Rotterdam. Here is a picture of the sprint, winding down on a Saturday night, with people working and playing. Joachim is in the bottom right corner, looking pensively at the laptop in front of us, while I'm explaining some code to him. It's Joachim as I knew him, in his element, surrounded by fellow enthusiasts.

http://faassen.n--tree.net/grokkerdam_joachim.jpg

I wrote the following message to his friends and family, and I want to share it with everybody. I wrote it for myself, as someone who knew Joachim and will miss him, and also for our community.

I know Joachim as a member of the Zope community. I want to share a little about what Joachim meant to us in the Zope community.

Zope is software. I hear that he talked a lot about Zope in his private life, so you might have heard of it. He talked about it even to people that don't know much about software. I understand this perfectly, as I have the same habit. Zope, to him, like to me, is more than just a boring tool you use in your work to build web applications. It's something you can be passionate about, like a craftsman can be about his tools. It's also a community: people you know, who you work with, and who you like. It's a community of people that have known each other for years.

I remember when I met Joachim for the first time. It was at the International Python Conference in Washington DC, in the United States, in late january in the year 2000. It feels very long ago to me now; so much has happened since then. I sat next to him during some talks about Zope. I think he had come to the conference for the same reason I had: to meet other people who work with Zope. Joachim was friendly, and we started talking to each other.

We didn't know then yet, at this first meeting, that we would see each other again on many occassions afterwards, in the years following. His friends and relatives must have noticed Joachim was frequently away at one Zope related event or another. He wasn't away. It's when he was with us! He became a regular face at Zope-related events. I saw him several times each year following the first meeting. He was there so often and so reliably that we joked with him: you again! you're always there!

It isn't a proper Zope event without Joachim there.

Joachim contributed to our community in many ways. One example is his hard work for several years to make sure that people could sign up and pay for the EuroPython conference, helping to make this conference a success.

I saw Joachim for the last time only a little over a week ago as I write this. He was participating in the Grokkerdam sprint, here in Rotterdam. It was yet another Zope-related event: a proper one, as Joachim was there! I sat together with Joachim for a while, working with him and talking to him. Joachim was always interested, always learning, always participating. We were talking about how to teach Zope a few new tricks, planning for the future... See you next time, we told each other, when he left for home.

A week later he was gone, suddenly. He was one of us, and well-liked. This is why I feel so sad now, knowing he passed away. I'm glad I got to see him that one last time.

We who knew him will miss him. Zope events, and EuroPython, won't be the same without him. We will have to get used to him not being there in person with us. But we will remember him. By thinking of him, he is not entirely lost to us. This way, we will still feel his friendly presence in the future.

Martijn Faassen, Rotterdam, the Netherlands

Chairman of the Zope Foundation

Grok 0.12 released!

It happened a few days already, but I hadn't mentioned it here yet: Grok 0.12 has been released! Thanks go to Jan-Wijbrand Kolman for again playing the role of release manager, and to all contributors that made this release possible!

Grok 0.12 is a release that brings two new features:

  • z3c.autoinclude. At the snow sprint last january I worked with Ethan Jucovy and Robert Marianski on this library, which Grok now uses by default. It implements another DRY (don't repeat yourself) feature for Grok: it takes away the need to manually include the ZCML of a package if you use it in setup.py. I was really fortunate to meet Ethan and Robert at the snow sprint, as they're great guys and they both became contributors to Grok afterward! Ethan has been very supportive in making sure z3c.autoinclude works well afterwards, and has expanded its feature set in response to a feature request from Martin Aspeli (with an eye on Plone).
  • viewlet support. Viewlets are a powerful construct from Zope 3 to flexibly configure and plug in "parts of pages". With Grok we've made it about as easy to create a viewlet as it is to create a view. Unfortunately we're still working on documentation for them, but this should be fixed soon. The feature was originally integrated into Grok by Kevin Smith. I had the pleasure of working on finalizing this feature with Tim Terlegård and Robert Marianski at the snow sprint. Tim was the one who pushed to get this one done and I'm very glad he did so. After the sprint various community members helped to test this and polish this up, so it should be in good shape.

It also has a host of bug fixes. We are now ready to start evolving Grok further. At the PyCon Grok sprint, Robert Marianski and Brandon Craig Rhodes worked on separating out some of Grok's core component architecture configuration bits out of Grok into its own independently reusable library: grokcore.component. Philipp von Weitershausen and Brandon will be merging the branch that uses this library into the Grok core soon.

Grok's evolution is quite interesting. While the way Grok works on the surface hasn't changed all that much since the fateful first Grok sprint back in 2006, Grok has undergone quite an evolution underneath. During the first Grok sprint, we set out to just make Grok as we envisioned it work. We wrote nice clean but somewhat ad-hoc code. After the sprint, I noticed some patterns in this code and abstracted out the concept of "grokkers". Grokkers allow Grok's behavior to be extended: you can register new grokkers to handle extension or application-specific registration for new base classes.

About a year ago, I refactored the way grokkers work into its own Python library, Martian, and made Grok work with this library. Martian offers a way to avoid meta-classes while still having automatic registration of classes. This is possible as Grok (and Zope 3) have an explicit, separate configuration phase: configuration is done separately from import time.

Meta-classes are a powerful tool, but can lead to complexity during initialization and surprises for developers who may get unexpected behavior from seem to be normal Python classes. Django uses them for registration, and sometimes a developer has to resort to self-admitted scary solutions. I recall having to resort to some magic code from PEAK to do something similarly scary when I was struggling with a problem in Five some years ago (Zope 2/Zope 3 integration layer, where Zope 2's acquisition metaclass was making trouble). Grok, with Martian, sidesteps the need for meta-classes to do configuration. The only meta-class in Grok code is the venerable and quite well-behaved Persistent (when you use the ZODB).

Grok's evolution continued. At the Neanderthal sprint last october, Godefroid Chapelle initiated a refactoring of Grok's standard grokkers to use Zope 3 configuration actions. The benefit of these is that conflicting registrations (two views with the same name for the same object, say) are detected, and this greatly enhanced compatibility of Grok-based code with Zope 3 based code as well. The action support was something we had been planning from the start but never finalized. After the sprint Philipp von Weitershausen picked up on this work and pushed it a lot further, completing it.

So now we see the step of taking some of the reusable component registrations for basic zope.component features such as adapters into a reusable library as well, grokcore.component. The next release of Grok will make use of this library. The library also make it possible to write Zope 3 libraries that use grokcore.component as a configuration mechanism, and can be dropped into straight Zope 3 applications (the reverse was already possible from the start). That should largely conclude Grok's development of full Zope 3 compatibility.

While all this was being done, Grok applications continued to work with only minor adjustments required. I really like how this worked out for Grok so far. We really are creating abstractions and reusable libraries the way it's supposed to be done: by doing it step-by-step and learning from code that already works.

A next step on this path is a better directive implementation for Martian, something I mentioned in an earlier blog entry. This should help us making grokkers easier to read, and help up maintain consistency. Most importantly, it should make it easier to expand our introspection tools for Grok, giving them knowledge about Grok directives. We will start on this project as part of the Summer of Code project of Uli Fouquet (with me as the mentor).