Zope Foundation in the Google Summer of Code

The Zope Foundation is very happy to announce it has five projects accepted by the Google Summer of Code. Thank you Google! In this article, I will introduce the projects, students and mentors.

Zope2 on Python2.5

student: ranjith babu kannikara

mentor: Sidnei da Silva

This project is a followup of the Zope 3 on Python 2.5 project that was successfully completed in last year's summer of code by another student from the same school, Nikhil N. It's safe to say this project is considered essential by everyone in our community.

Nikhil N already explored some of this porting project and Ranjith communicated with him about it, so we're off to a good start. Sidnei, the mentor, is very experienced in all things Zope and has been in our community for many years. He is also a long-standing contributor to Plone, the popular Zope-based CMS, and which should help make sure that Plone works on the Python 2.5 version of Zope as well.

http://code.google.com/soc/2008/zope/appinfo.html?csaid=3F42269E3CDF31BD

Improved replication for ZODB through ZEO Raid

student: Dirceu Pereira Tiegs

mentor: Christian Theune

ZEO Raid is an approach to eliminating the single point of failure that clustered ZODB projects traditionally have: the single ZEO server. ZEO Raid introduces a redundant array of these servers. Doing this should expand the scalability and robustness of Zope's powerful transactional object database technology further.

I know Dirceu already as he has been active in the Grok community. He wrote the great megrok.form library for Grok. Mentor Christian Theune is the original developer of ZEO Raid. He has been a very active member of the Zope community for many years, contributing to Zope 2, Zope 3, Plone, the ZODB, the German Zope User Group DZUG, and helping to found the Grok project, and much more.

http://code.google.com/soc/2008/zope/appinfo.html?csaid=480505ACAC256B7D

Flint: a simple Grok-based CMS

student: Luciano Ramalho

mentor: Leonardo Rochael Almeida

The Flint project aims to create a lightweight CMS based on the Grok web application framework. The idea is also to learn from this project to feed features and documentation back into Grok. I hope that besides an separately installable CMS, this project will also result in the development of some CRUD UI layers that can be easily be integrated into any Grok application.

Luciano is participating the second time as a student for the Zope Foundation. I had the pleasure of mentoring him last year in another Grok-related project. He continued to contribute to Grok after the conclusion of the summer of code project, and I had the great fortune of being able to meet him in person at the Neanderthal sprint at the offices of GfU Cyrus AG in Cologne last fall. This year he will be mentored by Leonardo Rochael Almeida, who is a contributor to Zope 2 and Zope 3, and an active participant in the Grok community. They know each other in person, so Leonardo can go over with a club to Luciano's house to keep him in check.

http://code.google.com/soc/2008/zope/appinfo.html?csaid=3B4AE1B198100838

OCQL a Query Language for ZODB (Phase 0)

Student: Charith Lakshitha Paranaliyanage

Mentor: Ádám Gröszer

This ambitious project will aim to add an object query language to the Zope object database. The ZODB is great as you can just store Python objects. Relational databases are great as you can just query everything. For the ZODB this is a bit more involved, and you're required to set up an index. Adding an object query language to the ZODB would help us get closer to the ease of querying a relational database for a database of Python objects.

Charith responded well in the application phase and wrote a good proposal. Zope 3 contributor and mentor Ádám was thorough in testing this prospective student by asking him to write code. Ádám already helped start a project to implement this earlier on and has looked up relevant academic papers, so we're off to a good start here. As someone with an occasional interest in such things I'm quite excited by this project and look forward to following it.

http://code.google.com/soc/2008/zope/appinfo.html?csaid=D03B177E4CFF8E51

Grok: Introspector

Student: Uli Fouquet

Mentor: Martijn Faassen (me!)

The Grok introspector project was begun by Uli in last year's summer of code, and since then we have gained experience and we have some new ideas. There is more to do: the component architecture lets us introspect all kinds of object associations that we don't display yet in the UI. In addition, it'd be good if we displayed what Grok directives were used in the introspector UI as well. We will develop a new package, tentatively named zope.introspector that will provide a common infrastructure for introspecor facilities. In this project we will work together with Martin Lundwall, who is going to work on a related project for the Plone Foundation, more about which I'll say below.

Student Uli contributed crucial introspector infrastructure to Grok in last year's summer of code. I also had the pleasure of getting to know Uli in person at a number of sprints and conferences last year. Last year his mentor was Philipp von Weitershausen, but this year I'll have the pleasure of mentoring him myself.

http://code.google.com/soc/2008/zope/appinfo.html?csaid=67681C063AA1A4BF

Wait! Extra bonus projects!

Our sister foundation the Plone Foundation also applied to the Google Summer of Code, and some of the work done there will surely flow back into Zope. Here is the list of their accepted five projects:

http://code.google.com/soc/2008/plone/about.html

I'd like to highlight two in particular:

Component Registry Introspector

Student: Martin Lundwall

Mentor: Lennart Regebro

Since there is a lot of commonality with the Grok inspector, during the proposal period Martin and Uli already started talking about joining forces. They will work together on zope.introspector, where Martin will focus on Plone integration and Uli will focus on Grok integration. Mentor Lennart Regebro is also a long-standing contributor in the Zope community. He's someone I've worked with in the past, and I'm very happy I'll be able to do so again for this project.

http://code.google.com/soc/2008/plone/appinfo.html?csaid=9772C1C3ABD45432

Buildout Builder - An automated buildout configuration system

Student: Kenneth Miller

Mentor: Raphael Ritz

This is another project to watch: a system to help buildout configurations is of benefit to all of us that use buildout, and moreover, a more personal interest: this system will be developed using Grok!

http://code.google.com/soc/2008/plone/appinfo.html?csaid=69983137B55A28A5

Another bonus project!

That doesn't conclude the bonus projects. There's another project, this time one hosted by the Python Software Foundation, that is of great relevance to Zope:

Improvement of Zope support on Jython

Student: Georgy Berdyshev

Mentor: Frank Joseph Wierzbicki

Zope Foundation backup mentor: Sidnei da Silva

The immediate goal is to port essential core Zope libraries to the Jython platform, with an eye on making something like Zope 3 and Grok run in Jython. Adjustements to the Zope libraries will be fed back to Zope; some C extensions will be rewritten in Python where this hasn't happened already.

Jython will gain a bunch of powerful libraries, and a lot of feedback on what is involved in porting a complicated Python framework to Jython.

For the Zope side, the hope is that in future you'd be able to deploy, say, your Grok application, on the Jython platform. This means we'd also have full access to the many java libraries. Another side effect is that this project would make it easier to port these libraries to other Python interpreter platforms as well. I wrote more about this here.

Mentor Frank Wierzbicki is project lead of the Jython project, recently hired by Sun to work full time on Jython. He is the best mentor we could possibly ask for this project.

It's safe to say the Zope community needs to back this up. Sidnei offered to be backup mentor for this project. Since he's already mentoring the Python 2.5 porting project, he is in a great position to offer useful insights in the Jython porting project as well. Sidnei's also well connected in the Zope community and knows the people to ask for help when needed. So Zope community members, if he, Frank or Georgy asks for your help, please help!

http://code.google.com/soc/2008/psf/appinfo.html?csaid=59C7870763174C10

Grok takes Zope 3 the rest of the way

My friend Lennart's blog entry Zope 3 rocks and rolls is fascinating, not least as it has one of the most misleading titles I've ever seen. Reading that title, you'd be surprised Lennart talks mostly about a series of difficulties encountered with Zope 3 and decision to switch to Plone for a project instead.

Only the first paragraph has positive things to say about Zope 3: it's a framework that in many ways tries to stay out of your way, letting you do your thing. A success story for Zope 3, though it sounds like many of the facilities in Zope 3 weren't actually used to the project's detriment. Hopefully most people will only read the title and the first paragraph, as the rest of the article goes into another project and how it failed to make use of Zope 3, and why.

This is interesting reading to me. I care about Zope 3 technologies. I think it is great stuff. I want people to use it more. I want the community of people who use it to grow, so we get more great stuff. We can do a lot to make this happen. We shouldn't sit back and think if we build great technology, they will come. We should take the attitude that if smart people don't use Zope technology, it's our fault. Yes, I can think of other reasons for non-uptake of Zope that have nothing to do with the faults of the Zope community, but I'm not so interested in those reasons. I am interested in things we can fix. The rest will have to take care of itself.

The bit in Lennart's blog about z3ext, a new Zope 3-based CMS which I know little about myself, is a good example of why first impressions are so important. Lennart tried to work with z3ext, but it had bugs. He didn't give up immediately: he helped fixed some bugs. It still wouldn't work for him. As a result, he gave up on z3ext even though the developers fixed more bugs later.

With Grok, we had a problem last year after we switched to using eggs for the Zope 3 packages it depends on, instead of relying on a single monolithic Zope 3 release. The splitting up of Zope 3 into many smaller packages is a good thing, as you can evolve dependencies more or less independently from each other, as separate projects. We also had some teething problems, though. We didn't have the versions of our dependencies hammered down enough. We could have a working Grok one day, but then someone would update some dependency somewhere, breaking some compatibility, typically in an innocuous way. As a result Grok suddenly failed to install properly. Most people who tried out Grok at such a point in time probably went away in frustration. We never heard from them in the first place. This is a loss of people to a community that the community cannot even see. Karl Fogel describes this disastrous problem quite well in his book Producing Open Source Software.

Some people who tried out Grok luckily were kind enough to let us know things didn't work. We were grateful, as those are the rare cases. One of the absolute priorities of the Grok project became to make it install out of the box, with a fixed set of versions for its dependencies. Jan-Wijbrand Kolman and Luciano Ramalho worked hard to fix this at the Neanderthal Sprint last October. Grok's installation story still isn't perfect, but it's gotten a lot better and should work for most people again.

Back to Lennart. After failing to use z3ext as a whole, Lennart then tried to use some z3ext modules instead. The nice thing about Zope 3 applications is that they are often composed of components you are, at least in theory, able to reuse independently. So he set up a Grok project and tried to use these components. Unfortunately this failed due to dependency skew. Since Grok fixed its dependencies last year, some Zope 3 packages have been changed and new releases have been made. One common superficial change that does break things is that some code got moved to another package. z3ext was relying on newer versions of its dependencies than Grok was, and thus z3ext couldn't be used with Grok. Lennart had to give up on that too.

Ugh! This breaks Grok's aim of being fully compatible with Zope 3. We should fix this. Some of the solution is already very close. The Zope 3 project also recognized the problem Grok saw, and a bit after Grok fixed its list of versions, Stephan Richter developed a system known as KGS, which maintains a list of "known good" dependencies for a Zope 3 installation that are known to work together. KGS has nice infrastructure for running the tests of all these packages together to make sure. Grok doesn't use the KGS list yet as we converged on a versions list earlier. Hopefully at the Grokkerdam sprint someone will look at using the newer list. If z3ext components make use of the same list, we should be fine in reusing these components.

One of the essential strategies to ensure your project can be built on is to release it; Lennart mentions this too. When people work with a released version, they know it doesn't change unless they decide to switch to a newer release. A release should include a stable set of versions for dependencies: we learned this lesson the hard way with Grok. Having this stability means you can start to build things on it. Release your projects! Zope 3 has been a moving target without a proper final release. Grok has suffered from this lack of clarity and instability: if Zope 3 had made a final 3.4.0 release half a year ago, I imagine Grok would have been using it now. Like Lennart I believe it was more than stable enough to do so back then, though in its defense it couldn't happen yet then as we had the egg teething problems to work out.

Lennart also talks about how religious doctesting is a problem in Zope 3. He says that doctests are often hard to read and go into low-level details of the package without giving the easy picture of how to use the code. That's true. Doctests often make for pretty sucky documentation.

But I still see the glass as being half-full here where Lennart sees it as a problem. A module with a doctest is usually further along on the path towards good documentation than one not doing doctesting. The documentation tends to suck as it often lacks context and is too heavy on the details, and there's no tutorial, but it's there. That's a whole lot better than having no documentation about how things go together at all. Realistically, no documentation is often the alternative for busy developers. I'll take flawed documentation any day! Incidentally I also think doctests help make tests easier to follow; that's worthwhile in itself.

The problem is not the doctesting. I think the religious doctesting is a good thing. The problem is that too often we see the doctest as an end point. We should add context, tutorial-style content, and start refactoring the doctests to become better documentation. I think we often cannot expect the developers themselves to do this. While of course I hope developers will improve their documentation, let's also be happy they got as far as to include a doctest. We can help take the package the rest of the way by writing a tutorial or an example.

One more thing I want to add: using a new component or library is often just plain hard. Even with perfect documentation it takes study. You can try to make it easier, but each new component will present its own challenges. This will never end. What we can do is make the common case easier, with good documentation, good APIs and sensible default behavior. By doing this, we actually create artifacts that make the component easier to learn for everybody. Some effort will always remain however. Programming is learning. No gain without some effort, especially where the component you want to use is rich and powerful. I know Lennart knows this, but I feel it's important to point it out.

Lennart's second Zope 3 project failed and he went to Plone with it. We can defend Zope 3: perhaps he was trying to be too ambitious and expected too much from Zope 3. The nice thing about Plone is that it's there, does a lot, and works out of the box. Lennart is intimately familiar with it too. Zope 3 is not about being an application that just works, but a "roll it yourself" toolkit. But that doesn't take away the problems with Zope 3 that we can fix.

Lennart concludes that Zope 3 isn't ready when you want to reuse other people's modules. That's true - it's half-way there, which I think is actually quite far along and a good basis to start from. Lennart says this is a problem of the Zope 3 community. I agree, but I also believe the Zope 3 community by itself does not have enough interest to really fix this.

I've stopped expecting a fix of this kind of thing from the Zope 3 community. As I see it, the Zope 3 community is primarily interested in technical problems, and write flexible, reusable components that they can reuse. They aren't very interested in actually doing the rest of the work to make sure other people can reuse their stuff. If they were, Zope 3 would have a proper website by now. Even experienced Zope developers that very familiar with Zope 3 technologies like Lennart, bounce off.

I've come to accept this attitude of the Zope 3 community and I actually think it's mostly fine - they should do what they are interested in and what they are good at. They should scratch their own itches. We should be happy they are already bringing their components half-way to reusability by others, which as I said before, is actually quite far. The amount of work to do that shouldn't be underestimated.

While the Zope 3 community is not fixing this, the wider Zope community is actualy busy fixing this. The project to fix it has a name: Grok. We care about attracting people to our project, and we got proof as we got a website: http://grok.zope.org

The Grok project cares about making it more easy to use powerful Zope 3 technology. Let the Zope 3 developers do what they are good at and produce interesting, powerful, flexible components. Grok will then happily polish them until they work for others. We will do the rest of the job: making Zope 3 safe for the common caveman. That's our mission. We are successful at it: in its relatively brief existence, Grok has untapped an amazing amount of wealth and has made Zope 3 technology a lot more accessible to developers.

There is a lot more wealth to untap however. Let's also recognize the problems we have. Lennart pointed out some of the things we still need to do. Let's continue this process. If you care about this too, please join us and help!

Grok security: more Grokkerdam sprint topics

In my previous article on the Grokkerdam sprint, I discussed various topics people may want to work on during the sprint. This blog entry is dedicated to some more topics, all security-related.

Porting Zope to different Pythons

Zope has been under development since 1996. It wasn't called Zope back then yet: it was called bobo, and later pieces were called Principia, and only in 1998 did it become Zope.

When Zope was born, there was just one Python implementation: CPython. CPython is important. It's the main line of Python evolution, has the most libraries available to it, and can integrate with all sorts of C and C++ libraries, and even other languages such as Java can be integrated with it.

Then there was another implementation, born in 1997: Jython (originally JPython), Python on the JVM. Unfortunately after a good start, Jython's development lagged behind CPython's for some time. More recently development of Jython picked up speed again, and received actual backing by Sun. Jython is very interesting as it can speak to Java libraries natively. Java has lots of libraries.

Meanwhile, in 2004, the originator of Jython, Jim Huginin, came up with another implementation of Python: IronPython, on top of Microsoft's common language runtime infrastructure. Microsoft hired him to work on it further and IronPython has come far since then. IronPython is also interesting as it can speak to lots of .NET platform libraries natively. In addition, IronPython runs on top of Mono too, meaning there is what looks like an open source implementation of this stack (no license flamewars please, though. I may be wrong). Not very open sourcy is that last I heard, Microsoft developers cannot accept patches to IronPython from the outside. Still, it's very interesting technology.

Then there's PyPy, in development possibly longer than IronPython actually, since 2003. PyPy is, in theory, very compatible with CPython. The PyPy Python interpreter can, in theory, be translated to run on the "C platform", on the JVM, and on the CLR, among other things. There is, in theory, potential for tremendous speed boosts once JIT technology is integrated into it, as is intended. It also allows, in theory, all kinds of interesting features in the interpreter that other Python implementations don't allow.

You noticed I used the words "in theory" a lot concerning PyPy. Unfortunately so far PyPy's coolness, though they have lots of working code, remains mostly theory. You can dream about PyPy but not really use it. The PyPy developers are not showing public signs of actually working towards a release we can all use. They are not actually showing public signs of working towards any release at all: their last release was over a year ago and a new release doesn't appear to be in sight. They seem to prefer developing in their SVN repository, organize sprints and present at conferences. In my book, if it's not actually used for real-world projects it can still be cool, but only theoretically cool. This makes PyPy rather a dark horse that may become useful one day. I hope it does, but I'm not holding my breath anymore.

This year, 2008, Python 3.0 will be released. It's actually the least compatible version of Python of the list so far, which is risky. It will most likely become important as significant amounts of people start using it as their main development platform.

The varieties of Python don't end there. There are Python interpreters that aim to be small in some sense, such as TinyPy and PyMite. There are the Python implementations geared towards translation to efficient C or C++ code: Pyrex and its spinoff Cython, and Shedskin. PyPy's RPython subset can also do things like this, but is only used for PyPy's implementation itself so far.

One other variety of Python that received lots of hype in the last week is the Google App Engine. It's of course CPython 2.x, but it doesn't allow write access to the file system, or opening sockets, nor the installation of libraries that contain C-language components (unless Google does it for you). If you want to store data, you will have to use Google's way to do it. This means that porting code to the App Engine is indeed that: a porting activity, just like to the other Python versions.

Zope, today, runs on CPython 2.x only. Given all these other Pythons around, this is a situation that probably can't and shouldn't last forever. When I say "Zope", I mean the Zope libraries as much as the framework itself - Zope has been moving to a reusable-library approach for some years now.

We have already seen the first work towards spreading Zope technology beyond CPython already. Some porting work was done, I believe by people from the Twisted team, who also use it, to translate one important core package (zope.interface) to Jython.

The Jython project is hoping to start a Google Summer of Code project this year to port more Zope 3 packages, such as zope.component to Jython. The Zope Foundation is very grateful to the Jython project for taking this initiative; I believe it's of great importance to Zope as a whole. So, we hope that project gets accepted. The Zope Foundation going to make sure the project is well-supported by the Zope community.

Hopefully the porting work to Jython can be done without the need to fork the packages involved. Forking packages is bad, so the hope is that the plain Python components can be tweaked to work with Jython while they remain compatible with CPython. This leaves the C-based components. Luckily Zope has maintained parallel Python versions of much of its C code, which will hopefully make it easier to port these packages. That said, it may well be that some drift has happened between the C versions and the Python versions, and the Python code may need to catch up. There is also C code that doesn't have a Python equivalent.

One of the hardest packages to port will be the ZODB, the Zope Object Database. It does significant meta-class trickery and has a significant amount of C code that doesn't have Python equivalents. Luckily most Zope libraries aren't dependent on the ZODB at all. We also devising ways for Zope 3 to run completely without the ZODB. Instead, object relational mappers (ORM) can be be used instead. This means that even without porting the ZODB, we can still use Zope much as it is.

There is another option that might help us allow the use of the ZODB on Jython: RelStorage. This is a ZODB backend that is backed by a relational database backend instead of by the usual big file in FileStorage, the most-commonly used ZODB backend. It may be that RelStorage is easier to port to the Jython platform than FileStorage, but perhaps not. Let the experts speak out!

Porting to the Jython platform will also teach us important lessons about porting to other platforms. If we can make Zope work on Jython, it may also be easier to make it work on IronPython. Or PyPy.

What about the current hot topic, the Google App Engine? Can we port Zope to that? Zope's C libraries won't work, just like with Jython. That's the same problem as we have with Jython! There is a synergy between these porting projects. Hopefully the Jython port (cross fingers that it'll happen!) will help make sure Zope works without the C extensions, and therefore also on the Google App Engine.

With the Google App Engine, we will also need to use Google's way to store data. We can integrate with that like we integrate with an ORM. There has also been some talk about porting the ZODB on top of it - the ZODB supports multiple backends, so why not Google's? I'd love to hear from Shane Hathaway, the author of RelStorage, to see whether RelStorage's approach could be used for this.

If we can make Zope run without the need for C extensions, and we have ZODB backends for the various platforms out there, we may be looking at a future where many run Zope applications on the Java platform, .NET, the Google App Engine, or maybe even PyPy.

Local European PyCons and EuroPython

There is a PyCon UK. There is a PyCon Italy. Just now I saw the announcement of a PyCon France. I understand why these conferences happen - local user communities will want to see conferences where most talks are in their own language and have a bigger selection of talks relevant to the local community.

But, being Dutch, these country-specific conferences are typically not things I'd go to, as I'm not based in these countries. I don't speak French or Italian. The Netherlands being a smaller European country, with a proportionally smaller group of Python users, I don't think I'll see a "PyCon the Netherlands" anytime soon either. The same likely holds true for many of the smaller European countries.

While I'm happy to see so much activity in the European Python community, it does worry me a little too - because it's not exactly the European Python community. There is already a conference for that community: EuroPython. EuroPython is more than for just the European Python community, in fact - it has traditionally been the most international Python conference there is. I enjoy meeting people from the larger international community. Open source is international.

The appearance of country-specific conferences is in itself a good sign: the Python community in Europe, and these countries in particular, has grown large enough to support them. It can be argued that these attract different attendees than EuroPython, people who would not have visited EuroPython anyway. These conferences, in itself, are of course a good thing. One can even argue that perhaps they will energize their local communities so much that more people will end up going to EuroPython.

But there is bound to be some overlap between the potential attendees of a country-specific conference and the potential attendees of EuroPython. Moreover, there will be an increased competition to attract international speakers for these local conferences as well, as it's unlikely these will want to go speak at four European Python-related conferences a year. So, local conferences in the UK, Italy and France may draw away visitors and good speakers from EuroPython. That would be a shame.

Analyzing the statistics of where EuroPython visitors come from doesn't tell us very much. While attendance from the Italy and UK, which had PyCons last year, was indeed down, attendance from many other European conferences was down as well, including France, where they didn't have a PyCon yet last year (though certainly they had user group meetings). The statistics don't support the increased attendance theory either, though: EuroPython last year wasn't one of the largest ones I've seen in terms of participants - against the trend of growing usage of Python overall.

One can attribute EuroPython's relatively small size to other causes. Perhaps the lower attendance is due to the somewhat haphazard organization in the run-up to the conference last year (it's not a complaint as I didn't help, and this year already looks to be better). Perhaps it was due to the location in Lithuania, which may appear to be out of the way to some people. I think if the location is the cause, people have entirely the wrong impression. I don't think Vilnius, Lithuania is harder to reach than, say, Gothenburg Sweden, where we've had the conference before. Vilnius is certainly not expensive compared to some of the previous locations EuroPython has been in.

I certainly don't want to begrudge anyone a conference in Python with their own community and own language. It would be a shame however to see EuroPython shrink further in the face of local conferences. An international conference can attract a really interesting set of attendees and speakers. We should also celebrate the international nature of our open-source community.

So, I hope my worries are unjustified and more people will visit EuroPython 2008 in bigger numbers. Speakers: prove my worries wrong and submit a talk: the call for participation is open! Local conference organizers: prove my worries wrong and visit EuroPython 2008 with a delegation. Promote EuroPython at your local conferences! Incentive: you might see speakers you will want to invite for your own local PyCon there. :)

Grokkerdam sprint topics

The Grokkerdam sprint is coming up in a few weeks. I've put some topics people may want to work on during the sprint on the wiki page. Since those topics are just one-liners, I figured I might expand a bit on them here. The sprinters are not required to pick these tasks of course, and can choose something else entirely. If you have ideas, pleases expand the list of tasks!

This blog post will discuss some of the sprint topics. I hope to write another blog entry soon that discusses the rest.

class MyView(grok.View):
   grok.require('zope.Public')

class MyView(grok.view):
   pass

permission = class_annotation(view, 'grok.require', 'zope.Public')

require_directive.get(view)

SQLAlchemy with Grok

Grok needs a great relational database integration story. Grok already has a great database story: by default, we use an object database: the ZODB. The ZODB is great as you can just store normal Python objects. You can persist complicated nested structures easily. But this entry isn't about the ZODB. Relational databases are also great. You can query tables every which way very easily. You can manage your data in a RDMS, a familiar system for many, and integrate with existing RDMS that already exist in many places.

Thanks to object relational mappers (ORMs), working with relational databases has become pretty easy as well. Python has a number of object relational wrappers available for it. Recently I started a project that uses Grok with a relational database. It was a good opportunity to investigate this topic more deeply, keeping in mind what might be useful for Grok. I evaluated two object relational mappers: Storm and SQLAlchemy. I don't claim to have done anything like an in-depth technical evaluation. I didn't. I don't believe there are many technical reasons to choose one or the other. In the end I went with SQLAlchemy for this project. I think it would make a good default integration story for Grok.

I looked at Storm. Storm is nice, and is used by two large parties in the Zope community: Canonical (where it originated) and Lovely Systems (who use it a lot). The Storm developers maintain Zope 3 integration together with Storm. In the Grok community, we also have Christian Klinger actively working on Grok integration. One thing bothered me though: Storm lacks the ability to generate database schemas from Python code.

I asked why that may be so. It turns out this lack is a deliberate choice, and the Storm developers have good reasons: usually, with large scalable relational databases, you don't want auto-generation of schemas. Control and flexibility is important in these cases, and being limited by a Python-based schema language would hinder a project and, possibly even worse, piss off the database admins.

When I approached the Storm mailing list about this topic I got friendly and helpful responses, but it was quite clear that schema-generation, while not completely out of the question (perhaps in an extension), isn't part of the Storm culture. It's like threading and Twisted - you can do it if you want, but if you talk to the developers they will keep reminding you that in most cases you shouldn't do it. It also reminded me of the Zope 3 attitude - control and flexibility are of paramount importance.

Grok emphasizes slightly different things however. We happily use the control and flexibility that Zope 3 offers, and don't want to lose it if we can help it, but we let ease of use weigh more heavily when we make choices. That said, the "control & flexibility" attitude is actually great for Grok - it's very nice to be able to build on such a flexible and well-engineered base!

SQLAlchemy seems to share the Zope 3 attitude as well: it offers a lot of flexibility and control. People are building more high level declarative systems on top of it, such as Elixir. SQLAlchemy also offers some extra features compared to Storm. The one that mattered to me was schema generation from Python code. I myself think schema generation can be quite useful. Not for large-scale relational databases, but projects often start out small, and more often stay small. If you're just trying to develop a small web application, it's very nice to be able to get started quickly and just write Python code.

This approach suited my project, and I think it will suit Grok as well. For something like Grok, making it easy to get started is very important, and to me that means it needs to include schema-generation. I realize that schema-generation is the wrong choice for many projects, but I don't think it's the wrong choice for all projects, and I didn't feel like rolling my own with Storm while it was already there for SQLAlchemy.

When evaluating Storm, we saw it had several strong connections to our Zope community. Such connections are valuable, as it means that if I, as a community member, have a question or a problem, there's a good chance I'll get an answer. It also means that useful integration code is likely to be written. What's interesting is that while Storm has buy-in from larger companies in our community, SQLAlchemy seems to have more activity in open-source land.

The evidence I have for this is that we have almost too many integration layers that connect Zope's transaction machinery to SQLAlchemy. I've used z3c.sqlalchemy and z3c.zalchemy (yes, they are different!) in small projects in the past. This time I felt it was time to try out collective.lead, born in the Plone world. Besides these, there are also people advocating WSGI-based integrations!

I actually had to use an unreleased branch of collective.lead (I hope there will be a release soon, and for your reference, it's elro-tpc), and it didn't work right with sqlite. Once I switched to PostgreSQL, it did work quite well. Laurence Rowe, developer of collective.lead, was very helpful in answering my questions. Don't ask me for technical reasons on why it might be better than the others, though! Some time in the future, we'll need to figure out which Zope 3/SQLAlchemy integration layer is right for Grok.

So what does Grok code that uses SQLAlchemy look like? I'll give just a sketch here - this is not a complete application. Actually besides the database setup code, my project's code ended up very similar to code that uses the ZODB - the patterns are the same.

Let's start with setting up the database. collective.lead needs us to register a global utility that includes the database URL, table definitions and the object/relational mappers:

from collective.lead import Database
from collective.lead.interfaces import IDatabase
import sqlalchemy as sa
from sqlalchemy import Table, Column

class MyDatabase(grok.GlobalUtility, Database):
    grok.provides(IDatabase)
    grok.name('mydb')

    _url = sa.engine.url.URL(drivername='postgres', database='foo')

    def _setup_tables(self, metadata, tables):
        user_table = Table(
          'user', metadata,
            Column('user_id', Integer, primary_key=True),
            Column('username', String(20), unique=True, index=True),
            Column('password', String(100)))
        metadata.create_all(self._engine)
        tables['user'] = user_table

    def _setup_mappers(self, tables, mappers):
         mappers['user'] = mapper(
            User, tables['user'],
         )

I find the use of underscores in collective.lead here a bit ugly. I actually hope we can make much of this utility go away again by defining some clever grokkers and using some declarative approach combining class and table definitions (Elixir is one option to explore). What the right way is needs careful thought (perhaps at the Grokkerdam sprint?).

Above we define a table user, then map it to the class User. We haven't defined this class yet, so let's do that now:

import grok

class User(grok.Model):
   pass

This is in fact an entirely normal Grok model. You can do everything with it that you can do with a normal model, connect views to it, adapters, traversers, etc. Just about the only thing that you probably shouldn't be doing is stuffing it into the ZODB - that would be rather confusing! I think eventually Grok wants to introduce a new base-class for ORMed classes, but for now, the grok.Model approach actually appears to work just fine. The fields in a record, as defined by the user table, become attributes on User instances.

Creating an object is a trifle more involved than just instantiating it and sticking it into a container, like you'd do with the ZODB, but still very easy. You can create new users like this:

from zope import component

session = component.getUtility(IDatabase, name='mydb').session
user = User()
session.save(User)

The getUtility call looks up the utility we defined above. It's a bit verbose to keep looking up the session this way, but this can be easily abstracted away into a small utility function. More high-level abstractions also are possible: there is some work going on involving ORM-based container classes, which act like Zope 3 containers (folders). Zope 3 containers in turn act much like Python dictionaries. Such a container should make this pattern even more similar to the ZODB pattern.

All the stuff you can do with Grok add and edit forms also works fine, including applyData. To make automatic form generation work, you do need to define a Zope interface for User to define the schema. Unfortunately defining the Zope 3 schema for User violates DRY (Don't Repeat Yourself) - you need to define the columns (which become attributes) in the user table definition and then also define them again, in a different way, in the IUser schema. This is another important topic to think about - perhaps we can devise a way to generate Zope 3 schemas from SQLAlchemy table definitions or vice versa. There are projects in the Plone world (Alchemist, collective.mercury) that offer such a facility, so we need to investigate those.

Grok makes it very easy to make custom traversers to set up your URL space. Let's combine it with a query example. If you want to make an object that acts as the base of all users in URL space (base/username), you could do this:

class UserContainer(grok.Model):
     def traverse(self, name):
         session = component.getUtility(IDatabase, 'mydb').session
         return session.query(User).filter(User.username == name).first()

This contains a query that looks for users by name. If it can be found, a User instance is returned. If not, SQLAlchemy's first() will return None, which is exactly what traverse wants too if the object cannot be found. Nice!

This code is similar to the code that would go into a generic ORM container (base)class that I talked about before. In fact, in my project, the code almost automatically started evolving other bits and pieces needed to create such a container, such as way to remove items, and a way to get all the items in the container (defined by a query). With use of zope.location.location.locate() you can make the results of such a query fit right into Grok, so you can also use view.url() on User instances to get their absolute URL, just like you can do with ZODB-backed objects.

In my project, I combined the ZODB with SQLAlchemy. The transaction machinery is integrated, so if you have an uncaught exception in your code both the ZODB transaction and the relational transaction are aborted (and rolled back) at the same time. A successful request commits both. This way I could store things that most easily fit in the ZODB in the ZODB, and use relational tables where I wanted to. In many systems, the filesystem is used directly where a relational databases don't work well, but then you lose the benefit of transactions. A combination of ZODB and RDMBS gives us quite interesting possibilities to explore.

In conclusion: the Grok community has various things to think about and improve concerning relational database integration - some of it I discussed here. Even without those features, using SQLAlchemy with Grok already felt very natural and easy. This is no doubt in large part thanks to SQLAlchemy's and Zope 3's flexibility.

I hope we can get some people to work on this during the Grokkerdam sprint and afterward, and soon make ORM integration a first-class citizen for Grok. It will add another important feature to Grok, and take away another reason not to use Grok (unfamiliarity with the ZODB). Once we have the basic integration done, we can explore building higher-level features. If you don't mind some DRY violations, it's not hard to create powerful CRUD interfaces with Grok and SQLAlchemy right now, but of course we want to eliminate the DRY violations.

No, you are smart enough for Zope

This blog entry on picking web frameworks is quite interesting. It doesn't give a lot of detail, and it acknowledges this, but it does show the struggles someone goes through trying to pick the right framework for the job (and one that fits his mind). Wyatt Baldwin considered, among other frameworks, Zope, and had the following to say:

I was having a hard time with the Pylons docs, and so I ended screwing around with Grok (which actually looks fairly interesting) and even took a look at the Zope 3 site. I’m sure Zope is really awesome or whatever, but it might as well suck. Every time I look at that site, I’m just like “WTF! This shit has been around for like five years!” Anyway, I might just not be smart enough for Zope.

It's very polite to say that the problem is that he's not smart enough for Zope, but of course that's not true. It's Zope's problem. If you cannot learn Zope while you can learn so many other technologies, that's a failure of the Zope community. My community.

Some in our community, including myself, are very much concerned with this problem. I believe in fact there is universal acknowledgment within the Zope community that this is a problem, but acknowledging a problem is one thing, fixing the problem is another. The process to fix this takes a while. There are multiple paths we've been taking:

• Grok: making Zope 3 more approachable. Grok attempts to make Zope 3 more easy to use and more easy to explain, and then tries to explain it. Even though Grok wasn't the final choice of this developer, Grok actually comes out not too badly in this blog entry! I think we've been doing a good job so far, but there's a lot more to do. Come join us and help us if you're interested.

• Finally fix our Zope web presence. This has been something we should've fixed years ago, but progress has been very slow. It's easy to get a bunch of web developers to debate what technology they should be using for a web site. It's also fairly easy to get some nice layouts for your website. It's hard to get good content written. When you ask for that, you tend to hear crickets all of a sudden. That said, in the background we are making real progress.

Zope has a community that's very much alive. We have much that is good. Our community is about 10 years old now. I've known quite a few of our community members for almost that long by now, and am happy to count them among my friends. We have a Zope Foundation, we have an industry partner network, we have a lot of sprints (Grokkerdam coming up soon!), we have regional user group meetings, including a big one every year in Germany, a significant presence at EuroPython, and we're active in the Google Summer of Code. I also feel comfortable in claiming the Plone community (including its own Plone Foundation) and the wildly successful Plone conferences as a very important part of the wider Zope community.

I think our community would've been even more vibrant if we paid more attention to attracting newcomers to Zope technologies. Plone in particular has been doing pretty well in attracting newcomers. We can learn from them, and from the other communities around web frameworks.

I believe passionately in this mission of attracting people to our platform, and thus I'm part of the Zope Foundation board (chairman, in fact), helping to shepherd our participation in the Summer of Code, and am part of the group of people who've started Grok. I'm excited about the potential of Zope technologies. We may have been negligent to attract newcomers, but we haven't been negligent otherwise: we've continued to build powerful and great tools. We're sitting on an a lot of amazing capabilities, and we need to make them more easy to use and let people know about them (the mission of Grok).

I see a lot of hopeful signs. I believe we've seen Zope turn a corner in the last year or so in the nature of perception in the wider Python community. People are talking about Zope technologies again. But we can do much, much more. Perhaps if we had done so earlier, Wyatt Baldwin and many others would be part of our community now, contributing, instead of moving on to another one.

Explicit is better than implicit, and what it means for Grok

This post was, of course, an April Fool's joke

Grok has been using patterns like "Don't repeat yourself" and "Convention over configuration" to make it more easy to work with Zope 3 code. It is now time to admit that this experiment has been a failure. Explicit is better than implicit, after all, and we'll put up with repeating ourselves a few times if we need to.

Consider the following example of Grok code in the module foo.py:

import grok

class MyModel(grok.Model):
    pass

class Index(grok.View):
    pass

You then place a template in the module foo_templates called index.pt and Grok will automatically associate the template. To add code that helps render the template, you simply add methods to Index and use them from the template.

All this looks nice and easy, but people do need to remember rules about base classes, and implicit association. Grok does offer more explicit directives to do so:

import grok

grok.templatedir('foo_templates')

class MyModel(grok.Model):
    pass

class Index(grok.View):
  grok.context(MyModel)
  grok.name('index')
  grok.template('foo')

This also seems like a nice approach at first. The problem with these directives is that they clutter up your Python code, and distract you from what is really going on. Instead of conveniently finding out (or modifying) how your view is hooked up to your model and your template in a separate XML file, you will have to look through the clutter of registrations mixed with your Python code.

Wouldn't it be much nicer to be able to write everything down explicitly and separately, like this:

import persistent
from zope.app.container.contained import Contained
from zope.publisher.browser import BrowserPage

class MyModel(Contained, persistent.Persistent):
   pass

class MyView(BrowserPage):
   pass

And then we have a separate configure.zcml file containing the following:

<configure
  xmlns:browser="http://namespaces.zope.org/browser"
  xmlns="http://namespaces.zope.org/zope">

  <browser:page>
    for=".foo.MyModel"
    name="index"
    class=".foo.MyView"
    template="foo_templates/index.pt"
    permission="zope.Public" />

</configure>

To encourage good security practices, we will make security pervasive. Whenever you have a method on a model that you want to be called, for instance bar on class MyModel, we should declare this in ZCML as well:

<class=".foo.MyModel">
  <require
    attributes="bar"
    permission="mypermission">
</class>

Programming against interface is a good thing. Instead of associating our view directly against MyModel, let's write a file interfaces.py that spells out the interface that MyModel implements:

from zope.interface import Interface

class IMyModel(Interface):
  def bar():
     "The famous bar method"

Now we change foo.py to use the interface:

import persistent
from zope.app.container.contained import Contained
from zope.publisher.browser import BrowserPage
from zope.interface import implements
from interfaces import IMyModel

class MyModel(Contained, persistent.Persistent):
  implements(IMyModel)

class MyView(BrowserPage):
  pass

And we change the ZCML to use that interface too:

<configure
  xmlns:browser="http://namespaces.zope.org/browser"
  xmlns="http://namespaces.zope.org/zope"

  <browser:page
    for=".interfaces.IMyModel"
    name="index"
    class=".foo.MyView"
    template="foo_templates/index.pt"
    permission="zope.Public"
</configure>

These abstractions are always a good thing as our application will undoubtedly grow. We have to refer to the name of the interface in a few extra places, but the code becomes more understandable as a result.

We can then also declare security against interface instead of implementation, which cuts down on the amount of writing we will have to do if we have more than one method or attribute we want to protect! Like this:

<class class=".foo.MyModel"
  <require
    interface=".interfaces.IMyModel"
    permission="mypermission" />
</class>

We will have to distinguish between a modification interface for MyModel and an access-only interface for MyModel so we can assign different permissions to different methods, though, for instance IReadModel and IWriteMyModel.

In conclusion, I think everybody can clearly see that being explicit is a good thing. The ZCML directives are separated out from the Python code, making both the Python code easier to understand and the way directives work very explicit and easier to remember. We make sure we keep control by having everything explicit. Security is also very explicit in the application, and as a result you can be secure from the start.

So, Grok the caveman can go back into his cave. In fact we are considering a next step very seriously: to get rid of the Python language and going for a more explicit language like, for instance, Java. It's only a small matter of rewriting our codebase. This April first, 2008, Grok unsmashes ZCML, giving it back its rightful, explicit, place in development.

Zope Foundation accepting student applications in the Google Summer of Code

Are you a student and are you interested to hack on Zope, the ZODB or Grok for the summer? Do you want to be helped by an expert mentor? And get money and a t-shirt? The Zope Foundation is participating in the Google Summer of Code for the second time this year, and the time for you to apply is now! The applications just opened yesterday, but deadline for student applications is monday, march 31, so there isn't a lot of time. We're really excited to be part of this program again, and we hope we get many good student applications.

What to do to apply?

• Sign up to our gsoc mailing list:

  http://mail.zope.org/mailman/listinfo/gsoc

• See our wiki page for project ideas:

  http://wiki.zope.org/gsoc/SummerOfCode2008

  Your own ideas are very welcome too! If you have an other project idea and you aren't sure there is interest, do discuss it on our gsoc mailing list. Introducing yourself there is smart anyway, as the mentors are listening!

• Then follow this guide to get signed up with Google and make an application:

  http://groups.google.com/group/google-summer-of-code-announce/web/guide-to-the-gsoc-web-app-for-student-applicants