The Stick

Some people blame Python 2.7. The core developers were way too nice to us Python developers by maintaining Python 2.x. If they had dropped support for Python 2.x years ago, Python 3 adoption would've been much faster. I think this argument is based on counterfactuals that are very hard to back up. I wish people would stop making these assertions, as it's impossible to prove either way. Even if this argument could be shown to be true, I for one am very glad that the core developers didn't throw the existing community into the deep end.

Somehow punishing those holdouts on Python 2 so they'll move I'll call the "stick" approach.

As Ian points out, this supposes some kind of moral authority for Python 3, or at least the progress of the Python language as a whole.

People who are using Python 2 and who don't port code are holding the language back. We should have some sort of allegiance to Python, and why aren't we getting off our behinds and do the work? It's not that hard! You're holding back The Python!

This kind of moral argument may work on those who are active members of the Python community, but it won't work those who are using Python as a tool for a job. I'll get back to them, as they're important.

The Default

Some people blame the Linux distributions. People new to Python apparently don't do the research, but instead type "python" only to discover Python 2 and then stick to it. Once Linux distros upgrade their Python, people will start using Python 3.

I can't believe that this is a major force holding everything back. Most Python beginners will check out python.org. Python 3 is quite widely available - it's in my Fedora installation and I'm two releases behind. Won't a beginner at least check out python.org? On Windows, installing Python 3 is no problem.

A more realistic problem is that of arcane system administration practices and obsolete Linux distributions making it harder to get Python 3 installed. I still don't think this is in any way a dominant factor.

The Carrot

Other people blame Python 3. The message is that Python 3 was immature early on, which is why people didn't adopt it, but that has changed. It's mature now, and people should adopt it now. If only Python 3 is made attractive enough, people will adopt it.

I think there is some truth to this. But the problem is that we've had about five years of people claiming that Python 3 is the superior version of Python, and it's still not commonly used.

There are other forces at work that I want to highlight here.

Dependencies

A common theory is that people aren't moving to Python 3 because their dependencies haven't moved to Python 3 yet.

What's a dependency? If you write code that depends on, say, the Reg library, and the Reg library only exists for Python 2, then you have to wait for the developer of the Reg library (me) to port to Python 3 first before you can port my own code. Or you have to dive in and port it yourself, hopefully somehow coordinating that with the original author.

This is the situation that holds you back from Python 3. I think this theory has a lot of merit. The problem is widely understood, which is why we see a lot of people urging us to port.

What seems to be widely neglected is the problem of the reverse dependencies.

Reverse Dependencies

I am sure that dependencies are an important factor, but what seems to be ignored in these discussions is the issue of reverse dependencies. Reverse dependencies have a gravity all their own.

What's a reverse dependency? If you are developing a cool new library, and you know that this library needs to be integrated into an existing Python 2 codebase, you need to make sure that this library works in Python 2.

Python 2, Python 3 or Polyglot?

There's a common response to this situation. You are writing a Python library, and you need it to work in a Python 2 context.

Writing the library in Python 3 is now out of the question. You can't enjoy the cleaner feel of Python 3. You can't use any of the cool new Python 3 features.

If you feel some moral obligation to The Python, or at least want to be cool and modern, you might consider writing polyglot code: code that works in Python 2 and Python 3. You gain for this some coolness points, potential users who use Python 3, and a harder to maintain codebase. You gain none of the benefits of Python 3.

You can also just write Python 2 code.

Now what is the better programming language to work with? Python 3, of course! But you can't use it. If you have to make the choice between Python 2 and Polyglot, Python 2 is the better programming language. So if you don't care about The Python, in this context you are better off writing Python 2.

The Gravity

Okay, forget about all those legacy codebases! If we write all our new code at least in Polyglot, we can start more new projects in Python 3.

I think that this does a disservice to those old projects. They were happy to upgrade to new Python 2.x versions, as that was doable with little effort. They might've picked Python before Python 3 came along and created this upgrade bump, or at least before Python 3 was mature (whatever point in time we pick where Python 3 is considered mature). But that's a moral argument. We have no allegiance to other people's codebases.

But those existing Python 2 codebases are the ones with business value to people. Those are the codebases with the bosses and customers who want new functionality, not a port to a new version of Python that involves an effort with no near-future benefits and the potential to introduce bugs. Those are the codebases with the money.

Those codebases have gravity. That's the gravity of Python 2.

Circular Dependencies

So now we have dependencies holding us back, and reverse dependencies holding us back. Those create a circular dependency structure. We know that those are the worst kind to deal with from a maintenance perspective!

Our code has dependencies. More dependencies are available for Python 2 than Python 3. Those dependencies that are polyglot are more likely to be battle tested in a Python 2 context than Python 3. Motivation not to move to Python 3.

The gravity caused by reverse dependencies written in Python 2 then pulls at people so that the situation remains that way. You will write your new library in Python 2 or Polyglot, because you have to integrate with Python 2 codebases. That is going to make more Python 2 dependencies available, not less.

Passing the porting buck down the Python 2 gravity well

The gravity field of Python 2 only pulls lightly on the core developers, who are high above us in the sky working on The Python. So after an upgrade path (2to3) was figured out, the porting buck was passed to the library developers.

The library developers are lower down in the gravity field, so 2to3 didn't work out so well for them. So they figured out how to write polyglot, and to the credit of the core developers, they made modifications in both Python 2 and Python 3 to make it easier to write polyglot. That works okay enough for many, so more and more polyglot libraries are appearing.

Now the porting buck is passing down the gravity field again, to those mired in existing Python 2 codebases. The force of bosses and customers and money is heavy. Porting is harder than for the library authors in all that weight, not helped by the fact that most applications have much less test coverage than libraries. What is going to happen? Are they going to port their codebases? Are they going to be able to do so in that gravity?

Escape Velocity?

How do we get out of this situation?

One option is for Python 3 to reach escape velocity. Python 3 the language becomes attractive enough, along with enough polyglot libraries being ported, that new projects will be written in Python 3. As we leave the gravity well old Python 2 codebases become just a pale blue dot in the cosmos. A Python 2 community is left behind.

I don't think that Python 3 reaching escape velocity is exactly friendly to those dealing with Python 2 every day for no fault of their own, a "bye bye we don't care about you anymore".

Another argument against escape velocity is that this is not healthy for our community; it would tear it apart. We're feeling some of the strains today.

Another argument against escape velocity is that this might cut off significant resources from the Python community. The high gravity Python 2 area that we're trying to leave behind has all the money. Is this wise?

So should we wish for escape velocity in the first place? Is it desirable?

Escape Velocity cannot be reached?

It's also possible that the Python 2 gravity field is too strong, circular dependencies will keep holding us back, and that escape velocity cannot be reached. This means we would be in this Python 2 versus Python 3 situation forever. That would suck.

Above I listed some arguments against going for escape velocity in the first place. What if we take those seriously?

A Way Forward

How to go forward then? I think it makes sense to work as hard as possible to lift those Python 2 codebases out of the gravity well.

The moral authority argument encouraging people to port their code is not going to work down in the gravity well, though. There interest in The Python is least; people are interested in functionality in actual applications.

We have to make the way forward for those in the gravity well as easy and obvious as possible. We know those codebases often don't have great test coverage. Code is not going to become polyglot, let alone Python 3, in one fell swoop. We can do this with libraries, but it works less well for aplications.

The keyword here is porting incrementally. Port to polyglot carefully, on a per-module basis, step by step, and write new modules in polyglot from the start, until finally you can switch the whole thing over.

Python 2.7 helps. There are tools to help. Recently I ran into Future, which to me is the most attractive approach yet. But will those tools work for those down in the gravity well? Will they even find them? Only things easy and obvious will reach them.

This is why I argue for more official Python 2 releases, where these things are built in. This stuff needs to be as obvious as possible.

And to make it even more obvious, we need Python 2.x releases where the deprecated stuff is removed, incrementally. Breaking code is making it as obvious as you can get! But you don't want to break it all at once, because then people will be inclined to give up before they even start.

This is why I think a continued future for Python 2.x would be in the community's interest, even in The Python's interest. I created a #python2.8 IRC channel on freenode to gauge interest and discuss this further.

Introduction

I'm happy to see the Python community starting the Python 3 conversation again. Alex Gaynor shares his analysis.

I agree with many of his points and am glad he was brave enough to take up this difficult topic in public.

Update: I see Ian Bicking has weighed in and he says things I very much agree with. Thanks Ian! Everybody should read Ian Bicking:

https://plus.google.com/+IanBicking/posts/iEVXdcfXkz7

I told you so

I've been thinking about this for a while. I brought my worries up in 2007, before Python 3 was released:

http://blog.startifact.com/posts/older/brief-python-3000-thoughts.html

I'll quote myself:

While I understand the need to be able to break backwards compatibility, I am worried about Python forking into two parallel versions (2.x and 3.x) for many, many years. Such a fork can't be good for the Python community.

This sounds a lot like what Alex is talking about today. While it's something to be able to say "I told you so", overall it just makes me slightly depressed. I'm sad I don't get excited when a new release of Python is made, like I used to be. Sometimes I don't even notice. Why read about exciting new features I won't be using any time soon?

I got feedback and wrote followups:

http://blog.startifact.com/posts/older/python-3-worries-feedback.html

http://blog.startifact.com/posts/older/the-purpose-to-my-whinging-about-the-transition-to-python-3.html

http://blog.startifact.com/posts/older/communicating-with-core-developers-on-the-python-3-transition.html

The feedback burned me quite heavily. I decided I was not going to actively do anything about Python 3 and would just develop Python 2 code and let others worry about the problem.

I shut up about Python 3 in public for years, until briefly in 2011, in response to a blog post by Armin Ronacher:

http://blog.startifact.com/posts/older/python-2-8--1.html

In early 2012 the now sadly deceased Aaron Swartz also talked about this topic, and I responded, saying "I told you so" (not Aaron personally):

http://blog.startifact.com/posts/older/cassandra-and-how-pypy-could-help-with-python-3.html

The path taken

In 2007 I was expecting that what the core developers were suggesting would continue to be the way to go: people would run 2to3 to convert Python 2 code to Python 3. I was skeptical then about this plan. Instead I hoped for what Aaron described: an interpreter where people could mix pure Python 3 modules with pure Python 2 modules.

Instead of 2to3, library authors started to work out ways to write Python 2 code in such a way that it would also run in Python 3. This was considered an absolutely unrealistic approach in the beginning, but gradually became the mainstream approach with support in Python 2.x itself.

Now much library code has been ported. That is going fairly well. But when I see such code I think it's scary and ugly: writing code that needs to work in both interpreters requires language knowledge that is more like that of an advanced C++ developer. It isn't Pythonic, at all.

Open source library and framework creators have some motivation to port. They're maintaining their stuff for a long time, and want to attract users. They also have the interest and skill level to do so. All this together may outweigh the cost of maintaining more complicated code.

The bind we're in

It's different for applications. An application developer usually has motivations not to port. Porting will introduce bugs in working applications, there's unclear benefit as the application will remain the same, and less libraries are available. It's also harder to port an application than a library, as libraries tend to have better test coverage. Finally applications exist for a reason, many of them are in-house to some organisation, and how do you motivate a budget to do the porting activity?

But if application developers don't port, why all the effort to port libraries then? You have many people continue to maintain and develop Python 2 applications. Some people write new applications in Python 3. As Alex says, not good for the community.

Python 2.8 as way forward?

Alex suggests an incremental upgrade of Python 2 adding more Python 3 features. I think this could be made to work. It might require a Python 2.8 while leaving the door open for a Python 2.9. Incremental backwards compatibility breaking can be part of the process: you could for instance drop old-style classes and tell people to adjust their code when upgrading to 2.8. While that's a relatively scary change, but most code will continue to work.

Perhaps that with the experience people gained writing 2 + 3 code we can even make the unicode path smooth enough.

The advantage of this approach compared to a magic Python 2 + 3 interpreter is that it would be possible to do incrementally based on the Python 2 codebase, with the Python 3 codebase in sight. That may involve a less scary development effort.

The main advantage is that by gradually dropping compatibility applications may move forward again, and messy 2+3 codebases can slowly be cleaned up as support for older versions of Python 2.x is dropped. I'll repeat that: a path forward for developers to Python 3 that has your code become cleaner, not uglier.

The closed door

The Python core developers somewhat gleefully slammed the door shut on Python 2.8 back in 2011, though. Recently celebrating the 5 year release of Python 3.0 I heard people say everything was going according to plan.

One quibble with Alex's post: the suggestion he makes that Python 2 should've been dropped in favor of Python 3 when Python 3 was first released, so that people would have urgency to port code, strikes me as unrealistic. I think it could have resulted in a justified rebellion among users and a fork of the Python interpreter. I'm glad that path was not taken. In fact since 2011 this has been the path taken, and I'm not sure it made any difference.

I hope the core developers will change their mind and take up the momentous unpleasant task of moving Python 2.x forward towards Python 3.x again. But if not, perhaps a brave group of volunteers will stand up and fork Python 2, and take the incremental steps forward. This will have to remain just an idle suggestion, as I'm not volunteering myself.

Unused UI Code

When the new Zope was built, we thought we were going to create something like the old Zope, where management and even development of web applications could be done using a through-the-web UI. The new Zope therefore consisted of two parts: a selection of libraries that could be used as a web framework (or the plumbing of one, in the case of Grok), and a selection of libraries built on top of that which implemented the web UI and its content.

The old Zope and Grok had started to use the new Zope as a collection libraries, but had no need for the user interface bits.

Why did we have to include the UI code if we didn't want it? Because the Zope libraries were all tangled up. To shift the metaphor from icebergs, they were like a ball of wool after a cat's done playing with it: all tangled up.

The New Zope as Libraries

The new Zope had originally been developed as a collection of libraries all in the same repository, in a giant tree of packages. Dependencies between parts of the tree had been rather freely introduced.

Then, around 2006, we became early adopters of setuptools, and split up these libraries into a collection of independent libraries, and published them on PyPI. We learned a lot along the way on how to do proper releases and how to manage such collections of libraries.

But now we had a collection of supposedly independent libraries with a large amount of dependencies to each other. What was worse, we had circular dependencies. This meant that just about all libraries were linked to all other libraries. Grab one Zope library off PyPI, get all of them. All 80 or so...

As Chris McDonough pointed out later, what we should have done is extract the libraries one by one, and for each give a clear purpose and documentation.

But it was too late. And at least the problem was clear now.

The Cave Sprint

So the beginning of 2009 I organized a small sprint. We'd just moved house and now had the room for it. I'm now in two minds about this sprint: while we did manage to make a good first step in cleaning up dependencies, it was also the beginning of the end of my involvement in Zope. Plus cleaning up dependencies is not a very rewarding creatively, and I think sprints should inspire as well as get work done.

During this sprint we had a discussion about the Zope project itself. The original leader of the Zope project, Jim Fulton, had slowly become less involved with it, and there was a gap that needed to be filled if we wanted to drive Zope forward. It had become difficult to make decisions.

Christian Theune and Wolfgang Schnerring convinced me we should engage the Zope project more actively instead of working around it with Grok. I was convinced, as I knew that the problem of simplifying the code base needed engagement with the Zope project.

Steering Zope

So after the sprint some of us organized a Zope Steering Group, and set up a lightweight system for technical decision making so we could hopefully move Zope forward. I think it did help.

The Zope community reformulated the new Zope as a Zope Toolkit, recognizing its status as the foundation of projects like Zope 2 and Grok. We rebranded the new Zope away from the confusingly named Zope 3, which for years had implied that the new Zope was going to be a true successor to the old Zope, something that it never quite became but confused people. On the technical level at the end of the year we were able to disconnect the UI-related packages from the rest, and had left a directed acyclic graph of foundational packages, without circular dependencies.

My hope was also that new leadership could infuse new energy in the Zope project beyond just maintenance tasks like cleaning up dependencies. The web was evolving, and could we get together and do exciting new things again? The answer was no, but that will have to wait for the next article in this series.

This blog entry is a part of a series on Zope and my involvement with it. Previous. Next.

Application Isolation

Morepath lets you create app objects like this:

app = morepath.App()

These app objects are WSGI applications, but also serve as registries for application configuration information. This configuration is specify used decorators. Typically apps consist of models and views:

@app.model(model=User, path='user/{username}',
           variables=lambda user: { 'username': user.username })
def get_user(username):
    return query_for_user(username)

@app.view(model=User)
def render_user(request, model):
    return "User: %s" % model.username

Here we've exposed the User model class under the path /user/{username}. When you go to such a URL, the default (unnamed) view will be found, and we've provided that too: it just renders "User: {username}".

What now if we have another app where we want to publish User in a different way? No problem, we can just create one:

other_app = morepath.App()
@other_app.model(model=User, path='different_path/{username}')
def get_user(username):
    return different_query_for_user(username)

@other_app.view(model=User)
def render_user(request, model):
    return "Differently Displayed User: %s" % model.username

Here we expose User to the web again, but use a different path and a different view. If you run this app (even in the same runtime), this will be separate.

This app isolation is nothing really special; it's kind of obvious that this is possible. But that's what we wanted. Let's look at a few more involved possibilities next.

Application Extension

Let's look at our first application app again. It exposes a single view for users (the default view). What now if we want to add a new functionality to this application so that we can edit users as well?

This is simple; we can add a new edit view to app:

@app.view(model=User, name='edit')
def edit_user(request, model):
    return 'Edit user: %s' % model.username

The string we return here is of course useless for a real edit view, but you get the idea.

But what if we have a scenario where there is a core application and we want to extend it without modifying it?

Why would this ever happen, you may ask? Well, it can, especially in more complex applications and reuse scenarios. Often you have a common application core and you want to be able to plug into it. Meanwhile, you want that core application to still function as before when used (or tested!) by itself. Perhaps there's somebody else who has created another extension of it.

This architectural principle is called the Open/Closed Principle in software engineering, and Morepath makes it really easy to follow it. What you do is create another app that extends the original:

extended_app = morepath.App(extends=[app])

And then we can add the view to the extended app:

@extended_app.view(model=User, name='edit')
def edit_user(request, model):
    return 'Edit user: %s' % model.username

Now when we publish extended_app using WSGI, the new edit view will be there, but when we publish app it won't be.

Kind of obvious, perhaps. Good. Let's move on.

Application Overrides

Now we get to a more exciting example: overriding applications. What if instead of adding an extension to a core application you want to override part of it? For instance, what if we want to change the default view for User?

Here's how we would do that:

@extended_app.view(model=User)
def render_user_differently(request, model):
    return 'Different view for user: %s' % model.username

We've now overridden the default view for User to a new view that renders it differently.

You can also do this for what is returned for model paths. We might for instance want to return a different user object altogether in our overriding app:

@extended_app.model(model=OtherUser, path='user/{username}')
def get_user_differently(username):
    return OtherUser(username)

To make OtherUser actually be published on the web under /user/{username} it either needs to be a subclass of User, for which we've already registered a default view, or we need to register a new default view for OtherUser.

Overriding apps actually doesn't look much different from how you build apps in the first place. Hopefully not so obvious that it's boring. Let's talk about something new.

Nesting Applications

Let's talk about application composition: nesting one app in another.

Imagine our user app allows users to have wikis associated with them. You would have paths like /user/faassen/wiki and /user/bob/wiki.

One approach might be to implement a wiki application within the user application we already have, along these lines:

@app.model(model=Wiki, path='user/{username}/wiki')
def get_wiki(username):
    return wiki_for_user(username)

@app.view(model=Wiki)
def wiki_default_view(request, model):
    return "Default view for wiki"

(this is massively simplified of course. we'd also have a Page model that's exposed on a sub-path under the wiki, with its own views, etc)

But this feels bad. Why?

• Why would we implement a wiki as part of our user app? Our wiki application should really be an app by itself, that we can use byitself and also test by itself.

• There's the issue of the username: it will appear in all paths that go to wiki-related models (the wiki itself, any wiki pages). But why should we have to care about the username of a user when we are thinking about wikis?

• It would also be nice if we can use the wiki app in other contexts as well, instead of only letting it be associated with users. What about associating a wiki app with a project instead, like you can do in github?

A separate app for wikis seems obvious. So let's do it. Here's the wiki app by itself:

wiki_app = morepath.App()

@wiki_app.model(model=Wiki, path='{wiki_id}')
def get_wiki(wiki_id):
    return query_wiki(wiki_id)

@wiki_app.view(model=Wiki)
def wiki_default_view(request, model):
    return "Default view for wiki"

This is an app that exposes wikis on URLs using wiki_id, like /my_wiki, /another_wiki.

But that won't work if we want to associate wikis with users. What if we want the paths we had before, like /user/faassen/wiki?

Morepath has a solution. We can mount the wiki app in the user app, like this:

@app.mount(app=wiki_app, path='user/{username}/wiki')
def mount_wiki(username):
    return {
       'wiki_id': get_wiki_id_for_username(username)
    }

We do need to adjust the wiki app a bit as right now it expects wiki_id to be in its paths, and the wiki id won't show up when mounted. This is a simple adjustment: we need to register the model so that its path is empty:

@wiki_app.model(model=Wiki, path='')
def get_wiki(wiki_id):
    return query_wiki(wiki_id)

But where does wiki_id come from now if not from the path? We already have it: it was determined when the app was mounted, and comes from the dictionary that we return from mount_wiki().

What if we want to use wiki_app by itself, as a WSGI app? That can be useful, also for testing purposes. It needs this wiki_id parameter now. We can construct this WSGI app from wiki_app by giving it a context explicitly:

wiki_app.context(wiki_id=5)

Application Reuse

Many web frameworks have mechanisms for overriding specific behavior and to support reusable applications. These tend to have been developed in an ad-hoc fashion as new needs arose.

Morepath instead has a general mechanism for supporting app extension and reuse. You use the same principles and APIs you already use to create new applications. Any normal Morepath app can without extra effort be reused. Anything registered in a Morepath app can be overridden. This is because Morepath builds on a powerful general configuration system.

This is because Morepath, like Grok or Pyramid, comes from the rich Zope heritage, where we've thought about this stuff. And Morepath wraps all that power in a small, easy, reusable little framework.

Let me know what you think!

Towards the First Grok Sprint

I got the idea for Grok in the summer of 2006. In the fall I still hadn't done anything yet about it. Then I gave a talk at a German Zope User Group (DZUG) conference (the video of that now appears to have sadly disappeared off the internet; if it's still somewhere I'd appreciate to get the link!). This was the series of conferences that a few years ago got broadened into the wider PyCon.DE (German PyCON) conferences. I chatted about Grok with Christian Theune of Gocept. He suggested we hold a sprint to build Grok at the Gocept offices.

So later on that fall I and my wife Felicia flew to Berlin and from there were taken by Christian to the Gocept offices in Halle. There were four of us at the sprint: myself, Christian, Philipp von Weitershausen, and Wolfgang Schnerring. The first three were very familiar with Zope both old and new. Wolfgang was new to it all, but had experience with Ruby on Rails, so could offer us an interesting different perspective.

Quadruple Programming

This sprint, we didn't do the pair programming that was pretty standard in Zope sprints, but quadruple programming. How did that work?

We used a video projector to project the screen of somebody's laptop onto the wall. The person at the keyboard was mostly either Philipp or Wolfgang. The rest would look at the projection and give directions. Christian made sure we didn't forget test cases by writing them down on a whiteboard. I sat or paced around and just talked. People who know me know I do that a lot anyway. I actually did not edit a single line of code during the entire weeklong sprint myself, but I'd seen all of it and was thoroughly familiar with it when I went home afterwards.

After a week of sprinting we got Grok from a design document to a working implementation. We built the traditional wiki application with it to try it all out in the end.

Starting Grok in a sprint like this made it a community project straight from the start. This was beneficial when we went home and had to cooperate online.

I wrote a report about the sprint and Grok to my blog back then; it may be interesting reading for some.

Grok the Caveman

None of the other sprinters liked my proposed name 'Grok' for the project. It was too nerdy, too obscure. Perhaps, it was proposed, we should name it something like "Easy Zope" (I forget the exact name proposed, but 'easy' was part of it). I protested and formulated what Philipp later dubbed "Faassen's law":

Don't use the name 'easy' or 'simple' in software, as it likely won't be and people will make fun of it.

Somehow during dinner we were discussing and imitating "Hulk talk", i.e. "Hulk smash". People are silly like that, especially people like me. I suddenly hit on the idea that Grok was actually a caveman, and that he talked like this: "me Grok smash ZCML". Everybody immediately liked this idea, as it had a sense of fun and fit the theme of Grok simplifying the new Zope.

We had the luck that my wife Felicia was there. She's a graphics designer, so we asked her Grok the Caveman, so we would have a mascot for the project. She made several drawings and showed them to the group. One of them was everybody's favorite. "But that one looks like me!" I protested. "Exactly," they said, grinning. And thus Grok the caveman was born.

Having a friendly caveman as a mascot/logo for your project actually helped attract people to it. Having such a character gives a project a face and gives people something to relate to. The web site theme was inspired by it, and even a stand-up wooden version of the caveman was made and used at various open source conventions.

I've been toying for a while with the idea of creating an elf character for the Obviel project...

While I did mention cave women in my original blog post on Grok, I wonder now whether our slogan "now even cavemen can use Zope 3" was sufficiently inclusive of women, given all the attention given to gendered language now... The word "cavepersons" would take away a bit of the charm of the ridiculously ahistorical but familiar caveman concept.

Grok Innovation

After I'd gone home from the first Grok sprint I could immediatetely pick up the code and continue working with it. I factored out a library, which contained what I think now was the most important innovation of Grok. This was the notion of scanning Python modules for things (classes, instances, decorators) that needed to be registered with the system. This, we discovered, allowed us to do metaclass-like things with Grok without the need for using actual metaclasses -- particularly picking up and registering relevant classes with the Zope configuration system. Avoiding metaclasses is good if you can get away with it, as they can lead to other unexpected behavior in subclasses. I called the library Martian, to fit the 'grok' theme.

When Chris McDonough started what later became the Pyramid project he used Martian to pick up configuration. Later on he reimagined Martian and he created the Venusian library, which I'm using today in Morepath. But that's another story.

Grok as an Open Source project

Grok as an open source project was moderately successful. A little community of people formed. It managed to attract a few people unfamiliar with Zope technology, and also made the new Zope technology more appealing to people who were already familiar with Zope. We got quite a few contributions, too.

We had regular Grok sprints that helped push Grok forward. We also participated in the Google summer of code for a few years, under the wider banner of the Zope project.

I was lucky enough that quickly after the initial creation of Grok I could start using Grok in my professional life on customer projects. I've been using it as the underpinnings in several large web applications since then.

Various contributors also refactored Grok technology so it could be plugged into the old Zope. From there the Plone community started using it; it's still widely used there today. This whole old Zope compatibility project was entirely pushed by contributors who weren't part of the original group -- a sign of open source success.

Grok and Zope

Grok was built on the new Zope. This gave Grok a running start and a built-in community to draw from, and this was definitely part of Grok's success. But many other Zope developers were not interested in Grok, and quite satisfied with the original ways of doing things. Grok could not have worked as a project to improve Zope itself -- there was insufficient consensus. To create something new, we had to go around the center.

Grok was successful at least for me personally: it made it possible for me to use the new Zope technology without much of the pain. But was Grok successful in being attractive to beginners? We certainly attracted some. But there were problems.

While Grok was simple enough on the surface, its learning curve was not as smooth as we'd have liked. Grok was too much like an iceberg; beneath the surface lurked the vast codebase of the new Zope. If we wanted to make Grok smaller and more comprehensible all the way down, we would need to fix this.

This blog entry is a part of a series on Zope and my involvement with it. Previous. Next.

Lost at CERN

In the summer of 2006 I went to the EuroPython conference, which that year was held in CERN, Switzerland. There was to be a Zope sprint ahead of the conference, so I came a few days earlier.

I got there in the evening. The next morning I tried to look for likely Python programmers. Normally this is not a difficult task: programming geeks tend to have a special look about them. But at CERN this doesn't work: many people at CERN look like they could be a programmer, and many of them of course are, but they were not there for EuroPython.

I got slightly lost in the complex, and was given directions by some gentlemen. They could've been janitors or Nobel prize winners, I will never know. CERN is a special place.

I finally concluded I had planned things a bit wrong, and had arrived at CERN one day earlier than everybody else. So I had a day to kill.

I remember the weather was gloriously sunny and warm. I was hanging out in the cafetaria for a while. I started thinking about my experiences with the new Zope, and its frustrations, and what it would take to turn it into a web framework without these frustrations. A web framework that I would like to use myself and would hopefully also be a bit less intimidating to beginners. Zope was starting to have a problem attracting new people -- the old Zope was out of date, and the new Zope was too intimidating.

The Idea of Grok

I decided to call this new framework Grok, as one has to give things a name. Grok is a term originally from Stranger in a Strange Land, a science fiction novel by Robert Heinlein, and is supposed to be a martian verb for "to understand/to become one with/to drink in". From there the word had made it into programming jargon.

Ruby on Rails had made popular the idea of "convention over configuration"; if you just use some conventions (naming, placing in modules, etc) for the entities in your system, the framework will know what they are without you having to explicitly and verbosely having to tell it. Since the new Zope had a problem with verbose configuration, I figured we could apply convention over configuration to make things more succinct.

In Grok you can use base classes (grok.View, grok.Model, etc) to signal what type of class you're definining, and the system will deduce from that how to register them with the Zope configuration system. Just base classes by themselves aren't enough though -- many configuration actions need parameters. We solved this with Grok directives.

For an example of a directive, You could associate a view with a model using the grok.context() directive. Grok directives are special calls you could make inside your class definition, like this:

class MyView(grok.View):
    grok.context(MyModel)

grok.context would magically apply to the class it is called in, in this case MyView, and set some information on it. These days we'd use a class decorator for this instead, but Python didn't have those yet then.

So far not much convention over configuration was going on. What I had done was found a way to integrate configuration information directly into Python classes so that you wouldn't have to maintain separate XML files anymore for configuration. Convention over configuration came in because you could leave out grok.context and it would then automatically associate the view with a model that happened to be defined in the same module.

It turned out later that convention over configuration in Grok was not as important as I had thought it was then. Some of the magic, even created very deliberately and layered over a worked-out configuration system, intimidated people and sometimes (though rarely) lead to unexpected behavior. The reduction of verbosity by doing configuration in Python near the code being configured was the main win; reducing that configuration further was less important.

When the other sprinters arrived at CERN, I explained some of these ideas to them and received some useful feedback. But it all remained just an idea.

After the Python conference, I was lucky enough to get to see the first web server (the web having been invented at CERN), and the Large Hadron Collider, which, they told us, was going to collide hadrons Real Soon Now.

This blog entry is a part of a series on Zope and my involvement with it. Previous. Next.

Introduction

I can't sleep because some weird Twitter interaction I had earlier. I shouldn't let it bother me, but it did. I received a retweet from someone I didn't know, saying: "Linux doesn't work. There. I said it."

There was a link to a blog entry describing various frustrating problems the author had with Linux, who was sincerely trying it out; applications that didn't do what was asked of them, things generally not cooperating, and intimations that others had experienced pain as well. Constructive feedback all. It also said Linux doesn't work. It said Mac OS X and Windows do work.

Heh, I thought, "Linux doesn't work" isn't really right. I've been using this Linux thing since 1995. Obviously Linux works for me. So I replied: "Heh, Linux works for me. Linux doesn't work for you."

Don't reply stuff to strangers on Twitter! The original person didn't reply (and hasn't yet. probably still resting in all innocence of this whole thing), but someone else jumped in and said I sounded dismissive and unhelpful. So I tried to clarify and defend myself. I didn't mean to dismiss the original writer's negative experiences with Linux, I was just nitpicky about language. "Linux doesn't work" sounded sensationalist to me. What works for one person might not work for another. This lead to a bevvy of tweets back and forth that make me understand Twitter rage a lot better. Maybe Twitter doesn't work. For me. Tonight.

More context

One IRC discussion later with yet more people, I understood the context a bit better. It turned out that the original author was a woman, something I had been completely unaware of, though I started to suspect something was up when the word "mansplaining" was uttered by yet another person. And here I thought I was just being a nitpicker. There are real issues women face in tech, and I had just obliviously stepped into a bit of a hornet's nest. I have more reading to do.

So after I calmed down I reviewed my words; getting caught up in unreasonable defensiveness in 140 line texts about nothing. I sent a few clarifications and "sorry didn't mean to be dismissive of your experience" back to the original writer. At the very least I could have been construed as "not constructive". Sure, it was a pretty mindless nitpicky tweet.

I'm sure this blog entry is part of that defensiveness too, but let's turn it into something constructive instead. Let's have a conversation about Linux, and why it works for me. And to tell some stories. Hopefully nothing sounds mansplainy.

A Very Brief History of Linux Troubles

I have been struggling with issues on Linux so long that I feel quite spoiled these days. Mostly when I install Linux things just work. I remember back in 2000 or so I would have to work hard getting my mouse to work. Back in 2004 or so I remember having to wrestle with nsdiswrapper to install a Windows driver just to make the wireless work. And I remember struggling to hear sound.

Back then I had to be my own sysadmin, or worse. Whatever is worse than being your own sysadmin.

Topsy-Turvy World

These days the hardware support is so much better. Generally after installation the graphics, sound and wireless work. I guess this astounds me so much given my past experience I'm too easily pleased, perhaps.

I was spoiled with Linux graphics support for some years. I'd been using the (properietary) NVidia drivers and they worked quite well on my previous laptop and my current desktop.

In fact, on my previous laptop it worked better than on Windows and Mac OS X, a rare story of a topsy-turvy world that I'll share. My wife got a new Mac OS X laptop at about the same time. It turned out almost all the hardware, from motherboard to video card, was the same as on my laptop. It was just my laptop was cheaper and didn't look very cool. But, you'd say, Mac OS X works out of the box, and with Linux you need to tinker a lot. True usually. Not this time.

With Linux, I could just install the NVidia driver from the repository, reboot, and things worked. On the dual boot Windows partition that I was maintaining for games and rarely used (I've since given it up), I had to find a hacked NVidia driver as for some reason laptop support wasn't officially in there. I think that got fixed since then. Annoying, but tolerable.

With Mac OS X, my wife would start a 3d application that she'd bought the Mac for, and that worked on the previous one, and after a few minutes the laptop would crash horribly. Hard reboot required. In desperation we eventually installed a separate Windows partition and it worked just fine there -- the hardware was fine.

No fix was possible; there was some bug somewhere in the NVidia driver. But it was impossible to upgrade. We had to wait half a year or so for Apple to create an update of their operating system including NVidia driver, and then buy the upgrade to get this bug fixed.

Topsy-turvy world. Not usually true. But a good story.

Present Day Troubles

So last year I got a new laptop, and mindlessly got NVidia again, as it had worked so well for me before, and then was thrust into a chaos of installation issues I just wasn't used to anymore. The reason is NVidia Optimus, a technology to let your laptop save power by using Intel graphics unless demanding 3d applications are run.

That really sucked. I was really frustrated. I didn't need nonsense like that. Eventually I got it sorted out, switching from Fedora back to Ubuntu, but it was painful. I must report that the support has much improved -- I recently reinstalled my laptop and while I still had to do some special installation, it started working then.

Then there was the wireless on the laptop. For some reason when not plugged in the wireless driver would magically decide it was going to go into Ultimate Power Saving Mode and slow my internet down to an absolute crawl. I found a fix which was to turn the powersaving off, but it was annoying. I can report again that this got fixed in more recent versions.

My desktop is usually pretty stable, but I got into the habit of using its suspend feature instead of shutting it down. That worked fine for a long time, but after one update the desktop would come back from suspend, pretend everything was fine for hours on an end, and then suddenly crash. I'm not used to hard freezes on Linux. I hear reports that this finally got fixed too, but I haven't dared to try yet, and switched to shutdowns. It boots up very fast these days too, anyway.

Why?

Why go through this trouble? While Linux is pretty good at it these days, generally speaking Windows and Mac OS X come with better hardware support. Why this Linux desktop? Is this free software ideology?

I don't think so. I'm not really a Linux advocate. I used to be more passionate about it. I do still appreciate the freedom it gives me, where I as a developer or power user can really dig down deep if I'm willing to invest the time.

But now that it's finally finally yes this year for real the year of the Linux desktop, I'm not going to try to convert people. I think software freedoms are important, and wish more people would care, but people care about convenience, which makes total sense, so am not an activist about it.

I'm not a Linux desktop developer either. If I were I'd want to use my own stuff, obviously, and others too, but that's not it.

The reason why Linux works for me is that I'm a developer. And I think Linux is a kick-ass system to develop on, especially web applications, where I know my code is likely to get deployed on Linux servers as well.

Back to 1995

Back in 1995 I first installed Linux on my PC at home. I remember lugging a lot of slackware floppies around, as I didn't have an Internet connection at home yet either. Why did I go through the trouble? Because Linux, for me who didn't have the money to buy software, made for a much better development environment than MSDOS. If I wrote a C program on MSDOS and I made a mistake, the machine would hang, and I'd had to reboot it. If I wrote a C program with Linux, and I made a mistake, I could shut down the process with control-C. Linux simply was a better operating system than MSDOS; it made better use of my hardware at the time. And it came with all these development tools.

Back to the present

Windows and Mac OS X have caught up in the stability department. While Windows Vista, the last Windows I'm somewhat familiar with, made you jump through hoops and reboot a thousand million times if you wanted to upgrade it after a year or more of no upgrades (something I went through last year. Hopelessly confusing and even scary to people unfamiliar with computers, like the ones I was helping), it generally does not crash all that often.

But with Linux I can download and install a huge range of programming environments and libraries and tools, from mainstream to obscure, within seconds. Windows and Mac OS X don't match that. Modern Linux package repositories are awesome for this purpose. I know I could get stuff like that set up with Mac OS X or even Windows, but it'd entail more tinkering than I have patience for. So the scales balance the other way, for me.

Otherwise it's a matter of familiarity and "makes no difference". I'm familiar with Linux. I think the modern deskop environments are pretty slick. Linux makes me comfortable. I'd feel lost if I have to use something else. The other stuff feels clunky. I'm sure I could get used to it, but I don't see a reason to.

The web browser works. That's important. I remember a period in the early 00s when Microsoft had won the browser wars, and Netscape 4.7 on Linux was aging and sites started to be developed for IE only. Thank goodness Firefox brought us out of that situation. The basic freedom of the web is pretty safe now in our multi-browser world.

Even playing games is getting so much better that I dumped Windows entirely. No more dual booting. Thanks Humble Indie Bundle, and thanks Steam! And Kerbal Space Program. I can't play all games available, but enough to keep me happy. It's really awesome, actually; I never thought it'd get this far.

It's now so much easier to maintain a Linux desktop, I barely have to touch a global config file anymore. I don't really feel like I have to play sysadmin anymore. But I'm sure my perspective on it is warped by the years of difficulty that has gone before.

Linux for non-developers?

Can the Linux deskop work for non-developers too? I think it does sometimes, and has. I suspect in some ways a modern Linux desktop can be less confusing in the amount of popups and warnings it gives you than at least Windows Vista could be. It's just that trouble starts when people want to install familiar Windows software. Learning the replacements takes time, and while some stuff will work better, some of it will be worse.

So let me end this with a story. About ten years ago, I was staying with my parents over Christmas. My mother's competent with computers, now so more so than ten years ago, but not an expert by any means.

I was playing with a Linux distribution that could be booted directly from cdrom, without installation, which was new at the time. I had it inserted into my parents laptop, and forgot to take it out when I went up to bed at night. The next morning I came downstairs to find my mother had started the computer, and it had booted into Linux. She was happily playing the games that came with it. I told her how amazing that was; mom, you're using Linux! My mom is using Linux. It just worked.

2003, the year of the Linux desktop. Briefly. We took the cdrom out. She uses Windows again. I haven't pushed Linux on her.

Because while I'm okay with being a sysadmin sometimes, I don't want to be her permanent tech support. That might indeed be worse than being your own sysadmin.

(not because it's my mom! she's great! and she doesn't read this blog, so I'm not just saying that. and tech support people! it's just not for me, that's all! thank goodness this isn't Twitter; didn't fit in 140 characters, this)

There. That's out of my system now. I shouldn't really let this stuff bother me, but what can you do when it does? Hopefully now I can finally sleep. A 7 year old and a 4 year old are visiting in a few hours, joining our toddler in generating energy and activity, demanding my attention. Good night! Zzzz.

Introduction

For a while now I've been working on Morepath. I thought I'd say a bit about it here.

Morepath is a Python web micro-framework with super powers. It looks much like your average Python micro-framework, but it packs some seriously power beneath the hood.

Web micro-framework battle wiki!

Let's start with sample code, a wiki implementation:

https://github.com/morepath/morepath_wiki/blob/master/mpwiki/wiki.py

This code is based on an example devised by Richard Jones for his web micro-framework battle presentation. When I asked him, he kindly shared his codebase with implementations for various micro-frameworks.

The Morepath wiki code uses the reusable module storage.py that Richard created which implements the wiki backend and page rendering. This is actually a little bit unfortunate for Morepath, as storage.py doesn't expose an object oriented API. There is no wiki Page class for instance, so I had to come up with a simple one myself, as Morepath is very model-driven. Perhaps I'll refactor storage.py in the future so I can better show off how Morepath can empower your models.

Configuration

Morepath at first sight looks quite a bit like a Flask application. You create an application object. You then use methods on the application object as decorators to configure it.

But there is a significant difference with Flask (or many other web frameworks, including Django) in that loading the configuration is not a import-time side-effect, but an explict step. Morepath is like Zope and Pyramid in this respect. The Pyramid documentation has an extensive section explaining why this is a good thing.

Routing to Models

Morepath's routing is different from any routing web framework I've encountered before. Morepath routes to models, not to views. In the wiki example the route to the wiki page model is done like this:

@app.model(model=Page, path='{name}',
           variables=lambda model: {'name': model.name})
def get_page(name):
    if not storage.wikiname_re.match(name):
        return None
    return Page(name)

This says that paths like /foo go to a Page object with the name foo. If no such wiki page exists, we return None, which will result in a 404 error.

Views

The routing code where the model is looked up is completely separate from presentation. That's done in the view. Here's the default view for Page (i.e. /FrontPage):

@app.html(model=Page)
def display(request, model):
    return wiki.render_page(model.name)

And here is the edit view (i.e. /FrontPage/edit):

@app.html(model=Page, name='edit', request_method='GET')
def edit_form(request, model):
    return wiki.render_edit_form(model.name)

Link generation

Because Morepath routes to models, it can construct a link for a model, which we see demonstrated for instance in a redirect response:

return redirect(request.link(model))

request.link() receives a model and optionally a view name. Using the variables argument on the @app.model decorator it can reconstruct the path for you from the model. You don't need to remember route names or what parameters go into them. Your links will continue to work if you change your app's routes completely.

Reuse and Flexibility

Morepath is powered by the generic function library Reg. This provides a general mechanism for composing, extending and overriding behavior.

In the wiki example, the Page model has a default display view, and edit and history views. If you were to create a subclass from Page called DiscussionPage, instances of DiscussionPage would automatically share these views. But you can specialize: if you want your DiscussionPage model to have a special way to display itself but still share edit and history, this is what you'd do:

@app.html(model=DiscussionPage)
def discussion_display(request, model):
    return "<h2>A different discussion page!</h2>"

With Morepath, you can bundle a set of behaviors such as views (and routes) in a custom framework application, and then let others reuse it in their own applications. But if in the same run-time you want to have another application that doesn't share any of this behavior, Morepath lets you do this too.

That's a lot of power in a small package. It's in fact a condensation of 15 years of my experience with Zope packed together into a micro-framework.

Development Status

I'm actively developing Morepath and looking for feedback. Documentation is very scarce right now, but that situation won't continue for long.

The Morepath code is here:

https://github.com/morepath/morepath

At PyCon DE earlier this month I gave a keynote speech where I go into some of the creative process and thinking behind Morepath. If you have more than an hour to spare, you can watch it.

There's an IRC channel, #morepath on freenode. Hope to see you there!