Cassandra and how PyPy could help with Python 3

It's probably something everything has experienced in one way or another, but I now know a little what Cassandra felt like. No, not the Cassandra database but the original Cassandra of Troy. Recently, Aaron Swartz posted this blog entry about Python 3. In it he proposes a Python runtime that can run both Python 2 and Python 3 at the same time, to help cross the chasm that exists between Python 2 and Python 3.

This is something I remember blogging about, and I think also talking to Guido about, back in 2007:

http://faassen.n--tree.net/blog/view/weblog/2007/6/22/0

The idea was rejected back then as far as I can recall. It would of course make, as Aaron also say, the Python runtime a lot more complicated. Recently Nick Coghlan told me in a comment here:

You don't know what a pleasure it is to be able to work on a code base that isn't weighed down by all the cruft that 2.x had accumulated. Things like the efficient Unicode representation coming in 3.3 would have been orders of magnitude harder in 2.x (which is the main reason why it's only happening now that at least some of the technical debt has been paid down).

I of course do understand what a pleasure it is to work on a clean codebase that isn't weighed down by cruft; I think any experienced programmer does. But we're moving cruft now:

• many developers of libraries seem to have concluded that 2 to 3 isn't working for them. Instead they are maintaining codebases that support both languages at the same time.

• this leads to cruft in the libraries.

So we have to consider whether it is worth it to shift cruft from the language implementation to the library implementations, and the burden of complexity to the developers of these libraries. This cruft will, of course, in a nebulous future, be removed from the library implementations again.

Adding the cruft and then removing it again is work. I'm a lazy programmer and I'm not ashamed of that; Larry Wall spoke wisdom when he called laziness a programmer's virtue. So I think trying to shame developers who contributed useful code into porting code to Python 3 is not exactly a noble effort. I've been through the Zope 2/Zope 3 mess (which made me more sensitive when Python 3 came along) and I've helped make those work together better - I've paid my developer dues in this area.

So I still think an interpreter that supported modules written in Python 2 and Python 3 at the same time, with perhaps a central index that marks which modules and packages are in which version in the language, would be our best hope to make the transition between Python 2 and Python 3 more smooth, and let lazy developers start to deal with this language. I am glad to see Aaron has similar ideas. There are serious, complicated, questions to be answered of course: how (or whether) to handle C extensions, for instance. But at least that complexity would be isolated in the interpreter and under control by one group of developers, and not be devolved onto library maintainers, or even application maintainers.

It'd be nice to see a sense of balance, and that the developers who wanted the change in the language were the ones who also took on the burden of the dual runtime. But I don't think that's going to happen.

Nobody likes a Cassandra. I don't want to be just a Cassandra.

I think the Python community's best hope to have a runtime happen that supports both Python 2 and Python 3 code at the same time is the PyPy project. So far PyPy's efforts to create a Python 3 implementation have gone the same way as CPython did: the codebase that implements Python in PyPy was forked. But I believe PyPy has the best interpreter infrastructure to try to tackle this problem.

Showing that even a lazy developer will spend a little effort talking when he thinks it might be useful, I brought up this idea on the Python mailing list a few months ago. I got the hopeful feedback that it should be possible to run the Python 3 and Python 2 interpreters in the same runtime with PyPy - it is a good starting point for such a project. And that then a lot of work would be required after that to integrate the two. It hasn't gone anywhere further since that discussion as far as I'm aware. But I hope the PyPy developers are at least thinking about it. It would, I think, make PyPy even more relevant to Python developers than it already is today.

Sometimes the PyPy project asks for monetary contributions for parts of their effort. If they asked for sponsorship of such a Python 2/Python 3 compatible runtime, then I'd be happy to help pay for it in my own little way, and perhaps others will too.

Python 2.8: +1

Armin Ronacher bravely shares his thoughts on Python 3.

He also says: "Can we reopen the option of doing a Python 2.8 if it makes porting easier?"

A big +1 from me. Declaring that a Python 2.8 is never going to happen is just bizarre from my perspective. What is this supposed to accomplish? Encourage people to port code to Python 3? And if so, is this really the right way to encourage people?

Why do I use the word bravely? Because when I shared my worries about the upgrade discontinuity between Python 2 and Python 3 back in 2007, I got burned pretty badly in the ensuing discussions. Please don't hurt Armin: what he's doing is very brave and I'm very glad someone is speaking up about this topic.

I did think, back then, that I had something to contribute to this discussion: I've been through the Zope 2 and Zope 3 mess as an active participant. It's been sort of resolved now. It took about 10 years. People are still using Zope 2 and that's what's called "Zope" today. People are using what was called Zope 3 too, as part of Zope, and as part of other systems with different names. The Zope community is now a bunch of pieces. Now Python is a programming language, not a web framework, and the situations are not identical at all (there are MANY points of difference), but it does inform my perspective on the situation the Python community finds itself in.

I've been somewhat reluctant to share my opinions about the Python 3 situation in public since then. But what the heck.

I know what else a "Never a Python 2.8" decision by the core team might encourage: a Python 2.x maintenance fork done by other people. And the resulting situation would be a mess. I hope we can avoid that.

P.S. Disclaimer: this is not to be construed as a criticism of the technical merits of Python 3, the efforts of involved developers to make code run on Python 3, and it's not to be construed as a criticism on the intelligence or wisdom of Python developers anywhere. Python is awesome! Please don't hurt me!

The New Hot Thing in Web Development: Client-side Templating Languages

Back in 2003 I found myself at a sprint (a hackaton) to help build a web framework. But that's not what I'm talking about right now.

For some reason I was messing around with JavaScript. This was before JavaScript was cool. Let's give an impression of this era.

JavaScript was this abomination of a language used only by front-end designers. Horrible code was written in it. The term AJAX hadn't been coined yet - that was years into the future. We didn't really have the browsers for it anyway, at least on Linux. Netscape 4.5 was the state of the art; Firefox had barely been born and was still called Phoenix. It was that dark moment in net history where it looked like Internet Explorer might have won the browser wars for good.

So I was messing around with JavaScript. Zope, which I was using, had this then fairly new template language called ZPT. I was wondering whether I could implement ZPT in JavaScript and do templates in the browser. I told other developers about it, and they all asked "why?" My answer was something like "I don't know man, it's just cool!". I remember I even worked on implementing unit tests for it in JavaScript. But I genuinely didn't know what I'd use it for; how was I even going to get data into it? JSON as a concept didn't exist yet either so I probably was thinking about using XML.

The project went nowhere, of course. The idea was way ahead of its time; so far ahead it was in fact still a crazy idea. I'm sure many people had thought of it; even back in 2003 people had been doing crazy things with JavaScript for years.

We flash forward to 2009. I was working on a project that involved a web application that needed a dynamic search interface. It involved JavaScript. I had noticed this client-side templating language called JSON-Template. I had already used its server-side Python implementation for another project to provide customizable email templates in an application. I figured we could write a little JavaScript framework to make it easy to use JSON-Template. So we did that. And we built the user interface, and it worked quite well, and the code for it was pretty easy to follow too, so that was good.

The idea wasn't crazy anymore: client-side templating languages were now useful! Over the course of the last year and a half a bunch of us refined the ideas of this browser-side framework some more, and this resulted into the creation of Obviel (which does much more than just support client-side templates).

Many client-side templating languages, JSON-Template included, are minimalistic. You combine a JSON object with a text file that has some templating instructions in it to render a template. JSON-Template doesn't even feature an expression language.

JSON-Template is a push only template language. Many templating languages (but not JSON-Template) are pull: allow you to embed calls to bits of application code within the template: some_object.get_something(). But get_something could do anything; call the database, raise exceptions, take a long time to execute, etc. If you write templates that call into the application's API, you can easily create a fairly tight coupling between application and template. This means the template becomes harder to test and debug and change; the bits of the application used by the template need to be available as well during testing and debugging. It also means the application becomes harder to change, because who knows what methods are being called by templates?

A push-only template language doesn't have such a problem. But how do you use such a minimalistic template language for complex scenarios where all sorts of decisions need to be made and data needs to be massaged? You can't do it in the template. Instead, you do the work in advance, in JavaScript or Python or Ruby or whatever language you are using. This means the logic is written in a full-featured programming language which was designed to implement complicated logic, instead of in a template language. It also means this logic is easier to test and debug in isolation, without the template language being involved. Loose coupling is awesome that way!

In fact, I believe many of the benefits of minimalistic push-only template languages exist on the server as well as the client, and would encourage people to write push-style templates also when writing server-side templates.

Let's go through some benefits of minimalistic push-only templates:

• they can be more secure, as there's no application code called from the template.

• they're more easy to test in isolation. The stuff going into the template is also more easy to test in isolation.

• you end up with less complicated templates that are easier to maintain. I don't think that means the complexity simply moves to another place -- complexity is actually reduced. This is because a full featured programming language will let you express complex logic better than a template language can; better often means simpler.

• since templates are sandboxed, you can let non-programmers edit templates in some cases, without them needing to know about your application's API. This is helpful if you have a customizable application.

• the template language can be made to be very fast indeed as the language hardly has to do anything.

• they can be run on the client (in the web browser) too, not just on the server.

Running a template language on the client instead of on the server has some important benefits as well:

• In this modern age, we build a lot of dynamic client-side applications. Templates can be very handy in helping to structure them, avoiding hard to maintain blobs of HTML-generating JavaScript.

• You almost have to use push strategies for templates, as you can't really avoid it - the data has to get to the client from the server. So doing things on the client can encourage more loose coupling in your codebase, and loose coupling is good.

• The performance of the template language becomes less important.

I'll go into the last statement a bit more deeply. It's a bit of a paradox: while minimalistic push-only template languages can render templates very quickly, running them on the client means they don't really have to be that fast after all.

On the server, if you have a thousand users, a thousand users are exercising your server-side template language engine. It can become a bottleneck.

On the client, there's only ever one user: the user of the web browser. If the template language is fast enough for that user, you're done. Your server-side application may have a thousand users more, but the template language performance is no longer a bottleneck. (Your JSON serializer is. but JSON serializers can be very fast)

So can we exploit that? Since our language doesn't need to be blazingly fast, can we do neat things that might take a bit more time? I think so. I've been thinking for a while now about a template language much like JSON-Template that would work with the HTML DOM as opposed to being text based like JSON-Template is. This would likely make it slower, but we can make it fast enough for a single user for sure. And then we can do "live" binding between the JSON object being rendered and the DOM tree rendered by the template. That would allow:

• update your data object and have the rendered template update in place, without re-rendering.

• imagine you're rendering an array of objects in your JSON, as some kind of HTML list or table. Each li or tr in fact is associated with an object in array. Now if you're setting up an event handler for each item in HTML, it's useful to know what underlying JSON object was used there, so you can use data that is in that object but not rendered in the event handler; an example of such data is a URL.

I'm sure I wasn't the first person to have such an idea, and I'd like to hear about implementations of it if you know any. (in fact, I created a hack based on JSON-Template back in 2009 that could do the second bit). I hope to eventually integrate such a templating language into Obviel. If you're interested in talking more about this, I hope to talk to you in the comments or on the Obviel mailing list!

Obviel and client-side templates

Armin Ronacher just wrote an interesting blog post.

Obviel is a JavaScript based glue toolkit that lets you assemble web applications on the client side. It integrates with client-side template languages; by default, Obviel works with JSON-template, which also has server-side implementations (in Python and other languages). Obviel is one approach to do some of what Armin is talking about. I've used Obviel to build a number of web applications already, and am looking for more to work on.

Last time I spoke about Obviel on this blog it didn't have any live demos. Now it has a few. We're working on more.

Obviel

WSGI: Bringing web frameworks closer together

Recently I have seen a sentiment espoused by some people in the Python web community that WSGI has failed to live up to one of this promises: bringing Python web frameworks closer together.

Let's look at history and see whether this is true.

Here's a snapshot of the Python web framework landscape in 2005:

• Zope had its own way of hooking up to web servers. Several ways. FastCGI, SCGI, Apache Rewrite rules, ProxyPass.

• Django had just emerged, with its own way of hooking up to web servers. Several ways too; I can find references to using mod_python, FastCGI, SCGI and AJP.

• TurboGears had also just emerged. TurboGears 1.x used CherryPy as its web server. A quick search turns up references to hooking it up to Apache using mod_python, and using mod_rewrite.

Let's look at the web framework landscape in 2011:

• Zope (in all its incarnations) can be hooked up to web servers to WSGI, and this is generally the preferred method.

  • Zope 2 comes with native WSGI support.

  • Grok has native WSGI support and uses paster as its development server.

  • BlueBream has native WSGI support and uses paster as its development server.

• Django has WSGI as the preferred method to hook up to web servers.

• TurboGears is now built around WSGI and uses paster as its development server.

• Both Pylons the web framework as well as Pyramid are built around WSGI, and both use paster as its development server.

[no claim is made that this represents all Python web frameworks out there]

Has this brought web frameworks closer together? The answer is clearly a resounding YES: if you know how to hook up a WSGI application to your web server of choice, you have basic knowledge to deal with all of these web frameworks. You can use Apache mod_wsgi with all of them, for instance.

In addition, only Zope 2 and Django do not use paster as their default development web server. We see a major component being shared between Grok, BlueBream, TurboGears, Pylons and Pyramid. This knowledge transfers between web frameworks.

So this sentiment is clearly wrong. Thank you to everybody who has helped create and push WSGI!

Other arguments have been made, for instance that WSGI middleware in particular isn't bringing Python web frameworks closer together. I believe that's wrong too, but I will leave defending that point to someone more familiar with that topic.

how to handle ideas

Over the years I gained some experience with how open source communities function. Sometimes these communities make it difficult for relative outsiders (or even insiders) to make a proposal or to share an idea.

Recently I made a proposal on a mailing list of a project. It's a very useful project, and there are fine, smart people working on it. But here are some of the responses I got:

• no! this is wrong! (no explanation)

• your use cases can be solved in another way. Therefore our project automatically doesn't need to tackle them.

• actually I don't use our own project for these use cases even though I know many others are. Instead I use an entirely different system, and you should too.

• do it yourself! solving these use cases for yourself is trivial! you just need to write your own tools, there is no documentation on how to do it, and there are a lot of details you need to worry about. Therefore this project is not going to handle your use cases, everybody is on their own on this one. But really it's trivial. trivial, really.

• what you are proposing is immoral.

I humbly submit that this is not a very constructive set of responses. It sounds lot like "go away". This will likely frustrate the person who made the proposal. They can:

• go away, therefore nobody got educated and nothing was improved.

• go away and not come back with new ideas.

• show their frustration, therefore putting the people in the project on the defensive. This rarely leads to a good discussion either.

These were smart, well-meaning people. I got more constructive responses too. Still, evidently it's easy for a group of such people to still give this kind of feedback.

Let's discuss what would be better responses to someone sharing an idea.

The best response for the proposer and the people who run the project of course would be:

• this is already solved: here's the solution

Sounds great, right? And it is, if it's true.

Unfortunately the desire to give this response on the part of the project tempts people to say this when it isn't exactly true. It's tempting to present "do it yourself" as "it's already solved". So the "here's the solution" part is important.

There's another good response for the proposer:

• hey, that's a good idea! let's work together to implement it!

This is less good for the people who run the project, as now they need to do work. Still, if this is a good idea, it'll be good for everybody, so nobody will really be complaining about this: they're in a community to get things solved together, after all. And in the best case scenario the proposer will actually join the project and help out.

But of course an idea is not always obviously a good idea. Perhaps the idea needs some tweaking. Perhaps the motivations behind the idea need to be better understood. So another great response to a proposal would be:

• hm, I'm not sure about this. let's analyze your use cases in some more detail.

So now the people in the project start asking questions, give suggestions, consider solution strategies. The idea will likely change in shape as things become more clear. The project, and the person making the proposal, may end up with new features and at the very least the project will have learned something. Incidentally this is also a great way to draw newcomers into a project.

Sometimes however a project will have to say no. If it's immediately obvious why no, this generally means this discussion has come up in the past. Perhaps it's time to document the decision on this in the project documentation? Then the project can point newcomers to the documentation. Perhaps they'll read the documentation before they even arrive on your mailing list. Or perhaps, just maybe, if this idea keeps coming up, is there something to it after all?

Here are some good ways to reject an idea:

• our project is not actually about fulfilling these use cases. Here are the project's goals, and here's why your idea won't fit.

• solving this might fit our project's goals, but would increase our maintenance burden to levels not acceptable to the project, and here's why. We weigh this against how common the use case really is.

If you handle newcomers and new ideas well, your project stands a better chance at growing, or at least remaining viable over time as people leave and new people come in.

Perhaps eventually your project will be so successful there will be masses of ill-formulated ideas that your project just can't handle. Then it's time to start coming up with other strategies to deal with the situation. Generally these would involve some form of layering or partitioning in the project, so that not everybody is swamped by everything all the time. Most projects aren't there yet, though, and even many successful projects will never grow to this size.

web frameworks considered useful

There is a a strain of thought in the Python web development community these days that considers web frameworks a bad idea. Even beginners are sometimes told: why do you need a framework anyway? Just build your app from scratch with WSGI! Or just compose your own out of existing libraries and tools!

I'm going to argue that frameworks are useful. I'm going to argue that we should normally not be telling beginners to avoid frameworks. And I'm going to argue that experienced developers should carefully consider whether their perspective isn't warped a little by their experience, and that frameworks can be useful for them too.

ME GROK BOOK

This is awesome. There is now a published book about Grok!

Thank you, Carlos, very much, for writing this book. That was a lot of work!

Why would you, Python developer, be interested in Grok? Why not learn about one of the myriad other Python web frameworks instead? Kevin Teague in a recent reddit thread put it very well, so I'll just quote him:

Forget all those Django, TurboGears, Web2Py, Pylons arguments, Grok is where it's at ... Why Grok?

• Schema and model are not tightly coupled: use objects from Zope Object Database, LDAP, SQLAlchemy, Web services, etc. and be able to auto-generate forms regardless of the persistence implementation.

• Convention-over-configuration: Introspects your code, so that you get the benefits of succinct, less-typing benefits convention, without forcing you to layout your software orthogonal to it's purpose. e.g. doesn't force "models" into a models module, "views" into a views module, if you have a Book model and Book view, you can put them in the same book.py file.

• Well packaged (mostly, Grok 1.1 provides a lot of fixes to the dependency tree), so easier to manage deployments. Also core parts are individually packaged, so re-useable outside of Grok (primarily of interest to Plone developers).

• Cavemen are way tougher than Ponies!

What does Grok the caveman himself have to say about a book on Grok? Or about any book, even, as he can't tell the difference?

ME GROK FIND BOOK!

BOOK STRANGE!

ME GROK LIKE HAS MANY PAGE

MANY MANY PAGE SQUIGGLE!

ME GROK THINK SQUIGGLE ARE FLY DROPPING?

BOOK GREAT MAGIC!

ME GROK TEAR OFF PAGE

PAGE NICE AND CHEWY TO EAT

PAGE GOOD MAKE FIRE WITH

PAGE GOOD WIPE BOT[censored]

Um, sorry about that. I think we should let more literate people review the book instead.

ME GROK JUST KIDDING!

ME GROK LOVE BOOK

ME GROK LEARN ALL ABOUT WEB APP

ME GROK WONDER WEB APP BETTER EAT THAN MAMMOTH?

Python-based configuration in a Grok near you

BFG 1.2 offers imperative configuration: doing configuration not in ZCML but in Python. The declarative configuration system is built on top of this. This is an interesting approach that has some merit.

It's interesting to compare this to Grok's configuration system, which also focuses on Python, not ZCML. Grok however offers a declarative configuration system in Python, not an imperative one.

It's important to note that from the start Grok's declarative configuration system allowed the user to be entirely explicit about how configuration happens. I don't know whether Chris was referring to Grok when he mentioned that in BFG no false choice between convention over configuration and ZCML is offered, but in case anyone had the wrong impression: Grok doesn't offer such a choice. You can be as explicit as you like. Relying on convention-based configuration patterns is an option when using Python-based configuration, but not an obligation.

Grok's declarative configuration system can be used in tests. If you have a component that you need to register, you can use grok.testing.grok_component to automatically configure it according to the standard Grok configuration rules.

Here is an example, a component:

import grok
class MyAdapter(grok.Adapter):
   grok.context(AdaptedFrom)
   grok.provides(IAdaptedTo)

This adapter when configured, be registered as allowing instances of class AdaptedFrom to objects providing interface IAdaptedFrom.

If we want to configure this adapter in our tests, we can use grok_component:

>>> from grok.testing import grok_component
>>> grok_component('MyAdapter', MyAdapter)
True

The grok_component function can also be imported from the reusable grokcore.component library, through grokcore.component.testing.

Interesting about this design is that Grok allows the use of the same configuration system for test-configuration as well as real configuration. There is no need to learn a different configuration approach when running tests.

Grok's focus on configuration as an area to challenge existing Zope assumptions is long-standing. Grok strikes a different balance than BFG as Grok aims to remain compatible with existing Zope-based systems.

Nothing's perfect. So, I'll close with some areas where we should improve Grok's configuration system:

• while it is very convenient to be able to register a component with the same configuration as it would have in a real application (a common case), we might need a way to support registering an existing component differently. Subclassing probably usually does provide this.

• we need to better promote grok_component as a well-understood tool by writing better documentation and more tests. Often I myself fall back on using the zope.component registration APIs in tests directly myself, and I need to understand why. Perhaps this is simply something to get used to, or perhaps the reasons are more fundamental and there is a BFG-like imperative configuration system at some level.

• we need to fix the API so we can avoid the first argument to grok_component, which is, if I recall it correctly, mostly useless.