Skip to main content

Cassandra and how PyPy could help with Python 3

It's probably something everything has experienced in one way or another, but I now know a little what Cassandra felt like. No, not the Cassandra database but the original Cassandra of Troy. Recently, Aaron Swartz posted this blog entry about Python 3. In it he proposes a Python runtime that can run both Python 2 and Python 3 at the same time, to help cross the chasm that exists between Python 2 and Python 3.

This is something I remember blogging about, and I think also talking to Guido about, back in 2007:

http://faassen.n--tree.net/blog/view/weblog/2007/6/22/0

The idea was rejected back then as far as I can recall. It would of course make, as Aaron also say, the Python runtime a lot more complicated. Recently Nick Coghlan told me in a comment here:

You don't know what a pleasure it is to be able to work on a code base that isn't weighed down by all the cruft that 2.x had accumulated. Things like the efficient Unicode representation coming in 3.3 would have been orders of magnitude harder in 2.x (which is the main reason why it's only happening now that at least some of the technical debt has been paid down).

I of course do understand what a pleasure it is to work on a clean codebase that isn't weighed down by cruft; I think any experienced programmer does. But we're moving cruft now:

  • many developers of libraries seem to have concluded that 2 to 3 isn't working for them. Instead they are maintaining codebases that support both languages at the same time.
  • this leads to cruft in the libraries.

So we have to consider whether it is worth it to shift cruft from the language implementation to the library implementations, and the burden of complexity to the developers of these libraries. This cruft will, of course, in a nebulous future, be removed from the library implementations again.

Adding the cruft and then removing it again is work. I'm a lazy programmer and I'm not ashamed of that; Larry Wall spoke wisdom when he called laziness a programmer's virtue. So I think trying to shame developers who contributed useful code into porting code to Python 3 is not exactly a noble effort. I've been through the Zope 2/Zope 3 mess (which made me more sensitive when Python 3 came along) and I've helped make those work together better - I've paid my developer dues in this area.

So I still think an interpreter that supported modules written in Python 2 and Python 3 at the same time, with perhaps a central index that marks which modules and packages are in which version in the language, would be our best hope to make the transition between Python 2 and Python 3 more smooth, and let lazy developers start to deal with this language. I am glad to see Aaron has similar ideas. There are serious, complicated, questions to be answered of course: how (or whether) to handle C extensions, for instance. But at least that complexity would be isolated in the interpreter and under control by one group of developers, and not be devolved onto library maintainers, or even application maintainers.

It'd be nice to see a sense of balance, and that the developers who wanted the change in the language were the ones who also took on the burden of the dual runtime. But I don't think that's going to happen.

Nobody likes a Cassandra. I don't want to be just a Cassandra.

I think the Python community's best hope to have a runtime happen that supports both Python 2 and Python 3 code at the same time is the PyPy project. So far PyPy's efforts to create a Python 3 implementation have gone the same way as CPython did: the codebase that implements Python in PyPy was forked. But I believe PyPy has the best interpreter infrastructure to try to tackle this problem.

Showing that even a lazy developer will spend a little effort talking when he thinks it might be useful, I brought up this idea on the Python mailing list a few months ago. I got the hopeful feedback that it should be possible to run the Python 3 and Python 2 interpreters in the same runtime with PyPy - it is a good starting point for such a project. And that then a lot of work would be required after that to integrate the two. It hasn't gone anywhere further since that discussion as far as I'm aware. But I hope the PyPy developers are at least thinking about it. It would, I think, make PyPy even more relevant to Python developers than it already is today.

Sometimes the PyPy project asks for monetary contributions for parts of their effort. If they asked for sponsorship of such a Python 2/Python 3 compatible runtime, then I'd be happy to help pay for it in my own little way, and perhaps others will too.

Python 2.8: +1

Armin Ronacher bravely shares his thoughts on Python 3.

He also says: "Can we reopen the option of doing a Python 2.8 if it makes porting easier?"

A big +1 from me. Declaring that a Python 2.8 is never going to happen is just bizarre from my perspective. What is this supposed to accomplish? Encourage people to port code to Python 3? And if so, is this really the right way to encourage people?

Why do I use the word bravely? Because when I shared my worries about the upgrade discontinuity between Python 2 and Python 3 back in 2007, I got burned pretty badly in the ensuing discussions. Please don't hurt Armin: what he's doing is very brave and I'm very glad someone is speaking up about this topic.

I did think, back then, that I had something to contribute to this discussion: I've been through the Zope 2 and Zope 3 mess as an active participant. It's been sort of resolved now. It took about 10 years. People are still using Zope 2 and that's what's called "Zope" today. People are using what was called Zope 3 too, as part of Zope, and as part of other systems with different names. The Zope community is now a bunch of pieces. Now Python is a programming language, not a web framework, and the situations are not identical at all (there are MANY points of difference), but it does inform my perspective on the situation the Python community finds itself in.

I've been somewhat reluctant to share my opinions about the Python 3 situation in public since then. But what the heck.

I know what else a "Never a Python 2.8" decision by the core team might encourage: a Python 2.x maintenance fork done by other people. And the resulting situation would be a mess. I hope we can avoid that.

P.S. Disclaimer: this is not to be construed as a criticism of the technical merits of Python 3, the efforts of involved developers to make code run on Python 3, and it's not to be construed as a criticism on the intelligence or wisdom of Python developers anywhere. Python is awesome! Please don't hurt me!

The New Hot Thing in Web Development: Client-side Templating Languages

Back in 2003 I found myself at a sprint (a hackaton) to help build a web framework. But that's not what I'm talking about right now.

For some reason I was messing around with JavaScript. This was before JavaScript was cool. Let's give an impression of this era.

JavaScript was this abomination of a language used only by front-end designers. Horrible code was written in it. The term AJAX hadn't been coined yet - that was years into the future. We didn't really have the browsers for it anyway, at least on Linux. Netscape 4.5 was the state of the art; Firefox had barely been born and was still called Phoenix. It was that dark moment in net history where it looked like Internet Explorer might have won the browser wars for good.

So I was messing around with JavaScript. Zope, which I was using, had this then fairly new template language called ZPT. I was wondering whether I could implement ZPT in JavaScript and do templates in the browser. I told other developers about it, and they all asked "why?" My answer was something like "I don't know man, it's just cool!". I remember I even worked on implementing unit tests for it in JavaScript. But I genuinely didn't know what I'd use it for; how was I even going to get data into it? JSON as a concept didn't exist yet either so I probably was thinking about using XML.

The project went nowhere, of course. The idea was way ahead of its time; so far ahead it was in fact still a crazy idea. I'm sure many people had thought of it; even back in 2003 people had been doing crazy things with JavaScript for years.

We flash forward to 2009. I was working on a project that involved a web application that needed a dynamic search interface. It involved JavaScript. I had noticed this client-side templating language called JSON-Template. I had already used its server-side Python implementation for another project to provide customizable email templates in an application. I figured we could write a little JavaScript framework to make it easy to use JSON-Template. So we did that. And we built the user interface, and it worked quite well, and the code for it was pretty easy to follow too, so that was good.

The idea wasn't crazy anymore: client-side templating languages were now useful! Over the course of the last year and a half a bunch of us refined the ideas of this browser-side framework some more, and this resulted into the creation of Obviel (which does much more than just support client-side templates).

Many client-side templating languages, JSON-Template included, are minimalistic. You combine a JSON object with a text file that has some templating instructions in it to render a template. JSON-Template doesn't even feature an expression language.

JSON-Template is a push only template language. Many templating languages (but not JSON-Template) are pull: allow you to embed calls to bits of application code within the template: some_object.get_something(). But get_something could do anything; call the database, raise exceptions, take a long time to execute, etc. If you write templates that call into the application's API, you can easily create a fairly tight coupling between application and template. This means the template becomes harder to test and debug and change; the bits of the application used by the template need to be available as well during testing and debugging. It also means the application becomes harder to change, because who knows what methods are being called by templates?

A push-only template language doesn't have such a problem. But how do you use such a minimalistic template language for complex scenarios where all sorts of decisions need to be made and data needs to be massaged? You can't do it in the template. Instead, you do the work in advance, in JavaScript or Python or Ruby or whatever language you are using. This means the logic is written in a full-featured programming language which was designed to implement complicated logic, instead of in a template language. It also means this logic is easier to test and debug in isolation, without the template language being involved. Loose coupling is awesome that way!

In fact, I believe many of the benefits of minimalistic push-only template languages exist on the server as well as the client, and would encourage people to write push-style templates also when writing server-side templates.

Let's go through some benefits of minimalistic push-only templates:

  • they can be more secure, as there's no application code called from the template.
  • they're more easy to test in isolation. The stuff going into the template is also more easy to test in isolation.
  • you end up with less complicated templates that are easier to maintain. I don't think that means the complexity simply moves to another place -- complexity is actually reduced. This is because a full featured programming language will let you express complex logic better than a template language can; better often means simpler.
  • since templates are sandboxed, you can let non-programmers edit templates in some cases, without them needing to know about your application's API. This is helpful if you have a customizable application.
  • the template language can be made to be very fast indeed as the language hardly has to do anything.
  • they can be run on the client (in the web browser) too, not just on the server.

Running a template language on the client instead of on the server has some important benefits as well:

  • In this modern age, we build a lot of dynamic client-side applications. Templates can be very handy in helping to structure them, avoiding hard to maintain blobs of HTML-generating JavaScript.
  • You almost have to use push strategies for templates, as you can't really avoid it - the data has to get to the client from the server. So doing things on the client can encourage more loose coupling in your codebase, and loose coupling is good.
  • The performance of the template language becomes less important.

I'll go into the last statement a bit more deeply. It's a bit of a paradox: while minimalistic push-only template languages can render templates very quickly, running them on the client means they don't really have to be that fast after all.

On the server, if you have a thousand users, a thousand users are exercising your server-side template language engine. It can become a bottleneck.

On the client, there's only ever one user: the user of the web browser. If the template language is fast enough for that user, you're done. Your server-side application may have a thousand users more, but the template language performance is no longer a bottleneck. (Your JSON serializer is. but JSON serializers can be very fast)

So can we exploit that? Since our language doesn't need to be blazingly fast, can we do neat things that might take a bit more time? I think so. I've been thinking for a while now about a template language much like JSON-Template that would work with the HTML DOM as opposed to being text based like JSON-Template is. This would likely make it slower, but we can make it fast enough for a single user for sure. And then we can do "live" binding between the JSON object being rendered and the DOM tree rendered by the template. That would allow:

  • update your data object and have the rendered template update in place, without re-rendering.
  • imagine you're rendering an array of objects in your JSON, as some kind of HTML list or table. Each li or tr in fact is associated with an object in array. Now if you're setting up an event handler for each item in HTML, it's useful to know what underlying JSON object was used there, so you can use data that is in that object but not rendered in the event handler; an example of such data is a URL.

I'm sure I wasn't the first person to have such an idea, and I'd like to hear about implementations of it if you know any. (in fact, I created a hack based on JSON-Template back in 2009 that could do the second bit). I hope to eventually integrate such a templating language into Obviel. If you're interested in talking more about this, I hope to talk to you in the comments or on the Obviel mailing list!

Obviel and client-side templates

Armin Ronacher just wrote an interesting blog post.

Obviel is a JavaScript based glue toolkit that lets you assemble web applications on the client side. It integrates with client-side template languages; by default, Obviel works with JSON-template, which also has server-side implementations (in Python and other languages). Obviel is one approach to do some of what Armin is talking about. I've used Obviel to build a number of web applications already, and am looking for more to work on.

Last time I spoke about Obviel on this blog it didn't have any live demos. Now it has a few. We're working on more.

Obviel

Introduction

This document is an introduction to Obviel from the perspective of how it was developed. For those impatient with history, you can just go to http://www.obviel.org and read the documentation there, however.

Genesis

A few years ago I worked on a dynamic web user interface on a project for Gocept, using JavaScript of course. There I invented a somewhat unusual approach to do:

  • use JSON Template to do client-side templating
  • the server would return a JSON object and the client side template would then be used to render it.

Callback Complexity

In the fall of 2010, I working on a few related customer projects to create complex web applications that needed a good UI. As they were applications, not web sites, I decided to create those UIs using a single-page approach. The server would just present a single web page with JavaScript to build and update the UI.

We decided to use jQuery for those projects. jQuery is very popular, with a vibrant community with a lot of components choose from. It would mean we would have to do more integration than we would need if we chose a more integrated framework such as YUI, which I also have experience with, but I was a bit tired of big frameworks. And jQuery's terse and powerful way to express things is attractive. And you just have to make a choice to see what happens at some point, right?

The JavaScript code in those applications was quickly growing out of control. We were trying to do proper REST: the client would just start with a single URL, which would resolve to a JSON object that would have links to other JSON objects on the server, which would in turn have links to other JSON objects. Proper REST is nice: you can decouple your client from the server more, because your client code doesn't depend on the specific URL structure of your server -- instead it just follows links.

But doing proper REST with JavaScript without taking special measures tends to result in endlessly nested callbacks in callbacks in callbacks. The code was too tightly coupled and hard to follow.

This made me think about the stuff I did at Gocept again. I felt there was quite a bit of potential in those patterns, potential I hadn't explored fully yet. So I added it to one of our projects and started to rewrite the UI to use it, to try to clean things up.

People

At this stage more people started to be involved. Ideas tend to get better with good input, and with usage by a larger group of people. I will forget some of the people involved in this project, for which I apologize, but I'm going to drop some names now.

The original idea was born at Gocept. I was doing the work for Christian Zagrodnick there, and I had some good conversations with him about it then. He also used the early code in a few of his own projects later on. Just as I had checked in the view stuff into the new project I was doing I found myself in Germany at Gocept again, and had further productive conversations with him about it. Thanks Zagy!

Going to Germany, where Gocept is based, from where I live, the Netherlands, means a train ride, and in the train I had the opportunity to bounce ideas off Jan-Wijbrand Kolman (and him ideas off me), something I always enjoy tremendously. The biggest thing that came out of that was Fanstatic, but that's the topic for another discussion. It also helped me think through what later became Obviel. Thanks JW!

The idea developed around this time was that the JSON returned by the server to JavaScript should have a type marker to describe what kind of JSON we are dealing with then. In Obviel we call this type marker an iface, short for interface. On the client, we could then hook up views to ifaces. This way if the client is confronted with some unknown JSON, it would know what to do with it by looking up the appropriate view for that iface. This helps with REST and therefore with loose coupling: the client code not only doesn't need to know what resources the server is presenting, but also would be able to know what resources are on what URLs; the resources would disclose this themselves. In this, "JavaScript views" (as we were calling it at the time) resemble dynamic view lookup like practiced in the Zope Component Architecture (don't you all run away now, though!).

Practice

New approaches get better when you try to actually use them in practice. You learn a lot. The customer projects I was doing were too big to do on my own, I was working with Izhar Firdaus on them. Izhar gave me a lot of invaluable feedback and improvements and came up with new patterns of using these "JavaScript views" (as we were calling them) throughout the projects that use Obviel; I had great pleasure working with him for over a year.

I think I already rewrote the system halfway back then, and I remember Izhar doing something similar. Eventually after some experience I sat down and started to write a document describing how I wanted this JavaScript views thing to work. So I got Guido Wesdorp involved. This meant Izhar and I were freed up to build more application specific code and gain experience with it.

Working with Guido in this way was a new experience: I had designed a framework but Guido was implementing it. Guido focused on making the code solid. I, Guido and Izhar together came up with various new approaches.

Forms

The projects we were working on needed a number of complex web forms. I have a lot of experience with server-side form libraries. Some time before I had been in a rich-client project with Jasper Op de Coul and he suggested creating a JavaScript form library instead. This would offer some benefits, such as inline validation for everything by default, and web-framework neutrality. While we didn't implement the idea at the time, it did stick in my mind.

Here we needed lots of complex forms again, and server-side validation with errors didn't really fit well in a rich client UI. So I sketched up a way to do JavaScript forms using a new technology jQuery datalink. The concept of Obviel forms is to maintain a JavaScript object with the form contents in parallel to the HTML form (an earlier conversion with Antonin Amand helped me develop this concept). When the user modifies the form, the JavaScript object is automatically updated. A form submission then simply submits that JavaScript data object as JSON to the server.

I had Guido go ahead and start building the form library for us.

Rewrites

"JavaScript views" had already gone through a few incarnations. I'd rewritten it at least once, Izhar had given it another shot, and Guido had rewritten it again.

Meanwhile Sylvain Viollon at Infrae had started using Obviel as well. He had decided to do its own rewrite, to my chagrin at the time, as now we had two parallel versions. But it turned out for the good: he had introduced some new ideas into his version that were very useful: when a view was rendered, it would be bound to the object it had just rendered, and the element it was rendered on. These bound views would allow a better structuring of view code than passing this information along as parameters all the time.

We kept learning however, and after a while, when Guido had already moved on from the project, several things in Obviel forms in particular started to be frustrating.

So I decided to rewrite Obviel forms cleaning it up. Among new things introduced were composite sub-forms and a generic repeat widget to allow the user to enter repeating sub-forms as well, and a way to maintain the inline error messages as a datalinked object as well. Govert Buijs, who had started to work on the projects with us, also helped work out advanced form validation scenarios.

On a roll, I decided to rewrite the Obviel core as well, introducing Sylvain's ideas about bound views. Along the way I also started using the new jQuery deferreds in its implementation (Sylvain then turned out to have done so in his version as well, in parallel).

Documentation

Now it is time to attract more people to Obviel. It's been battle tested in a number of big projects.

To attract people to a project you not only need good code. We do have good code I think: it's unit tested and coverage tested. But you also need great documentation. Along the way I and Guido had written bits of documentation about Obviel. It took me months of stops and start to get this documentation in the state it is now, so that newcomers to Obviel aren't lost.

So, if you are curious about Obviel and want to find out more, go to http://www.obviel.org

And give us feedback, please!

WSGI: Bringing web frameworks closer together

Recently I have seen a sentiment espoused by some people in the Python web community that WSGI has failed to live up to one of this promises: bringing Python web frameworks closer together.

Let's look at history and see whether this is true.

Here's a snapshot of the Python web framework landscape in 2005:

  • Zope had its own way of hooking up to web servers. Several ways. FastCGI, SCGI, Apache Rewrite rules, ProxyPass.
  • Django had just emerged, with its own way of hooking up to web servers. Several ways too; I can find references to using mod_python, FastCGI, SCGI and AJP.
  • TurboGears had also just emerged. TurboGears 1.x used CherryPy as its web server. A quick search turns up references to hooking it up to Apache using mod_python, and using mod_rewrite.

Let's look at the web framework landscape in 2011:

  • Zope (in all its incarnations) can be hooked up to web servers to WSGI, and this is generally the preferred method.
    • Zope 2 comes with native WSGI support.
    • Grok has native WSGI support and uses paster as its development server.
    • BlueBream has native WSGI support and uses paster as its development server.
  • Django has WSGI as the preferred method to hook up to web servers.
  • TurboGears is now built around WSGI and uses paster as its development server.
  • Both Pylons the web framework as well as Pyramid are built around WSGI, and both use paster as its development server.

[no claim is made that this represents all Python web frameworks out there]

Has this brought web frameworks closer together? The answer is clearly a resounding YES: if you know how to hook up a WSGI application to your web server of choice, you have basic knowledge to deal with all of these web frameworks. You can use Apache mod_wsgi with all of them, for instance.

In addition, only Zope 2 and Django do not use paster as their default development web server. We see a major component being shared between Grok, BlueBream, TurboGears, Pylons and Pyramid. This knowledge transfers between web frameworks.

So this sentiment is clearly wrong. Thank you to everybody who has helped create and push WSGI!

Other arguments have been made, for instance that WSGI middleware in particular isn't bringing Python web frameworks closer together. I believe that's wrong too, but I will leave defending that point to someone more familiar with that topic.

how to handle ideas

Over the years I gained some experience with how open source communities function. Sometimes these communities make it difficult for relative outsiders (or even insiders) to make a proposal or to share an idea.

Recently I made a proposal on a mailing list of a project. It's a very useful project, and there are fine, smart people working on it. But here are some of the responses I got:

  • no! this is wrong! (no explanation)
  • your use cases can be solved in another way. Therefore our project automatically doesn't need to tackle them.
  • actually I don't use our own project for these use cases even though I know many others are. Instead I use an entirely different system, and you should too.
  • do it yourself! solving these use cases for yourself is trivial! you just need to write your own tools, there is no documentation on how to do it, and there are a lot of details you need to worry about. Therefore this project is not going to handle your use cases, everybody is on their own on this one. But really it's trivial. trivial, really.
  • what you are proposing is immoral.

I humbly submit that this is not a very constructive set of responses. It sounds lot like "go away". This will likely frustrate the person who made the proposal. They can:

  • go away, therefore nobody got educated and nothing was improved.
  • go away and not come back with new ideas.
  • show their frustration, therefore putting the people in the project on the defensive. This rarely leads to a good discussion either.

These were smart, well-meaning people. I got more constructive responses too. Still, evidently it's easy for a group of such people to still give this kind of feedback.

Let's discuss what would be better responses to someone sharing an idea.

The best response for the proposer and the people who run the project of course would be:

  • this is already solved: here's the solution

Sounds great, right? And it is, if it's true.

Unfortunately the desire to give this response on the part of the project tempts people to say this when it isn't exactly true. It's tempting to present "do it yourself" as "it's already solved". So the "here's the solution" part is important.

There's another good response for the proposer:

  • hey, that's a good idea! let's work together to implement it!

This is less good for the people who run the project, as now they need to do work. Still, if this is a good idea, it'll be good for everybody, so nobody will really be complaining about this: they're in a community to get things solved together, after all. And in the best case scenario the proposer will actually join the project and help out.

But of course an idea is not always obviously a good idea. Perhaps the idea needs some tweaking. Perhaps the motivations behind the idea need to be better understood. So another great response to a proposal would be:

  • hm, I'm not sure about this. let's analyze your use cases in some more detail.

So now the people in the project start asking questions, give suggestions, consider solution strategies. The idea will likely change in shape as things become more clear. The project, and the person making the proposal, may end up with new features and at the very least the project will have learned something. Incidentally this is also a great way to draw newcomers into a project.

Sometimes however a project will have to say no. If it's immediately obvious why no, this generally means this discussion has come up in the past. Perhaps it's time to document the decision on this in the project documentation? Then the project can point newcomers to the documentation. Perhaps they'll read the documentation before they even arrive on your mailing list. Or perhaps, just maybe, if this idea keeps coming up, is there something to it after all?

Here are some good ways to reject an idea:

  • our project is not actually about fulfilling these use cases. Here are the project's goals, and here's why your idea won't fit.
  • solving this might fit our project's goals, but would increase our maintenance burden to levels not acceptable to the project, and here's why. We weigh this against how common the use case really is.

If you handle newcomers and new ideas well, your project stands a better chance at growing, or at least remaining viable over time as people leave and new people come in.

Perhaps eventually your project will be so successful there will be masses of ill-formulated ideas that your project just can't handle. Then it's time to start coming up with other strategies to deal with the situation. Generally these would involve some form of layering or partitioning in the project, so that not everybody is swamped by everything all the time. Most projects aren't there yet, though, and even many successful projects will never grow to this size.

web frameworks considered useful

There is a a strain of thought in the Python web development community these days that considers web frameworks a bad idea. Even beginners are sometimes told: why do you need a framework anyway? Just build your app from scratch with WSGI! Or just compose your own out of existing libraries and tools!

I'm going to argue that frameworks are useful. I'm going to argue that we should normally not be telling beginners to avoid frameworks. And I'm going to argue that experienced developers should carefully consider whether their perspective isn't warped a little by their experience, and that frameworks can be useful for them too.

Frameworks get in your way

I want to do something that should be simple, and would be simple if I just did it at a low level. But because the framework forces a certain way of working on me, it becomes needlessly difficult. I have to create workarounds, and it sucks.

Frameworks can get into your way, as they've made certain choices. How much they are constraining depends on the framework, your experience with it, and what you're trying to accomplish.

But if you picked the right framework for the job, regularly the task you're trying to perform is something that a framework makes very easy. The first time around you use a framework for something that it is good at, you're going to be impressed at how easy it was to accomplish your task. Before long however, you might become so used to the benefits that a framework brings that you won't even notice what the framework is doing for you anymore; it's now in the background. And humans are very attuned to pain, so the pain points are going to remain. You're going to remember the ugly workarounds better than the times when things just worked.

I think frameworks get in the way for beginners less than they do for experienced developers. Beginners need to learn the basics, and having a framework can be a useful guide to gain an understanding of the basics and how they go together. An experienced developer will understand all that already, and is more likely to work on more challenging projects where the choices that the framework made are not the right ones. But does that mean that the framework is wrong for all tasks, and that we should tell a beginner not to use them at all? And does it really mean that experienced developers shouldn't use frameworks at all anymore, either? What about those tasks where the pain is minimal? And what if you are working in a group that includes beginners? And may it be that your time is more productively spent improving the framework to make the pain go away, instead of avoiding it entirely?

Building from scratch feels more productive

A developer might argue to the last point: no, I am a lot more productive if I build something from scratch, and list a whole range of things they've built up in a short amount of time.

And of course building something from scratch can feel very liberating, and sometimes it is the right way to go. But frequently you will only feel more productive - if you're spending time to reimplement the features of the framework you discarded, you might be productive in implementing those features, but what about the task you're actually trying to accomplish? What about the web app you're trying to build? And how much code that only you understand will you end up trying to maintain?

There are cases where you may feel a lot more productive, because you're having more fun, when you actually aren't. We all know programmers like to reinvent wheels. Reusing existing code may be more painful, but it isn't necessarily less productive, although of course it can be.

And seriously, do we really think beginners without much experience of frameworks have enough knowledge to be able to build their web apps from scratch and do it well?

Don't use a framework, just reuse existing building blocks

Lots of people would agree with me that building things from scratch is not the way to go. But, they'll argue, people can just assemble the components they need for their application themselves, instead of using a framework.

And it's true: there are a lot of building blocks available these days, and that's good. There are libraries and middleware and components and so on to do everything from interfacing with a web server to talking to a database. Instead of picking a ready-made pre-assembled framework, you can instead pick and choose the best of breed components that are out there and use those to develop your application. Since you are doing it, it may be better suited to your requirements than any pre-assembled framework can offer. You'll understand it better too.

I believe there is a lot of value in this approach. The Python web development world has been moving in this direction for a while; there's the emphasis on WSGI, and there is the emphasis on easily distributing and combining libraries. I'll note that the Python web framework development has been moving in this direction too, because web framework developers are also aware of the benefit of sharing useful components with each other.

But does this mean we should recommend to everyone all the time to just ignore frameworks and assemble the bits themselves? No.

The obvious case where this is a bad idea is in the case of beginners. A beginner will have little understanding of how components can go together. A beginner will have no way to evaluate components. A beginner is usually much better off to use a pre-assembled framework where these choices have already been made. The beginner can find information in a one-stop shop: they'll learn about the template language and the approach to databases of the framework in one place, and they'll see not just descriptions of the bits and pieces but also how they go together. This means that they won't have to do this integration themselves anymore; they'll see examples. There's also a community where they can ask questions.

These reasons for a beginner to use a pre-assembled framework also frequently apply to more experienced developers. You don't always have time to go out and assemble components together, and figure out how to make it work. Sure, what you might end up with might be slightly better suited to a particular task than a pre-assembled framework, but is it really always worth the effort? Do you really want to make all those choices each time, for each app you build? And you'll still be stuck with the maintenance burden - the glue code you wrote of course, but also the assembly itself: what if there are newer releases of the components you are using?

Finally, these smaller components are often quite complex frameworks too - and if you integrate them yourself there might be less unity of vision than a good framework can offer, and the whole might end up harder to understand.

The burden of assembling and integrating best of breed components can be shared: that's what the developers of a framework do. And if you base your work on a pre-assembled framework, it's likely to be less work for you to upgrade, because the framework developers will have taken care that the components in the framework work together in newer versions. There is also a larger chance that people will write extensions and documentation for the this same assembly, and that is very likely something you may benefit from in the future.

So it often makes sense to share the assembling and integration of components in an open source fashion just like it makes sense to share the components themselves. An assembly is not just a collection of loose parts, it can be a new thing, with a vision of it own.

Conclusion

All the points above are true: frameworks can get into your way, using a framework (especially when wrestling with it) can feel less productive than building something new from scratch, and reusing components yourself is often more flexible.

But frameworks can also take care of a lot of issues for you, even though after a while you only feel it when they are in the way. Frameworks make choices for you so you don't have to make them all the time. Frameworks can save time. Frameworks offer an integrated whole so you won't have to worry about the rest of the world for a while. Good frameworks will also be flexible enough to handle a huge amount of tasks more than adequately.

So, there is a place for web frameworks. It makes sense to recommend web frameworks to beginners. It also makes sense for experienced developers to consider using a web framework. It's not always the right tool for the job, but it often is. By all means let's discuss the particular pain points of frameworks, doing things without a framework and assembling components yourself. But let's not forget the benefits that web frameworks bring to many of us.

ME GROK BOOK

This is awesome. There is now a published book about Grok!

Thank you, Carlos, very much, for writing this book. That was a lot of work!

Why would you, Python developer, be interested in Grok? Why not learn about one of the myriad other Python web frameworks instead? Kevin Teague in a recent reddit thread put it very well, so I'll just quote him:

Forget all those Django, TurboGears, Web2Py, Pylons arguments, Grok is where it's at ... Why Grok?

  • Schema and model are not tightly coupled: use objects from Zope Object Database, LDAP, SQLAlchemy, Web services, etc. and be able to auto-generate forms regardless of the persistence implementation.
  • Convention-over-configuration: Introspects your code, so that you get the benefits of succinct, less-typing benefits convention, without forcing you to layout your software orthogonal to it's purpose. e.g. doesn't force "models" into a models module, "views" into a views module, if you have a Book model and Book view, you can put them in the same book.py file.
  • Well packaged (mostly, Grok 1.1 provides a lot of fixes to the dependency tree), so easier to manage deployments. Also core parts are individually packaged, so re-useable outside of Grok (primarily of interest to Plone developers).
  • Cavemen are way tougher than Ponies!

What does Grok the caveman himself have to say about a book on Grok? Or about any book, even, as he can't tell the difference?

ME GROK FIND BOOK!

BOOK STRANGE!

ME GROK LIKE HAS MANY PAGE

MANY MANY PAGE SQUIGGLE!

ME GROK THINK SQUIGGLE ARE FLY DROPPING?

BOOK GREAT MAGIC!

ME GROK TEAR OFF PAGE

PAGE NICE AND CHEWY TO EAT

PAGE GOOD MAKE FIRE WITH

PAGE GOOD WIPE BOT[censored]

Um, sorry about that. I think we should let more literate people review the book instead.

ME GROK JUST KIDDING!

ME GROK LOVE BOOK

ME GROK LEARN ALL ABOUT WEB APP

ME GROK WONDER WEB APP BETTER EAT THAN MAMMOTH?

Python-based configuration in a Grok near you

BFG 1.2 offers imperative configuration: doing configuration not in ZCML but in Python. The declarative configuration system is built on top of this. This is an interesting approach that has some merit.

It's interesting to compare this to Grok's configuration system, which also focuses on Python, not ZCML. Grok however offers a declarative configuration system in Python, not an imperative one.

It's important to note that from the start Grok's declarative configuration system allowed the user to be entirely explicit about how configuration happens. I don't know whether Chris was referring to Grok when he mentioned that in BFG no false choice between convention over configuration and ZCML is offered, but in case anyone had the wrong impression: Grok doesn't offer such a choice. You can be as explicit as you like. Relying on convention-based configuration patterns is an option when using Python-based configuration, but not an obligation.

Grok's declarative configuration system can be used in tests. If you have a component that you need to register, you can use grok.testing.grok_component to automatically configure it according to the standard Grok configuration rules.

Here is an example, a component:

import grok
class MyAdapter(grok.Adapter):
   grok.context(AdaptedFrom)
   grok.provides(IAdaptedTo)

This adapter when configured, be registered as allowing instances of class AdaptedFrom to objects providing interface IAdaptedFrom.

If we want to configure this adapter in our tests, we can use grok_component:

>>> from grok.testing import grok_component
>>> grok_component('MyAdapter', MyAdapter)
True

The grok_component function can also be imported from the reusable grokcore.component library, through grokcore.component.testing.

Interesting about this design is that Grok allows the use of the same configuration system for test-configuration as well as real configuration. There is no need to learn a different configuration approach when running tests.

Grok's focus on configuration as an area to challenge existing Zope assumptions is long-standing. Grok strikes a different balance than BFG as Grok aims to remain compatible with existing Zope-based systems.

Nothing's perfect. So, I'll close with some areas where we should improve Grok's configuration system:

  • while it is very convenient to be able to register a component with the same configuration as it would have in a real application (a common case), we might need a way to support registering an existing component differently. Subclassing probably usually does provide this.
  • we need to better promote grok_component as a well-understood tool by writing better documentation and more tests. Often I myself fall back on using the zope.component registration APIs in tests directly myself, and I need to understand why. Perhaps this is simply something to get used to, or perhaps the reasons are more fundamental and there is a BFG-like imperative configuration system at some level.
  • we need to fix the API so we can avoid the first argument to grok_component, which is, if I recall it correctly, mostly useless.