Recently new HTTP 1.1 RFCs were published that obsolete the old HTTP 1.1 RFCs. They are extensively rewritten.

Unfortunately the WSGI PEP 3333 refers to something only made explicit in the old version of the RFCs, but which is much harder to find in the new versions of the RFCs. I thought I'd leave a report of my investigations here so that others who may run into this in the future can find it.

WSGI is a protocol that's like HTTP but isn't quite HTTP. In particular WSGI defines its own iterator-based way to send larger responses out in smaller parts. It therefore cannot deal with so-called "hop-by-hop" headers, which try to control this behavior on a HTTP level. The WSGI spec says a WSGI application must not generate such headers.

This is relevant when you're dealing with a WSGI-over-HTTP proxy. This is a special WSGI application that talks to an underlying HTTP server. It presents itself as a normal WSGI application.

The underlying HTTP server could very well be sending out stuff like such as Transfer-Encoding: chunked. The WSGI spec does not allow a WSGI application to send them out though, so a WSGI proxy must strip these headers out.

So what headers are to be stripped out? The WSGI spec refers to section 13.5.1 in now-obsolete RFC 2616.

This nicely lists hop-by-hop headers:

• Connection

• Keep-Alive

• Proxy-Authenticate

• Proxy-Authorization

• TE

• Trailers

• Transfer-Encoding

• Upgrade

That RFC also says:

"All other headers defined by HTTP/1.1 are end-to-end headers."

and then confusingly:

"Other hop-by-hop headers MUST be listed in a Connection header, (section 14.10) to be introduced into HTTP/1.1 (or later)."

which one is it, HTTP 1.1? I guess that's one of the reasons this text got rewritten.

In the new rewritten version of HTTP 1.1, this list is gone. Instead it specifies for some headers (such as TE and Upgrade) that these should be added to the Connection field. A HTTP proxy can then strip out the headers listed in Connection, and then also strip out Connection itself.

Confusingly, while the new RFC 7230 refers to the concept of 'hop-by-hop' early on, and also say this in the change notes in A.2:

"Also, "hop-by-hop" header fields are required to appear in the Connection header field; just because they're defined as hop- by-hop in this specification doesn't exempt them."

it doesn't actually say any headers are hop-by-hop anywhere else. Instead it mandates some headers should be added to Connection.

But wait: Transfer-Encoding is not to be listed in the Connection header, as it's not hop-by-hop. At least, not anymore. I've seen it described as 'hopX-by-hopY', but not in the RFC. This is, I think, because a HTTP proxy could let these through without having to remove them. But not for a WSGI over HTTP proxy: it MUST remove Transfer-Encoding, as WSGI applications have no such concept.

I think the WSGI PEP should be updated in terms of the new HTTP RFC. It should make explicit that some headers such as Transfer-Encoding must not be specified by a WSGI app, and that no headers that must be listed in Connection can be specified by a WSGI app, or something like that.

Relevant mailing list thread:

http://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/thread.html#msg1710

It's pretty common for an open source project to have a "contrib" directory as part of its project structure. This contains useful code donated to the project by outsiders. It seems innocuous. A contrib section, why not?

I don't like contrib. A contrib directory gives the signal that "yes, we carry this source code around, but it's not really part of our project". What does that mean? Why is it even part of your project at all then? Why isn't this code distributed in library form instead? I'd much prefer the project to be smaller instead, as in that case I wouldn't have to worry about the contrib code at all.

Perhaps in the case of your project, placing code in contrib doesn't really mean "it's not really part of our project". Perhaps the code in contrib is meant to be a fully supported part of project's codebase. If so, why use the name "contrib" at all? It doesn't signal anything functional -- it only signals something about origins, which is why people should suspect any claim that it's a fully integral part of the project. Projects, instead of dumping something in contrib, just put that code in its appropriate place and really own it.

Here are some stories on how you can go wrong with naming, especially in open source software.

I had a fun time at EuroPython 2014 in Berlin last week. It was a very well organized conference and I enjoyed meeting old friends again as well as meeting new people. Before I went I was a bit worried with the amount of attendees it'd feel too massive; I had that experience at a PyCon in the US a few years ago. But I was pleasantly surprised it didn't -- it felt like a smaller conference, and I liked it.

Another positive thing that stood out was a larger diversity; there seemed to be more people from central and eastern Europe there than before, and most of all, there were more women. It was underscored by a 13 year old girl giving a lightning talk -- that was just not going to happen at EuroPython 5 years ago.

This is a very positive trend and I hope it continues. I know it takes a lot of work on the part of the organizers to get this far.

I gave a talk at EuroPython myself this year, and I think it went well:

I just released Morepath 0.4.1. This fixes a regression with Python 3 compatibility and has a few other minor tweaks to bring test coverage back up to 100%.

I had broken Python 3 support in Morepath 0.4. I'm still not in the habit of running 'tox' before a release, so I find out about these problems too late.

I'll go into a bit of detail about this issue, as it's a mildly amusing example of writing Python code being more complicated than it should be.

Morepath 0.4 broke in Python 3 because I introduced a metaclass for the morepath.App class. I usually avoid metaclasses as they are a source of unpredictability and complexity, but the best solution I saw here was one. It's a very limited one.

One task of the metaclass is to attach to the class with Venusian. Venusian is a library that lets you write decorators that don't execute during import time but later. This is nice as import time side effects can be a source of trouble.

Venusian also lets you attach a callback to a Python object (such as a class) outside of a decorator. That's what I was doing; attaching to a class, in my metaclass.

Venusian determines in what context the decorator was called, such as module-level and class-level, so you can use that later. For this it inspects the Python stack frame of its caller.

My first attempt to make the metaclass work in Python 3 was to use the with_metaclass functionality from the future compatibility layer. I am using this library anyway in Reg, which is a dependency of Morepath, so using it would not introduce a new dependency for Morepath.

Unfortunately after making that change my tests broke in both Python 2 and Python 3. That's not an improvement over having the tests being broken in just Python 2!

It appears that with_metaclass introduces a new stack frame into the mix somewhere, which breaks Venusian's assumptions. Now Venusian's attach has a depth argument to determine where in the stack to check, so I increased the stack depth by one and ran the tests again. Less tests broke than before, but quite a few still did. I think the cause is that the stack depth of with_metaclass is just not consistent for whatever reason.

Digging around in the future package I saw it includes a copy of six, another compatibility layer project. six has a name close to my heart -- long ago I originated the Five project for compatibility between Zope 2 and Zope 3.

That copy of six had another version of with_metaclass. I tried using future.util.six.with_metaclass, and hey, it all started working suddenly. All tests passed, in both Python 2 and Python 3. Yay!

Okay then, I figured, I don't want to depend on a copy of six that just happens to be lying about in future. It's not part of its public API as far as I understand. So I figured I should introduce a new dependency for Morepath after all, on six. It's not a big deal; Morepath's testing dependencies include WebTest, and this already has a dependency on six.

But when I pulled in six proper, I got a newer version of it than the one in future.util.six, and it caused the same test breakages as with future. Argh!

So I copied the code from old-six into Morepath's compat module. It's a two-liner anyway. It works for me. Morepath 0.4.1 done and released.

But I don't know why six had to change its version, and why future's version is different. It worries me -- they probably have good reasons. Are those reasons going to break my code at some point in the future?

Being a responsible open source citizen, I left bug reports about my experiences in the six and future issue trackers:

https://bitbucket.org/gutworth/six/issue/83/with_meta-and-stack-frame-issues#comment-11125428

https://github.com/PythonCharmers/python-future/issues/75

I much prefer writing Python code. Polyglot is an inferior programming language as it introduces complexities like this. But Polyglot is what we got.

I've just released Morepath 0.4!

Morepath 0.4 is a Python web framework that's small ("micro") and packs a lot of power. There are a lot of facilities for application reuse. And as opposed to most web frameworks, it actually has some intelligence about generating hyperlinks to objects.

Morepath 0.4 has a breaking change to the way application reuse works. Don't worry, you can fix your code by making a few minor changes. In short, Morepath application objects are now classes, not instances, and you can instantiate this class to get a WSGI object. See the CHANGES for a lot of details on what happened and what you need to do.

The big win is that application reuse in Morepath has become Python subclassing, and that making a WSGI application (even a parameterized one) is just instantiating the class.

The other win is that Morepath gained even more extensibility features, namely the ability for Morepath extension to introduce new Morepath directives (the decorators you see everywhere in Morepath examples). But I can't talk too much about that until I document them properly.

Along with the new Morepath, I've also made the initial release of BowerStatic (announcement). BowerStatic is the WSGI framework that lets you easily include bower-installed resources in your web page and do the right thing with caching (forever, thank you, but on a separate URL for each version).

How does that relate to Morepath, you may ask? Well, today I've also released the Morepath integration for BowerStatic, more.static. I've described in the Morepath documentation what to do to get it working in your Morepath project. The reason Morepath 0.4 had the breaking change was in part to support more.static, which needed the ability to introduce a new Morepath directive among other things.

I've been working on something new these last few weeks that I'd like to share with you: BowerStatic.

BowerStatic is Python WSGI framework for easily serving and including components that you manage by using Bower.

Let's unpack that:

• BowerStatic serves static resources (JS, CSS, etc) for you. It makes sure those resources are cached when they can be, and that the cache is automatically busted when you change them.

• BowerStatic can serve third-party components you can easily install using bower install. It can also serve locally developed components for you.

• BowerStatic can automatically create inclusions for static resources in your web page; it inserts <script> and <link> tags for you automatically

• It is a WSGI-based framework, meaning that you should be able to integrate it easily into any WSGI-based Python web application. So if you use Pyramid, Flask or Morepath: BowerStatic is for you!

Look at the BowerStatic website for much more information about all this. And check out the source. Contributions are welcome!

BowerStatic was heavily inspired by another project that I helped create: Fanstatic. For more information on how things led up to BowerStatic, read the history section in the docs.

I am not done yet. I still need to work on some of the cache busting behavior, improve error reporting. But soon it will be time for the first release.

Deeper integration with the Morepath web framework is also coming. I have a good idea of what know what it will look like. Look out for more.static soon!

Today I've released Morepath 0.3!

Morepath is a Python web framework that lets you write simple, flexible code for the modern web, and makes linking to things as easy as it should be.

New in Morepath 0.3 is the ability to write path directives that absorb the rest of the path. This is useful when you want to integrate a web server with a frontend that does its own client-side routing using the HTML 5 history API. In that case the server-side code needs an easy way to absorb all sub-paths so it can pass them to the frontend router.

Another feature is the ability to mark views as internal. This allows you to write a view that is not published on the web but is for reuse within other views.

Besides this Morepath 0.3 contains a number of bugfixes and documentation improvements. See CHANGES for more details.

If you like Morepath, please get in touch with me, either through the Morepath issue tracker or the #morepath IRC channel on Freenode. Let me know if there's interest in starting a mailing list too!

I've just released Morepath 0.2! Morepath is your friendly neighborhood Python web microframework with super powers.

Major changes include:

• Python 3 support thanks to a generous contribution by Alec Munro.

• Documented security infrastructure.

And various smaller tweaks. You can also read the changelog.

What makes Morepath special? You can read about its superpowers and read how it compares to other web frameworks, but here are some highlights:

• Route to model, then look up views. This leads to better link generation, better generic views, better reusable code.

• Have apps that can extend other apps, combine apps together. Extend and override anything, even Morepath behavior itself, but don't affect other apps in the same runtime.

It's lots of power contained in a small codebase.

(If you followed the Morepath link above, ignore the 0.1 in the title bar there: those are the 0.2 docs, not 0.1. I've now fixed the doc building configuration to include the version number from the package itself.)

Thanks to an awesome contribution by Alec Munro, Morepath, your friendly neighborhood Python micro framework with super powers, has just gained Python 3 support!

Developing something new while juggling the complexities of Python 2 and Python 3 in my head at the same time was not something I wanted to do -- I wanted to focus on my actual goals, which was to create a great web framework.

So then I had to pick one version of Python or the other. Since my direct customer use cases involves integrating it with Python 2 code, picking Python 2 was the obvious choice.

But now that Morepath has taken shape, taking on the extra complexity of supporting Python 3 is doable. The Morepath test coverage is quite comprehensive, and I had already configured tox (so I could test it with PyPy). Adding Python 3.4 meant patiently going through all the code and adjusting it, which is what Alec did. Thank you Alec, this is great!

Morepath's dependencies (such as WebOb) already had Python 3 support, so credit goes to their maintainers too (thanks Chris McDonough in particular!). This includes the Reg library, which I polyglotted to support Python 3 myself a few months ago.

All this doesn't take away from my opinion that we need to do more to support the large Python 2 application codebases. They are much harder to transition to Python 3 than well-tested libraries and frameworks, for which the path was cleared in the last 5 years or so.

[update: this is still in git; the Morepath 0.1 release is Python 2 only. But it will be included in the upcoming Morepath 0.2 release]