I had a fun time at EuroPython 2014 in Berlin last week. It was a very well organized conference and I enjoyed meeting old friends again as well as meeting new people. Before I went I was a bit worried with the amount of attendees it'd feel too massive; I had that experience at a PyCon in the US a few years ago. But I was pleasantly surprised it didn't -- it felt like a smaller conference, and I liked it.
Another positive thing that stood out was a larger diversity; there seemed to be more people from central and eastern Europe there than before, and most of all, there were more women. It was underscored by a 13 year old girl giving a lightning talk -- that was just not going to happen at EuroPython 5 years ago.
This is a very positive trend and I hope it continues. I know it takes a lot of work on the part of the organizers to get this far.
I gave a talk at EuroPython myself this year, and I think it went well:
I just released Morepath 0.4.1. This fixes a regression with Python 3 compatibility and has a few other minor tweaks to bring test coverage back up to 100%.
I had broken Python 3 support in Morepath 0.4. I'm still not in the habit of running 'tox' before a release, so I find out about these problems too late.
I'll go into a bit of detail about this issue, as it's a mildly amusing example of writing Python code being more complicated than it should be.
Morepath 0.4 broke in Python 3 because I introduced a metaclass for the morepath.App class. I usually avoid metaclasses as they are a source of unpredictability and complexity, but the best solution I saw here was one. It's a very limited one.
One task of the metaclass is to attach to the class with Venusian. Venusian is a library that lets you write decorators that don't execute during import time but later. This is nice as import time side effects can be a source of trouble.
Venusian also lets you attach a callback to a Python object (such as a class) outside of a decorator. That's what I was doing; attaching to a class, in my metaclass.
Venusian determines in what context the decorator was called, such as module-level and class-level, so you can use that later. For this it inspects the Python stack frame of its caller.
My first attempt to make the metaclass work in Python 3 was to use the with_metaclass functionality from the future compatibility layer. I am using this library anyway in Reg, which is a dependency of Morepath, so using it would not introduce a new dependency for Morepath.
Unfortunately after making that change my tests broke in both Python 2 and Python 3. That's not an improvement over having the tests being broken in just Python 2!
It appears that with_metaclass introduces a new stack frame into the mix somewhere, which breaks Venusian's assumptions. Now Venusian's attach has a depth argument to determine where in the stack to check, so I increased the stack depth by one and ran the tests again. Less tests broke than before, but quite a few still did. I think the cause is that the stack depth of with_metaclass is just not consistent for whatever reason.
Digging around in the future package I saw it includes a copy of six, another compatibility layer project. six has a name close to my heart -- long ago I originated the Five project for compatibility between Zope 2 and Zope 3.
That copy of six had another version of with_metaclass. I tried using future.util.six.with_metaclass, and hey, it all started working suddenly. All tests passed, in both Python 2 and Python 3. Yay!
Okay then, I figured, I don't want to depend on a copy of six that just happens to be lying about in future. It's not part of its public API as far as I understand. So I figured I should introduce a new dependency for Morepath after all, on six. It's not a big deal; Morepath's testing dependencies include WebTest, and this already has a dependency on six.
But when I pulled in six proper, I got a newer version of it than the one in future.util.six, and it caused the same test breakages as with future. Argh!
So I copied the code from old-six into Morepath's compat module. It's a two-liner anyway. It works for me. Morepath 0.4.1 done and released.
But I don't know why six had to change its version, and why future's version is different. It worries me -- they probably have good reasons. Are those reasons going to break my code at some point in the future?
Being a responsible open source citizen, I left bug reports about my experiences in the six and future issue trackers:
I much prefer writing Python code. Polyglot is an inferior programming language as it introduces complexities like this. But Polyglot is what we got.
I've just released Morepath 0.4!
Morepath 0.4 is a Python web framework that's small ("micro") and packs a lot of power. There are a lot of facilities for application reuse. And as opposed to most web frameworks, it actually has some intelligence about generating hyperlinks to objects.
Morepath 0.4 has a breaking change to the way application reuse works. Don't worry, you can fix your code by making a few minor changes. In short, Morepath application objects are now classes, not instances, and you can instantiate this class to get a WSGI object. See the CHANGES for a lot of details on what happened and what you need to do.
The big win is that application reuse in Morepath has become Python subclassing, and that making a WSGI application (even a parameterized one) is just instantiating the class.
The other win is that Morepath gained even more extensibility features, namely the ability for Morepath extension to introduce new Morepath directives (the decorators you see everywhere in Morepath examples). But I can't talk too much about that until I document them properly.
Along with the new Morepath, I've also made the initial release of BowerStatic (announcement). BowerStatic is the WSGI framework that lets you easily include bower-installed resources in your web page and do the right thing with caching (forever, thank you, but on a separate URL for each version).
How does that relate to Morepath, you may ask? Well, today I've also released the Morepath integration for BowerStatic, more.static. I've described in the Morepath documentation what to do to get it working in your Morepath project. The reason Morepath 0.4 had the breaking change was in part to support more.static, which needed the ability to introduce a new Morepath directive among other things.
I've been working on something new these last few weeks that I'd like to share with you: BowerStatic.
BowerStatic is Python WSGI framework for easily serving and including components that you manage by using Bower.
Let's unpack that:
I am not done yet. I still need to work on some of the cache busting behavior, improve error reporting. But soon it will be time for the first release.
Deeper integration with the Morepath web framework is also coming. I have a good idea of what know what it will look like. Look out for more.static soon!
Today I've released Morepath 0.3!
Morepath is a Python web framework that lets you write simple, flexible code for the modern web, and makes linking to things as easy as it should be.
New in Morepath 0.3 is the ability to write path directives that absorb the rest of the path. This is useful when you want to integrate a web server with a frontend that does its own client-side routing using the HTML 5 history API. In that case the server-side code needs an easy way to absorb all sub-paths so it can pass them to the frontend router.
Another feature is the ability to mark views as internal. This allows you to write a view that is not published on the web but is for reuse within other views.
Besides this Morepath 0.3 contains a number of bugfixes and documentation improvements. See CHANGES for more details.
If you like Morepath, please get in touch with me, either through the Morepath issue tracker or the #morepath IRC channel on Freenode. Let me know if there's interest in starting a mailing list too!
I've just released Morepath 0.2! Morepath is your friendly neighborhood Python web microframework with super powers.
Major changes include:
And various smaller tweaks. You can also read the changelog.
It's lots of power contained in a small codebase.
(If you followed the Morepath link above, ignore the 0.1 in the title bar there: those are the 0.2 docs, not 0.1. I've now fixed the doc building configuration to include the version number from the package itself.)
Developing something new while juggling the complexities of Python 2 and Python 3 in my head at the same time was not something I wanted to do -- I wanted to focus on my actual goals, which was to create a great web framework.
So then I had to pick one version of Python or the other. Since my direct customer use cases involves integrating it with Python 2 code, picking Python 2 was the obvious choice.
But now that Morepath has taken shape, taking on the extra complexity of supporting Python 3 is doable. The Morepath test coverage is quite comprehensive, and I had already configured tox (so I could test it with PyPy). Adding Python 3.4 meant patiently going through all the code and adjusting it, which is what Alec did. Thank you Alec, this is great!
Morepath's dependencies (such as WebOb) already had Python 3 support, so credit goes to their maintainers too (thanks Chris McDonough in particular!). This includes the Reg library, which I polyglotted to support Python 3 myself a few months ago.
All this doesn't take away from my opinion that we need to do more to support the large Python 2 application codebases. They are much harder to transition to Python 3 than well-tested libraries and frameworks, for which the path was cleared in the last 5 years or so.
[update: this is still in git; the Morepath 0.1 release is Python 2 only. But it will be included in the upcoming Morepath 0.2 release]
Guido recently felt he needed to re-empathize that there will be no Python 2.8. The Python developers have been very clear for years that there will never be a Python 2.8.
At the Python language summit there were calls for a Python 2.8. Guido reports:
We (I) still don't want to do a 2.8 release, and I don't want to accelerate 3.5, but I do think we should make things better for people who have to straddle Python 2 and 3 in a single codebase, by developing more tools, and by security and possibly installer updates to 2.7 (PEP 466).
At his keynote at PyCon, he said it again:
A very good thing happened to recognize the reality that Python 2.7 is still massively popular: the end of life date for Python 2.7 was changed by Guido to 2020 (it was 2015). In the same change he felt he should repeat there will be no Python 2.8:
+There will be no Python 2.8.
The call for Python 2.8 is strong. Even Guido feels it!
People talk about a Python 2.8, and are for it, or, like Guido, against it, but rarely talk about what it should be. So let's actually have that conversation.
Why talk about something that will never be? Because we can't call for something, nor reject something if we don't know what it is.
Python 2.8 could be different things. It could be a Python 2.x release that reduces some pain points and adds features for Python 2 developers independent from what's going on in Python 3. It makes sense, really: we haven't had a new Python 2 feature release since 2010 now. Those of us with existing large Python 2 codebases haven't benefited from the work the language developers have done in those years. Even polyglot libraries that support Python 2 and 3 both can't use the new features, so are also stuck with a 2010 Python. Before Python 2.7, the release cycle of Python has seen a new compatible release every 2 years or less. The reality of Python for many of its users is that there has been no feature update of the language for years now.
But I don't want to talk about that. I want to talk about Python 2.8 as an incremental upgrade path to Python 3. If we are going to add features to Python 2, let's take them from Python 3. I want to talk about bringing Python 2.x closer to Python 3. Python 2 might never quite reach Python 3 parity, but it could still help a lot if it can get closer incrementally.
In the discussion about Python 3 there is a lot of discussion about the need to port Python libraries to Python 3. This is indeed important if you want the ability to start new projects on Python 3. But many of us in the trenches are working on large Python 2 code bases. This isn't just maintenance. A large code base is alive, so we're building new features in Python 2.
Such a large Python codebase is:
You can argue that I'm overstating the risks of porting. But we need to face it: many codebases written in Python 2 have low automatic test coverage. We don't like to talk about it because we think everybody else is better at automated testing than we are, but it's the reality in the field.
We could say, fine, they can stay on Python 2 forever then! Well, at least until 2020. I think this would be unwise, as these organizations are paying a lot of developers money to work on Python code. This has an effect on the community as a whole. It contributes to the gravity of Python 2.
Those organizations, and thus the wider Python community, would be helped if there was an incremental way to upgrade their code bases to Python 3, with easy steps to follow. I think we can do much more to support such incremental upgrades than Python 2.7 offers right now.
Besides helping Python 2 code bases go further step by step, Python 2.8 can also help those of us who are maintaining polyglot libraries, which work in both Python 2 and Python 3.
If a Python 2.8 backported Python 3 features, it means that polyglot authors can start using those features if they drop Python 2.7 support right there in their polyglot libraries, without giving up Python 2 compatibility. Python 2.8 would actually help encourage those on Python 2.7 codebases to move towards Python 3, so they can use the library upgrades.
Of course dropping Python 2.x support entirely for a polyglot library will also make that possible. But I think it'll be feasible to drop Python 2.7 support in favor of Python 2.8 much faster than it is possible to drop Python 2 support entirely.
I've seen Python 3 developers say: but we've done all we could with Python 2.7 already! What do you want from a Python 2.8?
And that's a great question. It's gone unanswered for far too long. We should get a lot more concrete.
What follows are just ideas. I want to get them out there, so other people can start thinking about them. I don't intend to implement any of it myself; just blogging about it is already breaking my stress-reducing policy of not worrying about Python 3.
Anyway, I might have it all wrong. But at least I'm trying.
Here's a paradox: I think that in order to make an incremental upgrade possible for Python 2.x we should actually break existing Python 2.x code in Python 2.8! Some libraries will need minor adjustments to work in Python 2.8.
I want to do what the from __future__ pattern was introduced for in the first place: introduce a new incompatible feature in a release but making it optional, and then later making the incompatible feature the default.
Python 2.7 lets you do from __future__ import something to get the interpreter behave a bit more like Python 3. In Python 2.8, those should be the default behavior.
In order to encourage this and make it really obvious, we may want to consider requiring these in Python 2.8. That means that the interpreter raises an error unless it has such a from __future__ import there.
If we go for that, it means you have to have this on the top of all your Python modules in Python 2.8:
absolute_import appears to be uncontroversial, but I've seen people complain about both division and print_function. If people reject Python 3 for those reasons, I want to make clear I'm not in the same camp. I believe that is confusing at most a minor inconvenience with a dealbreaker. I think discussion about these is pretty pointless, and I'm not going to engage in it.
I've left out unicode_literals. This is because I've seen both Nick Coghlan and Armin Ronacher argue against them. I have a different proposal. More below.
What do we gain by this measure? It's ugly! Yes, but we've made the upgrade path a lot more obvious. If an organisation wants to upgrade to Python 2.8, they have to review their imports and divisions and change their print statements to function calls. That should be doable enough, even in large code bases, and is an upgrade path a developer can do incrementally, maybe even without having to convince their bosses first. Compare that to an upgrade to Python 3.
We can't do everything with the old future imports. We want to allow more incremental upgrading. So let's introduce a new future import.
New-style classes, that is classes that derive from object, were introduced in Python 2 many years ago, but old-style classes are still supported. Python 3 only has new-style classes. Python 2.8 can help here by making new style classes the default. If you import from __future3__ import new_classes at the top of your module, any class definition in that module that looks like this:
class Foo: pass
is interpreted as a new-style class.
This might break the contract of the module, as people may subclass from this class and expect an old-style class, and in some (rare) cases this can break code. But at least those problems can be dealt with incrementally. And the upgrade path is really obvious.
Why did I write __future3__ and not __future__? Because otherwise we can't write polyglot code that is compatible in Python 2 and Python 3.
Python 3.4 doesn't support from __future__ import new_classes. We don't want to wait for a Python 3.5 or Python 3.6 to support this, even there is even any interest in supporting this among the Python language developers at all. Because after all, there won't be a Python 2.8.
That problem doesn't exist for __future3__. We can easily fake a __python3__ module in Python 3 without being dependent on the language developers. So polyglot code can safely use this.
Back to the magic moment of Nick Coghlan and Armin Ronacher agreeing.
Let's have a from __future3__ import explicit_literals.
This forces the author to be entirely explicit with string literals in the module that imports it. "foo" and 'foo' are now errors; the module won't import. Instead the module has to be explicit and use b'foo' and u'foo' everywhere.
What does that get us? It forces a developer to think about string literals everywhere, and that helps the codebase become incrementally more compatible with Python 3.
This import line does two things:
I took this idea from the Python future module, which makes Python 3 style str and bytes (and much more) available in Python 2.7. I've modified the idea as I have the imaginary power to change the interpreter in Python 2.8. Of course anything I got wrong is my own fault, not the fault of Ed Schofield, the author of the future module.
To ensure you still have access to Python 2 bytes (really str) just in case you still need it, we need an additional import:
from __past__ import bytes as oldbytes
oldbytes` can be called with Python 2 str, Python 2 bytes and Python 3 bytes. It rejects a Python 3 str. I'll talk about why it can be needed in a bit.
Yes, __past__ is another new namespace we can safely support in Python 3. It would get more involved in Python 3: it contains a forward port of the Python 2 bytes object. Python 3 bytes have less features than Python 2 bytes, and this has been a pain point for some developers who need to work with bytes a lot. Having a more capable bytes object in Python 3 would not hurt existing Python 3 code, as combining it with a Python 3 string would still result in an error. It's just an alternative implementation of bytes with more methods on it.
This is the equivalent import for getting the Python 3 bytes object.
So what happens when we somehow combine a Python 3 str/bytes with a Python 2 str/bytes/unicode? Let's think about it.
The future module by Ed Schofield forbids py3bytes + py2unicode, but supports other combinations and upcasts them to their Python 3 version. So, for instance, py3str + py2unicode -> py3str. This is a consequence of the way it tries to make Python 2 string literals work a bit like they're Python 3 unicode literals. There is a big drawback to this approach; a Python 3 bytes is not fully compatible with APIs that expect a Python 2 str, and a library that tried to use this approach would suffer API breakage. See this issue for more information on that.
I think since we have the magical power to change the interpreter, we can do better. We can make real Python 3 string literals exist in Python 2 using __future3__.
I think we need these rules:
So while we upcast existing Python 2 unicode strings to Python 3 str we refuse any other combination.
Why not let people combine Python 2 str/bytes with Python 3 bytes? Because the Python 3 bytes object is not compatible with the Python 2 bytes object, and we should refuse to guess and immediately bail out when someone tries to mix the two. We require an explicit Python 2 str call to convert a Python 3 bytes to a str.
This is assuming that the Python 3 str is compatible with Python 2 unicode. I think we should aim for making a Python 3 string behave like a subclass of a Python 2 unicode.
We can now start using Python 3 str and Python 3 bytes in our Python 2 codebases, incrementally upgrading, module by module.
Libraries could upgrade their internals to use Python 3 str and bytes entirely, and start using Python 3 str objects in any public API that returns Python 2 unicode strings now. If you're wrong and the users of your API actually do expect str-as-bytes instead of unicode strings, you can go deal with these issues one by one, in an incremental fashion.
For compatibility you can't return Python 3 bytes where Python 2 str-as-bytes is used, so judicious use of __past__.str would be needed at the boundaries in these cases.
People who have ported their code to Python 2.8 and have turned on all the __future3__ imports incrementally will be in a better place to port their code to Python 3. But to offer a more incremental step, we can have a Python 2.9 that requires the __future3__ imports introduced by Python 2.8. And by then we might have thought of some other ways to smoothen the upgrade path.
I've just released Morepath 0.1! This is Morepath's first ever proper release. Hurray! If you've been waiting for a release before trying Morepath, now is the time to stop waiting!
Morepath is your friendly neighborhood Python web framework with super powers.
The docs for 0.1 are here: http://morepath.readthedocs.org/en/0.1/
The docs are quite complete in describing Morepath's abilities and how to use them, though undoubtedly things could be improved. Let me know if you find any sections where things are unclear or missing!
There are still some docs floating around that I intend to integrate into the main documentation. One bit involves the permission and authentication docs; I previously blogged a description.
You can integrate Morepath with SQLAlchemy or the ZODB using more.transaction. I've just made a 0.1 release of that too. See this blog entry for more information on what's going on with that, and here is example code.
There is a #morepath channel on freenode IRC.
If you have an issue or feature request, you can use the issue tracker on Github:
You can also contact me directly; contact information is at the bottom of this weblog. I'm happy to hear from you!
Morepath is now transitioning into a phase where its actual use will drive its further development. I expect others (you?) will come up with new and interesting ways to use the facilities and abstractions that Morepath offers. I'm looking forward to discovering new and better ways to do web development with you!
Morepath is the friendly neighborhood Python web framework with super powers that I'm working on.
Let me start by stating that Werkzeug and WebOb are extremely similar libraries. There are some minor differences in the details of the Request and the Response object API, but the capabilities are pretty equivalent. It was easy to switch from one to the other.
I am primarily interested in the Request and Response wrappers for WSGI, and my second interest is in the lower-level APIs to handle HTTP.
Werkzeug exposes and documents lower level APIs for HTTP processing. WebOb does not have so much layering, and does not expose a low-level HTTP API.
I like Werkzeug better here. I noticed the lack in WebOb in one point in Morepath: dealing with basic authentication. While Werkzeug exposed an API for parsing the authentication header (parse_authorization_header), WebOb did not and I had to steal code from Pyramid. It would be nice if WebOb included more of such lower-level utility APIs.
Werkzeug contains a routing implementation, which always bothered me a little; I have my own routing implementation in Morepath and do not want Werkzeug's. WebOb focuses on just request and response handling and is a better fit for Morepath here.
During my testing I ran into a point where I wanted to test cookies. Werkzeug offers handy test utilities that take care of this. WebOb does not do it out of the box. But I quickly found WebObToolkit, which does offer a Client class with the same capabilities as the one in Werkzeug. I could easily convert my tests from Werkzeug to WebOb doing this, and only have to have a test dependency on WebObToolkit.
Werkzeug exposes URL parameters in a args attribute of the Request object, and WebOb instead offers a GET attribute.
I find the name of the GET attribute slightly wrong, as you can have URL parameters for a POST request as well.
Werkzeug exposes a parsed form in its form attribute, whereas WebOb uses the POST attribute for this. This is also confusing, as it contains the POST body for other HTTP methods as well, such as PUT. In addition with form I get the immediate clue that it's a parsed form, whereas with POST I don't get this clue and in fact I had to check the manual to verify it only can contain form data.
WebOb also offers params, which is GET and POST combined, but Morepath needs specific access, not this combined one. Werkzeug calls this values.
It's easy enough to learn this and only a minor annoyance. Still, I wonder whether it'd be worth it for WebOb to introduce args and form as aliases for GET and POST and then perhaps deprecate the old style.
As discussed, WebOb is a bit faster for my use cases than Werkzeug. I suspect a lot of the performance in WebOb has to do with the optimization efforts by Chris McDonough, who uses WebOb in Pyramid.
Werkzeug's performance issues may be a regression due to compatibility code for Python 3 -- much of it seems to be due to an excessive amount of isinstance calls that probably have to do with string processing.
Both WebOb and Werkzeug are Python 3 compatible, though the way WebOb introduced this compatibility evidently avoided performance issues.
While Morepath looks like Flask, it is quite similar to Pyramid under the hood in many details.
When I announced the switch to WebOb from Werkzeug I got some positive feedback -- of course I might've gotten equivalent positive feedback from the Flask folk if I'd switched the other way around; it's impossible to say. I do know that in the Pyramid world there seems to be a bit more of a culture of sharing generic libraries than there is in the Flask world.
People already expressed interest in sharing code between Pyramid libraries and Morepath libraries, and this should now be easier as the request and response objects are shared.
This should in particular make it easier to write tweens in such a way that they work in Pyramid and Morepath both. Tweens are an idea I took from Pyramid and a tween function has the same API in Pyramid and Morepath -- it takes a request and returns a response.
This would involve some refactoring of tween factory code however (or a compatibility layer), as the way tweens are created is different.
One thing that bothered me with Werkzeug are the many mixins provided that you can include in the Request and Response objects. It was never quite clear to me what mixins Morepath should be using exactly, except in one case, where I had to involve CommonResponseDescriptorsMixin to make sure the content_type header got set properly on the response -- which I found out only after some debugging.
I don't really see the point of all these mixins; in theory you could include just the functionality you need, but in practice the extra functionality does not really hurt on the original Request and Response objects itself, and I just get confused as what I should use.
WebOb does offer BaseRequest versus Request, where Request adds the AdhocAttrMixin, which seems to maintain all non-webob attributes on the Request in the WSGI environment for some reason. Once I saw the performance drawback that brought, I quickly started using BaseRequest instead.
Werkzeug has a built-in debug server with some interesting capabilities. WebOb does not. I hadn't used the debug server myself with Morepath yet, though I had integrated it, so I didn't feel terrible in replacing it with the server of wsgiref for development purposes. Still, I should look around in WSGI/WebOb land to see whether I can find something similar. Anyone have any ideas?
Werkzeug implements HTTP exception classes, and WebOb does too. This means you can raise a HTTP exception and have the framework catch it and use it as a HTTP response. Very convenient, and I use it in Morepath.
But Werkzeug actually documents the HTTP exception classes available, and I can link to them with the Morepath documentation using intersphinx.
WebOb does not offer API documentation for its exception classes, and I had to look at the source. It would be nice if WebOb included API documentation for these.
The two frameworks are pretty equivalent. There are not really very strong reasons to pick one over the other.
Werkzeug does a bit more, which is sometimes nice and something more than I need. Werkzeug also has better API documentation. On the other hand it offers a complex system of mixins.
WebOb is faster, and a bit closer to the goldilocks zone for the purposes of Morepath: not too much, and only rarely too little. It should also not be hard to improve WebOb in the areas where Werkzeug is nicer. There's also the hope of more code sharing with the Pyramid ecosystem.
Hopefully this article will be helpful to those trying to figure out what WSGI request/response implementation to use, and also to the maintainers of Werkzeug and WebOb themselves.
Let me know what you think!