Racing the Morepath: SQLAlchemy Integration    Posted: 2014-02-27 21:20


I felt like I was racing on the Morepath today. My goal was to see how to integrate Morepath with a database. To make this goal practical, I looked into integrating Morepath with SQLAlchemy. To go faster, I borrowed ideas and code liberally from Pyramid.

Tweens

This morning I borrowed the idea of tweens from Pyramid. A tween is basically much like a WSGI framework component, but one that knows about the web framework -- it gets the web framework's request, and it can send the web framework's response, among other things. Now you can write this with Morepath:

@app.tween_factory()
def get_tween_factory(app, handler):
    def wrapping_handler(request, mount):
         reponse = handler(request, mount)
         response.headers['foo'] = 'bar'
         return response
    return wrapping_handler

You can plug in a function that generate a wrapper around the original request handler.

more.transaction

This allows all sorts of neat stuff, including, for Pyramid, pyramid_tm, which integrates Pyramid with the transaction module.

So I ported over this code to Morepath in the form of more.transaction.

To use it in your Morepath app, simply do:

from more.transaction import transaction_app

app = morepath.App(extends=[transaction_app])

What happens now? Morepath will automatically commit the transaction if there is no error (like a 500 error, say).

This means that Morepath now has integration with databases that use the transaction module, such as the ZODB and also SQLALchemy (using zope.sqlalchemy for transaction integration).

[update after someone blindly complains after seeing the word "zope" in a package name... Please don't do that.]

Moreover, you can use multiple such databases in the same application. You can modify the ZODB and a relational database in an application, and be secure that if anything fails during the request handling, none of the databases will changed -- both transactions will be aborted. There's a lot of goodness in the transaction module.

Morepath settings infrastructure

It turns out pyramid_tm is configurable in various ways. It allows you to set the number of attempts it will try to commit in the face of conflicts, for instance. To support this, I had to build Morepath's settings infrastructure; just the part where you can have settings at all, not loading them from a config file -- that's for later.

Here's an example of the settings for the transaction app in more.transaction:

@app.setting_section(section='transaction')
def get_transaction_settings():
    return {
        'attempts': 1,
        'commit_veto': default_commit_veto
        }

These are the defaults defined by more.transaction, but they can be easily overridden in your own app (by writing the same code as above with different values for the configuration).

When I started to write Morepath's settings infrastructure I wrote quite a bit of code involving combining and extending settings, only to throw it away by replacing it with much shorter code that builds on Morepath's config engine that I already use for its other directives. Nice!

morepath_sqlalchemy

Now that I had all the pieces I needed to put them together to demonstrate SQLAlchemy integration: morepath_sqlalchemy.

This uses more.transaction, zope.sqlalchemy and SQLAlchemy together. It's all done here, but I'll summarize the important bits of integration here:

from sqlalchemy.orm import scoped_session, sessionmaker
# SQLAlchemy session
Session = scoped_session(sessionmaker())

from zope.sqlalchemy import register
# register session with transaction module
register(Session)

import morepath
from more.transaction import transaction_app

# create our app, extending transaction_app.
# this means we get Morepath transaction integration
# and default settings for it.
app = morepath.App(extends=[transaction_app])

Quite a day

It's all still rough, needs polishing and documenting, but the foundations are now there. Quite the day of coding! I couldn't have done it without the Pyramid project from which I could borrow many an idea and piece of code.

Now I know Morepath can be integrated with any kind of database -- if transaction module integration is there, like for SQLAlchemy and the ZODB, it is very easy: just use more.transaction. But even if not, it should now be possible to write a tween that does the trick.

Tweens allow other neat things too: I think I saw Pyramid's custom error view system is based on a tween, and I still need custom error views in Morepath...

Interested? Hope to hear from you! Join #morepath on freenode IRC, drop me an email, or leave a comment on the issue tracker.

Comments

The Centre Cannot Hold    Posted: 2014-02-26 15:05


Things fall apart; the centre cannot hold

(The Second Coming by Yeats)

Last time I talked about how I went back to the center of the Zope project. Over the course of the year following, we managed to refactor the Zope Toolkit, clean up the dependency structure, and we could drop many of its unwanted dependencies.

I had hoped that with new leadership, a steering group, we could also reinvigorate the Zope project itself. Could we get together and do exciting new things again?

The answer was no.

Chris McDonough

In late march 2009, I visited PyCon, in Chicago. There I had a conversation with Chris McDonough, who was working on what was to become the Pyramid web framework. He and I had a conversation about the dependency cleanup project I had started and that had been making waves on the zope-dev mailing list. Some beer was involved, and some miscommunication. Chris was skeptical that the cleanup project would succeed within the year, which confused me a bit, as we had already made a lot of progress.

But I was talking about cleaning up circular dependencies and the ability to lose a lot of the code. Chris was talking about making libraries with a clear purpose and documentation.

And while we got better dependencies at the end of the year, I failed.

Burning out on Zope

The details of the straw that broke the camel's back (though I hardly have the stamina of a camel) are immaterial. Suffice it to say that when practical disagreements happened our steering group did not function.

So in early 2010 I realized I was putting more into the project than I was getting out of it. I was running out of steam. It was costing me emotionally. So I stopped going to the Zope mailing lists. While we had made progress on matters where there was consensus, maintenance seemed the only consensus that could still be found.

Consensus on the boring stuff is not enough when the web is changing. And the web was changing, as it always is. A lot of the innovation on the web was happening on the client-side, in JavaScript, but Zope had no client-side story. People had moved into different directions, and the community had fractured.

The dependency cleanup was just about the only progress being made -- what about my personal goals? Where was the creativity, the getting together to do new interesting things? It wasn't there. Instead we had a bunch of opinionated people who couldn't agree enough to get anything but basic maintenance work done, and stumbled doing even that.

I'd been involved in Zope heavily, also serving on the Zope Foundation board for some years, including as its chairman. I decided I had to pull back from all of it.

The Zope Summit

In September 2010 I found myself at a Zope Developer Summit, which had been organized in part because of my urging. I had been heavily invested in Zope, and had been for more than 10 years. I had used Zope, benefited from Zope, contributed to Zope, redefined Zope and built on Zope. I had learned from Zope. I had been first board member and then chairman of the Zope Foundation.

I had hoped that a summit could get things moving again. Talk about cool new things that we might do together.

Zope was in trouble. The codebase would live on. It still does. But it is in maintenance mode - it doesn't do much that is new.

I came to the Zope Summit with a gloomy heart. That did not help my mood at the summit -- sorry.

I think most of us left the summit with the feeling not enough had been accomplished. The future of Zope was disappearing. Zope had lost its power to adapt. The web changes, but Zope didn't anymore.

This was the end of Zope, for me. I still use it in the shape of Grok to this day, but it is not the same.

Life After Zope

But this is not the end. The code still continues, and is being used, though is mostly in maintenance mode.

And there is life after Zope. Next I will talk a bit about what came after.

This blog entry is a part of a series on Zope and my involvement with it. Previous.

Comments

Breaking Morepath Changes    Posted: 2014-01-22 17:30


I'm slowly heading to a first release of the Morepath web framework, but right now I can still change anything without breaking any significant code. So I took the opportunity to do so.

What's Morepath? Morepath is your friendly neighborhood web framework with superpowers. Read more here.

These changes are in fact less big than some refactorings I do to Morepath frequently, but they break the public API of Morepath, so they're big in that sense.

These are the changes:

  • The @app.model directive was renamed to @app.path, as I realized the directive is describing a path more than a model. Here's what it looks like now:

    @app.path(model=Document, path='documents/{id}')
    def get_document(id):
        return query_document(id)
    

    The name is justified as such: just like the view directive describes a view, the path directive describes a path. Paths and views are related to each other by model class. The word model was rather overused in Morepath anyway.

  • The @app.view directive decorates a function that's a view. It used to get request, model parameters. I've changed this to self, request, reversing the order. This to make it clearer to people that a view is really much like a method, and to free up the word model some more. Here's what it looks like:

    @app.view(model=Document)
    def document_default(self, request):
        return "Hello document: %s" % self.id
    

    self is not a reserved word in Python, so I figured this was a good place to use it, even though document_default really is a function, not a method. But since it's a generic function it's like a method anyway.

    The lookup of the view is still done giving request a greater weight than model, like in Pyramid. That's mostly an implementation detail in Morepath. In Pyramid this matters a lot more, but in Morepath there really isn't anything done yet with different request classes.

  • The @app.root directive is now gone. It wasn't pulling its weight anymore, as it had become just an alias for @app.path with a path parameter of "/". This is what it looks like now:

    @app.path(model=Root, path='/')
    def get_root():
        return the_root
    

(by the way, if the docs on the Morepath website don't update for you, do a shift reload. I'm not sure how long it takes for the cache to expire.)

Want to talk to me about Morepath? Leave a comment here, drop me an email, use the Morepath issue tracker or join the #morepath IRC channel on freenode. Hope to hear from you!

Comments

Morepath Update    Posted: 2014-01-20 13:26


I've been hard at work on Morepath recently, and I thought I'd give an update. Morepath is your friendly neighborhood Python web framework with super powers.

Models and linking

URL parameters are now a first-class citizen in Morepath model-link generation, meaning that you can generate a link like:

/document?id=100

as easily as a link like this:

/document/100

with the same invocation:

request.link(doc)

It just depends on how you register the document.

Morepath can now also automatically decode the string "100" into the int 100 and encode it back again, and this is extensible to other kinds of objects.

Much more about this can be found in the model and linking section in the Morepath documentation.

Views

Views also have had their own powers and capabilities for a while.

Writing a generic view is as easy as writing any view in Morepath. You can extend view matching with new predicates, and composing views programmatically is as easy as calling request.view().

What's new are the docs. Read much more about views.

Comparing Morepath with other web frameworks

There are so many web frameworks, why should you even consider Morepath? I attempted to give a bit of background on how Morepath is the same and how it is different, and why, in the Comparison with other Web Frameworks document.

Feedback

Even a little bit of feedback can be enormously helpful to me, so I invite everybody again to leave a comment, drop me a mail, or join us on the tiny but growing #morepath IRC channel on freenode!

Join the Morepath in the early days and gain "I was there at the time" bragging rights!

Comments

How to do REST with Morepath    Posted: 2014-01-09 02:20


Another day, another piece of Morepath documentation. Morepath is your friendly neighborhood Python microframework with super powers.

Morepath lets you write RESTful web services using the same tools you'd use to write any web applications, not with a separate framework. That's how it should be: traditional multi-page web applications tend to be eminently RESTful, after all. I think it's a good idea to make what's behind your modern rich client-side applications in JavaScript RESTful too: that's what I've been doing for some years now.

Lots of people claim to do REST, and they're all wrong! Okay, well, not all of them, but many of them are. They do all the rest of REST, except the very important bit called HATEOAS, which has the scariest acronym ever, I can barely even spell it, but basically means you link your REST resources to each other, just like you'd link web pages. Morepath does HATEOAS out of the box, because Morepath is good at paths. It's in the name.

Join me on the Morepath and read how to do REST with it!

http://morepath.readthedocs.org/en/latest/rest.html

As always feedback is very welcome. Even a little bit of feedback can mean a lot to me. It gives me ideas. Leave a comment here, or join the ever burgeoning #morepath IRC channel on Freenode. Seriously. I counted like 5 people there today! Including me. I'm totally not lonely!

Have fun reading!

Comments

Morepath Security    Posted: 2014-01-07 15:50


In this post I'd like to discuss the security infrastructure of the Morepath web framework. I hope that I can showcase some of the interesting features of Morepath security, generate a bit of feedback for me, and also serve as the start of documentation on this topic. Finally, it's also useful for me to get an overview of where gaps in my implementation still exist.

What it should do

So what do I mean by "security"? I mean infrastructure in the Morepath web framework that helps you make sure that the web resources published by your application are only accessible by those people who are allowed to by your application. If you're not allowed you'll get a HTTP error.

Right now that error is HTTP Unauthorized (401) but I think I'll shortly change it to make it HTTP Forbidden (403), and make HTTP Unauthorized optional when HTTP Basic Authentication is in use. But I need to add a bit of infrastructure to make that possible first.

What does this look like in practice?

First we need to make sure that the application has an identity policy: a system that takes the HTTP request and established a claimed identity for it. For basic authentication for instance it will extract the username and password. The claimed identity can be accessed by looking at the identity attribute on the request object.

This is how you install an identity policy into a Morepath app:

from morepath.security import BasicAuthIdentityPolicy

@app.identity_policy()
def get_identity_policy():
    return BasicAuthIdentityPolicy()

Let's say we want two permissions in our application, view and edit. We define those as plain Python classes:

class ViewPermission(object):
    pass

class EditPermission(object):
    pass

Since they're classes they could even inherit from each other and form some kind of permission hierarchy, but we'll keep things simple here.

Now we can protect views with those permissions. Let's say we have a Document model that we can view and edit:

@app.html(model=Document, permission=ViewPermission)
def document_view(request, model):
    return "<p>The title is: %s</p>" % model.title

@app.html(model=Document, permission=EditPermission)
def document_edit(request, model):
    return "some kind of edit form"

This says that we only want to allow access to document_view if the identity has ViewPermission on the Document model, and only allow access to document_edit if the identity has EditPermission on the Document model.

But how do we establish permissions for models at all? We can do this with the permission directive. Let's give absolutely everybody view permission:

@app.permission(model=Document, permission=ViewPermission)
def document_view_permission(identity, model, permission)
    return True

We can implement any permission policy we like. Let's say a user has EditPermission on Document if it's in a list allowed_users on that document:

@app.permission(model=Document, permission=EditPermission)
def document_edit_permission(identity, model, permission):
    return identity.userid in model.allowed_users

Morepath Super Powers Go!

What if we don't want to have to define permissions on a per-model basis? In our application, we may have a generic way to check for the edit permission. We can easily do it with Morepath, as it knows about inheritance:

@app.permission(model=object, permission=EditPermission)
def has_edit_permission(identity, model, permission):
    return has_generic_edit_permission(identity, model)

This permission function is registered for model object, so will be valid for all models in our application. We can of course still make exceptions for particular models:

@app.permission(model=SpecialModel, permission=EditPermission)
def special_model_edit_permission(identity, model, permission):
    return exceptional_edit_permission_check(identity, model)

Variation is not only on model, but can also on permission or identity. For instance, we can register a permission policy for when the user has not logged in yet, i.e. has no claimed identity:

@app.permission(model=object, permission=EditPermission, identity=None)
def has_edit_permission_not_logged_in(identity, model, permission):
    return False

This permission check works in addition to the ones we specified above. In short you can be as generic or specific as you like.

Login forms

We've tackled a lot, but not yet how a user actually logs in to establish an identity in the first place. We need a view that gets triggered when the user logs in, for instance by submitting a login form. I'll sketch out some code here to give you an idea:

import morepath

@app.root()
class Root(object):
    pass

@app.html(model=Root, name='login')
def login_attempt(request, model):
    username = request.form['username']
    password = request.form['password']
    # check whether user has password, using password hash and database
    if not user_has_password(username, password):
        return "Sorry, login failed"
    # okay, user is known, so modify response remembering that user
    @request.after
    def remember_security(response):
        # create identity object
        identity = morepath.Identity(username)
        # modifies response, setting identity
        morepath.remember(response, request, identity)
   # redirect to original page after successful login attempt
   return morepath.redirect(request.form['came_from'])

This would work well for session, cookie or ticket-based authentication: after the identity is established once it is remembered on the response object, so that it is automatically sent with each new request.

For HTTP basic authentication this form-based authentication approach wouldn't work, of course. We'd instead need to issue a HTTP Unauthorized response, causing the browser to ask the user for a username/password combination, which it would then send along for each subsequent request. The remember operation is a no-op for basic auth.

Note: Currently this is in fact broken code as I haven't enabled the implicit generic function implementation lookup yet for Morepath implementations. It's an easy fix as I have the infrastructure for it in Reg, but I need to add it to my todo list.

Theory: Identity/permits

Security in Morepath has two parts:

  • establish someone's claimed identity.
  • use that claimed identity to see whether access to a Morepath view is permitted.

This looks very much like, but isn't quite, a separation between authentication and authorization. There are subtle differences.

Establishing identity only establishes a claimed identity and does not verify (with, say, a database), that this identity is in fact still recognized by the system (the user may be removed), or that this identity is even as it is claimed.

In the case of some identity policies and some applications this is in fact enough: if someone manages to claim an identity, then it is the identity, without the need to access a database. This is for instance the case with cryptographic ticket based authentication systems such as the one implemented by mod_auth_tkt. If someone comes along with the right encrypted auth tkt cookie, we know that's the identity we gave to this user earlier when they first logged in. No database verification check is needed.

Then there's the next step: to see whether a claimed identity is permitted to access a view on a model. This can be seen as the authorization step, and will normally have to access some kind of database to do so. It may however do something in addition when accessing the database: verify that the claimed identity is still valid. This is done in this step because it can sometimes be established whether someone has access and whether the claimed identity is valid by a single database query.

What have these changes accomplished? We've made sure that establishing a claimed identity can be done without touching a database; only checking whether that identity is permitted something has to touch a database, possibly only once.

I'm grateful to Chris McDonough, creator of the Pyramid web framework, who wrote a very useful postmortem on Pyramid's design and what he'd do differently if he could change things. That was extremely useful when I come up with Morepath's security system, so thanks again, Chris! Also inspirational was my long familiarity with Zope's security system, and I hope to have distilled some of it down to the minimum. Of course anything I got wrong is my own fault.

Identity Verification?

I think the case of Morepath it would in fact be easy to let users specify an app-specific verify function, which will be called with the identity and can use the database to establish whether an identity is claimed. This would be easy to integrate into Morepath, and doesn't require the miniframeworks that Chris bother him in his postmortem. It could look like this (for something like basic auth):

@app.function(generic.verify, object)
def verify_identity(identity):
     return validate_password(identity.userid, identity.password)

I could add a bit of sugar for this in the form of a directive:

@app.verify()
def verify_identity(identity):
     return validate_password(identity.userid, identity.password)

The default verify would just return True, so it would still work for identity policies and applications that have no need for identity verification. Identification verification would not be part of identity policy, but part of Morepath's security infrastructure, as it would be entirely application specific.

More identity policies

Right now Morepath implements only a single identity policy, and that's BasicAuthIdentityPolicy for HTTP basic auth. I hope I will get time in the future to port some of the more interesting functionality from Pyramid's authentication module; the AuthTktAuthenticationPolicy looks interesting in particular. These could then be made available in a Morepath extension module. If you are interested in helping this porting activity I'd be thrilled though -- get in touch with me, please!

Conclusion

I'm pretty pleased with the flexibility of the Morepath security system: you can be fine-grained or generic when you need to. It's all built on top of the Morepath foundations of directive-based configuration and Reg generic functions, and it fits.

The identity system should be able to cope well with any kind of authentication system you can throw at it, allowing you to write generic code and swapping in a different one with ease. The only oddball is the perennial exception of HTTP Basic Auth, but at least Morepath can deal with it.

There may seem to be many concepts involved at first, but I think in fact they've been kept down to an amount that you can still keep in your head after you get used to it.

I hope to get feedback on this, and I also hope people will play with it, so we can smoothen out any kinks quickly. Please let me know what you think! Join the #morepath IRC channel on freenode!

Comments

the Gravity of Python 2    Posted: 2014-01-06 17:30


Recently there have been some discussions about Python 3 adoption. Adoption rates aren't great. There are a number of theories on why Python 3 is not adopted more and what should be done about it. I'll expand a bit on the analysis by Ian Bicking and add my own. I'll offer a few suggestions on what might help.

The Stick

Some people blame Python 2.7. The core developers were way too nice to us Python developers by maintaining Python 2.x. If they had dropped support for Python 2.x years ago, Python 3 adoption would've been much faster. I think this argument is based on counterfactuals that are very hard to back up. I wish people would stop making these assertions, as it's impossible to prove either way. Even if this argument could be shown to be true, I for one am very glad that the core developers didn't throw the existing community into the deep end.

Somehow punishing those holdouts on Python 2 so they'll move I'll call the "stick" approach.

As Ian points out, this supposes some kind of moral authority for Python 3, or at least the progress of the Python language as a whole.

People who are using Python 2 and who don't port code are holding the language back. We should have some sort of allegiance to Python, and why aren't we getting off our behinds and do the work? It's not that hard! You're holding back The Python!

This kind of moral argument may work on those who are active members of the Python community, but it won't work those who are using Python as a tool for a job. I'll get back to them, as they're important.

The Default

Some people blame the Linux distributions. People new to Python apparently don't do the research, but instead type "python" only to discover Python 2 and then stick to it. Once Linux distros upgrade their Python, people will start using Python 3.

I can't believe that this is a major force holding everything back. Most Python beginners will check out python.org. Python 3 is quite widely available - it's in my Fedora installation and I'm two releases behind. Won't a beginner at least check out python.org? On Windows, installing Python 3 is no problem.

A more realistic problem is that of arcane system administration practices and obsolete Linux distributions making it harder to get Python 3 installed. I still don't think this is in any way a dominant factor.

The Carrot

Other people blame Python 3. The message is that Python 3 was immature early on, which is why people didn't adopt it, but that has changed. It's mature now, and people should adopt it now. If only Python 3 is made attractive enough, people will adopt it.

I think there is some truth to this. But the problem is that we've had about five years of people claiming that Python 3 is the superior version of Python, and it's still not commonly used.

There are other forces at work that I want to highlight here.

Dependencies

A common theory is that people aren't moving to Python 3 because their dependencies haven't moved to Python 3 yet.

What's a dependency? If you write code that depends on, say, the Reg library, and the Reg library only exists for Python 2, then you have to wait for the developer of the Reg library (me) to port to Python 3 first before you can port my own code. Or you have to dive in and port it yourself, hopefully somehow coordinating that with the original author.

This is the situation that holds you back from Python 3. I think this theory has a lot of merit. The problem is widely understood, which is why we see a lot of people urging us to port.

What seems to be widely neglected is the problem of the reverse dependencies.

Reverse Dependencies

I am sure that dependencies are an important factor, but what seems to be ignored in these discussions is the issue of reverse dependencies. Reverse dependencies have a gravity all their own.

What's a reverse dependency? If you are developing a cool new library, and you know that this library needs to be integrated into an existing Python 2 codebase, you need to make sure that this library works in Python 2.

Python 2, Python 3 or Polyglot?

There's a common response to this situation. You are writing a Python library, and you need it to work in a Python 2 context.

Writing the library in Python 3 is now out of the question. You can't enjoy the cleaner feel of Python 3. You can't use any of the cool new Python 3 features.

If you feel some moral obligation to The Python, or at least want to be cool and modern, you might consider writing polyglot code: code that works in Python 2 and Python 3. You gain for this some coolness points, potential users who use Python 3, and a harder to maintain codebase. You gain none of the benefits of Python 3.

You can also just write Python 2 code.

Now what is the better programming language to work with? Python 3, of course! But you can't use it. If you have to make the choice between Python 2 and Polyglot, Python 2 is the better programming language. So if you don't care about The Python, in this context you are better off writing Python 2.

The Gravity

Okay, forget about all those legacy codebases! If we write all our new code at least in Polyglot, we can start more new projects in Python 3.

I think that this does a disservice to those old projects. They were happy to upgrade to new Python 2.x versions, as that was doable with little effort. They might've picked Python before Python 3 came along and created this upgrade bump, or at least before Python 3 was mature (whatever point in time we pick where Python 3 is considered mature). But that's a moral argument. We have no allegiance to other people's codebases.

But those existing Python 2 codebases are the ones with business value to people. Those are the codebases with the bosses and customers who want new functionality, not a port to a new version of Python that involves an effort with no near-future benefits and the potential to introduce bugs. Those are the codebases with the money.

Those codebases have gravity. That's the gravity of Python 2.

Circular Dependencies

So now we have dependencies holding us back, and reverse dependencies holding us back. Those create a circular dependency structure. We know that those are the worst kind to deal with from a maintenance perspective!

Our code has dependencies. More dependencies are available for Python 2 than Python 3. Those dependencies that are polyglot are more likely to be battle tested in a Python 2 context than Python 3. Motivation not to move to Python 3.

The gravity caused by reverse dependencies written in Python 2 then pulls at people so that the situation remains that way. You will write your new library in Python 2 or Polyglot, because you have to integrate with Python 2 codebases. That is going to make more Python 2 dependencies available, not less.

Passing the porting buck down the Python 2 gravity well

The gravity field of Python 2 only pulls lightly on the core developers, who are high above us in the sky working on The Python. So after an upgrade path (2to3) was figured out, the porting buck was passed to the library developers.

The library developers are lower down in the gravity field, so 2to3 didn't work out so well for them. So they figured out how to write polyglot, and to the credit of the core developers, they made modifications in both Python 2 and Python 3 to make it easier to write polyglot. That works okay enough for many, so more and more polyglot libraries are appearing.

Now the porting buck is passing down the gravity field again, to those mired in existing Python 2 codebases. The force of bosses and customers and money is heavy. Porting is harder than for the library authors in all that weight, not helped by the fact that most applications have much less test coverage than libraries. What is going to happen? Are they going to port their codebases? Are they going to be able to do so in that gravity?

Escape Velocity?

How do we get out of this situation?

One option is for Python 3 to reach escape velocity. Python 3 the language becomes attractive enough, along with enough polyglot libraries being ported, that new projects will be written in Python 3. As we leave the gravity well old Python 2 codebases become just a pale blue dot in the cosmos. A Python 2 community is left behind.

I don't think that Python 3 reaching escape velocity is exactly friendly to those dealing with Python 2 every day for no fault of their own, a "bye bye we don't care about you anymore".

Another argument against escape velocity is that this is not healthy for our community; it would tear it apart. We're feeling some of the strains today.

Another argument against escape velocity is that this might cut off significant resources from the Python community. The high gravity Python 2 area that we're trying to leave behind has all the money. Is this wise?

So should we wish for escape velocity in the first place? Is it desirable?

Escape Velocity cannot be reached?

It's also possible that the Python 2 gravity field is too strong, circular dependencies will keep holding us back, and that escape velocity cannot be reached. This means we would be in this Python 2 versus Python 3 situation forever. That would suck.

Above I listed some arguments against going for escape velocity in the first place. What if we take those seriously?

A Way Forward

How to go forward then? I think it makes sense to work as hard as possible to lift those Python 2 codebases out of the gravity well.

The moral authority argument encouraging people to port their code is not going to work down in the gravity well, though. There interest in The Python is least; people are interested in functionality in actual applications.

We have to make the way forward for those in the gravity well as easy and obvious as possible. We know those codebases often don't have great test coverage. Code is not going to become polyglot, let alone Python 3, in one fell swoop. We can do this with libraries, but it works less well for aplications.

The keyword here is porting incrementally. Port to polyglot carefully, on a per-module basis, step by step, and write new modules in polyglot from the start, until finally you can switch the whole thing over.

Python 2.7 helps. There are tools to help. Recently I ran into Future, which to me is the most attractive approach yet. But will those tools work for those down in the gravity well? Will they even find them? Only things easy and obvious will reach them.

This is why I argue for more official Python 2 releases, where these things are built in. This stuff needs to be as obvious as possible.

And to make it even more obvious, we need Python 2.x releases where the deprecated stuff is removed, incrementally. Breaking code is making it as obvious as you can get! But you don't want to break it all at once, because then people will be inclined to give up before they even start.

This is why I think a continued future for Python 2.x would be in the community's interest, even in The Python's interest. I created a #python2.8 IRC channel on freenode to gauge interest and discuss this further.

Comments

#python2.8 discussion channel on freenode    Posted: 2014-01-06 14:00


Fellow Pythoneers,

I've started an informal channel "#python2.8" on freenode IRC. It's to discuss the potential for a Python 2.8 version -- to see whether there is interest in it, what it could contain, how it could facilitate porting to Python 3, who would work on it, etc. If you are interested in constructive discussion about a Python 2.8, please join.

I realize that if there is actual code created, and if it's not under the umbrella of the PSF, it couldn't be called "Python 2.8" due to trademark reasons. But that's premature - let's have some discussions first to see whether anything can happen.

Hope to see you there for some discussion!

Comments

Alex Gaynor on Python 3    Posted: 2013-12-30 21:13


Introduction

I'm happy to see the Python community starting the Python 3 conversation again. Alex Gaynor shares his analysis.

I agree with many of his points and am glad he was brave enough to take up this difficult topic in public.

Update: I see Ian Bicking has weighed in and he says things I very much agree with. Thanks Ian! Everybody should read Ian Bicking:

https://plus.google.com/+IanBicking/posts/iEVXdcfXkz7

I told you so

I've been thinking about this for a while. I brought my worries up in 2007, before Python 3 was released:

http://blog.startifact.com/posts/older/brief-python-3000-thoughts.html

I'll quote myself:

While I understand the need to be able to break backwards compatibility, I am worried about Python forking into two parallel versions (2.x and 3.x) for many, many years. Such a fork can't be good for the Python community.

This sounds a lot like what Alex is talking about today. While it's something to be able to say "I told you so", overall it just makes me slightly depressed. I'm sad I don't get excited when a new release of Python is made, like I used to be. Sometimes I don't even notice. Why read about exciting new features I won't be using any time soon?

I got feedback and wrote followups:

http://blog.startifact.com/posts/older/python-3-worries-feedback.html

http://blog.startifact.com/posts/older/the-purpose-to-my-whinging-about-the-transition-to-python-3.html

http://blog.startifact.com/posts/older/communicating-with-core-developers-on-the-python-3-transition.html

The feedback burned me quite heavily. I decided I was not going to actively do anything about Python 3 and would just develop Python 2 code and let others worry about the problem.

I shut up about Python 3 in public for years, until briefly in 2011, in response to a blog post by Armin Ronacher:

http://blog.startifact.com/posts/older/python-2-8--1.html

In early 2012 the now sadly deceased Aaron Swartz also talked about this topic, and I responded, saying "I told you so" (not Aaron personally):

http://blog.startifact.com/posts/older/cassandra-and-how-pypy-could-help-with-python-3.html

The path taken

In 2007 I was expecting that what the core developers were suggesting would continue to be the way to go: people would run 2to3 to convert Python 2 code to Python 3. I was skeptical then about this plan. Instead I hoped for what Aaron described: an interpreter where people could mix pure Python 3 modules with pure Python 2 modules.

Instead of 2to3, library authors started to work out ways to write Python 2 code in such a way that it would also run in Python 3. This was considered an absolutely unrealistic approach in the beginning, but gradually became the mainstream approach with support in Python 2.x itself.

Now much library code has been ported. That is going fairly well. But when I see such code I think it's scary and ugly: writing code that needs to work in both interpreters requires language knowledge that is more like that of an advanced C++ developer. It isn't Pythonic, at all.

Open source library and framework creators have some motivation to port. They're maintaining their stuff for a long time, and want to attract users. They also have the interest and skill level to do so. All this together may outweigh the cost of maintaining more complicated code.

The bind we're in

It's different for applications. An application developer usually has motivations not to port. Porting will introduce bugs in working applications, there's unclear benefit as the application will remain the same, and less libraries are available. It's also harder to port an application than a library, as libraries tend to have better test coverage. Finally applications exist for a reason, many of them are in-house to some organisation, and how do you motivate a budget to do the porting activity?

But if application developers don't port, why all the effort to port libraries then? You have many people continue to maintain and develop Python 2 applications. Some people write new applications in Python 3. As Alex says, not good for the community.

Python 2.8 as way forward?

Alex suggests an incremental upgrade of Python 2 adding more Python 3 features. I think this could be made to work. It might require a Python 2.8 while leaving the door open for a Python 2.9. Incremental backwards compatibility breaking can be part of the process: you could for instance drop old-style classes and tell people to adjust their code when upgrading to 2.8. While that's a relatively scary change, but most code will continue to work.

Perhaps that with the experience people gained writing 2 + 3 code we can even make the unicode path smooth enough.

The advantage of this approach compared to a magic Python 2 + 3 interpreter is that it would be possible to do incrementally based on the Python 2 codebase, with the Python 3 codebase in sight. That may involve a less scary development effort.

The main advantage is that by gradually dropping compatibility applications may move forward again, and messy 2+3 codebases can slowly be cleaned up as support for older versions of Python 2.x is dropped. I'll repeat that: a path forward for developers to Python 3 that has your code become cleaner, not uglier.

The closed door

The Python core developers somewhat gleefully slammed the door shut on Python 2.8 back in 2011, though. Recently celebrating the 5 year release of Python 3.0 I heard people say everything was going according to plan.

One quibble with Alex's post: the suggestion he makes that Python 2 should've been dropped in favor of Python 3 when Python 3 was first released, so that people would have urgency to port code, strikes me as unrealistic. I think it could have resulted in a justified rebellion among users and a fork of the Python interpreter. I'm glad that path was not taken. In fact since 2011 this has been the path taken, and I'm not sure it made any difference.

I hope the core developers will change their mind and take up the momentous unpleasant task of moving Python 2.x forward towards Python 3.x again. But if not, perhaps a brave group of volunteers will stand up and fork Python 2, and take the incremental steps forward. This will have to remain just an idle suggestion, as I'm not volunteering myself.

Comments

Morepath Documentation Starting to Take Shape    Posted: 2013-12-19 23:04


For a while now I've been working on documenting Morepath, the Python web framework with super powers. There is now a significant dose of documentation available. Perhaps it'll be interesting holiday reading for someone! If not, no problem: I'll be back next year with more!

Hopefully this will help people get an idea of what Morepath is all about. I'm very excited about how it's been taking shape!

Let me know what you think!

Comments

Contents © 2005-2014 Martijn Faassen | Twitter | Github | Gittip