How to do REST with Morepath    Posted: 2014-01-09 02:20


Another day, another piece of Morepath documentation. Morepath is your friendly neighborhood Python microframework with super powers.

Morepath lets you write RESTful web services using the same tools you'd use to write any web applications, not with a separate framework. That's how it should be: traditional multi-page web applications tend to be eminently RESTful, after all. I think it's a good idea to make what's behind your modern rich client-side applications in JavaScript RESTful too: that's what I've been doing for some years now.

Lots of people claim to do REST, and they're all wrong! Okay, well, not all of them, but many of them are. They do all the rest of REST, except the very important bit called HATEOAS, which has the scariest acronym ever, I can barely even spell it, but basically means you link your REST resources to each other, just like you'd link web pages. Morepath does HATEOAS out of the box, because Morepath is good at paths. It's in the name.

Join me on the Morepath and read how to do REST with it!

http://morepath.readthedocs.org/en/latest/rest.html

As always feedback is very welcome. Even a little bit of feedback can mean a lot to me. It gives me ideas. Leave a comment here, or join the ever burgeoning #morepath IRC channel on Freenode. Seriously. I counted like 5 people there today! Including me. I'm totally not lonely!

Have fun reading!

Comments

Morepath Security    Posted: 2014-01-07 15:50


In this post I'd like to discuss the security infrastructure of the Morepath web framework. I hope that I can showcase some of the interesting features of Morepath security, generate a bit of feedback for me, and also serve as the start of documentation on this topic. Finally, it's also useful for me to get an overview of where gaps in my implementation still exist.

What it should do

So what do I mean by "security"? I mean infrastructure in the Morepath web framework that helps you make sure that the web resources published by your application are only accessible by those people who are allowed to by your application. If you're not allowed you'll get a HTTP error.

Right now that error is HTTP Unauthorized (401) but I think I'll shortly change it to make it HTTP Forbidden (403), and make HTTP Unauthorized optional when HTTP Basic Authentication is in use. But I need to add a bit of infrastructure to make that possible first.

What does this look like in practice?

First we need to make sure that the application has an identity policy: a system that takes the HTTP request and established a claimed identity for it. For basic authentication for instance it will extract the username and password. The claimed identity can be accessed by looking at the identity attribute on the request object.

This is how you install an identity policy into a Morepath app:

from morepath.security import BasicAuthIdentityPolicy

@app.identity_policy()
def get_identity_policy():
    return BasicAuthIdentityPolicy()

Let's say we want two permissions in our application, view and edit. We define those as plain Python classes:

class ViewPermission(object):
    pass

class EditPermission(object):
    pass

Since they're classes they could even inherit from each other and form some kind of permission hierarchy, but we'll keep things simple here.

Now we can protect views with those permissions. Let's say we have a Document model that we can view and edit:

@app.html(model=Document, permission=ViewPermission)
def document_view(request, model):
    return "<p>The title is: %s</p>" % model.title

@app.html(model=Document, permission=EditPermission)
def document_edit(request, model):
    return "some kind of edit form"

This says that we only want to allow access to document_view if the identity has ViewPermission on the Document model, and only allow access to document_edit if the identity has EditPermission on the Document model.

But how do we establish permissions for models at all? We can do this with the permission directive. Let's give absolutely everybody view permission:

@app.permission(model=Document, permission=ViewPermission)
def document_view_permission(identity, model, permission)
    return True

We can implement any permission policy we like. Let's say a user has EditPermission on Document if it's in a list allowed_users on that document:

@app.permission(model=Document, permission=EditPermission)
def document_edit_permission(identity, model, permission):
    return identity.userid in model.allowed_users

Morepath Super Powers Go!

What if we don't want to have to define permissions on a per-model basis? In our application, we may have a generic way to check for the edit permission. We can easily do it with Morepath, as it knows about inheritance:

@app.permission(model=object, permission=EditPermission)
def has_edit_permission(identity, model, permission):
    return has_generic_edit_permission(identity, model)

This permission function is registered for model object, so will be valid for all models in our application. We can of course still make exceptions for particular models:

@app.permission(model=SpecialModel, permission=EditPermission)
def special_model_edit_permission(identity, model, permission):
    return exceptional_edit_permission_check(identity, model)

Variation is not only on model, but can also on permission or identity. For instance, we can register a permission policy for when the user has not logged in yet, i.e. has no claimed identity:

@app.permission(model=object, permission=EditPermission, identity=None)
def has_edit_permission_not_logged_in(identity, model, permission):
    return False

This permission check works in addition to the ones we specified above. In short you can be as generic or specific as you like.

Login forms

We've tackled a lot, but not yet how a user actually logs in to establish an identity in the first place. We need a view that gets triggered when the user logs in, for instance by submitting a login form. I'll sketch out some code here to give you an idea:

import morepath

@app.root()
class Root(object):
    pass

@app.html(model=Root, name='login')
def login_attempt(request, model):
    username = request.form['username']
    password = request.form['password']
    # check whether user has password, using password hash and database
    if not user_has_password(username, password):
        return "Sorry, login failed"
    # okay, user is known, so modify response remembering that user
    @request.after
    def remember_security(response):
        # create identity object
        identity = morepath.Identity(username)
        # modifies response, setting identity
        morepath.remember(response, request, identity)
   # redirect to original page after successful login attempt
   return morepath.redirect(request.form['came_from'])

This would work well for session, cookie or ticket-based authentication: after the identity is established once it is remembered on the response object, so that it is automatically sent with each new request.

For HTTP basic authentication this form-based authentication approach wouldn't work, of course. We'd instead need to issue a HTTP Unauthorized response, causing the browser to ask the user for a username/password combination, which it would then send along for each subsequent request. The remember operation is a no-op for basic auth.

Note: Currently this is in fact broken code as I haven't enabled the implicit generic function implementation lookup yet for Morepath implementations. It's an easy fix as I have the infrastructure for it in Reg, but I need to add it to my todo list.

Theory: Identity/permits

Security in Morepath has two parts:

  • establish someone's claimed identity.
  • use that claimed identity to see whether access to a Morepath view is permitted.

This looks very much like, but isn't quite, a separation between authentication and authorization. There are subtle differences.

Establishing identity only establishes a claimed identity and does not verify (with, say, a database), that this identity is in fact still recognized by the system (the user may be removed), or that this identity is even as it is claimed.

In the case of some identity policies and some applications this is in fact enough: if someone manages to claim an identity, then it is the identity, without the need to access a database. This is for instance the case with cryptographic ticket based authentication systems such as the one implemented by mod_auth_tkt. If someone comes along with the right encrypted auth tkt cookie, we know that's the identity we gave to this user earlier when they first logged in. No database verification check is needed.

Then there's the next step: to see whether a claimed identity is permitted to access a view on a model. This can be seen as the authorization step, and will normally have to access some kind of database to do so. It may however do something in addition when accessing the database: verify that the claimed identity is still valid. This is done in this step because it can sometimes be established whether someone has access and whether the claimed identity is valid by a single database query.

What have these changes accomplished? We've made sure that establishing a claimed identity can be done without touching a database; only checking whether that identity is permitted something has to touch a database, possibly only once.

I'm grateful to Chris McDonough, creator of the Pyramid web framework, who wrote a very useful postmortem on Pyramid's design and what he'd do differently if he could change things. That was extremely useful when I come up with Morepath's security system, so thanks again, Chris! Also inspirational was my long familiarity with Zope's security system, and I hope to have distilled some of it down to the minimum. Of course anything I got wrong is my own fault.

Identity Verification?

I think the case of Morepath it would in fact be easy to let users specify an app-specific verify function, which will be called with the identity and can use the database to establish whether an identity is claimed. This would be easy to integrate into Morepath, and doesn't require the miniframeworks that Chris bother him in his postmortem. It could look like this (for something like basic auth):

@app.function(generic.verify, object)
def verify_identity(identity):
     return validate_password(identity.userid, identity.password)

I could add a bit of sugar for this in the form of a directive:

@app.verify()
def verify_identity(identity):
     return validate_password(identity.userid, identity.password)

The default verify would just return True, so it would still work for identity policies and applications that have no need for identity verification. Identification verification would not be part of identity policy, but part of Morepath's security infrastructure, as it would be entirely application specific.

More identity policies

Right now Morepath implements only a single identity policy, and that's BasicAuthIdentityPolicy for HTTP basic auth. I hope I will get time in the future to port some of the more interesting functionality from Pyramid's authentication module; the AuthTktAuthenticationPolicy looks interesting in particular. These could then be made available in a Morepath extension module. If you are interested in helping this porting activity I'd be thrilled though -- get in touch with me, please!

Conclusion

I'm pretty pleased with the flexibility of the Morepath security system: you can be fine-grained or generic when you need to. It's all built on top of the Morepath foundations of directive-based configuration and Reg generic functions, and it fits.

The identity system should be able to cope well with any kind of authentication system you can throw at it, allowing you to write generic code and swapping in a different one with ease. The only oddball is the perennial exception of HTTP Basic Auth, but at least Morepath can deal with it.

There may seem to be many concepts involved at first, but I think in fact they've been kept down to an amount that you can still keep in your head after you get used to it.

I hope to get feedback on this, and I also hope people will play with it, so we can smoothen out any kinks quickly. Please let me know what you think! Join the #morepath IRC channel on freenode!

Comments

the Gravity of Python 2    Posted: 2014-01-06 17:30


Recently there have been some discussions about Python 3 adoption. Adoption rates aren't great. There are a number of theories on why Python 3 is not adopted more and what should be done about it. I'll expand a bit on the analysis by Ian Bicking and add my own. I'll offer a few suggestions on what might help.

The Stick

Some people blame Python 2.7. The core developers were way too nice to us Python developers by maintaining Python 2.x. If they had dropped support for Python 2.x years ago, Python 3 adoption would've been much faster. I think this argument is based on counterfactuals that are very hard to back up. I wish people would stop making these assertions, as it's impossible to prove either way. Even if this argument could be shown to be true, I for one am very glad that the core developers didn't throw the existing community into the deep end.

Somehow punishing those holdouts on Python 2 so they'll move I'll call the "stick" approach.

As Ian points out, this supposes some kind of moral authority for Python 3, or at least the progress of the Python language as a whole.

People who are using Python 2 and who don't port code are holding the language back. We should have some sort of allegiance to Python, and why aren't we getting off our behinds and do the work? It's not that hard! You're holding back The Python!

This kind of moral argument may work on those who are active members of the Python community, but it won't work those who are using Python as a tool for a job. I'll get back to them, as they're important.

The Default

Some people blame the Linux distributions. People new to Python apparently don't do the research, but instead type "python" only to discover Python 2 and then stick to it. Once Linux distros upgrade their Python, people will start using Python 3.

I can't believe that this is a major force holding everything back. Most Python beginners will check out python.org. Python 3 is quite widely available - it's in my Fedora installation and I'm two releases behind. Won't a beginner at least check out python.org? On Windows, installing Python 3 is no problem.

A more realistic problem is that of arcane system administration practices and obsolete Linux distributions making it harder to get Python 3 installed. I still don't think this is in any way a dominant factor.

The Carrot

Other people blame Python 3. The message is that Python 3 was immature early on, which is why people didn't adopt it, but that has changed. It's mature now, and people should adopt it now. If only Python 3 is made attractive enough, people will adopt it.

I think there is some truth to this. But the problem is that we've had about five years of people claiming that Python 3 is the superior version of Python, and it's still not commonly used.

There are other forces at work that I want to highlight here.

Dependencies

A common theory is that people aren't moving to Python 3 because their dependencies haven't moved to Python 3 yet.

What's a dependency? If you write code that depends on, say, the Reg library, and the Reg library only exists for Python 2, then you have to wait for the developer of the Reg library (me) to port to Python 3 first before you can port my own code. Or you have to dive in and port it yourself, hopefully somehow coordinating that with the original author.

This is the situation that holds you back from Python 3. I think this theory has a lot of merit. The problem is widely understood, which is why we see a lot of people urging us to port.

What seems to be widely neglected is the problem of the reverse dependencies.

Reverse Dependencies

I am sure that dependencies are an important factor, but what seems to be ignored in these discussions is the issue of reverse dependencies. Reverse dependencies have a gravity all their own.

What's a reverse dependency? If you are developing a cool new library, and you know that this library needs to be integrated into an existing Python 2 codebase, you need to make sure that this library works in Python 2.

Python 2, Python 3 or Polyglot?

There's a common response to this situation. You are writing a Python library, and you need it to work in a Python 2 context.

Writing the library in Python 3 is now out of the question. You can't enjoy the cleaner feel of Python 3. You can't use any of the cool new Python 3 features.

If you feel some moral obligation to The Python, or at least want to be cool and modern, you might consider writing polyglot code: code that works in Python 2 and Python 3. You gain for this some coolness points, potential users who use Python 3, and a harder to maintain codebase. You gain none of the benefits of Python 3.

You can also just write Python 2 code.

Now what is the better programming language to work with? Python 3, of course! But you can't use it. If you have to make the choice between Python 2 and Polyglot, Python 2 is the better programming language. So if you don't care about The Python, in this context you are better off writing Python 2.

The Gravity

Okay, forget about all those legacy codebases! If we write all our new code at least in Polyglot, we can start more new projects in Python 3.

I think that this does a disservice to those old projects. They were happy to upgrade to new Python 2.x versions, as that was doable with little effort. They might've picked Python before Python 3 came along and created this upgrade bump, or at least before Python 3 was mature (whatever point in time we pick where Python 3 is considered mature). But that's a moral argument. We have no allegiance to other people's codebases.

But those existing Python 2 codebases are the ones with business value to people. Those are the codebases with the bosses and customers who want new functionality, not a port to a new version of Python that involves an effort with no near-future benefits and the potential to introduce bugs. Those are the codebases with the money.

Those codebases have gravity. That's the gravity of Python 2.

Circular Dependencies

So now we have dependencies holding us back, and reverse dependencies holding us back. Those create a circular dependency structure. We know that those are the worst kind to deal with from a maintenance perspective!

Our code has dependencies. More dependencies are available for Python 2 than Python 3. Those dependencies that are polyglot are more likely to be battle tested in a Python 2 context than Python 3. Motivation not to move to Python 3.

The gravity caused by reverse dependencies written in Python 2 then pulls at people so that the situation remains that way. You will write your new library in Python 2 or Polyglot, because you have to integrate with Python 2 codebases. That is going to make more Python 2 dependencies available, not less.

Passing the porting buck down the Python 2 gravity well

The gravity field of Python 2 only pulls lightly on the core developers, who are high above us in the sky working on The Python. So after an upgrade path (2to3) was figured out, the porting buck was passed to the library developers.

The library developers are lower down in the gravity field, so 2to3 didn't work out so well for them. So they figured out how to write polyglot, and to the credit of the core developers, they made modifications in both Python 2 and Python 3 to make it easier to write polyglot. That works okay enough for many, so more and more polyglot libraries are appearing.

Now the porting buck is passing down the gravity field again, to those mired in existing Python 2 codebases. The force of bosses and customers and money is heavy. Porting is harder than for the library authors in all that weight, not helped by the fact that most applications have much less test coverage than libraries. What is going to happen? Are they going to port their codebases? Are they going to be able to do so in that gravity?

Escape Velocity?

How do we get out of this situation?

One option is for Python 3 to reach escape velocity. Python 3 the language becomes attractive enough, along with enough polyglot libraries being ported, that new projects will be written in Python 3. As we leave the gravity well old Python 2 codebases become just a pale blue dot in the cosmos. A Python 2 community is left behind.

I don't think that Python 3 reaching escape velocity is exactly friendly to those dealing with Python 2 every day for no fault of their own, a "bye bye we don't care about you anymore".

Another argument against escape velocity is that this is not healthy for our community; it would tear it apart. We're feeling some of the strains today.

Another argument against escape velocity is that this might cut off significant resources from the Python community. The high gravity Python 2 area that we're trying to leave behind has all the money. Is this wise?

So should we wish for escape velocity in the first place? Is it desirable?

Escape Velocity cannot be reached?

It's also possible that the Python 2 gravity field is too strong, circular dependencies will keep holding us back, and that escape velocity cannot be reached. This means we would be in this Python 2 versus Python 3 situation forever. That would suck.

Above I listed some arguments against going for escape velocity in the first place. What if we take those seriously?

A Way Forward

How to go forward then? I think it makes sense to work as hard as possible to lift those Python 2 codebases out of the gravity well.

The moral authority argument encouraging people to port their code is not going to work down in the gravity well, though. There interest in The Python is least; people are interested in functionality in actual applications.

We have to make the way forward for those in the gravity well as easy and obvious as possible. We know those codebases often don't have great test coverage. Code is not going to become polyglot, let alone Python 3, in one fell swoop. We can do this with libraries, but it works less well for aplications.

The keyword here is porting incrementally. Port to polyglot carefully, on a per-module basis, step by step, and write new modules in polyglot from the start, until finally you can switch the whole thing over.

Python 2.7 helps. There are tools to help. Recently I ran into Future, which to me is the most attractive approach yet. But will those tools work for those down in the gravity well? Will they even find them? Only things easy and obvious will reach them.

This is why I argue for more official Python 2 releases, where these things are built in. This stuff needs to be as obvious as possible.

And to make it even more obvious, we need Python 2.x releases where the deprecated stuff is removed, incrementally. Breaking code is making it as obvious as you can get! But you don't want to break it all at once, because then people will be inclined to give up before they even start.

This is why I think a continued future for Python 2.x would be in the community's interest, even in The Python's interest. I created a #python2.8 IRC channel on freenode to gauge interest and discuss this further.

Comments

#python2.8 discussion channel on freenode    Posted: 2014-01-06 14:00


Fellow Pythoneers,

I've started an informal channel "#python2.8" on freenode IRC. It's to discuss the potential for a Python 2.8 version -- to see whether there is interest in it, what it could contain, how it could facilitate porting to Python 3, who would work on it, etc. If you are interested in constructive discussion about a Python 2.8, please join.

I realize that if there is actual code created, and if it's not under the umbrella of the PSF, it couldn't be called "Python 2.8" due to trademark reasons. But that's premature - let's have some discussions first to see whether anything can happen.

Hope to see you there for some discussion!

Comments

Alex Gaynor on Python 3    Posted: 2013-12-30 21:13


Introduction

I'm happy to see the Python community starting the Python 3 conversation again. Alex Gaynor shares his analysis.

I agree with many of his points and am glad he was brave enough to take up this difficult topic in public.

Update: I see Ian Bicking has weighed in and he says things I very much agree with. Thanks Ian! Everybody should read Ian Bicking:

https://plus.google.com/+IanBicking/posts/iEVXdcfXkz7

I told you so

I've been thinking about this for a while. I brought my worries up in 2007, before Python 3 was released:

http://blog.startifact.com/posts/older/brief-python-3000-thoughts.html

I'll quote myself:

While I understand the need to be able to break backwards compatibility, I am worried about Python forking into two parallel versions (2.x and 3.x) for many, many years. Such a fork can't be good for the Python community.

This sounds a lot like what Alex is talking about today. While it's something to be able to say "I told you so", overall it just makes me slightly depressed. I'm sad I don't get excited when a new release of Python is made, like I used to be. Sometimes I don't even notice. Why read about exciting new features I won't be using any time soon?

I got feedback and wrote followups:

http://blog.startifact.com/posts/older/python-3-worries-feedback.html

http://blog.startifact.com/posts/older/the-purpose-to-my-whinging-about-the-transition-to-python-3.html

http://blog.startifact.com/posts/older/communicating-with-core-developers-on-the-python-3-transition.html

The feedback burned me quite heavily. I decided I was not going to actively do anything about Python 3 and would just develop Python 2 code and let others worry about the problem.

I shut up about Python 3 in public for years, until briefly in 2011, in response to a blog post by Armin Ronacher:

http://blog.startifact.com/posts/older/python-2-8--1.html

In early 2012 the now sadly deceased Aaron Swartz also talked about this topic, and I responded, saying "I told you so" (not Aaron personally):

http://blog.startifact.com/posts/older/cassandra-and-how-pypy-could-help-with-python-3.html

The path taken

In 2007 I was expecting that what the core developers were suggesting would continue to be the way to go: people would run 2to3 to convert Python 2 code to Python 3. I was skeptical then about this plan. Instead I hoped for what Aaron described: an interpreter where people could mix pure Python 3 modules with pure Python 2 modules.

Instead of 2to3, library authors started to work out ways to write Python 2 code in such a way that it would also run in Python 3. This was considered an absolutely unrealistic approach in the beginning, but gradually became the mainstream approach with support in Python 2.x itself.

Now much library code has been ported. That is going fairly well. But when I see such code I think it's scary and ugly: writing code that needs to work in both interpreters requires language knowledge that is more like that of an advanced C++ developer. It isn't Pythonic, at all.

Open source library and framework creators have some motivation to port. They're maintaining their stuff for a long time, and want to attract users. They also have the interest and skill level to do so. All this together may outweigh the cost of maintaining more complicated code.

The bind we're in

It's different for applications. An application developer usually has motivations not to port. Porting will introduce bugs in working applications, there's unclear benefit as the application will remain the same, and less libraries are available. It's also harder to port an application than a library, as libraries tend to have better test coverage. Finally applications exist for a reason, many of them are in-house to some organisation, and how do you motivate a budget to do the porting activity?

But if application developers don't port, why all the effort to port libraries then? You have many people continue to maintain and develop Python 2 applications. Some people write new applications in Python 3. As Alex says, not good for the community.

Python 2.8 as way forward?

Alex suggests an incremental upgrade of Python 2 adding more Python 3 features. I think this could be made to work. It might require a Python 2.8 while leaving the door open for a Python 2.9. Incremental backwards compatibility breaking can be part of the process: you could for instance drop old-style classes and tell people to adjust their code when upgrading to 2.8. While that's a relatively scary change, but most code will continue to work.

Perhaps that with the experience people gained writing 2 + 3 code we can even make the unicode path smooth enough.

The advantage of this approach compared to a magic Python 2 + 3 interpreter is that it would be possible to do incrementally based on the Python 2 codebase, with the Python 3 codebase in sight. That may involve a less scary development effort.

The main advantage is that by gradually dropping compatibility applications may move forward again, and messy 2+3 codebases can slowly be cleaned up as support for older versions of Python 2.x is dropped. I'll repeat that: a path forward for developers to Python 3 that has your code become cleaner, not uglier.

The closed door

The Python core developers somewhat gleefully slammed the door shut on Python 2.8 back in 2011, though. Recently celebrating the 5 year release of Python 3.0 I heard people say everything was going according to plan.

One quibble with Alex's post: the suggestion he makes that Python 2 should've been dropped in favor of Python 3 when Python 3 was first released, so that people would have urgency to port code, strikes me as unrealistic. I think it could have resulted in a justified rebellion among users and a fork of the Python interpreter. I'm glad that path was not taken. In fact since 2011 this has been the path taken, and I'm not sure it made any difference.

I hope the core developers will change their mind and take up the momentous unpleasant task of moving Python 2.x forward towards Python 3.x again. But if not, perhaps a brave group of volunteers will stand up and fork Python 2, and take the incremental steps forward. This will have to remain just an idle suggestion, as I'm not volunteering myself.

Comments

Morepath Documentation Starting to Take Shape    Posted: 2013-12-19 23:04


For a while now I've been working on documenting Morepath, the Python web framework with super powers. There is now a significant dose of documentation available. Perhaps it'll be interesting holiday reading for someone! If not, no problem: I'll be back next year with more!

Hopefully this will help people get an idea of what Morepath is all about. I'm very excited about how it's been taking shape!

Let me know what you think!

Comments

Back to the Center    Posted: 2013-12-17 14:10


Grok the web framework was simple enough on its surface, but like an iceberg, beneath its simple surface lurked a vast codebase. This made it harder to understand Grok. We also were depending on a lot of code Grok was in fact not using. That unused code was still being loaded, distracting, and even at one point interacting with Grok to generate a security bug.

To simplify Grok we had to lose that code. This meant we couldn't go around the center of the Zope project anymore. We had to go back to the center.

Unused UI Code

When the new Zope was built, we thought we were going to create something like the old Zope, where management and even development of web applications could be done using a through-the-web UI. The new Zope therefore consisted of two parts: a selection of libraries that could be used as a web framework (or the plumbing of one, in the case of Grok), and a selection of libraries built on top of that which implemented the web UI and its content.

The old Zope and Grok had started to use the new Zope as a collection libraries, but had no need for the user interface bits.

Why did we have to include the UI code if we didn't want it? Because the Zope libraries were all tangled up. To shift the metaphor from icebergs, they were like a ball of wool after a cat's done playing with it: all tangled up.

The New Zope as Libraries

The new Zope had originally been developed as a collection of libraries all in the same repository, in a giant tree of packages. Dependencies between parts of the tree had been rather freely introduced.

Then, around 2006, we became early adopters of setuptools, and split up these libraries into a collection of independent libraries, and published them on PyPI. We learned a lot along the way on how to do proper releases and how to manage such collections of libraries.

But now we had a collection of supposedly independent libraries with a large amount of dependencies to each other. What was worse, we had circular dependencies. This meant that just about all libraries were linked to all other libraries. Grab one Zope library off PyPI, get all of them. All 80 or so...

As Chris McDonough pointed out later, what we should have done is extract the libraries one by one, and for each give a clear purpose and documentation.

But it was too late. And at least the problem was clear now.

The Cave Sprint

So the beginning of 2009 I organized a small sprint. We'd just moved house and now had the room for it. I'm now in two minds about this sprint: while we did manage to make a good first step in cleaning up dependencies, it was also the beginning of the end of my involvement in Zope. Plus cleaning up dependencies is not a very rewarding creatively, and I think sprints should inspire as well as get work done.

During this sprint we had a discussion about the Zope project itself. The original leader of the Zope project, Jim Fulton, had slowly become less involved with it, and there was a gap that needed to be filled if we wanted to drive Zope forward. It had become difficult to make decisions.

Christian Theune and Wolfgang Schnerring convinced me we should engage the Zope project more actively instead of working around it with Grok. I was convinced, as I knew that the problem of simplifying the code base needed engagement with the Zope project.

Steering Zope

So after the sprint some of us organized a Zope Steering Group, and set up a lightweight system for technical decision making so we could hopefully move Zope forward. I think it did help.

The Zope community reformulated the new Zope as a Zope Toolkit, recognizing its status as the foundation of projects like Zope 2 and Grok. We rebranded the new Zope away from the confusingly named Zope 3, which for years had implied that the new Zope was going to be a true successor to the old Zope, something that it never quite became but confused people. On the technical level at the end of the year we were able to disconnect the UI-related packages from the rest, and had left a directed acyclic graph of foundational packages, without circular dependencies.

My hope was also that new leadership could infuse new energy in the Zope project beyond just maintenance tasks like cleaning up dependencies. The web was evolving, and could we get together and do exciting new things again? The answer was no, but that will have to wait for the next article in this series.

This blog entry is a part of a series on Zope and my involvement with it. Previous. Next.

Comments

Morepath App Reuse    Posted: 2013-12-03 15:10


I'm working on a new Python web microframework called Morepath, as I've mentioned before. Here's the code and here's a draft quickstart. Morepath is a microframework with a difference: it's small and easy to learn like the others, but has special super powers under the hood.

One of those super powers is Reg, which along with Morepath's model/view separation makes it easy to write reusable views. But in this article I'll talk about another super power: Morepath's application reuse patterns.

We'll talk about how Morepath lets you isolate applications, extend and override applications, and compose applications together. Morepath tries to make these possibilities simple, even obvious. Morepath strives to make the possible as obvious as possible.

Application Isolation

Morepath lets you create app objects like this:

app = morepath.App()

These app objects are WSGI applications, but also serve as registries for application configuration information. This configuration is specify used decorators. Typically apps consist of models and views:

@app.model(model=User, path='user/{username}',
           variables=lambda user: { 'username': user.username })
def get_user(username):
    return query_for_user(username)

@app.view(model=User)
def render_user(request, model):
    return "User: %s" % model.username

Here we've exposed the User model class under the path /user/{username}. When you go to such a URL, the default (unnamed) view will be found, and we've provided that too: it just renders "User: {username}".

What now if we have another app where we want to publish User in a different way? No problem, we can just create one:

other_app = morepath.App()
@other_app.model(model=User, path='different_path/{username}')
def get_user(username):
    return different_query_for_user(username)

@other_app.view(model=User)
def render_user(request, model):
    return "Differently Displayed User: %s" % model.username

Here we expose User to the web again, but use a different path and a different view. If you run this app (even in the same runtime), this will be separate.

This app isolation is nothing really special; it's kind of obvious that this is possible. But that's what we wanted. Let's look at a few more involved possibilities next.

Application Extension

Let's look at our first application app again. It exposes a single view for users (the default view). What now if we want to add a new functionality to this application so that we can edit users as well?

This is simple; we can add a new edit view to app:

@app.view(model=User, name='edit')
def edit_user(request, model):
    return 'Edit user: %s' % model.username

The string we return here is of course useless for a real edit view, but you get the idea.

But what if we have a scenario where there is a core application and we want to extend it without modifying it?

Why would this ever happen, you may ask? Well, it can, especially in more complex applications and reuse scenarios. Often you have a common application core and you want to be able to plug into it. Meanwhile, you want that core application to still function as before when used (or tested!) by itself. Perhaps there's somebody else who has created another extension of it.

This architectural principle is called the Open/Closed Principle in software engineering, and Morepath makes it really easy to follow it. What you do is create another app that extends the original:

extended_app = morepath.App(extends=[app])

And then we can add the view to the extended app:

@extended_app.view(model=User, name='edit')
def edit_user(request, model):
    return 'Edit user: %s' % model.username

Now when we publish extended_app using WSGI, the new edit view will be there, but when we publish app it won't be.

Kind of obvious, perhaps. Good. Let's move on.

Application Overrides

Now we get to a more exciting example: overriding applications. What if instead of adding an extension to a core application you want to override part of it? For instance, what if we want to change the default view for User?

Here's how we would do that:

@extended_app.view(model=User)
def render_user_differently(request, model):
    return 'Different view for user: %s' % model.username

We've now overridden the default view for User to a new view that renders it differently.

You can also do this for what is returned for model paths. We might for instance want to return a different user object altogether in our overriding app:

@extended_app.model(model=OtherUser, path='user/{username}')
def get_user_differently(username):
    return OtherUser(username)

To make OtherUser actually be published on the web under /user/{username} it either needs to be a subclass of User, for which we've already registered a default view, or we need to register a new default view for OtherUser.

Overriding apps actually doesn't look much different from how you build apps in the first place. Hopefully not so obvious that it's boring. Let's talk about something new.

Nesting Applications

Let's talk about application composition: nesting one app in another.

Imagine our user app allows users to have wikis associated with them. You would have paths like /user/faassen/wiki and /user/bob/wiki.

One approach might be to implement a wiki application within the user application we already have, along these lines:

@app.model(model=Wiki, path='user/{username}/wiki')
def get_wiki(username):
    return wiki_for_user(username)

@app.view(model=Wiki)
def wiki_default_view(request, model):
    return "Default view for wiki"

(this is massively simplified of course. we'd also have a Page model that's exposed on a sub-path under the wiki, with its own views, etc)

But this feels bad. Why?

  • Why would we implement a wiki as part of our user app? Our wiki application should really be an app by itself, that we can use byitself and also test by itself.
  • There's the issue of the username: it will appear in all paths that go to wiki-related models (the wiki itself, any wiki pages). But why should we have to care about the username of a user when we are thinking about wikis?
  • It would also be nice if we can use the wiki app in other contexts as well, instead of only letting it be associated with users. What about associating a wiki app with a project instead, like you can do in github?

A separate app for wikis seems obvious. So let's do it. Here's the wiki app by itself:

wiki_app = morepath.App()

@wiki_app.model(model=Wiki, path='{wiki_id}')
def get_wiki(wiki_id):
    return query_wiki(wiki_id)

@wiki_app.view(model=Wiki)
def wiki_default_view(request, model):
    return "Default view for wiki"

This is an app that exposes wikis on URLs using wiki_id, like /my_wiki, /another_wiki.

But that won't work if we want to associate wikis with users. What if we want the paths we had before, like /user/faassen/wiki?

Morepath has a solution. We can mount the wiki app in the user app, like this:

@app.mount(app=wiki_app, path='user/{username}/wiki')
def mount_wiki(username):
    return {
       'wiki_id': get_wiki_id_for_username(username)
    }

We do need to adjust the wiki app a bit as right now it expects wiki_id to be in its paths, and the wiki id won't show up when mounted. This is a simple adjustment: we need to register the model so that its path is empty:

@wiki_app.model(model=Wiki, path='')
def get_wiki(wiki_id):
    return query_wiki(wiki_id)

But where does wiki_id come from now if not from the path? We already have it: it was determined when the app was mounted, and comes from the dictionary that we return from mount_wiki().

What if we want to use wiki_app by itself, as a WSGI app? That can be useful, also for testing purposes. It needs this wiki_id parameter now. We can construct this WSGI app from wiki_app by giving it a context explicitly:

wiki_app.context(wiki_id=5)

Application Reuse

Many web frameworks have mechanisms for overriding specific behavior and to support reusable applications. These tend to have been developed in an ad-hoc fashion as new needs arose.

Morepath instead has a general mechanism for supporting app extension and reuse. You use the same principles and APIs you already use to create new applications. Any normal Morepath app can without extra effort be reused. Anything registered in a Morepath app can be overridden. This is because Morepath builds on a powerful general configuration system.

This is because Morepath, like Grok or Pyramid, comes from the rich Zope heritage, where we've thought about this stuff. And Morepath wraps all that power in a small, easy, reusable little framework.

Let me know what you think!

Comments

Implementing Grok    Posted: 2013-12-02 13:20


Towards the First Grok Sprint

I got the idea for Grok in the summer of 2006. In the fall I still hadn't done anything yet about it. Then I gave a talk at a German Zope User Group (DZUG) conference (the video of that now appears to have sadly disappeared off the internet; if it's still somewhere I'd appreciate to get the link!). This was the series of conferences that a few years ago got broadened into the wider PyCon.DE (German PyCON) conferences. I chatted about Grok with Christian Theune of Gocept. He suggested we hold a sprint to build Grok at the Gocept offices.

So later on that fall I and my wife Felicia flew to Berlin and from there were taken by Christian to the Gocept offices in Halle. There were four of us at the sprint: myself, Christian, Philipp von Weitershausen, and Wolfgang Schnerring. The first three were very familiar with Zope both old and new. Wolfgang was new to it all, but had experience with Ruby on Rails, so could offer us an interesting different perspective.

Quadruple Programming

This sprint, we didn't do the pair programming that was pretty standard in Zope sprints, but quadruple programming. How did that work?

We used a video projector to project the screen of somebody's laptop onto the wall. The person at the keyboard was mostly either Philipp or Wolfgang. The rest would look at the projection and give directions. Christian made sure we didn't forget test cases by writing them down on a whiteboard. I sat or paced around and just talked. People who know me know I do that a lot anyway. I actually did not edit a single line of code during the entire weeklong sprint myself, but I'd seen all of it and was thoroughly familiar with it when I went home afterwards.

After a week of sprinting we got Grok from a design document to a working implementation. We built the traditional wiki application with it to try it all out in the end.

Starting Grok in a sprint like this made it a community project straight from the start. This was beneficial when we went home and had to cooperate online.

I wrote a report about the sprint and Grok to my blog back then; it may be interesting reading for some.

Grok the Caveman

None of the other sprinters liked my proposed name 'Grok' for the project. It was too nerdy, too obscure. Perhaps, it was proposed, we should name it something like "Easy Zope" (I forget the exact name proposed, but 'easy' was part of it). I protested and formulated what Philipp later dubbed "Faassen's law":

Don't use the name 'easy' or 'simple' in software, as it likely won't be and people will make fun of it.

Somehow during dinner we were discussing and imitating "Hulk talk", i.e. "Hulk smash". People are silly like that, especially people like me. I suddenly hit on the idea that Grok was actually a caveman, and that he talked like this: "me Grok smash ZCML". Everybody immediately liked this idea, as it had a sense of fun and fit the theme of Grok simplifying the new Zope.

We had the luck that my wife Felicia was there. She's a graphics designer, so we asked her Grok the Caveman, so we would have a mascot for the project. She made several drawings and showed them to the group. One of them was everybody's favorite. "But that one looks like me!" I protested. "Exactly," they said, grinning. And thus Grok the caveman was born.

Having a friendly caveman as a mascot/logo for your project actually helped attract people to it. Having such a character gives a project a face and gives people something to relate to. The web site theme was inspired by it, and even a stand-up wooden version of the caveman was made and used at various open source conventions.

I've been toying for a while with the idea of creating an elf character for the Obviel project...

While I did mention cave women in my original blog post on Grok, I wonder now whether our slogan "now even cavemen can use Zope 3" was sufficiently inclusive of women, given all the attention given to gendered language now... The word "cavepersons" would take away a bit of the charm of the ridiculously ahistorical but familiar caveman concept.

Grok Innovation

After I'd gone home from the first Grok sprint I could immediatetely pick up the code and continue working with it. I factored out a library, which contained what I think now was the most important innovation of Grok. This was the notion of scanning Python modules for things (classes, instances, decorators) that needed to be registered with the system. This, we discovered, allowed us to do metaclass-like things with Grok without the need for using actual metaclasses -- particularly picking up and registering relevant classes with the Zope configuration system. Avoiding metaclasses is good if you can get away with it, as they can lead to other unexpected behavior in subclasses. I called the library Martian, to fit the 'grok' theme.

When Chris McDonough started what later became the Pyramid project he used Martian to pick up configuration. Later on he reimagined Martian and he created the Venusian library, which I'm using today in Morepath. But that's another story.

Grok as an Open Source project

Grok as an open source project was moderately successful. A little community of people formed. It managed to attract a few people unfamiliar with Zope technology, and also made the new Zope technology more appealing to people who were already familiar with Zope. We got quite a few contributions, too.

We had regular Grok sprints that helped push Grok forward. We also participated in the Google summer of code for a few years, under the wider banner of the Zope project.

I was lucky enough that quickly after the initial creation of Grok I could start using Grok in my professional life on customer projects. I've been using it as the underpinnings in several large web applications since then.

Various contributors also refactored Grok technology so it could be plugged into the old Zope. From there the Plone community started using it; it's still widely used there today. This whole old Zope compatibility project was entirely pushed by contributors who weren't part of the original group -- a sign of open source success.

Grok and Zope

Grok was built on the new Zope. This gave Grok a running start and a built-in community to draw from, and this was definitely part of Grok's success. But many other Zope developers were not interested in Grok, and quite satisfied with the original ways of doing things. Grok could not have worked as a project to improve Zope itself -- there was insufficient consensus. To create something new, we had to go around the center.

Grok was successful at least for me personally: it made it possible for me to use the new Zope technology without much of the pain. But was Grok successful in being attractive to beginners? We certainly attracted some. But there were problems.

While Grok was simple enough on the surface, its learning curve was not as smooth as we'd have liked. Grok was too much like an iceberg; beneath the surface lurked the vast codebase of the new Zope. If we wanted to make Grok smaller and more comprehensible all the way down, we would need to fix this.

This blog entry is a part of a series on Zope and my involvement with it. Previous. Next.

Comments

Grok: the Idea    Posted: 2013-11-28 14:00


It's been a month since I posted an entry in this ongoing series, but I got positive feedback and the story is not over yet, so here's a new entry! I hope you enjoy it!

Lost at CERN

In the summer of 2006 I went to the EuroPython conference, which that year was held in CERN, Switzerland. There was to be a Zope sprint ahead of the conference, so I came a few days earlier.

I got there in the evening. The next morning I tried to look for likely Python programmers. Normally this is not a difficult task: programming geeks tend to have a special look about them. But at CERN this doesn't work: many people at CERN look like they could be a programmer, and many of them of course are, but they were not there for EuroPython.

I got slightly lost in the complex, and was given directions by some gentlemen. They could've been janitors or Nobel prize winners, I will never know. CERN is a special place.

I finally concluded I had planned things a bit wrong, and had arrived at CERN one day earlier than everybody else. So I had a day to kill.

I remember the weather was gloriously sunny and warm. I was hanging out in the cafetaria for a while. I started thinking about my experiences with the new Zope, and its frustrations, and what it would take to turn it into a web framework without these frustrations. A web framework that I would like to use myself and would hopefully also be a bit less intimidating to beginners. Zope was starting to have a problem attracting new people -- the old Zope was out of date, and the new Zope was too intimidating.

The Idea of Grok

I decided to call this new framework Grok, as one has to give things a name. Grok is a term originally from Stranger in a Strange Land, a science fiction novel by Robert Heinlein, and is supposed to be a martian verb for "to understand/to become one with/to drink in". From there the word had made it into programming jargon.

Ruby on Rails had made popular the idea of "convention over configuration"; if you just use some conventions (naming, placing in modules, etc) for the entities in your system, the framework will know what they are without you having to explicitly and verbosely having to tell it. Since the new Zope had a problem with verbose configuration, I figured we could apply convention over configuration to make things more succinct.

In Grok you can use base classes (grok.View, grok.Model, etc) to signal what type of class you're definining, and the system will deduce from that how to register them with the Zope configuration system. Just base classes by themselves aren't enough though -- many configuration actions need parameters. We solved this with Grok directives.

For an example of a directive, You could associate a view with a model using the grok.context() directive. Grok directives are special calls you could make inside your class definition, like this:

class MyView(grok.View):
    grok.context(MyModel)

grok.context would magically apply to the class it is called in, in this case MyView, and set some information on it. These days we'd use a class decorator for this instead, but Python didn't have those yet then.

So far not much convention over configuration was going on. What I had done was found a way to integrate configuration information directly into Python classes so that you wouldn't have to maintain separate XML files anymore for configuration. Convention over configuration came in because you could leave out grok.context and it would then automatically associate the view with a model that happened to be defined in the same module.

It turned out later that convention over configuration in Grok was not as important as I had thought it was then. Some of the magic, even created very deliberately and layered over a worked-out configuration system, intimidated people and sometimes (though rarely) lead to unexpected behavior. The reduction of verbosity by doing configuration in Python near the code being configured was the main win; reducing that configuration further was less important.

When the other sprinters arrived at CERN, I explained some of these ideas to them and received some useful feedback. But it all remained just an idea.

After the Python conference, I was lucky enough to get to see the first web server (the web having been invented at CERN), and the Large Hadron Collider, which, they told us, was going to collide hadrons Real Soon Now.

This blog entry is a part of a series on Zope and my involvement with it. Previous. Next.

Comments

Contents © 2005-2014 Martijn Faassen | Twitter | Github | Gittip