mstform: a form library for mobx-state-tree

Introduction

So I've written a new web form library: mstform.

The first form library I ever wrote was called ZFormulator and I released it in 1999. My first frontend form library was Obviel Forms and it I wrote it in 2012 or so. I've been at it for a while.

Much has changed in web development since 1999, and I've learned a thing or two along the way as well. So a little while ago, almost against my will (though I admit I also enjoy it a lot), I helped to create another form library, one for the frontend again. It integrates with mobx-state-tree. You'd probably use React to render the form, though you don't have to.

A whirlwind introduction to mobx-state-tree

So what, you may ask, is mobx-state-tree, and why should I want to use it in my React application? mobx-state-tree is a state management library - it manages the state you show in your frontend UI. It's built on top of another React state management library named MobX. MobX is magic. You define your model classes, and you mark a few properties and methods so that MobX can observe them. You also mark your React components as observers. It's easy. After that when you change your model instances your React UI will automatically and efficiently update itself.

As I said, mobx-state-tree (MST to make it less of a mouthful) is built on MobX. The way you mark your React components as observers is identical. What it then adds is the following: it forces you to define your models in a very particular way and then it gives you a ton of features in return.

Here's a tiny example:

const Animal = types
  .model("Animal", {
    name: types.string,
    hunger: types.integer
  })
  .views(self => ({
    get isSated() {
      return self.hunger === 0;
    }
  }))
  .actions(self => ({
    feed(amount) {
      self.hunger -= amount;
      if (self.hunger < 0) {
        self.hunger = 0;
      }
    }
  }));

Here we have a model Animal. It has two properties, name and hunger. There's also a special view property we have defined with a getter, isSated. We've also defined an action which manipulates the properties.

MST then gives you some interesting features:

  • You get run-time checks if you put in the wrong type of value. With TypeScript you also get compile-time checks.

  • You can automatically serialize this to JSON and deserialize it from JSON.

  • You can install hooks to monitor any changes to properties.

  • There is a mechanism for defining references between models, and the JSON serializer works with it.

mstform builds on MST to help you create web forms.

Rendering a web form: you're on your own

In the past, for a library like Formulator or Grok or Obviel Forms, I made sure it could render your form. That's nice to have when you have simple forms - you just write a form description or perhaps a schema, and boom, the system automatically shows a form. Django's form system works like that too.

The problem with automatically rendering a form is that any complex form will have special UI. Perhaps you want to display a piece of text between two widgets. Perhaps you want to render repeating sub-forms in a certain way. Perhaps you want to layout the form so that it responds to the width of the browser window. I don't know. If you automatically render a form and that's the default way of using the form library, the form library has to supply concepts and hooks so that you can influence the form rendering process. These hooks are typically difficult to use for the developer and are incomplete, as it's difficult to anticipate everything.

With mstform, I let all of that go. It doesn't try to render forms. It isn't in control of rendering your form at all. It would be fairly straightforward to build something on top of mstform that did this for you, but it's not a priority. You have React. You have your own form components or use some UI library that already provides them. mstform instead makes it easy to integrate with these components.

What does mstform do then?

If mstform doesn't render your form, what does it do then? It manages the form contents - the form state. In fact, an earlier name for this library was FormState, until I discovered there already was another library out there with that name.

mstform in its essence lets you define a form that represents a MST model. Here is an example for the Animal model we defined above:

import { Form, Field, converters } from "mstform";

const form = new Form(Animal, {
    name: new Field(converters.string),
    hunger: new Field(converters.integer)
});

You can then use this form to manage the state of instances of that model:

// use MST to create an instance of an animal
const elephant = Animal.create({name: "Elephant", hunger: 3});

// now make a form state for elephant
const formState = form.state(elephant);

You can access fields from the form state:

const nameField = formState.field('name');

Once you have a field, you can render it with React. Here is how you could render an input text:

import { Component }, React from "react";
import { observer } from "mobx-react";

@observer
class Input extends Component {
    render() {
        const { type, field } = this.props;
        return <input type={type} {...field.inputProps} />;
    }
}

You can do a lot of other things with the field accessor too: get its error message with field.error, check whether the field is required with field.required, check whether the field is empty with field.isEmpty and so on.

What does that give you?

Why should I use MST with mstform instead of rolling my own, you may ask?

Here we come to features of mstform:

Convert form input

The form input is often just a bunch of texts, but you have a data model underneath. For instance, if I enter an integer in a text input it is a string. mstform converts this into an integer.

Show edit forms

Say you already already have form content, because you just loaded it from the backend and deserialized it for instance, you need to convert that content back again to its display state. For instance, render an integer as a string so it can be displayed in a text input.

Client-side validation

Even after the conversion was successful, you want to validate that the value fulfills some criteria, such as being within a certain range. You still need to do validation on the backend to be sure, but this way you can show errors right away.

Error handling

An easy way to show errors, both in conversion and validation. We can also show backend-generated errors, to which we will get in a bit.

Repeated forms and sub forms

You can express nested and repeated structures in your models easily in the form.

Backend integration

You can define how your form is saved to the backend. You can only submit a form if it's valid. The backend can return validation errors and warnings which can be displayed in the form.

Server-side validation

You can also set it up so that your backend generates validation errors during user interaction. This way the backend can remain in charge of validation as the single source of truth, while you still dynamically display errors right away.

Form access states

Make your fields required, disabled, hidden and read-only. Disable individual fields or a whole sub-form at once. You have to write rendering code that can handle these states as mstform doesn't do rendering, but once you have it your forms become very flexible indeed.

Modify the underlying object

You can modify the underlying MST model instance with your own code and the form automatically updates itself. Modify a field value, add a new repeated item, the works. It's just that simple.

Derived values between fields

A field can have a default value that is based on the values filled in with other fields.

Support for different kinds of React input components

Some components get the value with onChange, others get the value as event.target.value. Some components display their value with value, others with checked. mstform converters have defaults, but you can override them to suit your needs.

Support for internationalized decimal input

Decimal numbers tend to have quite a few differences between countries. For instance, in the US you use periods for the fraction, and commas for the thousands, but in the Netherlands it's the other way around. mstform has a special decimal parser built-in which takes care of that.

TypeScript support

mstform is written in TypeScript and exports TypeScript type definitions for your development convenience.

For more, consult the mstform documentation.

Conclusion

Setting up a form with mstform takes a bit of effort, though it's not difficult. Once you do, a lot of code you might write as custom and failure-prone React components goes away, while you retain a lot of flexibility. Your users edit a form, and a proper MST instance sits under it. You can immediately sends its JSON to the backend. You can load up new form contents from the backend just as easily.

Is this the last form library I will ever write? Experience teaches me not to make such promises. But mstform does pack a lot of my experience with web forms in it. I hope you'll give it a try!

Credits

mstform was created by myself and team of people at a customer of mine named ISProjects. I am very grateful to ISProjects for their great support in this and quite a few other interesting projects I got to work on for them. And they're looking for developers!

Seven Years: A Very Personal History of the Web

Introduction

Humans are storytellers. As anyone who knows me can confirm, I definitely enjoy the activity of telling stories. We process and communicate by telling stories. So let me tell you all a story about my life as a web developer the last 7 years. It may just be self-reflection, but hopefully it's also useful for some others. Perhaps we can see my story as a highly personal history of the web.

I always say that what pulls me to software development most is creativity. I am creative; I can't help myself. I enjoy thinking about creativity. So this is also going to be about my creative trajectory over the last 7 years.

Why 7 years? Because in early 2010 I decided to withdraw myself from a software development community I had been involved in for about 12 years previously, and I took new paths after that. It is now early 2017; time to take stock.

Letting go of an intense involvement with a software development project, up to the point where it became part of my identity, was difficult. Perhaps that's a geeky thing to say, but so it is. I needed to process it.

Life after Zope

Zope was a web framework before that concept existed. The modern web framework only materialized sometime in the year 2005, but Zope had been ahead of its time. I was involved with the Zope community from when it was first released, in 1998. I learned a lot from Zope and its community. I made many valued connections with people that last until this day.

Zope helped shape who I am and how I think, especially as a web developer. In 2013 I wrote a retrospective that went into the history of Zope and my involvement with it.

But I did not just process it by writing blog posts. I also processed it creatively.

Frameworks

So I am a web developer. Back in 2010 I saw some people argue that the time of the web framework had passed. Instead, developers should just gather together a collection of building blocks and hook it up to a server using a standard API (WSGI in the case of Python). This would provide more flexibility than a framework ever could.

In a "X considered Y" style post I argued that web frameworks should be considered useful. Not that many people needed convincing, but hey, why not?

I wrote:

The burden of assembling and integrating best of breed components can be shared: that's what the developers of a framework do. And if you base your work on a pre-assembled framework, it's likely to be less work for you to upgrade, because the framework developers will have taken care that the components in the framework work together in newer versions. There is also a larger chance that people will write extensions and documentation for this same assembly, and that is very likely something you may benefit from in the future.

I follow the ReactJS community. The React community definitely is a community that lets you self-assemble a framework out of many parts. This gives that ecosystem flexibility and encourages creativity -- new approaches can be tried and adopted quickly. I like it a lot.

But figuring out how to actually start a React-based project had become a major effort. To get a good development platform, you needed to learn not only about React but also about a host of packaging and build tools: CommonJS and Webpack and npm and Babel and so on. That's quite intimidating and plain work.

So some React developers realized this and created create-react-app which makes it easy to start a working example, with minimal boilerplate code, and with suggestions on how to expand from there. It's a build framework for React that gathers good software in one place and makes it easy to use. It demonstrates how frameworks can make life easier for developers. It even goes a step further and allows you to opt out of the framework once you need more control. Now that's an interesting idea!

Client-side JS as Servant to the Server UI

So frameworks are useful. And in late 2010 I had an idea for a new one. But before I go into it, I will go on a digression on the role of client-side JavaScript on the web.

This is how almost all JavaScript development used to be done and how it's still done today in many cases: the server framework generates the HTML for the UI, and handles all the details of UI interaction in request/response cycles. But sometimes this is not be enough. More dynamic behavior on the client-side is needed. You then write a little bit of JavaScript to do it, but only when absolutely necessary.

This paradigm makes JavaScript the ugly stepsister of whatever server server programming language is used; a minor servant of Python, Ruby, Java, PHP or whatever. The framework is on the server. JavaScript is this annoying thing you have to use; a limited, broken language. As a web developer you spend as little as time possible writing it.

In short, in this paradigm JavaScript is the servant to the server, which is in charge of the UI and does the HTML generation.

But JavaScript had been gathering strength. The notion of HTTP-based APIs had attracted wide attention through REST. The term AJAX had been coined by 2005. Browsers had become a lot more capable. To exploit this more and more JavaScript needed to be written.

jQuery was first released in 2006. jQuery provided better APIs over sucky browser APIs, and hid incompatibilities between them. Its core concept is the selector: you select things in the web page so you can implement your dynamic behavior. Selectors fit the server-dominant paradigm very well.

Client-side JS as Master of the UI

By 2010, the notion of single-page web application (SPA) was in the air. SPAs promised more powerful and responsive UIs than server-side development could accomplish. The backend is a HTTP API.

This is a paradigm shift: the server framework lets go of its control of the UI entirely, and client-side JavaScript becomes its new master. It encourages a strong separation between UI on the client and business logic on the server. This brings a big benefit: the unit of UI reuse is on the client, not spread between client and server. This makes reuse of UI components a lot easier.

By 2010, I had played with client-side template languages already. I was about to build a few large web applications, and I wanted them to be dynamic and single page. But client-side JavaScript could easily become a mess. I wanted something that would help organize client-side code better. I wanted a higher-level framework.

The idea

So there we finally get to my idea: to create a client side web framework, by bringing over concepts from server frameworks to the client, and to see what happens to them. Cool things happen! We started with templates and then moved to MVC. We created a notion of components you could compose together. We created a client-side form library based on that. In 2011 we released this as Obviel.

For a little while in 2010, early 2011, I thought I was the only one with this cool idea of a client-side web framework. It turns out that I was not: it was a good idea, and so many people had the same idea at about the same time. Even before we released Obviel, I started to hear about Backbone. Ember and Angular soon followed.

I continued working on Obviel for some time. I created a template language with a built-in i18n system for it, and a client-side router. Almost nobody seemed to care.

In 2011 and 2012 we built a lot of stuff with Obviel. In the beginning of 2013 those projects were done. Obviel didn't get any traction in the wider community. It was a project I learned a lot from, so I don't regret it. I can claim deep experience in the area of client-side development.

React

I went to my first JS conference in September of 2013. I had originally submitted a talk about Obviel to it, but it wasn't accepted. Everybody was promoting their shiny new client-side framework by that time.

So was Facebook. Pete Hunt gave a talk about React. This was in fact only the second time React had been introduced to a wider audience. Apparently it went over a lot better than the first time. It certainly made an impression on me: there were some fascinating new ideas in it. The React community became ferment with fascinating new ideas. At the conference I talked to people about another idea I'd had: a client framework that helps coordinate client/server communication; maybe sort of like a database, with transactions that commit UI state to the server? Nobody seemed to know of any at the time. Uh oh. If nobody has the idea at the same time, then it might be a bad one?

Then from the React community came Flux and Redux and Relay and Mobx. I let go of Obviel and started to use React. There is a little irony there: my move to client-side frameworks had started with templates, but React actually let go of them.

The Server in Modern Client-side times

In early 2013 I read an interesting blog post which prompted me to write Modern Client-Side Times, in which I considered the changed role of the server web framework if it was to be the servant to JavaScript instead of its master.

I wrote a list of what tasks remain for the server framework:

What remains is still significant, however:

  • serving up JSON on URLs with hyperlinks to other URLs

  • processing POSTs of JSON content (this may include parts of form validation)

  • traversal or routing to content

  • integrating with a database (SQLAlchemy in this case)

  • authentication - who are you?

  • authorization - who can access this URL?

  • serve up an initial web page pulling in a whole bunch of static resources (JS, CSS)

I also wrote:

Much of what was useful to a server side web framework is still useful. The main thing that changes is that what goes over the wire from server to client isn't rendered HTML anymore. This is a major change that affects everything, but much does stay the same nonetheless.

I didn't know at the time of writing that I would be working on just such a server web framework very soon.

On the Morepath

In 2013 I put some smaller pieces I had been playing with for a while together and created Morepath, a server Python web framework. I gave an over-long keynote at PyCON DE that year to describe the creative processes that had gone behind it. I gave a more focused talk at EuroPython 2014 that I think works better as an introduction.

I announced Morepath on my blog:

For a while now I've been working on Morepath. I thought I'd say a bit about it here.

Morepath is a Python web micro-framework with super powers. It looks much like your average Python micro-framework, but it packs some seriously power beneath the hood.

One of the surprises of Morepath was the discovery that a web framework that tries to be good at being a REST web server actually works very well as a server web framework as well. That does make sense in retrospect: Morepath is good at letting you build REST services, therefore it needs to be good at HTTP, and any HTTP application benefits from that, no matter whether they render their UI on the client or the server. Still, it was only in early 2015 that Morepath gained official support for server templates.

2014 was full with Morepath development. I announced it at EuroPython. It slowed down a little in 2015, then picked up speed again in 2016.

I'm proud that Morepath is micro in implementation, small in its learning surface, but macro in power. The size of Morepath is another surprise: Morepath itself is currently a little over 2000 lines of Python code, but it does a lot, helped by the powerful Reg (<400 lines) and Dectate (<750 lines) libraries. Morepath offers composable, overridable, extensible applications, an extensible configuration system, an extensible view dispatch system, automated link generation, built-in powerful permission rule system, and lots more. Morepath is like Flask, but with a nuclear fusion generator inside. Seriously. Take a look.

The Future?

Over the last few years Morepath has become a true open source project; we have a small team of core contributors now. And in late 2016 Morepath started to gain a bit more attention in the wider world. I hope that continues. Users that turn into contributors are invaluable for an open source project.

There was a mention of Morepath in an Infoworld article, I was interviewed about it for Podcast__init__, and was also interviewed about it for an upcoming episode of Talk Python to Me.

Ironically I've been writing some Django code lately. I'm new to Django (sort of). I have been reintroduced to the paradigm I started to leave behind 7 years ago. With standard Django, the server rules and JavaScript is this adjunct that you use when you have to. The paradigm works, and for some projects it may be the best approach, but it's definitely not my preferred way to work anymore. But I get to help with architecture and teach a bit so I'll happily take the Django on board.

The Django management UI is cool. It makes me want to implement the equivalent for Morepath with PonyORM and React and Mobx. Or something. Want to help?

I've been itching to do something significant on the client-side again. It's been a little while since I got to do React. I enjoyed attending React Europe 2015 and React Europe 2016. I played with React Native for a bit last year. I want to work with that stuff again.

The space where client and server interacts is fertile with creative potential. That's what I've found with Obviel and React on the client, and with Morepath on the server side. While GraphQL replaces the REST paradigm that Morepath is based around (oops!), I'd enjoy working with it too.

Where might the web be going? I like to think that by being creative I sometimes get to briefly peek into its near future. I hope I can continue to be creative for the next 7 years, as I really enjoy it.

I'm a freelancer, so the clients I work for in part shape my creative future. Hint. Let me know if you have something interesting for me to work on.

Looking for new challenges

Fair warning: In this blog post I aim to sell myself. I'm looking for an exciting and challenging new freelance engagement.

I'm a software developer and I have been one professionally for about 20 years now. I have deep experience, and I continue to learn and create. I know what real-world codebases look like, and I know software development is also about people. I think I can offer lot of value. I can develop software for you, and I can also help you improve the way you develop software.

I see myself as a creative developer -- I want to invent things to improve life for myself and others. Creativity is what attracts me to software development the most. Creativity is transformative. If you need a bit of transformation in your software, talk to me.

I'm a web application developer. I've focused on web development since the late 90s. In that period I've seen the web grow from static websites and with a few server-side Perl scripts thrown in to the dynamic application platform it is today. I started developing web applications with the server-side, the only game in town then, but over the last 10 years I've shifted more and more to the client-side and JavaScript, where much of the creativity is today. I am however still very much at home on the server as well. I've done React, I've done REST, I've done hypermedia APIs, I've dug into GraphQL. If you need a web developer with deep insight in the whole stack, look no further.

I've focused on Python for the better part of my software development career. I came to Python early; I have seen Python grow from a small language with no name recognition to the enormously popular language it is today. I greatly enjoy using it. I've also criticized it where I felt it was needed, painful as it was. If you need a very experienced Python developer, contact me.

But in the last decade I've started to use JavaScript more and more. Thanks to the React community I've learned a few new things about functional programming; it's exciting to see it become relevant to the web. I haven't stopped looking at interesting new languages. I'm a developer who can look beyond a single programming language.

I have repeatedly demonstrated I can take a large piece of software and transform it:

  • Back in the day, I rejuvenated the Zope web framework with Five, a technology that is still in use by the Plone CMS today.

  • I took libxml2, a huge C library that was difficult to use for Python developers, and created the most powerful XML library in the Python world: lxml. I used the predecessor to Cython to do this (in 2004!).

  • I created a simpler, better way to use Zope with Grok. I've helped add web capabilities to existing codebases with a deep history.

I can help you open up your codebase to new possibilities.

I can build on other people's foundations, but I can also build new foundations. In the last few years I've created Morepath, a web framework that compresses the power I expect into a few thousand lines of Python code.

I'm not afraid to say I've also created many a thing that went nowhere. In 2010 I created Obviel, a client-side web framework. I took concepts like model/view separation and templates and a form library and brought them to JavaScript and the client. I thought I was doing something new in 2010, something that people hadn't really thought of yet. It turns out everybody had the same idea about the same time. Backbone burst upon the scene and Obviel never got any traction. Now I'm a frontend web framework hipster; I was doing them before it was cool. It was worth it, because I gained deep insights in what makes a frontend framework tick.

These days I prefer to use React for front-end application development. React is awesome. With React Native a mere web developer such as myself can even build a real mobile app. Want a developer that loves React but is also tempered by experience? Look no further.

I'm looking for a new challenge. I want to help build, create and transform. I work from home (in the Netherlands) so I can enjoy my family and garden, but I can certainly come visit you on a regular basis. I write code, I write docs, I write tests, I give talks, I give training, and I review your code. I am open, constructively critical and honest. I contribute a bit of insight here and there. My services are not cheap, but they are worth it.

In summary, here I am: a very experienced, creative web developer who looks a little bit beyond what's in front of him.

Think you have an interesting challenge for me? Please drop me a mail at faassen@startifact.com and let's talk.

Morepath 0.16 released!

I'm proud to announce the release of Morepath 0.16. Morepath_ is a Python web framework that is easy to use and lightweight but grows with you when your project demands more.

Morepath 0.16 is one of the biggest releases of Morepath in a while. I want to discuss a few of the highlights of this release here.

Reg Rewritten

Morepath uses the predicate dispatch library Reg for its view lookup system and other behavior. We've rewritten Reg once again. For most Morepath users nothing changes, except that Reg is faster which also makes Morepath faster. If you want to use Reg directly, the new registration API makes it easier to use.

With Reg you can control the context in which dispatch takes place: this allows multiple separate configurations of dispatch in the same runtime. To control context, previously we used an implicit global lookup object, or an explicit but not very Pythonic lookup argument. Those are all gone. If you need multiple dispatch contexts in an application, you can define dispatch methods which derive their context from their class. This change allowed us to simplify Reg considerably and increase its performance.

This work was done by Stefano Taschini in collaboration with myself. Thanks Stefano!

New pip-based build system

This only affects us Morepath developers, but it's a significant change, so I want to highlight it here. We have a nice core team of contributors now and I hope we can attract more, after all.

I've been a happy buildout user over the years, so of course I used it for Morepath's development setup as well. But for a Python-only project like Morepath, pip can now do what buildout does. Since many more Python programmers are familiar with pip, and we want to make it as easy as possible for someone to start contributing, we've taken the plunge and entirely replaced buildout with pip. Even a buildout guy such as myself has been appreciating the results.

We've updated our developer documentation to reflect the changes, so it's easy to find how to do common things. The build environment for the Reg and Dectate libraries were used to use pip as well.

This work was done by Henri Hulski. Thanks Henri!

Other significant changes

  • I took a good look at Traject's routing system with an eye on performance and refactored it.

  • We realized that the directive directive was a bit too magic for its own good. I changed Dectate so that new Morepath configuration directives are now defined directly on the App class using the dectate.directive function. This breaks some code if you define new directives, but it's easy to fix.

  • Our extensive documentation has had a reorganization of its table of contents.

Look at the detailed changelog for much more information, including upgrade notes.

Performance increase

I benchmarked Morepath quite frequently during this development cycle. To make benchmarking easier, I created a new benchmarking tool called howareyou. It can not only benchmark Morepath, but can also benchmark other web frameworks -- Michael Merickel has in fact been using it already to help optimize Pyramid. You can find the howareyou tool here . The origins of this tool ultimately go back to work by wheezy.web creator Andriy Kornatskyy.

Morepath uses Webob for its request and response implementation. I learned quite a lot about Webob performance characteristics during this development cycle. This allowed me to make performance tweaks in Morepath.

It also let me detect that Webob's development version had some performance regressions that affected both Pyramid and Morepath. I'm very grateful to Bert Regeer for picking up so quickly and thoroughly on my reports of performance problems in Webob, and the Webob development version is currently actually slightly faster than release 1.6.1.

I talked about Morepath's performance history recently in my article Is Morepath Fast yet?. There we had peaked at about 19000 requests per second (on a synthetic benchmark) for the development version. I am happy to announce that we've managed to increase performance even more in our 0.16 release. It's now more than 28000 requests per second!

Morepath performance over time

Let's compare Morepath with some other carefully selected frameworks:

Morepath performance compared

Cool, Morepath 0.16 is actually faster than Pyramid at this point in time! I don't expect it to last long given that the Pyramid devs already using howareyou to optimize Pyramid, but it's nice to have such a moment. And I deliberately didn't include Falcon, Bottle or wheezy.web in this comparison, as that would rather spoil the effect. Do remember these are somewhat silly, synthetic benchmarks. It's rare indeed that Python web framework overhead is going to affect real world performance, but at least Morepath isn't the slowest one, right?

Enjoy!

I hope you all enjoy the fresh new release. Do get in touch with us!

Is Morepath Fast Yet?

Morepath is a Python web framework. But is it fast enough for your purposes?

Does performance matter?

Performance is one of the least important criteria you should use when you pick a Python web framework. You're using Python for web development after all; you are already compromising performance for ease of use.

But performance makes things seem easy. It boils down the whole choice between web frameworks to a single seemingly easy to understand variable: how fast is this thing? All web frameworks are the same anyway, right? (Wrong). We don't want the speed of our application be dragged down by the web framework. So we should just pick the one that is fastest. Makes total sense.

It makes total sense until you take a few minutes to think about it. Performance, sure, but performance doing what? Performance is notoriously difficult to measure. Sending a single "hello world" response? Parsing complex URLs with multiple variables in them? HTML template rendering? JSON serialization? Link generation? What aspect of performance matters to you depends on the application you're building. Why do we worry so much about performance and not about features, anyway?

Choosing a web framework based on performance makes no sense for most people. For most applications, application code dominates the CPU time spent. Pulling stuff out of a database can take vastly more time than rendering a web response.

What matters

So it makes sense to look at other factors when picking a web framework. Is there documentation? Can it do what I need it to do? Will it grow with me over time? Is it flexible? Is it being maintained? What's the community like? Does it have a cool logo?

Okay, I'm starting to sound like someone who doesn't want to admit the web framework I work on, Morepath, is atrociously slow. I'm giving you all kinds of reasons why you should use it despite its performance, which you would guess is pretty atrocious. It's true that the primary selling point of Morepath isn't performance -- it's flexibility. It's a micro-framework that is easy to learn but that doesn't let you down when your requirements become more complex.

Morepath performance

I maintain a very simple benchmark tool that measures just one aspect of performance: how fast a web framework at the Python WSGI level can generate simple "hello world" responses.

https://github.com/faassen/welcome-bench

I run it against Morepath once every while to see how we're doing with performance. I actually care more about what the framework is doing when Morepath generates the response than I care about the actual requests per second it can generate. I want Morepath's underlying complexity to be relatively simple. But since performance is so easy to think about I take advantage of that as a shortcut. I treat performance as an approximation of implementation complexity. Plus it's cool when your framework is fast, I admit it.

The current Morepath development version takes about 5200 ms per 100,000 requests, which translates to about 19200 requests per second. Let's see how that compares to some of the friendly competition:

                ms     rps  tcalls  funcs
django       10992    9098     190     85
flask        15854    6308     270    125
morepath      5204   19218     109     80

So at this silly benchmark, Morepath is more than twice as fast as Django and more than three times faster than Flask!

Let me highlight that for marketing purposes and trick those who aren't reading carefully:

Morepath is more than 2 times faster than Django and more than 3 times faster than Flask

Yay! End of story. Well, I gave a bit of a selective view just now. Here are some other web frameworks:

                ms     rps  tcalls  funcs
bottle        2172   46030      53     31
falcon        1539   64961      26     24
morepath      5204   19218     109     80
pyramid       3920   25509      72     57
wheezy.web    1201   83247      25     23

I'm not going to highlight that Bottle is more than two times faster at this silly benchmark nor that Falcon is more than three times faster. Let's not even think about wheezy.web.

I think this performance comparison actually highlights my point that in practice web framework performance is usually irrelevant. People aren't flocking to wheezy.web just because it's so fast. People aren't ignoring Flask because it's comparatively slow. I suspect many are surprised are surprised Flask is one of the slowest frameworks in this benchmark, as it's a lightweight framework.

Flask's relatively slow performance hasn't hurt its popularity. This demonstrates my point that web framework performance isn't that important overall. I don't fully understand why Flask is relatively slow, but I know part of the reason is werkzeug, its request/response implementation. Morepath is actually doing a lot more sophisticated stuff underneath than Flask and it's still faster. That Pyramid is faster than Morepath is impressive, as what it needs to do during runtime is similar to Morepath.

Let's look at the tcalls column: how many function calls get executed during a request. There is a strong correlation between how many Python function calls there are during a request and requests per second. This is why performance is a decent approximation of implementation complexity. It's also a clear sign we're using an interpreted language.

How Morepath performance has changed

So how has Morepath's performance evolved over time? Here's a nice graph:

Morepath performance over time

So what does this chart tell us? Before its 0.1 release when it still used werkzeug, Morepath was actually about as slow as Flask. After we switched to webob, Morepath became faster than Flask, but was still slower than Django.

By release 0.4.1 a bunch of minor improvements had pushed performance slightly beyond Django's -- but I don't have a clear idea of the details. I also don't understand exactly why there's a performance bump for 0.7, though I suspect it has to do with a refactor of application mounting behavior I did around that time -- while that code isn't exercised in this benchmark, it's possible it simplified a critical path.

I do know what caused the huge bump in performance in 0.8. This marked the switch to Reg 0.9, which is a dispatch library that is used heavily by Morepath. Reg 0.9 got faster, as this is when Reg switched to a more flexible and efficient predicate dispatch approach.

Performance was stable again until version 0.11, when it went down again. In 0.11 we introduced a measure to make the request object sanitize potentially dangerous input, and this cost us some performance. I'm not sure what caused the slight performance drop in 0.14.

And then there's a vast performance increase in current master. What explains this? Two things:

  • We've made some huge improvements to Reg again. Morepath benefits because it uses Reg heavily.

  • I cheated. That is, I found work that could be skipped in the case no URL parameters are in play, as in this benchmark.

Skipping unnecessary work was a legitimate improvement of Morepath. The code now avoids accessing the relatively expensive GET attribute on the webob request, and also avoids a for loop through an empty list and a few if statements. In Python, performance is sensitive to even a few extra lines of code on the critical path.

But when you do have URL parameters, Morepath's feature that lets you convert and validate them automatically is pretty nice -- in almost all circumstances you should be glad to pay the slight performance penalty for the convenience. Features are usually worth their performance cost.

So is Morepath fast yet?

Is Morepath fast yet? Probably. Does it matter? It depends. What benchmark? But for those just skimming this article, I'll do another sneaky highlight:

Morepath is fast. Morepath outperforms the most popular Python web frameworks while offering a lot more flexibility.

Introducing Bob Strongpinion

I posted an article about programming yesterday. (punctuated equilibrium in software development) In it I try to share some insights I've had about software development, and illustrate it with a library I work on that I think is interesting. I posted it to reddit /r/programming, obviously hoping to get a bit of attention with it. I think the topic is interesting to other developers, and writing it took a bit of time. Besides, I'm human enough to want positive attention.

My reddit post quickly sank from sight with nary an upvote. That got me thinking about what kind of posts I could make that would draw more attention. I joked a bit that having strong opinions boldly spoken gets more attention -- but I should blame the topic and my writing more than anything else. To blame the outside world makes you stop learning from it, and I want to learn.

In engineering there are always trade offs, and it's important to be aware of them. It is also nicer to be respectful of works that people have put a lot of time and effort in. And if you disregard them you are also less likely to learn and grow. So I won't go out there and say Morepath, the Python web framework I work on, is better than, say, Flask or Django or Pyramid. It depends on what you're doing. When I compare Morepath to other web frameworks, I probably don't excite you; it may in fact be sleep inducing.

I think Morepath is great. I think it solves problems that many Flask and Django developers don't even fully realize they actually have. That's in fact part of the problem I have in explaining it. But I also recognize that in many circumstances other frameworks are the better choice, for a variety of reasons.

But sometimes I wish I was more bold in expressing my opinions than I usually am. Sometimes I wish I was more like Bob Strongpinion.

Who is Bob Strongpinion?

Bob Strongpinion has strong opinions and conviction. He would blog saying all other web frameworks suck compared to Morepath. He'd have a lot blog posts with eye-catching titles:

Django was yesterday. The future belongs to Morepath.

10 Reasons Morepath is just plain better.

You're doing configuration wrong. This library gets it right. (Dectate)

Single dispatch is over. Get ready for predicate dispatch. (Reg)

Why routing to the model instead of the view is winning.

Bob Strongpinion posts a lot of articles with lists, and they're all 10 items explaining that what he likes is plain better. Bob Strongpinion doesn't believe in trade offs in engineering. There's better, and there's worse, and what he likes is just plain better. Bob Strongpinion knows there's ONE tool that's right for EVERY job.

If someone doesn't agree with Bob Strongpinion's choice of tools, they're either stupid, or more likely, evil. Bob Strongpinion may not have more experience in a problem domain than you do, but it's the RIGHT experience so he's still right.

When Bob Strongpinion makes a pronouncement in a comment thread, the case is closed. Disagree with him and incur his righteous wrath.

Engineering projects with Bob Strongpinion on it always succeed as he picks the right tools. And when they don't, it's someone else's fault. When Bob Strongpinion doesn't get to use his favorite tools it's no wonder the project failed.

When Bob Strongpinion uses an operating system, it's because it's the best one for everyone, unless they're too wimpy, so he's still elite. Bob Strongpinion definitely believes systemd is a great conspiracy to destroy Linux.

Conclusion

I think Bob Strongpinion would get a lot more upvotes on reddit. He'd get attention; some of it negative, but a lot of it would be positive.

"Yeah, Strongpinion! You tell the truth about software development!"

And he's always right. Obvious comparisons to certain public figures come to mind. You can see why his mindset would be comfortable, so it's understandable why Bob Strongpinion lives among us.

I sometimes wish I could be more like Bob Strongpinion when I promote my own work. As you can see from the above, I can channel him pretty well. I snuck him in while rejecting him, how sneaky! The Strongpinion must be strong in me. I've even done a list of 10 bullet points before. It got some upvotes.

But in the end I keep choosing not to be him.

Punctuated Equilibrium in Software

Punctuated equilibrium

Punctuated equilibrium is a concept in evolution theory. It was developed to explain a feature of the fossil record: biological species appear quite suddenly and then tend to be in relative stasis for a long period, only undergoing very gradual changes. The species is in equilibrium with its environment. Then suddenly this stasis is punctuated: there is a relatively brief period where a large series of changes occur. This results in the evolution of a new species. The rapid changes can be brought on by changes in the environment, or by a lucky mutation in a single individual that opens up a whole set of possibilities for subsequent changes.

I've noticed that software too can evolve with a pattern of punctuated equilibrium. I'll illustrate this using a Python library that I work on: Reg. To explain how it evolved I need to go into some detail about it. Luckily, Reg is quite an interesting little library.

Reg is predicate dispatch implementation for Python. It didn't start out that way, but that's what it is now. The Morepath web framework, which I also work on, uses Reg to enable some powerful features. I'll refer to Morepath a few places in this article as it provides use cases for Reg.

Reg ancestry

The ancestor of Reg is the zope.interface library, which was created around the year 2002 by Jim Fulton. It is still in very active use by large projects like the Pyramid web framework and the Plone CMS.

zope.interface lets you define interfaces for Python objects. It's similar to the Python abc module, though zope.interface came earlier and is more general.

zope.interface also implements a registry that maps interface (or type) to interface to support an adapter lookup pattern: you can adapt an object of an interface (or type) to an object with an interface you want.

In a web application you could for instance look up a HTML view interface (API) for a given content object such as a document or a someone's address, or whatever other type of content object you may have in your system. We'll give an example of this in code when we get to Reg.

The genesis of zope.interface took a few years and involved a predecessor project. Like Reg itself it underwent evolution by punctuated equilibrium in its early years. I describe this a bit in the Reg history document.

I try to keep history documents describing the evolution of various projects I work on, as I think they can provide insight into a project beyond what the rest of the documentation can bring. If you like software history, see Morepath history, Dectate history and Reg history. (The Reg history overlaps with this article, so if you're curious to learn more, do check it out later.)

After 2002 zope.interface became stable: its API hasn't changed much, and neither has its implementation. There were a few small tweaks here and there, in particular to add Python 3 compatibility, but that's it. At some point around 2009 I made some proposals to improve its API, but got nowhere. That's when I started playing around with the idea to reimplement it for myself.

The genesis of Reg

It often takes a period of experimentation and play to create something new. It's important during this phase not to think too much about immediate practical goals. Focus on a few core features that interest you; don't worry about it covering everything. Banish any thoughts about backwards compatibility and how to upgrade existing large code bases; that would be detrimental to the spirit of playfulness.

"Why are you reimplementing zope.interface, Martijn?"

"Just for fun, I don't expect anyone to use this."

After a few years of occasional play with various ideas I had concerning zope.interface, they finally started to come together in 2013. The goal of Reg at the time was straightforward: it was to be like zope.interface, but with an implementation I could understand, and with a streamlined API.

I'm going to show sample code now. Be aware that the sample code in this article may be somewhat fictional for educational purposes.

Reg initially worked like this:

# the view API
class IView(reg.Interface):
    def __call__(self):
        "If you call this, you get a web representation."

# register implementation of the view API for Document and
# HTTP Request classes
@IView.register(Document, Request)
class DocumentToViewAdapter:
    def __init__(self, doc, request):
        self.doc = doc
        self.request = request

    def __call__(self):
        return "<p>%s</p>" % self.doc.content

# can register other implementations, for example for Address and
# Request

# create instances we can look up for
doc = Document()
request = Request()

# look up the view adapter for a specific object, in this case a document
# The specific implementation you find depends on the class of doc and
# request arguments
view = IView.adapt(doc, request)
# and get the representation
html = view()

Here we define an IView interface. You can then register adapters for this view that take parameters (the object and a HTTP request) and turn it into an object that can create a HTML representation of the parameters.

Major transition to generic functions

I worked with the zope.interface version of Reg for a while in the context of the Morepath web framework. This gave me some practical experience. I also talked about Morepath and Reg in public and got some feedback. Even minimal feedback is great; it sets thoughts into motion. I quickly realized that Reg's API could be simplified if I centered it around generic functions with multiple dispatch instead of interfaces and adapters. Something like this:

# generic function definition instead of interface
@reg.generic
def view(obj, request):
    """"If you call this, you get a web representation."""
    raise NotImplemented

# an implementation for Document and Request
@view.register(Document, Request)
def document_view(obj, request):
    return "<p>%s</p>" % obj.content

# get representation for document by calling view()
html = view(doc, request)

This looks simpler. The interface definition is gone, the adapter class is gone. We just have a function that dispatches on the type of its arguments.

Reg worked like this for over a year. It was stable. I didn't foresee any more large changes.

Major transition to predicate dispatch

Meanwhile...

I created a predicate registry implementation. This lived inside of a module in Reg, but it could as well have been in a totally different library: the code was unrelated to the rest of Reg.

The use case for this predicate registry was Morepath view lookup. The predicate system let you register objects by a selection of keys. You could look up a view based on the request_method attribute (HTTP GET, POST, etc) of the request object, for instance, not just by type.

Two things came together in the fall of 2014:

  • I realized that it was annoying that the multiple dispatch system automatically dispatched on all arguments to the function -- in many cases that wasn't required.

  • I needed the predicate system to understand about types and inheritance. The multiple dispatch system in Reg understood types but not predicates, and the predicate dispatch system understood predicates but not types.

Then I realized that if I generalized Reg and turned it into a predicate dispatch system, I could actually unify the two systems. The dialectic (thesis, antithesis, synthesis) is a strong force for creativity in software development

With predicate dispatch you can dispatch on any aspect of the arguments to a function; not just its class but also any attribute of it. You can still do multiple dispatch as before: dispatch on the type of an argument is now just be a special case. Since arguments now needed a description of what predicate they dispatch on, I could also have arguments that are ignored by the dispatch system altogether.

This is when I finally understood some of the reasoning behind the PEAK-Rules library, which is a Python predicate dispatch implementation that predates Reg by many years. Almost everything has been implemented before, but with reimplementation you gain understanding.

With that insight, the equilibrium was punctuated, and Reg underwent rapid change again. Now it looked like this:

# dispatch function instead of generic function
# note how we explicitly name the arguments we want to match on
# (obj, request) in the predicates, and how we match on the
# request_method attribute. match_instance still matches on type.
@reg.dispatch(reg.match_instance('obj'),
              reg.match_key('request_method',
                            lambda request: request.request_method))
def view(obj, request):
    raise NotImplemented

# an implementation for GET requests of documents
@view.register(obj=Document, request_method='GET')
def document_view(obj, request):
    return "<p>%s</p>" % obj.content

# get representation for document by calling view()
html = view(doc, request)

When we define a dispatch function, we can now precisely describe on what aspects of the arguments we want to do dispatch. When we register an implementation, we can use more than just the types of the arguments. We can also have arguments that do not play a role in dispatch at all.

This system allows Morepath to have views looked up by the type of the instance being represented, the last name in the path (/edit, /details), the HTTP request method (GET, POST, etc), and the type of object in the POST body, if any.

I succeeded in making predicates participate in the cache that I already had to speed up multiple dispatch, so this change both simplified and increased performance.

Major transition to dispatch methods

After this huge change, Reg was stable again for almost 2 years. I didn't think it needed further major changes. I was wrong.

The trigger was clear this time, as it was a person. Stefano Taschini, who started contributing to the Morepath web framework project. Stefano's a very smart guy, so I'm doing my best to learn from him. Listen hard, even if your impulse, like mine, is to defend your design decisions. I was lucky that Stefano started to think about Reg. So while Reg seemed outwardly stable, the pressure for change was slowly building up.

In the summer of 2016 Stefano and I had a lot of discussions and created a few branches of Reg and Morepath. All that work has now landed in master of Reg and Morepath. The implementation of Reg is simpler and more explicit, and its performance has been increased. Yet again we had a major punctuation in the evolution of Reg.

I mentioned before how the code samples in this article are somewhat fictional. One fiction is the way you register implementations in Reg. It didn't actually work this way until now. Instead of this:

@view.register(obj=Document, request_method='GET')
def document_view(obj, request):
    return "<p>%s</p>" % obj.content

until very recently, you'd write something like this:

r = reg.Registry()

def document_view(obj, request):
    return "<p>%s</p>" % obj.content

r.register(view, document_view, obj=Document, request_method='GET')

So, we used to have an explicit registry object in Reg. This was there because of a use case of Reg that I haven't told you about yet: we need it to support separate dispatch contexts in the same run-time. Morepath uses this let you compose a larger web application out of multiple smaller ones, each with their own context.

To control which context Reg used you could pass in a special magic lookup parameter to each dispatch function call:

view(doc, request, lookup=registry)

Dispatch implementations needed access to the context too. They could get to it by defining a magic lookup argument in their signature:

def document_view(obj, request, lookup):
    return "<p>%s</p>" % process_content(obj.content, lookup=lookup)

If you didn't specify the lookup, an implicit thread-local lookup was used.

All this wasn't ideal. During the creation of Reg I was fully aware of Python's mantra "explicit is better than implicit", but I didn't know a better way to make context-specific dispatch calls work. I tried my best to at least isolate the implicit behavior in a well-controlled portion of Reg, and to allow a fully explicit option with lookup arguments, but the machinery to support all this was more complex than I'd wish.

When Stefano and I discussed this we came up with the following ideas:

  • Remove multiple registries. Instead allow simple registration on dispatch functions as we've already seen in the examples above. Each function keeps its own private registry. Stefano pushed hard for this while I was resistant, but he was right.

  • To control context, introduce the notion of dispatch methods. Build dispatch methods on dispatch functions.

A dispatch method is associated with a context class:

class Context:
    @reg.dispatch_method(reg.match_instance('obj'))
    def foo(self, obj):
        raise NotImplementedError()

You can register implementations with the method:

@Context.foo.register(obj=Document)
def implementation(self, obj):
    ...

When you call the dispatch method you call it in its context:

c = Context()
c.foo(doc)

Each subclass of the context class creates a new context, with a fresh set of registrations:

# a completely new and clean context
class NewContext(Context):
    pass

Instead of a magic lookup argument to call a generic function in a particular context, you simply use self as the instance of the context. This fits Python a lot better and is faster as well. Magical lookup arguments were gone. Thread-local implicit context was gone too. All is well. With Stefano on board now, Reg's bus factor has doubled too.

A new period of stasis?

Large changes create room for further changes. We've already seen a lot of follow-on changes, especially in the area of performance, and I think we haven't seen the end of this yet. I am starting to understand now why PEAK-Rules has AST manipulation code. We may not quite have reached a point of equilibrium yet.

But after that performance engineering, surely Reg won't need any further drastic changes anymore? I can't think of any. But I've been here several times before. zope.interface is assumed to be done; Reg isn't. If you assume a project is done that could become a self-fulling prophecy and cause the project to stagnate before its time.

Dare to change

Reg is a piece of software that sits at the lower levels of our software stack. Morepath is on top of it, and applications built with it are on top of that. I've been impressed by how much of the underlying codebase of Morepath we've been able to change without breaking Morepath applications much.

Of course the amount of code written with Morepath is insignificant compared to that written with web frameworks like Django or Flask or Pyramid, so we can still afford to be be bold -- now, when the community is still small, before many more people join us, is the time to make changes. That is why we can play with a cool technique like predicate dispatch that while not new, is still unfamiliar to many. It is also a creative challenge to make the unfamiliar approachable.

If you're interested in any of this and want to talk to us, the Morepath devs are one click away.

Self-serving mercenary statement: if you need a developer and like what you hear, talk to me -- I'm on the lookout for interesting projects.

Morepath 0.15 released!

Today the Morepath developers released Morepath 0.15 (CHANGES).

What is Morepath? Morepath is a Python web framework that is small, easy to learn, extensively documented, and insanely powerful.

This release is a smaller release without big visible changes, or big hood changes. Instead it polishes a lot of stuff. It also continues the trend with contributions from multiple core developers.

This release prepares the way for the next Morepath release. To this end we've deprecated a number of APIs. We are preparing a big change to the underlying Reg predicate dispatch library APIs that should make it less implicit, less magic, and make it perform slightly better. Stay tuned!

Impressions of React Europe 2016

Last week I went to the React Europe conference 2016 in Paris. It was a lot of fun and inspirational as well. I actually hadn't used React for about 6 months because I've been focusing on server-side stuff, in particular Morepath, but this really makes me want to go and work with it again (I'm available). I especially enjoy the creativity in the community.

In this post I want to give my impression of the conference, and highlight some talks that stood out to me. There are actually too many to highlight here: I thought the talk quality of this conference was very high. I also appreciated the wide range of topics -- not everything was about React directly. More of that, please!

Travel

I was quite worried about travel this year. I'm in the Netherlands, so it all should be so easy: hop in a train to go to Rotterdam, a 45 minute ride. Then take the Thalys that speeds from Rotterdam to Paris in about 3 hours. In total it takes about 4 hours. Awesome.

But it's strike season in France. Railway strikes were threatened. And then there was a railway strike in Belgium, through which the train passes, on the day I was to travel. Uh oh. I already got some warnings in the days in advance about possible train cancellations due to the strikes. But my train was still going.

But travel in the Netherlands at least wasn't disrupted, so I wasn't worried about that. I made it in time to the normal intercity train that brings me from my home town, Tilburg, to Rotterdam. Found a comfortable seat. All ready to go. Then an announcement: please leave the train as it can go no further. A cargo train had broken down ahead of us. Argh!

In the end I managed to get to Rotterdam and catch a later Thalys, and made it to Paris, just 2 hours later than I'd planned.

I was also worried about announced strikes in the Paris metro system on the day of the conference. Getting around in Paris is very convenient with the metro, but not if it isn't going. In the end the metro wasn't affected.

What I did not anticipate was the whole flood situation, to the point where they had to move parts of the inventory of the Louvre. But Paris is a big city and the floods did not affect me.

So in the end what I worried about never happened and stuff happened that I didn't worry about at all.

Hackathon and MobX

Like last year there was a hackathon one day ahead of the conference at the Mozilla offices in Paris.

Last year's hackathon was special: I met up with Lee Bannard and we worked on reselect, which became quite a popular little library for use with Redux. You might enjoy my story on that.

I was very happy to see Lee again at this year's hackathon. We didn't create any new code this time; we spent most of our time learning about MobX, which I first heard about that day. We met Devin Pastoor at the hackathon. He already had a little app that used MobX that he wanted to work on together. Lee and myself helped a little with it but then got distracted trying to figure out how MobX's magic works.

MobX is a state management library, typically used with React, that takes a different approach than the now dominant library for this, Redux. Mobx lets you use normal OOP style objects with state and references in your client-side model. Unlike Redux is does not require you to normalize your state. MobX observes changes to your objects automatically and is very clever about only updating the parts of the UI that are affected.

This gives MobX different tradeoffs than Redux. I haven't used MobX in practice at all, but I would say MobX is less verbose than Redux, and you get more performance out of the box automatically. I also think it would be easier for newcomers to adopt. On the other hand Redux's focus on the immutable state constraint simplifies testing and debugging, and opened up a rich ecosystem of extensions. Redux's implementation is also a lot simpler. People want a simple answer to "what is better", but these really are tradeoffs: which way is the right way for you and applications depends on who you are and what you are working on.

Sorry folks, Lee and I created no thing new this time. But we had fun.

Dan Abramov: The Redux Journey

Dan Abramov, one of my open source heroes, gave a very interesting talk where he talked about the quick and wild ride Redux has been on since last year. Redux was indeed everywhere at this conference, and people were building very cool stuff on top of it.

Dan explained how the constraints of the Redux architecture, such as reducers on immutable state, also lead to its stand-out features, such as simple debugging and persisting and sharing state. He also spoke about how Redux's well-defined minimal contracts help its extension and middleware ecosystem.

Dan's talk is interesting to anyone who is interested in framework design, even if you don't care about React or Redux at all.

Watch "The Redux Journey"

Eric Vicenti: Native Navigation for Every Platform

This talk was about using a Redux-style approach to manage navigation state in mobile apps, so that the app can react appropriately to going "back", and helps with links within the app as well as it responding to links from other apps. Web folks have deep experience with links and is an example of how bringing web technology to mobile app development can also help make mobile app development easier.

Watch "Native Navigation for Every Platform"

Lin Clark: A cartoon guide to performance in React

Lin Clark gave a great talk where she explained why React can be fast, and how you can exploit its features to help it along. Explaining complex topics well to make them seem simple is hard, and so I appreciated how well she accomplished it.

If you are new to React this is a really good talk to watch!

Watch "A cartoon guide to performance in React"

Christopher Chedeau: Being Successful at Open Source

Christopher described what strategies Facebook uses when they open source stuff. I especially liked the question they made sure to ask: "What did you struggle with?". Not "what do you want?" as that can easily devolve into a wishlist discussion, but specifically asking about problems that newcomers had. One fun anecdote: the way to make a FAQ go away was not writing more documentation but changing an error message.

I also liked the clever "fake it until you make it" approach to making a community appear more active than it is in the early stages, so that it actually becomes active. One trick they used is to ask people to blog about React, then publish all those links on a regular basis.

As an individual developer who occasionally open sources stuff I must point out rule 0 for open source success is "be a huge company with lots of resources like Facebook". Without those resources it is a much bigger struggle to build an open source community. (It also doesn't help that with Morepath I picked a saturated space: Python web frameworks. It's hard to convince people it's innovative. But that was the same problem React faced when it was first released.) (UPDATE: of course Facebook-level resources are not required for open source success, there are a lot of counter examples, but it sure helps. The talk mentions a team of multiple people engaging the community through multiple routes. A single individual can't replicate that at the start.)

Nevertheless, open source success for React was by no means guaranteed, and React helped make Facebook's reputation among developers. They made the React open source community really work. Kudos.

Watch "Being Successful at Open Source"

Dan Schafer: GraphQL at Facebook

I liked Dan Schafer's talk: a nice quick recap of why GraphQL is the way it is, some clear advice on how to deal with authorization in GraphQL, then a nice discussion on how to implement efficient queries with GraphQL, and why GraphQL cache keys are the way they are. Clear, focused and pragmatic, while still going into to the why of things, and without overwhelming detail.

Watch "GraphQL at Facebook"

Jeff Morrison: A Deepdive into Flow

This talk was about the implementation of Flow, which is a type checker for JavaScript code. This is a very complex topic involving compiler technology and type inferencing, and was still amazingly clear. The talk gave me the nice illusion I actually understand how Flows works. It also makes me want to try Flow again and integrate it into my development tool chain.

Flow is a general JavaScript tool. It comes out of Facebook but is not directly connected to React at all, even more so than something like GraphQL. I really appreciated that the organizers included talks like this and mixed things up.

Watch "A Deepdive into Flow"

Cheng Lou: On the Spectrum of Abstraction

This talk, which isn't about React, really stood out for me, and from what I heard also resonated with others at the conference. It tied neatly into the themes Dan Abramov already set up in his opening talk about Redux. Dan told me later this was not actually coordinated. The ideas are just in the air, and this speaks for the thoughtfulness of the React community.

Cheng Lou's talk was a very high level talk about the benefits and the costs of abstraction. This is something I care about a lot as a developer: how do I avoid over-engineering and under-engineering (I've written about it before), and solve problems at the right level? Software has many forces on many levels pulling at it, from end-users to low-level details, and how do you balance out these forces? Engineering is so much about dealing with tradeoffs. How do you even communicate about this?

The next day I had an interesting chat with Cheng Lou about his talk, where he discussed various things he had to cut out of his talk so it wouldn't be too long. He also mentioned Up and Down the Ladder of Abstraction by Bret Victor, so that is now on my reading list.

I highly recommend this talk for anyone interested in these topics.

Watch "On the Spectrum of Abstraction"

Preethi Kasireddy: Going from 0 to full-time software engineer in 6 months

This was a 5 minute lightning talk with a personal story: how overwhelming software development is to a newcomer and how it can nonetheless be learned. During the talk I was sitting next to someone who was relatively new to software development himself and I could see how much this talk resonated with him.

Preethi Kasireddy also encouraged more experienced developers to mentor newcomers. I've found myself that mentoring doesn't have to take a lot of time and can still be hugely appreciated. It's fun to do as well.

A new developer is often insecure as there are just so many things to grasp, and experienced developers seem to know so much. Ironically I sometimes feel insecure as an older, more experienced developer as well, when I see people like Preethi learn software development as quickly as they do. I certainly took more time to get where they are.

But I'm old enough to have gotten used to intimidatingly smart younger people too. I can keep up. The Internet overall helps with learning: the resources on the Internet for a new developer may be overwhelming, but they are also of tremendous value. Preethi called for more intermediate-level resources. I am not sure this series I wrote counts; I suspect Preethi is beyond it, but perhaps others will enjoy it.

(Video not up yet! I'll update this post when it is.)

Jonas Gebhardt: Evolving the Visual Programming Environment with React

This was another one of those non-React talks I really appreciated. It is related to React as it is both inspired by functional programming patterns and component-based design, but it's really about something else: a UI to construct programs by connecting boxes with arrows.

There are many of these around. Because these don't seem to ever enter the daily life of a programmer, I tend to be skeptical about them.

But Jonas Gebhardt acknowledged the prior art, and the approach he described is pragmatic. An open world approach in the web browser, unlike many of the "we are the world" sandbox implementations from the past. Annotated React components can serve as the building blocks. He even sketched out an idea on how to connect UI input and output to custom user interfaces in the end.

So I came away less skeptical. This approach has potential and I'd like to see more.

Watch "Evolving the Visual Programming Environment with React"

Bonnie Eisenman: React Native Retrospective

I really like retrospectives. This was an excellent talk about the history of React Native over the course of the last year and a half. React Native is the technology that lets you use JavaScript and React to develop native iPhone and Android apps. Bonnie Eisenman also wrote a book about it.

React Native is a potential game changer to me as it lets people like me use our deep web development experience to build phone apps. The talk made me excited to go and play with React Native, and I'm sure I wasn't the only one. In a chat afterwards, Bonnie confirmed that was a goal of her talk, so mission accomplished!

Watch "React Native Retrospective"

Phil Holden: subdivide and redux-swarmlog

Phil Holden gave a 5 minute lightning talk, but please give him more space next time. He discussed Subdivide, an advanced split pane layout system for React, and then also discussed another mind-blowing topic: using WebRTC to create a peer to peer network between multiple Redux frontends, so that they share actions. This lets users share data without a server being around. This he packaged as a library in a package called redux-swarmlog.

I've been thinking about peer to peer serverless web applications for some years as I believe they have the potential to change the web, and Phil's talk really reignited that interest. Peer to peer is hard, but the technology is improving. Later that day, I had the pleasure of having a brief chat with Phil about such wild topics. Thanks Phil for the inspiration!

(Video not up yet! I'll update this post when it is.)

Andrew Clark: Recomposing your React application

Andrew Clark is Internet-famous to me, as he created Flummox, the Flux state management framework I used before switching to Redux (Andrew in fact co-created Redux). In this talk he discusses recompose, a library he wrote that helps you do sophisticated things with pure, stateless function components in React. I need to play with it and see whether it fits in my React toolbox. Andrew also described the interesting techniques recompose uses to help reduce the overhead of small composed functions -- this highlights the properties you gain when you stick to the pure function constraint.

Watch "Recomposing your React application"

Jafar Husain: Falcor: One Model Everywhere

When multiple development teams have a similar idea at about the same time, that may be a sign the idea is a good one. This happened to me when I came up with a client-side web framework few years ago, thought I was onto something new, and then Backbone emerged, followed by many others.

Jafar Husain in this well-done talk described how Falcor and GraphQL were a similar solution to similar problems. Both Falcor and GraphQL let the client be in control of what data it demands from the server. He then highlighted the differences between Falcor and GraphQL, where he contrasted Falcor's more lightweight approach to GraphQL's more powerful but involved focus on schemas. It's tradeoffs again: which fits best depends on your use cases and team.

Watch "Falcor: One Model Everywhere"

Laney Kuenzel & Lee Byron: GraphQL Future

This was a wide-ranging talk that went into various issues that GraphQL team at Facebook is trying to solve, mostly centered about the need to receive some form of immediate update when state on the server changes. Laney and Lee presented various solutions in a various states of readiness, from mostly untested ideas to stuff that is already deployed in production at Facebook. Very interesting in you're interested in GraphQL at all, and also if you're interested in how smart people tackle problems.

Watch "GraphQL Future

Constructive feedback

In my blog post last year I was clear I enjoyed the conference a lot, but also engaged in a little bit of constructive criticism. I don't presume that the React Europe organizers directly responded to my feedback, but let's see how they did anyway and give a bit more feedback here. My intent with this feedback is to do my bit to make a great conference even better.

Temperature

Last year the conference was in early July in Paris and it was 40 degrees Celsius. The React Europe team responded by shifting conference a month earlier. It was not too hot: problem solved.

Hackathon

Last year the hackathon assumed people were going to compete in a contest by default instead of cooperate on cool projects. This year they were very clear that cooperation on cool projects was encouraged. Awesome!

Still, I found myself walking around Paris with a friend on Friday night trying to find a quiet place so we could look at some code together. We enjoyed the conversation but we didn't find such a place in the end.

This is why I prefer the approach Python conferences take: a 1-3 day optional sprint for people to participate in after the conference has ended. Why I like afterwards better:

  • You can get involved in cool projects you learned about during the conference.

  • You can get to know people you met during the conference better.

  • Since there is no pressure it's a good way to wind down. Speakers can participate without stressing out about a talk they will be giving soon.

Facebook speakers

Many of the speakers at this conference work for Facebook. They gave excellent talks: thank you. I understand that having a lot of speakers from Facebook is natural for a conference on React, as that's where it originated. (and Facebook hires people from the community). But this is an open source community. While I realize you'd take on more unknown quantities and it would be more difficult to keep up the quality of the talks, I would personally enjoy hearing a few more voices not from Facebook next year.

Gender diversity

Last year I spoke about a bit gender diversity at the conference. This year there were more female speakers than last year (keep it up!), but male voices were still the vast majority. Women speakers are important in helping women participants feel more welcome in our conferences and our community. We can still do a lot better: let's learn from PyCon US.

Back home

The train ride back home on Saturday morning was as it should: uneventful. Left the hotel around 9 am the morning, was back home around 2:30 pm. I came home tired but inspired, as it should be after a good conference. Thanks so much to the organizers and speakers for the experience! I hope you have enjoyed my little contribution.

Morepath 0.14 released!

Today we released Morepath 0.14 (CHANGES).

What is Morepath? Morepath is a Python web framework that is powerful and flexible due to its advanced configuration engine (Dectate) and an advanced dispatch system (Reg), but at the same time is easy to learn. It's also extensively documented!

The part of this release that I'm the most excited about is not technical but has to do with the community, which is growing -- this release contains significant work by several others than myself. Thanks Stefano Taschini, Denis Krienbühl and Henri Hulski!

New for the community as well is that we have a web-based and mobile-supported chat channel for Morepath. You can join us with a click.

Please join and hang out!

Major new features of this release:

  • Documented extension API

  • New implementation overview.

  • A new document describing how to test your Morepath-based code.

  • Documented how to create a command-line query tool for Morepath configuration.

  • New cookiecutter template to quickly create a Morepath-based project.

  • New releases of various extensions compatible with 0.14. Did you know that Morepath has more.jwtauth, more.basicauth and more.itsdangerous extensions for authentication policy, more.static and more.webassets for static resources, more.chameleon and more.jinja2 for server templating languages, more.transaction to support SQLAlchemy and ZODB transactions and more.forwarded to support the Forwarded HTTP header?

  • Configuration of Morepath-based applications is now simpler and more explicit; we have a new commit method on application classes and applications get automatically committed during runtime if you don't do it first.

  • Morepath now performs host header validation to guard against header poisoning attacks.

  • New defer_class_links directive. This helps in a complicated app that is composed of multiple smaller applications that want to link to each other using the request.class_link method introduced in Morepath 0.13.

  • We've refactored both the publishing/view system and the link generation system. It's cleaner now under the hood.

  • Introduced an official deprecation policy as we prepare for Morepath 1.0, along with upgrade instructions.

Interested? Feedback? Let us know!