Punctuated equilibrium

Punctuated equilibrium is a concept in evolution theory. It was developed to explain a feature of the fossil record: biological species appear quite suddenly and then tend to be in relative stasis for a long period, only undergoing very gradual changes. The species is in equilibrium with its environment. Then suddenly this stasis is punctuated: there is a relatively brief period where a large series of changes occur. This results in the evolution of a new species. The rapid changes can be brought on by changes in the environment, or by a lucky mutation in a single individual that opens up a whole set of possibilities for subsequent changes.

I've noticed that software too can evolve with a pattern of punctuated equilibrium. I'll illustrate this using a Python library that I work on: Reg. To explain how it evolved I need to go into some detail about it. Luckily, Reg is quite an interesting little library.

Reg is predicate dispatch implementation for Python. It didn't start out that way, but that's what it is now. The Morepath web framework, which I also work on, uses Reg to enable some powerful features. I'll refer to Morepath a few places in this article as it provides use cases for Reg.

Reg ancestry

The ancestor of Reg is the zope.interface library, which was created around the year 2002 by Jim Fulton. It is still in very active use by large projects like the Pyramid web framework and the Plone CMS.

zope.interface lets you define interfaces for Python objects. It's similar to the Python abc module, though zope.interface came earlier and is more general.

zope.interface also implements a registry that maps interface (or type) to interface to support an adapter lookup pattern: you can adapt an object of an interface (or type) to an object with an interface you want.

In a web application you could for instance look up a HTML view interface (API) for a given content object such as a document or a someone's address, or whatever other type of content object you may have in your system. We'll give an example of this in code when we get to Reg.

The genesis of zope.interface took a few years and involved a predecessor project. Like Reg itself it underwent evolution by punctuated equilibrium in its early years. I describe this a bit in the Reg history document.

I try to keep history documents describing the evolution of various projects I work on, as I think they can provide insight into a project beyond what the rest of the documentation can bring. If you like software history, see Morepath history, Dectate history and Reg history. (The Reg history overlaps with this article, so if you're curious to learn more, do check it out later.)

After 2002 zope.interface became stable: its API hasn't changed much, and neither has its implementation. There were a few small tweaks here and there, in particular to add Python 3 compatibility, but that's it. At some point around 2009 I made some proposals to improve its API, but got nowhere. That's when I started playing around with the idea to reimplement it for myself.

The genesis of Reg

It often takes a period of experimentation and play to create something new. It's important during this phase not to think too much about immediate practical goals. Focus on a few core features that interest you; don't worry about it covering everything. Banish any thoughts about backwards compatibility and how to upgrade existing large code bases; that would be detrimental to the spirit of playfulness.

"Why are you reimplementing zope.interface, Martijn?"

"Just for fun, I don't expect anyone to use this."

After a few years of occasional play with various ideas I had concerning zope.interface, they finally started to come together in 2013. The goal of Reg at the time was straightforward: it was to be like zope.interface, but with an implementation I could understand, and with a streamlined API.

I'm going to show sample code now. Be aware that the sample code in this article may be somewhat fictional for educational purposes.

Reg initially worked like this:

# the view API
class IView(reg.Interface):
    def __call__(self):
        "If you call this, you get a web representation."

# register implementation of the view API for Document and
# HTTP Request classes
@IView.register(Document, Request)
class DocumentToViewAdapter:
    def __init__(self, doc, request):
        self.doc = doc
        self.request = request

    def __call__(self):
        return "<p>%s</p>" % self.doc.content

# can register other implementations, for example for Address and
# Request

# create instances we can look up for
doc = Document()
request = Request()

# look up the view adapter for a specific object, in this case a document
# The specific implementation you find depends on the class of doc and
# request arguments
view = IView.adapt(doc, request)
# and get the representation
html = view()

Here we define an IView interface. You can then register adapters for this view that take parameters (the object and a HTTP request) and turn it into an object that can create a HTML representation of the parameters.

Major transition to generic functions

I worked with the zope.interface version of Reg for a while in the context of the Morepath web framework. This gave me some practical experience. I also talked about Morepath and Reg in public and got some feedback. Even minimal feedback is great; it sets thoughts into motion. I quickly realized that Reg's API could be simplified if I centered it around generic functions with multiple dispatch instead of interfaces and adapters. Something like this:

# generic function definition instead of interface
@reg.generic
def view(obj, request):
    """"If you call this, you get a web representation."""
    raise NotImplemented

# an implementation for Document and Request
@view.register(Document, Request)
def document_view(obj, request):
    return "<p>%s</p>" % obj.content

# get representation for document by calling view()
html = view(doc, request)

This looks simpler. The interface definition is gone, the adapter class is gone. We just have a function that dispatches on the type of its arguments.

Reg worked like this for over a year. It was stable. I didn't foresee any more large changes.

Major transition to predicate dispatch

Meanwhile...

I created a predicate registry implementation. This lived inside of a module in Reg, but it could as well have been in a totally different library: the code was unrelated to the rest of Reg.

The use case for this predicate registry was Morepath view lookup. The predicate system let you register objects by a selection of keys. You could look up a view based on the request_method attribute (HTTP GET, POST, etc) of the request object, for instance, not just by type.

Two things came together in the fall of 2014:

• I realized that it was annoying that the multiple dispatch system automatically dispatched on all arguments to the function -- in many cases that wasn't required.

• I needed the predicate system to understand about types and inheritance. The multiple dispatch system in Reg understood types but not predicates, and the predicate dispatch system understood predicates but not types.

Then I realized that if I generalized Reg and turned it into a predicate dispatch system, I could actually unify the two systems. The dialectic (thesis, antithesis, synthesis) is a strong force for creativity in software development

With predicate dispatch you can dispatch on any aspect of the arguments to a function; not just its class but also any attribute of it. You can still do multiple dispatch as before: dispatch on the type of an argument is now just be a special case. Since arguments now needed a description of what predicate they dispatch on, I could also have arguments that are ignored by the dispatch system altogether.

This is when I finally understood some of the reasoning behind the PEAK-Rules library, which is a Python predicate dispatch implementation that predates Reg by many years. Almost everything has been implemented before, but with reimplementation you gain understanding.

With that insight, the equilibrium was punctuated, and Reg underwent rapid change again. Now it looked like this:

# dispatch function instead of generic function
# note how we explicitly name the arguments we want to match on
# (obj, request) in the predicates, and how we match on the
# request_method attribute. match_instance still matches on type.
@reg.dispatch(reg.match_instance('obj'),
              reg.match_key('request_method',
                            lambda request: request.request_method))
def view(obj, request):
    raise NotImplemented

# an implementation for GET requests of documents
@view.register(obj=Document, request_method='GET')
def document_view(obj, request):
    return "<p>%s</p>" % obj.content

# get representation for document by calling view()
html = view(doc, request)

When we define a dispatch function, we can now precisely describe on what aspects of the arguments we want to do dispatch. When we register an implementation, we can use more than just the types of the arguments. We can also have arguments that do not play a role in dispatch at all.

This system allows Morepath to have views looked up by the type of the instance being represented, the last name in the path (/edit, /details), the HTTP request method (GET, POST, etc), and the type of object in the POST body, if any.

I succeeded in making predicates participate in the cache that I already had to speed up multiple dispatch, so this change both simplified and increased performance.

Major transition to dispatch methods

After this huge change, Reg was stable again for almost 2 years. I didn't think it needed further major changes. I was wrong.

The trigger was clear this time, as it was a person. Stefano Taschini, who started contributing to the Morepath web framework project. Stefano's a very smart guy, so I'm doing my best to learn from him. Listen hard, even if your impulse, like mine, is to defend your design decisions. I was lucky that Stefano started to think about Reg. So while Reg seemed outwardly stable, the pressure for change was slowly building up.

In the summer of 2016 Stefano and I had a lot of discussions and created a few branches of Reg and Morepath. All that work has now landed in master of Reg and Morepath. The implementation of Reg is simpler and more explicit, and its performance has been increased. Yet again we had a major punctuation in the evolution of Reg.

I mentioned before how the code samples in this article are somewhat fictional. One fiction is the way you register implementations in Reg. It didn't actually work this way until now. Instead of this:

@view.register(obj=Document, request_method='GET')
def document_view(obj, request):
    return "<p>%s</p>" % obj.content

until very recently, you'd write something like this:

r = reg.Registry()

def document_view(obj, request):
    return "<p>%s</p>" % obj.content

r.register(view, document_view, obj=Document, request_method='GET')

So, we used to have an explicit registry object in Reg. This was there because of a use case of Reg that I haven't told you about yet: we need it to support separate dispatch contexts in the same run-time. Morepath uses this let you compose a larger web application out of multiple smaller ones, each with their own context.

To control which context Reg used you could pass in a special magic lookup parameter to each dispatch function call:

view(doc, request, lookup=registry)

Dispatch implementations needed access to the context too. They could get to it by defining a magic lookup argument in their signature:

def document_view(obj, request, lookup):
    return "<p>%s</p>" % process_content(obj.content, lookup=lookup)

If you didn't specify the lookup, an implicit thread-local lookup was used.

All this wasn't ideal. During the creation of Reg I was fully aware of Python's mantra "explicit is better than implicit", but I didn't know a better way to make context-specific dispatch calls work. I tried my best to at least isolate the implicit behavior in a well-controlled portion of Reg, and to allow a fully explicit option with lookup arguments, but the machinery to support all this was more complex than I'd wish.

When Stefano and I discussed this we came up with the following ideas:

• Remove multiple registries. Instead allow simple registration on dispatch functions as we've already seen in the examples above. Each function keeps its own private registry. Stefano pushed hard for this while I was resistant, but he was right.

• To control context, introduce the notion of dispatch methods. Build dispatch methods on dispatch functions.

A dispatch method is associated with a context class:

class Context:
    @reg.dispatch_method(reg.match_instance('obj'))
    def foo(self, obj):
        raise NotImplementedError()

You can register implementations with the method:

@Context.foo.register(obj=Document)
def implementation(self, obj):
    ...

When you call the dispatch method you call it in its context:

c = Context()
c.foo(doc)

Each subclass of the context class creates a new context, with a fresh set of registrations:

# a completely new and clean context
class NewContext(Context):
    pass

Instead of a magic lookup argument to call a generic function in a particular context, you simply use self as the instance of the context. This fits Python a lot better and is faster as well. Magical lookup arguments were gone. Thread-local implicit context was gone too. All is well. With Stefano on board now, Reg's bus factor has doubled too.

A new period of stasis?

Large changes create room for further changes. We've already seen a lot of follow-on changes, especially in the area of performance, and I think we haven't seen the end of this yet. I am starting to understand now why PEAK-Rules has AST manipulation code. We may not quite have reached a point of equilibrium yet.

But after that performance engineering, surely Reg won't need any further drastic changes anymore? I can't think of any. But I've been here several times before. zope.interface is assumed to be done; Reg isn't. If you assume a project is done that could become a self-fulling prophecy and cause the project to stagnate before its time.

Dare to change

Reg is a piece of software that sits at the lower levels of our software stack. Morepath is on top of it, and applications built with it are on top of that. I've been impressed by how much of the underlying codebase of Morepath we've been able to change without breaking Morepath applications much.

Of course the amount of code written with Morepath is insignificant compared to that written with web frameworks like Django or Flask or Pyramid, so we can still afford to be be bold -- now, when the community is still small, before many more people join us, is the time to make changes. That is why we can play with a cool technique like predicate dispatch that while not new, is still unfamiliar to many. It is also a creative challenge to make the unfamiliar approachable.

If you're interested in any of this and want to talk to us, the Morepath devs are one click away.

Self-serving mercenary statement: if you need a developer and like what you hear, talk to me -- I'm on the lookout for interesting projects.

Travel

I was quite worried about travel this year. I'm in the Netherlands, so it all should be so easy: hop in a train to go to Rotterdam, a 45 minute ride. Then take the Thalys that speeds from Rotterdam to Paris in about 3 hours. In total it takes about 4 hours. Awesome.

But it's strike season in France. Railway strikes were threatened. And then there was a railway strike in Belgium, through which the train passes, on the day I was to travel. Uh oh. I already got some warnings in the days in advance about possible train cancellations due to the strikes. But my train was still going.

But travel in the Netherlands at least wasn't disrupted, so I wasn't worried about that. I made it in time to the normal intercity train that brings me from my home town, Tilburg, to Rotterdam. Found a comfortable seat. All ready to go. Then an announcement: please leave the train as it can go no further. A cargo train had broken down ahead of us. Argh!

In the end I managed to get to Rotterdam and catch a later Thalys, and made it to Paris, just 2 hours later than I'd planned.

I was also worried about announced strikes in the Paris metro system on the day of the conference. Getting around in Paris is very convenient with the metro, but not if it isn't going. In the end the metro wasn't affected.

What I did not anticipate was the whole flood situation, to the point where they had to move parts of the inventory of the Louvre. But Paris is a big city and the floods did not affect me.

So in the end what I worried about never happened and stuff happened that I didn't worry about at all.

Hackathon and MobX

Like last year there was a hackathon one day ahead of the conference at the Mozilla offices in Paris.

Last year's hackathon was special: I met up with Lee Bannard and we worked on reselect, which became quite a popular little library for use with Redux. You might enjoy my story on that.

I was very happy to see Lee again at this year's hackathon. We didn't create any new code this time; we spent most of our time learning about MobX, which I first heard about that day. We met Devin Pastoor at the hackathon. He already had a little app that used MobX that he wanted to work on together. Lee and myself helped a little with it but then got distracted trying to figure out how MobX's magic works.

MobX is a state management library, typically used with React, that takes a different approach than the now dominant library for this, Redux. Mobx lets you use normal OOP style objects with state and references in your client-side model. Unlike Redux is does not require you to normalize your state. MobX observes changes to your objects automatically and is very clever about only updating the parts of the UI that are affected.

This gives MobX different tradeoffs than Redux. I haven't used MobX in practice at all, but I would say MobX is less verbose than Redux, and you get more performance out of the box automatically. I also think it would be easier for newcomers to adopt. On the other hand Redux's focus on the immutable state constraint simplifies testing and debugging, and opened up a rich ecosystem of extensions. Redux's implementation is also a lot simpler. People want a simple answer to "what is better", but these really are tradeoffs: which way is the right way for you and applications depends on who you are and what you are working on.

Sorry folks, Lee and I created no thing new this time. But we had fun.

Dan Abramov: The Redux Journey

Dan Abramov, one of my open source heroes, gave a very interesting talk where he talked about the quick and wild ride Redux has been on since last year. Redux was indeed everywhere at this conference, and people were building very cool stuff on top of it.

Dan explained how the constraints of the Redux architecture, such as reducers on immutable state, also lead to its stand-out features, such as simple debugging and persisting and sharing state. He also spoke about how Redux's well-defined minimal contracts help its extension and middleware ecosystem.

Dan's talk is interesting to anyone who is interested in framework design, even if you don't care about React or Redux at all.

Watch "The Redux Journey"

Eric Vicenti: Native Navigation for Every Platform

This talk was about using a Redux-style approach to manage navigation state in mobile apps, so that the app can react appropriately to going "back", and helps with links within the app as well as it responding to links from other apps. Web folks have deep experience with links and is an example of how bringing web technology to mobile app development can also help make mobile app development easier.

Watch "Native Navigation for Every Platform"

Lin Clark: A cartoon guide to performance in React

Lin Clark gave a great talk where she explained why React can be fast, and how you can exploit its features to help it along. Explaining complex topics well to make them seem simple is hard, and so I appreciated how well she accomplished it.

If you are new to React this is a really good talk to watch!

Watch "A cartoon guide to performance in React"

Christopher Chedeau: Being Successful at Open Source

Christopher described what strategies Facebook uses when they open source stuff. I especially liked the question they made sure to ask: "What did you struggle with?". Not "what do you want?" as that can easily devolve into a wishlist discussion, but specifically asking about problems that newcomers had. One fun anecdote: the way to make a FAQ go away was not writing more documentation but changing an error message.

I also liked the clever "fake it until you make it" approach to making a community appear more active than it is in the early stages, so that it actually becomes active. One trick they used is to ask people to blog about React, then publish all those links on a regular basis.

As an individual developer who occasionally open sources stuff I must point out rule 0 for open source success is "be a huge company with lots of resources like Facebook". Without those resources it is a much bigger struggle to build an open source community. (It also doesn't help that with Morepath I picked a saturated space: Python web frameworks. It's hard to convince people it's innovative. But that was the same problem React faced when it was first released.) (UPDATE: of course Facebook-level resources are not required for open source success, there are a lot of counter examples, but it sure helps. The talk mentions a team of multiple people engaging the community through multiple routes. A single individual can't replicate that at the start.)

Nevertheless, open source success for React was by no means guaranteed, and React helped make Facebook's reputation among developers. They made the React open source community really work. Kudos.

Watch "Being Successful at Open Source"

Dan Schafer: GraphQL at Facebook

I liked Dan Schafer's talk: a nice quick recap of why GraphQL is the way it is, some clear advice on how to deal with authorization in GraphQL, then a nice discussion on how to implement efficient queries with GraphQL, and why GraphQL cache keys are the way they are. Clear, focused and pragmatic, while still going into to the why of things, and without overwhelming detail.

Watch "GraphQL at Facebook"

Jeff Morrison: A Deepdive into Flow

This talk was about the implementation of Flow, which is a type checker for JavaScript code. This is a very complex topic involving compiler technology and type inferencing, and was still amazingly clear. The talk gave me the nice illusion I actually understand how Flows works. It also makes me want to try Flow again and integrate it into my development tool chain.

Flow is a general JavaScript tool. It comes out of Facebook but is not directly connected to React at all, even more so than something like GraphQL. I really appreciated that the organizers included talks like this and mixed things up.

Watch "A Deepdive into Flow"

Cheng Lou: On the Spectrum of Abstraction

This talk, which isn't about React, really stood out for me, and from what I heard also resonated with others at the conference. It tied neatly into the themes Dan Abramov already set up in his opening talk about Redux. Dan told me later this was not actually coordinated. The ideas are just in the air, and this speaks for the thoughtfulness of the React community.

Cheng Lou's talk was a very high level talk about the benefits and the costs of abstraction. This is something I care about a lot as a developer: how do I avoid over-engineering and under-engineering (I've written about it before), and solve problems at the right level? Software has many forces on many levels pulling at it, from end-users to low-level details, and how do you balance out these forces? Engineering is so much about dealing with tradeoffs. How do you even communicate about this?

The next day I had an interesting chat with Cheng Lou about his talk, where he discussed various things he had to cut out of his talk so it wouldn't be too long. He also mentioned Up and Down the Ladder of Abstraction by Bret Victor, so that is now on my reading list.

I highly recommend this talk for anyone interested in these topics.

Watch "On the Spectrum of Abstraction"

Preethi Kasireddy: Going from 0 to full-time software engineer in 6 months

This was a 5 minute lightning talk with a personal story: how overwhelming software development is to a newcomer and how it can nonetheless be learned. During the talk I was sitting next to someone who was relatively new to software development himself and I could see how much this talk resonated with him.

Preethi Kasireddy also encouraged more experienced developers to mentor newcomers. I've found myself that mentoring doesn't have to take a lot of time and can still be hugely appreciated. It's fun to do as well.

A new developer is often insecure as there are just so many things to grasp, and experienced developers seem to know so much. Ironically I sometimes feel insecure as an older, more experienced developer as well, when I see people like Preethi learn software development as quickly as they do. I certainly took more time to get where they are.

But I'm old enough to have gotten used to intimidatingly smart younger people too. I can keep up. The Internet overall helps with learning: the resources on the Internet for a new developer may be overwhelming, but they are also of tremendous value. Preethi called for more intermediate-level resources. I am not sure this series I wrote counts; I suspect Preethi is beyond it, but perhaps others will enjoy it.

(Video not up yet! I'll update this post when it is.)

Jonas Gebhardt: Evolving the Visual Programming Environment with React

This was another one of those non-React talks I really appreciated. It is related to React as it is both inspired by functional programming patterns and component-based design, but it's really about something else: a UI to construct programs by connecting boxes with arrows.

There are many of these around. Because these don't seem to ever enter the daily life of a programmer, I tend to be skeptical about them.

But Jonas Gebhardt acknowledged the prior art, and the approach he described is pragmatic. An open world approach in the web browser, unlike many of the "we are the world" sandbox implementations from the past. Annotated React components can serve as the building blocks. He even sketched out an idea on how to connect UI input and output to custom user interfaces in the end.

So I came away less skeptical. This approach has potential and I'd like to see more.

Watch "Evolving the Visual Programming Environment with React"

Bonnie Eisenman: React Native Retrospective

I really like retrospectives. This was an excellent talk about the history of React Native over the course of the last year and a half. React Native is the technology that lets you use JavaScript and React to develop native iPhone and Android apps. Bonnie Eisenman also wrote a book about it.

React Native is a potential game changer to me as it lets people like me use our deep web development experience to build phone apps. The talk made me excited to go and play with React Native, and I'm sure I wasn't the only one. In a chat afterwards, Bonnie confirmed that was a goal of her talk, so mission accomplished!

Watch "React Native Retrospective"

Phil Holden: subdivide and redux-swarmlog

Phil Holden gave a 5 minute lightning talk, but please give him more space next time. He discussed Subdivide, an advanced split pane layout system for React, and then also discussed another mind-blowing topic: using WebRTC to create a peer to peer network between multiple Redux frontends, so that they share actions. This lets users share data without a server being around. This he packaged as a library in a package called redux-swarmlog.

I've been thinking about peer to peer serverless web applications for some years as I believe they have the potential to change the web, and Phil's talk really reignited that interest. Peer to peer is hard, but the technology is improving. Later that day, I had the pleasure of having a brief chat with Phil about such wild topics. Thanks Phil for the inspiration!

(Video not up yet! I'll update this post when it is.)

Andrew Clark: Recomposing your React application

Andrew Clark is Internet-famous to me, as he created Flummox, the Flux state management framework I used before switching to Redux (Andrew in fact co-created Redux). In this talk he discusses recompose, a library he wrote that helps you do sophisticated things with pure, stateless function components in React. I need to play with it and see whether it fits in my React toolbox. Andrew also described the interesting techniques recompose uses to help reduce the overhead of small composed functions -- this highlights the properties you gain when you stick to the pure function constraint.

Watch "Recomposing your React application"

Jafar Husain: Falcor: One Model Everywhere

When multiple development teams have a similar idea at about the same time, that may be a sign the idea is a good one. This happened to me when I came up with a client-side web framework few years ago, thought I was onto something new, and then Backbone emerged, followed by many others.

Jafar Husain in this well-done talk described how Falcor and GraphQL were a similar solution to similar problems. Both Falcor and GraphQL let the client be in control of what data it demands from the server. He then highlighted the differences between Falcor and GraphQL, where he contrasted Falcor's more lightweight approach to GraphQL's more powerful but involved focus on schemas. It's tradeoffs again: which fits best depends on your use cases and team.

Watch "Falcor: One Model Everywhere"

Laney Kuenzel & Lee Byron: GraphQL Future

This was a wide-ranging talk that went into various issues that GraphQL team at Facebook is trying to solve, mostly centered about the need to receive some form of immediate update when state on the server changes. Laney and Lee presented various solutions in a various states of readiness, from mostly untested ideas to stuff that is already deployed in production at Facebook. Very interesting in you're interested in GraphQL at all, and also if you're interested in how smart people tackle problems.

Watch "GraphQL Future

Constructive feedback

In my blog post last year I was clear I enjoyed the conference a lot, but also engaged in a little bit of constructive criticism. I don't presume that the React Europe organizers directly responded to my feedback, but let's see how they did anyway and give a bit more feedback here. My intent with this feedback is to do my bit to make a great conference even better.

Back home

The train ride back home on Saturday morning was as it should: uneventful. Left the hotel around 9 am the morning, was back home around 2:30 pm. I came home tired but inspired, as it should be after a good conference. Thanks so much to the organizers and speakers for the experience! I hope you have enjoyed my little contribution.