Reg, Now With More Generic!

A month ago I first announced the Reg library for Python. After posting it I got a comment asking why I didn't just use simplegeneric or PEP 443? One important reason is that I wasn't aware them. The other reason is that I want some special features. But it did get me thinking. Thanks commenter yesvee!

PEP 443

So what's PEP 443? It proposes functionality that lets you create generic functions that dispatch on their first argument (single dispatch):

from functools import singledispatch
@singledispatch
def fun(arg):
    return "The default"

@fun.register(int)
def int_fun(arg):
    return "Int argument"

@fun.register(list)
def list_fun(arg)
    return "List argument"

When you now call fun() it will dispatch to different implementations dependent on the type of the first argument:

>>> fun(1)
"Int argument"

>>> fun([])
"List argument"

>>> fun('foo')
"The default"

The advantage: dispatch on type, which is useful to extend classes from the outside, but just functions to the end user.

The Zope Component Architecture

Reg is the Zope Component Architecture (ZCA) reimagined. The ZCA is interface-centric. Objects are looked up by interface, and an interface looks a lot like a class. Here's what it looked like with Reg (which has a simpler registration API):

import reg

class IFunctionality(reg.Interface)
    pass

def int_fun(arg):
    return "Int argument"

def list_fun(arg)
    return "List argument"

r = reg.Registry()
r.register(IFunctionality, [int], int_fun)
r.register(IFunctionality, [list], list_fun)

And then this is how you'd use it:

>>> IFunctionality.adapt(1)
"Int argument"
>>> IFunctionality.adapt("List argument")
"List argument"

This API was in fact slightly less good as you could accomplish with the ZCA itself:

>>> IFunctionality(1)
"Int argument"
>>> IFunctionality("List argument")
"List argument"

This is because I wanted to keep Reg simpler, and the ZCA has to do quite a lot of wizardy to make the latter functionality work. But I actually wanted it in Reg. And that starts to look suspiciously like a generic function implementation.

So I started thinking about rewriting Reg so it presented itself much like generic functions as in PEP 443, but with special Reg sauce added. I talked it over with a few people at PyCon DE too. Thanks for helping me clarify my thinking!

All New Generic Reg

So last week I refactored Reg so be generic function based. Gone is lookup by interface, in is generic function calling. Here's what it looks like with Reg today:

import reg

@reg.generic
def fun(arg):
    return "The default"

def int_fun(arg):
    return "Int argument"

def list_fun(arg)
    return "List argument"

r = reg.Registry()
r.register(fun, [int], int_fun)
r.register(fun, [list], list_fun)

And using it looks like a normal function call, just like in PEP 443:

>>> fun(1)
"Int argument"

Reg doesn't deal with interfaces anymore. Interfaces tend to scare off Python developers as they're a new concept, and I realize they just weren't necessary anymore. Now people using your Reg-based APIs just use functions, like with any other API.

Why not use the decorator approach to register generic function implementations? Because Reg allows more than one registry. While writing this I realize that's not a good answer: I can provide this in Reg if I make the decorator a method on Registry. Stay tuned!

But anyway, Morepath, the web framework built on Reg that I'm working on, already supplies a decorator like this that integrates with its configuration engine.

Why Reg instead of PEP 443?

So what's the special Reg sauce? Why not just use PEP 443? Reg offers a few facilities that I needed in Morepath:

  • Multiple dispatch. PEP 443 offers single dispatch on the first argument. Reg implements multiple dispatch, on multiple arguments. This is handy when you're implementing a web framework such as Morepath, which dispatches on request and model.

  • You can isolate one set of generic implementation function implementations from another one in the same runtime. This is because Reg allows more than one registry in which these can be registered.

  • Registries can also be composed together. This allows a core framework to register some implementations and application code some more.

  • The registry to use can be implicit, or you can be totally explicit about what registry to use when you call a generic function.

  • Get all the generic functions that would apply to arguments. Register other things than functions and still look them up. I use these advanced use cases in Morepath in a few places.

  • Predicate support. The Pyramid web framework lets you register views not only by type, but also by other things, such as HTTP request method (GET, POST, etc). I've put support for this kind of thing in Reg itself, and I use it in Morepath. This bit is still underdocumented, please stay tuned!

Luckily you don't have to commit to Reg. You could implement your API using the more modest PEP 443 implementation first, and then switch to Reg should you discover its features would be helpful to you. You'd have to change how you register the implementations of generic functions, but not the API itself.

The New Reg Docs

I've updated and extended the Reg docs. Read the usage docs and check out the all new API documentation!

This should help you get started using Reg.

Road to Morepath

Reg is foundational to Morepath, the up and coming Python micro web framework with super powers. I've adjusted Morepath to use the new generic Reg. The morepath code is online. Stay tuned for docs!

The New Zope as a Web Framework

Using the New Zope

In 2004 the new Zope had started make it back into the old Zope through the Five project. In 2005 I also had the opportunity to use the new Zope in its pure, unadulterated form in a new project.

The overall experience of coding in the promised land of the new Zope was mixed. On the one hand, if you wanted to write Python code it was clearly a much cleaner environment than the old Zope. The new Zope also was great at supporting all kinds of forms of extensibility in a clean way through its component and configuration systems.

On the other hand, I found myself having to edit too many files at once to get things done. You'd have the actual Python modules that contain the application code, a Python module that described the API of the various objects in your system (interfaces.py) and a XML file that described how these objects plugged into the system and with each other (configure.zcml).

I'm a lazy developer. I don't want to edit three files where I can edit only one.

Why was this problem not as big in the old Zope after we added the new Zope technologies to it? An important reason for this was the new Zope's security system, not shared by the old Zope.

The new Zope had a transparent proxy system that made sure you could only access an attribute on a Python object if you had permission for it, otherwise you'd get an AttributeError. And you'd give yourself permission by adding it to the configure.zcml (or the interfaces.py, where it could then be picked up from). If you forgot, you'd be staring at an AttributeError.

I found this pretty hard to use myself. All together the new Zope slowed down the feedback cycle for me, making it hard for me to stay in the flow when coding.

Let's make it clear: I also really appreciated the qualities of the new Zope. But I thought that these complexities made life harder for myself and also would hinder the adoption of the new Zope by other developers. It was different enough conceptually already, and its verbosity could be quite intimidating to many.

Rise of the web frameworks

In 2005 the shape of web development was undergoing a significant change. While there had been web frameworks for years, they were becoming more popular, and the concept was finally coming into a more clear focus. This was in a large part due to the rise of Ruby on Rails. In the Python world Django and TurboGears were making a big splash.

Ruby on Rails attracted developers from Java who were really happy to find reduced complexity of configuration. And here we were with the new Zope adding more of it. How was the new Zope going to compete as a web framework?

I saw these problems, but many of the other developers of the new Zope did not share my concerns. Different developers have different strengths, and the developers of the new Zope were pretty smart ones. Many of them had no problem editing 3 files at once and adding lots of configuration XML to configure.zcml. They could keep all of this in their heads and remain productive. Or, as they argued, doing it this way gained you something too and they didn't want to lose that.

It was my problem. So how was I going to fix this? In 2006, I decided to go around the center, and start a new project to try to turn the new Zope into a modern web framework: Grok.

This blog entry is a part of a series on Zope and my involvement with it. Previous. Next.

Jim Fulton, Zope Architect

Introduction

I hope to have shown that Zope in 1998 was massively creative: traversal based object publishing, a web UI, a pluggable architecture, a NoSQL object database and acquisition were all in the original Zope. When Zope renewed itself in the early 2000s we added a component system, an explicit configuration system, and many other things.

Much of the credit for all this innovation goes to the chief architect of Zope, Jim Fulton. Many other people contributed, of course, but that doesn't take away from Jim's creativity -- he helped others be creative too. Creativity is very important to me in my life as a developer, and I respect it immensely in him.

Meeting Jim

In the year 2000 I went to the International Python Conference in Washington DC. This was the first time I'd been to any Python conference. Python was a smaller community then. People knew each other. They even knew me, even though I hadn't done much yet; my main contribution to Python were some silly posts on the Python newsgroup, and some activity surrounding Zope.

That's also where I met Jim for the first time. Zope was a big happening thing in the Python community at the time; the attendance to the Python conference had grown enormously thanks to Zope. Many people wanted to talk to Jim as a result. Still it was easy to have a chat with him.

The first time I really got to work with Jim was over the phone. I had created a form library for Zope called Formulator, and he phoned me to talk about how we could reimagine this for the new Zope. It felt special to have someone so central to Zope to call me.

Out these conversations eventually came the Python schema system zope.schema and the various form libraries built on top of it. I only contributed a little bit of code to these at best -- Jim wrote much of it. Still I was proud that some of my ideas were taken up and put in the new Zope, thanks to Jim.

Jim at Sprints

The new Zope in the heady days of the early 2000s was developed using sprints. A sprint was a few days when people would come together somewhere with a laptop and an internet connection and work on things together that are less immediately pressing than the urgent needs of the business day. A time to take a step back, enjoy each other's company, have fun and be creative. I've sprinted at conference locations and in offices, but also in a castle, in a cabin in the forest, and on a snow-covered mountain, and at home. Tres Seaver wrote an article on the origin of sprinting that is worth a read.

The first time I got to work with Jim was in Berlin at tiny sprint (just 4 or 5 people) before a Zope meeting there, in 2001. I forgot what we worked on - I think we discussed forms again too.

Jim was everywhere during those days. I saw him at many sprints and conferences and had the pleasure of working with him often. He was friendly, smart, and engaging. He helped channel lots of people's creativity energies into the new Zope project. I learned a lot from my interactions with Jim, and enjoyed them tremendously.

Jim and me

In the years afterwards Zope was moving less quickly. Jim had less time for Zope, as there are of course still customer projects to work on. Jim's reduced presence on the Zope project was sorely felt; he was missed.

I still saw Jim on occassion at conferences, and worked with him for a while when we were both board members of the Zope Foundation.

Jim is an inspiration to me, and a massive influence on the way I think about software. We learned from each other -- me more from him, as he was the more experienced developer, but he also listened to me. Thank you, Jim.

We didn't always agree -- I wanted more automatic XML-free configuration in Grok, Jim liked the original explicit ZCML; Jim prefers widgets that hide HTML on the client-side, I like client-side HTML template languages. And so on. So sometimes we argued, but we respected each other.

After Zope

In 2010 Jim and I last saw each other at the Zope Summit. At the end, we had a stupid fight about a stupid thing -- a result primarily, I think, of my frustrations with Zope, which didn't help my mood. I'm a passionate software developer, and seeing one of my passions in trouble made me gloomy. I was hurt by that fight. I think we both got hurt.

We didn't have a chance to work it out then. I had to rush for the train back home and looked for Jim to say goodbye, but missed him. Then we found ourselves on different continents again.

Afterwards, Jim tried to contact me to talk on the phone, but we didn't get around to it. Perhaps I wasn't ready for it. We both emailed each other again several times afterwards, but we still didn't get to talk. Life is distracting. I blame myself for not trying harder.

I'll try to call him tonight. I will keep on trying this time.

This blog entry is a part of a series on Zope and my involvement with it. Previous. Next.

Renewing Zope

Introduction

The Zope community in late 2001 had been around for a few years already, almost an eternity in web time back then. Still, we weren't afraid to do bold new things. Perhaps too bold, as we'll see.

Zope Rewrite

We were learning new, better ways to do web development. Led by Jim Fulton, lead architect of Zope, we set out to rewrite Zope, fix mistakes, explore new innovations, make Zope cleaner, easier, and more powerful. I say we now, as by that time I and many others were fully engaged in this project to build its next generation.

One way that this engagement was fostered was by sprints. The Zope community pioneered the sprints now widely adopted in the Python community; we would gather together in real life to work on the new Zope. At such a sprint you would often find yourself working with Jim Fulton himself, so they were quite the draw.

In the new Zope, we would have more powerful and more clean extensibility and reuse mechanisms. Zope components would be developed as normal Python packages. We would do away with the weird magic of acquisition. We would make the glueing together of components into applications an explicit, coherent notion instead of a collection of ad-hoc APIs.

Central to this was the Zope Component Architecture, which introduced the notion of an explicit interface, along with a way to register and look up components that would enrich models (we called them views and adapters). It could also fulfill the role of a service locator, and provided the basis for an event system.

Besides sprint-driven development we were also early adopters of strongly test-driven development.

This exciting renewal of Zope was done as a complete rewrite. That's scary; rewrites have a habit of never being completed, and you risk the second system effect. But in many ways we succeeded -- we had a capable, very powerful and much cleaner web framework in the new Zope. (There were flaws, as it was not easier, just difficult in new ways, but I'll go into that later.)

So we had an exciting new Zope. We had a practical problem, however. Almost all of the real work done with Zope was done with the old Zope. This new, better Zope was held up as the future, but was tantalizingly out of reach for those of us with large codebases built on top of the old Zope.

Five: Redefining Zope

I've written a lot of code over the years, some of it generally useful, most of it not. But I believe some of my biggest contributions to Zope were not in the form of code, but projects that helped redefine what Zope was all about, giving people new ways to use it, and giving the project new directions.

The first such redefinition I attempted was Five, a name I came up with by adding together the version numbers of the old Zope (2) and the new (3). It was cute, and stuck in people's mind, and such things can matter.

The new Zope was out of reach to users of the old Zope. The community had been led to believe that the new Zope would be the next Zope. We thought that at some point in the nebulous future there would be a transition to the next system, some form of compatibility, some way to transfer code from the old to the new system. How this would work was left unspecified.

I grew doubtful about whether it would work at all. It was in the future and seemed to stay there. And I wanted to use the new stuff.

So in 2004 I took a different approach. I worked out a way to add the code of the new Zope to the old, using the new Zope as a library, extending the old Zope: Five. Through Five, the new ways of developing became available in the old system. We could use the new Zope, right then and there, in the context of the old Zope. We could use the better ways of doing things without giving up our existing codebases.

By 2006 the Five technology was merged into the core of the old Zope: Five became an integral part of it. Finally in 2009 we fixed the confusing, now deceitful naming, which was still suggesting after all these years that Zope 3, the new Zope, was the next Zope, which it was not. We let the new Zope evaporate into a puff of libraries. The next Zope was no more; there was just Zope. Its code lives on, in the old Zope, and various successor frameworks.

The Wisdom of Hindsight

Was the Five solution ideal? It was not. It made the old Zope more complex. While code written in the new way was more clean and maintainable, we didn't gain all of the cleanliness, because the old stuff was still there. The transition from old to new ways of doing things would be a very long one. The Plone developers, for instance, adopted the new ways many years ago, but are only now finally able to remove some of the old ways from the next version of their CMS.

Should we have done things differently? In retrospect, we should perhaps not have created the new Zope at all. Instead we could have embarked on a project to renew the old Zope from within, directly. That would have been very difficult as well, though. We might have missed out on something too. A clean slate, after all, lets you take a step back from what you're doing, and helps you think things through more clearly. We benefited from that as well.

The new Zope project was flawed, and the approach to merge it into the old Zope generated difficulty as well as opportunity. But through these projects Zope did continue to evolve, renew itself, and stay relevant.

This blog entry is a part of a series on Zope and my involvement with it. Previous. Next.

Object Publishing

Introduction

A final early Zope innovation that I want to talk about is object publishing. Object publishing is one of those ideas that seemed weird at the time. In fact, in its execution Zope would still be considered weird by many today. But in an important way the idea of publishing code on URLs is now absolutely the norm.

The Web in 1998

Let's go back to 1998, when Zope was born. Let's look at a URL:

http://example.com/dir/file.html

If you saw this URL in 1998, you knew that /dir/file.html refers to a file on the filesystem on a web server somewhere. That is, there is an actual directory dir, and an actual file file.html. Where the root of the website would be on the filesystem was up to the configuration of the web server, but once you're inside this root paths in URLs get translated to paths in the filesystem.

A URL could refer to a static HTML file as above. You could also build dynamic web applications. You'd do this by putting files in your web server that were CGI scripts. When someone accesses the file, the script would be executed and produce HTML content. So script.cgi or script.pl or script.py or something.

Zope's way

Zope was different. The path in a URL in Zope did not map to a file in the filesystem anymore, but to a piece of Python code. Zope was one of the first systems to do so. With Zope, the path was interpreted as a series of steps to a Python method. So, the path:

/a/b/c

would translate to:

root['a']['b']['c']()

or possibly:

root.a.b.c()

or a combination thereof. The concept is called object publishing and the process by which the right method to call is found is called traversal, as an object graph is traversed.

URLs served from a Zope server looked clean, without complicated URL parameters to encode state, and without file extensions. The URLs looked so clean and unadorned that in fact they sometimes confused people who were expecting a .cgi or .pl or .php or .asp or .cfm. Bizarrely enough for a little while we had an effort to bring them back so people would be less confused.

The Web today

A variation of Zope's approach is now the norm. Most web development frameworks in Python and other languages don't map URL paths to files anymore, but to code. If you use a routing system like the one in Django, for instance, a URL is not resolved to a file, but to a function that typically queries an object from the database and then represents it as HTML or JSON.

While the mapping of URL paths to code is now the norm, Zope's particular way of doing traversal is still weird to many. Zope's way fits well with the ZODB, as the ZODB is natively an object graph that can be traversed. Zope's way doesn't fit other database structures as well.

Zope's approach is weird, but it's still worth examining. A benefit of this traversing approach is that it gives URLs to models, not only views or controllers; making explicit something that is implicit in more modern routing-to-view web frameworks. This means that not only can a URL be resolved to a model (and only then a view/controller), but you can also construct a URL with just a model. This is very powerful and, in the RESTful, client-side web of today more relevant than ever. 1

This blog entry is a part of a series on Zope and my involvement with it. Previous. Next.

1

I myself have explored combining routing and traversal in my older Traject library and the so-new-oh-no-I-gotta-write-the-docs Morepath web framework.

The Weirdness of Zope

Besides the use of Python, what were some of the other weirdnesses of early Zope?

Web UI

Zope featured a web-based user interface, which was rare thing at the time. The user interface was a hybrid between a CMS and a development environment. Using the web interface, people were able to construct and configure applications using HTML templates, scripts and SQL statements.

The Zope UI was an interesting experiment. It dramatically lowered the entry barrier for web development. It attracted a lot of people to Zope. This helped the Zope community grow, but also led to problems, as the next barrier to real Python web development was way too high.

In the end neither the CMS nor the development environment were sufficient. The Zope community moved on to build its own CMSes. I helped create one called Silva, but the really big one is Plone. Both are still around. Plone has always had a very vibrant community, something I witnessed again first hand last year at the Plone conference.

Extensibility

Although the APIs were sometimes weird, you could extend Zope with new components written in Python; just modules in files. Over time the Zope community -- at least the developers who could -- slowly moved away from the web user interface to write more and more code the Python way as Zope extensions. Writing code through a web user interface was too cumbersome -- it had a deliberately crippled Python for security reasons, and the code you wrote that way was hard to maintain too; integration with a version control system was difficult, for instance.

A rich ecosystem of new Zope components written in Python emerged from the Zope community. One of the first pieces of open source software that I wrote was a form component called Formulator that you could use to construct web forms using the Zope web user interface. People still use Formulator to this day.

Extensibility remained a focus of the Zope project throughout its history. In a future post I'll talk about some of the extensibility mechanisms that were introduced later.

ZODB

Another innovation of Zope was the Zope Object Database (ZODB). The ZODB is a Python object database, offering near transparent persistence for Python objects. Over time it grew clustering support and multiple storage backends. It is nearly magic: you write your Python class and derive from the Persistent base class, and as long as you attach instances of that class to a persistent root, they will continue to exist in the database automatically.

The ZODB was actually sometimes problem commercially -- try to explain to a customer that they should trust their data not to their precious SQL database but to this weird Python database. Imaging doing this years before the NoSQL movement emerged -- if you said "database" at the time, people thought relational database with SQL. NoSQL has certainly made it a lot easier now.

The ZODB is still around, and is very useful, though it does have some drawbacks. Its strength, the integration with the Python language, is also a source of some of its weaknesses. The ZODB is almost too intimate with the Python language, making it harder to use data in it outside of Python.

The intimacy between database and Python classes also makes it more cumbersome to change the structure of Python code -- changing a Python class can easily break its existing instances in the database if you aren't careful.

The ZODB also doesn't support a built-in query engine, though it does offer the tools to build one. I was one of the people who did (hurry.query), but a query engine is really one of those things you want to be taken care of by dedicated maintainers.

Acquisition

Zope also innovated on something that is too weird to go into much detail about here: acquisition. Store an object in an attribute of another, and it will gain all the attributes and method of its container. While sometimes useful it was a language innovation that went too far, and had too many unforeseen interactions. I see acquisition as an educational but failed experiment. If you want to read the documentation, or if you have a strange desire to shoot yourself in the foot, Acquisition is still available as a library on PyPI today.

This blog entry is a part of a series on Zope and my involvement with it. Previous. Next.

The Rise of Zope

Zope at its birth in 1998 was neither a content management system nor a web framework. Zope was born too early; before we understood either very well.

Zope was like a futuristic space alien from another dimension. In many ways it was advanced far beyond its time. It was weird. Some of its weird ideas didn't work out, but there are also ideas in there that are worthy of further consideration even today. Some of its innovations were weird at the time but have now become the new normal.

Zope during its early days was big news, with a big community and a big codebase. It had been a commercial product at first, and then was open sourced by its creators -- a move that seems unremarkable now, but was so unprecedented at the time it gained wide-spread media attention.

One of its then-weird ideas that turned out to be a good idea was to write Zope in Python. Python at the time was still an obscure programming language. Zope was enormously influential on the Python community. Zope pushed the Python language beyond where it had been pushed before. People flocked to Zope, and some went on to Python. Zope caused the attendance of Python conferences to swell. In Europe, the EuroPython series of conferences was kicked off, in part, by people who used Zope.

This caused some friction, to be sure. Even today, many years later, passions can still be raised by Zope among Python developers. The Python language community puts an emphasis on clarity, and Zope was using Python in strange new ways, not always clear or clean.

This blog entry is a part of a series on Zope and my involvement with it. The first entry. Next.

My Exit from Zope

Last week I gave a keynote speech at PyCon DE. It was about a number of topics, too many topics, in fact (I went over my allotted time, something I regret). One topic was described by a friend as me "coming out of the closet" about my exit from Zope.

So let's come out of the closet on my blog too. A few years ago, I stopped my involvement with Zope. I gave up hope about Zope becoming more creative again as a web framework, and stopped my futile efforts to push it in that direction. For me, the days of Zope are over. Leaving Zope was difficult for me -- I am a passionate software developer and I had been involved with Zope since its inception as an open source project in 1998.

In fact, I still use Zope, through Grok, though Grok is now in maintenance mode as well. I use Zope a lot conceptually, when I approach new problem domains like programming the web browser (Obviel) or creating a new server-side web framework (morepath). The lessons and patterns I learned from Zope will never go away. Though I don't work on it anymore, Zope is still on my mind.

In a few [edit: more than a few!] upcoming blog entries I will talk a bit about Zope and my involvement in it. Perhaps it will be of interest to some.

I will post the entries in this series here:

Reg: Component Architecture Reimagined

Reg is a new Python library for registering and looking up objects. It provides clever registries to let you build extensible applications, libraries and frameworks.

Why should you want to use Reg at all? The best way to find out is to read the Reg docs.

The documentation is a work in progress, and especially API documentation is still lacking (read the interfaces.py module in the source). It should however give you a good overview of the basic usage of Reg.

The Reg code is here on github:

https://github.com/morepath/reg

The project is currently in pre-alpha, with no release yet. The documented behavior is however expected to be stable.

Let me know what you think!

Background

Reg is heavily inspired by the Zope Component Architecture (ZCA) as implemented in zope.interface and zope.component, though shares no code in common with Reg, and no knowledge of the ZCA is needed in order to use Reg. The ZCA is a venerable implementation, going back more than a decade. Use cases for the ZCA have evolved, some added, some dropped away, and we have a lot more experience. Learning from this, Reg provides many of the same features but with a lean, modern API and implementation.

I must mention Chris McDonough, creator of the Pyramid web framework in this - he's been leading by example reimplementing (reimagining) Zope concepts in a leaner, clearer, better way for years now. I was originally skeptical of this approach, but history has proved him right. Kudos, Chris! I've learned a lot from you.

I've been working on Reg for some years now, on and off, in different incarnations. Some Plone developers might remember my Crom lightning talk back at the Plone Conference 2012 in Arnhem - Reg is a further evolution of this. See the history section of the documentation for more information.

Road to Morepath

Reg is foundational to Morepath, the up and coming Python micro web framework with super powers. Stay tuned!

JSConf EU 2013 impressions

Yesterday I returned from JSConf EU in Berlin. I enjoyed the conference and I'll share some of my impressions here. It's not going to be a complete report; I met a lot of people and had a lot of conversations.

This was my first JSConf. I've been to a lot of Python conferences in the past. I was a bit nervous as for the first time in years I was going to a conference where I didn't really know anybody, but that is also an important reason why I was going - since a large part of what I do now in web development is JavaScript, I wanted to engage with its community.

Travel to Berlin

It turns out it's really easy to travel to Berlin from where I live in the Netherlands; go to the small (and thus less stressful) airport Eindhoven nearby by train, fly for an hour, and land in Berlin.

Since I had a bad experience with delayed check-in luggage earlier this year and it was just a weekend trip this time I decided to I travel very light; just my laptop backpack with clothes stuffed in.

The plane was very full, many with maximum-size carry-on luggage. I was in the plane early so had my backpack stowed already. Until a lady who absolutely did not want her suitcase to be put in the hold pulled out my backpack and tried to fit it into a space elsewhere too small for it, and then gave it to me, putting her suitcase in the now open space. Problem solved!

This attempt to saddle me with my backpack and her problem wasn't really appreciated by me, and I had to insist she take her suitcase out again. Later on she was still arguing with the flight crew.

In the taxi ride from the airport to the hotel I had a little chat with the taxi driver, in German. I understand German pretty well, but my spoken German is pretty broken. I have no idea how it came up, but suddenly the driver asked:

"So do you think the Americans landed on the moon?"

Being a fan of scientific skepticism had prepared me for this sudden question, and I explained that they had left a mirror on the moon that you could bounce lasers off, and that faking the moon landings would have had to have been a conspiracy of monumental proportions with professionally-interested-in-the-truth scientists and engineers and the Soviet Union both being involved.

"So they did land on the moon then," he said quietly, seeming to marvel at the thought. And it is a marvel.

My broken German had been up to the job, which amazes me still. Next he asked me what programming languages I used!

Talks: overall

The choice of talks was interesting; many of the talks were not so much about JavaScript technology and discussed other human topics associated with it -- art, ethics, creativity, design, usability. This led to a mixture that helped the whole being more inspirational than if all the talks had just been about tech all the time.

I will discuss some of the talks that stood out of me. This is necessarily going to be selective - there were plenty of great talks I'm not going to mention, and many more that I missed altogether as there were two parallel tracks.

I've tried to put in links to slides where I could find them. Videos should also eventually appear on the JSConf EU youtube channel.

Talks and People: Day 1

Parsing, Compiling and Static Metaprogramming - Patrick Dubroy

Patrick discussed some basic principles of parsing and discussed two projects:

  • esprima, a JavaScript parsing library that I had become aware of because of my explorations of the Polymer project, which uses it in its implementation of the MDV templating system. It was very helpful to see Patrick present a few examples of how it can be used.

  • PEG.js, a parser generator (along the lines of yacc or bison) for JavaScript. I hadn't been really aware of this project before, but from the examples it looks like it's really easy to construct parsers with it. I'll enjoy adding this as a tool to my toolbox.

I had a few interesting conversations with Patrick afterward throughout the conference. Thanks!

slides

A comparison of the two-way binding in AngularJS, EmberJS and KnockoutJS - Marius Gundersen

This talk was of particular interest to me because of my interest in client-side web framework design in general, and my interest in adding two-way binding to Obviel in particular.

He compared approaches in various ways:

  • Observable properties (versus dirty checking)

  • Asynchronous response to changes

  • Computed properties

  • Dynamic dependencies (recomputation only happening when needed)

  • Performance (each has different tradeoffs)

Marius is Norwegian by the way; I kept meeting friendly Norwegians at this conference - they seemed particularly well represented.

slides

Security talks

I saw three security related talks at this conference. I need to read some of the references the speakers give in their slides, and maybe you should too:

  • Towards a post-XSS world - Mike West (slides)

  • JavaScript crypto. Ugly ducking with good reason? - Martin Boßlet (slides)

  • Security First - Adam Baldwin (can't find the slides yet) (this was on day 2 actually)

UXing without a UXer - Chrissy Welsh

I had the pleasure to chat with Chrissy during the sunday night party. It reminded me that again how much usability in UIs is related to doing good API and protocol design (and also language design), which after all is also in a large part a usability exercise.

Chrissy's text to hang on the wall: "what is the one thing you want your product to do well" is also a relevant question to library and framework authors. Even if we're handling multiple use cases.

When doing API design I benefit from years of experience and also from being one of the intended users myself, a benefit that I don't have for many of the application UIs that I helped to develop. Chrissy's tips help with that.

slides

The web experience in the autistic spectrum - Natalia Berdys

This talk made the case for having people with Asperger's syndrome do your web site usability testing for you, or at least to think about them when you design a web site. The usability issues that confuse a person with Asperger's are in fact also usability issues for everybody else; it can just be experienced more intensely. This talk was a nice change of perspective on a topic that everybody is familiar with in one way or another: how usable web sites are.

(can't find the slides yet)

/be

There was a talk somewhat mysteriously marked /be in the schedule of day 1. This turned out to be a talk by Brendan Eich, creator of JavaScript!

He very, very rapidly discussed various upcoming new features in JavaScript. High time to have a module system and standard sugar for prototype-based class definition!

He followed it up with an asm.js demo. This involves running various first-person C++ 3d shooters within the web browser. Even though I was already aware of this before, it's still a totally amazing demonstration.

Besides making for cool demos and running C++ code in your browser, I expect the asm.js technology to have an enormous impact on client-side development in the longer term. One thing that it might do is open up the browser platform for other programming languages besides JavaScript; another thing it might do is to let browsers move more and more of their implementation from C++ into JavaScript - how much native code will be left in the long term?

slides

Talks and People: Day 2

Plight of the butterfly - everything you wanted to know about Object.observe() - Addy Osmani

Day two for me started with Addy Osmani's talk about Object.observe(), one of the most exciting technologies coming to JavaScript.

Object.observe() provides a way to observe changes in JavaScript objects from the outside - a technology that in the future will provide the foundation for a wide range of technologies such as two-way binding to templates, persistence technologies (Addy demonstrated a simple JavaScript object database) and server interaction (a topic I'm also very interested in).

I was vaguely familiar with Addy's name from my explorations of the Polymer project, but I realize now he also wrote several books and is behind the TodoMVC project. Addy's talk was a rapid-fire discussion of a lot of interesting aspects of Object.observe() that I certainly will want to refer back to in the future. I chatted with Addy about some framework design issues later on. I hope I was coherent at all, and I appreciate his time.

(can't find the slides yet)

Promises and Generators: control flow utopia - Forbes Lindesay

I'm finally quite familiar with promises; after having heard about them in Twisted years ago and never bothering to wrap my head around them, and again in MochiKit (a pioneering JavaScript client-side library), I finally started using them in the context of AJAX, using the implementation in jQuery.

I thought I was familiar with generators from Python, but I hadn't thought about their implications for async programming at all (even though I knew the Twisted folks had). During this talk I also realized I needed to understand yield expressions (as opposed to just the yield statement) better, in both Python and JavaScript!

A very enlightning talk that prodded me into the realization that I don't even understand all about generators in Python yet either. I have certainly been coming to async development the long away round!

slides

Acceptable in the 80s - Revisiting Microworlds - Jason Frame

This was an extremely interesting exploration of some important older ideas in UIs that we have lost somewhat since the 1980s.

Jason showed off a prototype desktop environment implemented in a web browser that allows you to construct ad-hoc applications from smaller components visually; he could throw an editor and a file loader together into an editor for instance by throwing some lines here and there. It involves building blocks that are larger than GUI widgets but smaller than applications that can be combined by non-programmers.

I think many of these ideas could also be applied to web application development as well, and I need to ponder this more.

(I'm not sure there were slides; it was a long demo, really. Wait for the video!)

Make world less shit. NOW. - Natalia Buckley

Natalia Buckley gave an essay-like (no slides, articulate) presentation (wonderfully titled) about the interactions between software and ethics. She discussed unintended consequences of unexplored human bias in software, and how software and culture interact both ways. It was welcome reminder to continue to examine my own biases and how they can affect the software I create, and others through it.

(no slides, I hope she will post an essay somewhere)

Building Live HTML and Omniscient Debuggers in Brackets - Kevin Dangoor & Peter Flynn

Kevin Dangoor and Peter Flynn came to JSConf EU in a relaxed mood: they were going to attend the conference and didn't need to do any presentations.

This turned out to be a misunderstanding. Preparing for a talk at the last minute is a rather stressful activity, but the talk went very well.

Brackets is web code editor project by Adobe with some very interesting features.

This is a tool that has a huge range of exciting technologies under the hood that I definitely need to explore: live updating of rendered HTML as you edit, an an omniscient debugger (a debugger that knows a lot more about your program than the state of it at your break point), type inferencing JavaScript to help with command-line completion, and a lot more. Very cool!

I had a number of very pleasant and also informative conversations with Kevin Dangoor at the conference. Kevin has been doing (and influencing) JavaScript for years but I know him from the old days as the creator of the TurboGears web framework in Python. After a chat with Kevin late sunday night I finally started grokking those generator expressions I mentioned before. And also thanks for the reference to the very interesting frb binding library! Also a shoutout to Peter Flynn, the other speaker; I had fun chatting during relax.js yesterday.

(no slides yet, perhaps wait for the video as it is also a demo)

I have a dreamcode: build apps, not backends - Alex Feyerke

Alex Feyerke gave a talk discussing technologies that let you develop rich-client JavaScript applications without having to develop the backend for them. It turns out there's a lot of activity in this space - turn-key generic backends that you can start building front-end apps on immediately. I definitely need to look into this more. (I've been wildly speculating for a while about going one step further: applications without servers at all, just peer to peer between browsers.)

slides

Stop breaking the web - Tom Dale

This was a wide-ranging and amusing talk about making URLs work for single-page web apps.

It also touched upon a few interesting client-side framework issues from the perspective of Ember. Each time I learn something about another client-side web framework I recognize parallel evolution in Obviel, which helps validate the concepts in Obviel. Everybody does it all slightly differently, with different tradeoffs. That's where it gets interesting and where we can learn from each other.

(can't find the slides yet)

Rethinking best practices - Pete Hunt

Pete Hunt (who it turns out remembers me from the Grok days long ago, woohoo!) had a very thought-provoking talk from the perspective of the new React client-side web framework. His arguments against the use of templating in web frameworks are interesting but do not yet convince me. His description of how React side-steps the whole data binding issue by doing something more like a real-time 3d engine does, in particular gaining efficiency by re-rendering components but only sending diffs into the real browser DOM were extremely thought-provoking.

I like to have my thoughts in this area provoked, so I want to thank Pete for doing so.

Incidentally, the Brackets environment does this diffing of the HTML DOM as well in an interesting case of parallel evolution.

1024+ seconds of JS wizardry - Martin Kleppe

I only caught the tail end of this entertaining talk, but I still want to mention it, as it was a lot of fun and I'm going to look up the video when it's posted to see the it all. Martin was trying to write write the shortest code possible and still do amusing and interesting things. It was "crazy", which is a compliment in this context. It already made it onto hackernews.

People First - Adam Brault

The closing keynote to close the conference was by Adam Brault; high-impact, emotional, sometimes, in fact, to me, uncomfortable in its emotion. JSConf EU seems to be wearing its heart on its sleeve a bit more than Python conferences are - different.

The talk was about putting people first (in small ways as in big ways); it's good to be reminded of that in a world of technology.

The talk made me think about the personal and the emotional and how it could fit in my own talks.

I shouldn't do these things, and certainly not disclose it here, but I pulled off a successful heckle ("hey, we're the ones that stayed here!") when Adam tried to share blame for bad stuff in early North America with present-day Europeans. Adam was very gracious about it.

(And in all seriousness, I agree Europeans actually do share in the blame anyway, as much as anyone in the present day can share in the blame for acts before they were born.)

(Historically speaking, us Europeans that stayed in Europe managed to screw up our own continent just fine; no moral superiority there.)

By the way, given that Adam Baldwin (Security First talk) and Adam Brault (People First talk) are both from the same company (&yet), how are we going to put both people and security first, guys? Are you both part of OM or something?

slides

JSConf as a community event

JSConf people really like to party; perhaps even more so than the Plone community does, if that is possible. By the time I arrived there already had been one party the night before. I appreciated that the noise level was kept low enough so that you could have a conversation with people. I also really appreciated the relax.js brunch on the morning after the conference.

JSConf organizers clearly understand that a conference is about more than just talks, but is also a way to create, sustain and celebrate community. Much appreciated.

One thing I must mention here is that I'm glad the conference was deliberately kept small. The rather difficult procedure to buy a ticket initially made me think that the attendance would be huge, but instead I found a cozy conference where it was easy to interact with people, including speakers, and find them again later. I was very happy about that; I've experienced a PyCon in the US with huge attendance before and it was just too hard to find people there. I heard that in recent years the attendance at EuroPython has grown. That makes me happy for the health of the Python community in Europe, but it's a bit regretful the conference might lose some of its charm due to the scale.

The crowd at JSConf was pretty diverse, but still younger than what I'm used to at Python conferences. I'm used to seeing a wider mix of ages around. I found myself being the guy dating myself by telling stories about the olden days. I didn't mind, and I've been telling stories since the olden days anyway. Turns out that I did know a few of the people at the conference, and others recognized me from the Python community. Nice, but it was easy to make new connections too.

I liked the length of the conference at 2 days. I was rather horrified to see that EuroPython is now 5 days of talks. That's just too much for me; after 3 days I'm totally done with talks. I think the JSConf approach of being very selective about talks is better.

There are two Python conference traditions that I missed: lightning talks and sprints.

Lightning talks are fast (5 minute) talks in very quick succession. Talks are submitted during the conference itself by the attendees, until there are not more slots; there is no selection process. If someone is boring that's okay, as they're gone in 5 minutes. It's a good way to get a quick message across, draw attention to a project, or just entertain. Doing a lightning talks track would probably not fit in a 2 day conference though.

A "sprint" is a slightly Python specific term for "hackathon". After a conference interested people can stay around for up to three days more and hack with others on open source projects they find interesting. It's a great way to learn more about a project and get to know other developers better.

Travel back

After the first Berlin taxi driver raised my expectations high, the taxi driver on the way back to the airport yesterday turned out to be very disappointing. He was on the headset in the phone speaking Turkish (I think) with a friend all the time.

One last story. I was in the Berlin Tegel airport restrooms doing my business. Then I heard someone call out "Is anybody there?" in English, with an American accent. I wasn't sure he was talking to me, though the place was otherwise empty. He called out again: "Can anybody help me?"

"What's going on?" I asked, wary.

"I'm stuck in this stall. Can you help me open it?"

I went to his stall and opened the door. I had expected that to be difficult, but it wasn't locked and just opened like any other door. Turns out that the door handle on the inside was missing, making the door, once closed, as good as locked for the guy inside.

"Thanks!"

"No problem."

"Really thanks, man, I would've been in there forever."

I smiled. "No problem."

Sometimes you can help by just opening a door for someone.