Random Rust Impressions

I have been using Rust for some years now for hobby projects. Recently I also had the opportunity to use it professionally for a while. My background is mostly in dynamically typed languages like Python and JavaScript, though I have played with various other languages over the years. I thought I'd share some of my impressions of Rust.

Learning Rust

I'd had my eye on Rust for years until I decided to give it more serious attention in 2019. I used it for various hobby projects; I created half of an artificial life simulation that never got anywhere. I played around with the Bevy game engine and the Rapier physics engine. I wrote a little driver that can send images to an e-paper display over USB. I wrote a plugin for the i3 window manager that uses async Rust and Tokio; it's complete overkill and could just as well be sync Python, but it's cool that it uses almost zero CPU or memory.

This year I wrote an artificial life simulation that actually does work: Apilar. I made it run on multiple cores, and hooked it up with a web frontend written in TypeScript using Axum.

Then I delved into making it even faster by generating machine code using LLVM in a project called Aleven; while Aleven works it's far from integrated into an artificial life simulation yet.

Finally I was paid to work professionally on Iroh for a little while, which really deepened my understanding of several aspects of Rust, especially in the areas of async and testing.

So based on that experience, here are some impressions.

A Long Strange Trip Through Type Systems

I was exposed to static types in the 1990s through Pascal, C and C++.

Then I learned Python in 1998 and with it the joy of dynamic types. Dynamic typing has great benefits: it's often less verbose and it allows software to be very malleable. During test driven development, dynamic typing is really useful when you write test doubles.

Around 2018 I went back into the world of static types again with TypeScript, which has amazing features like mapped types. Though I had played a bit with Haskell, TypeScript was the first modern type system I used in the real world and I enjoy it a lot. I haven't used Python's gradual type system a lot yet, though in the little experience I've had with it, it feels similar to TypeScript.

Just like I had discovered that writing tests and writing documentation can sharpen my thinking and the code I design, I now discovered that writing down types explicitly can do so too. And having static types really helps various development tools, too.

Even when writing code in a dynamically typed language like Python I was thinking in types; running a little type system in my head. But that doesn't let the computer help you, and sometimes that help can be quite nice.

But TypeScript's type system offers an escape. You don't have to make it work perfectly; you can smuggle in an any here or there. With Rust there is no any. With Rust there is no escape.

That leads to a particular game I call Type Whack-A-Mole.

Type Whack-A-Mole

The term "Whac-a-mole" (or "Whack-a-mole") is used colloquially to depict a situation characterized by a series of repetitious and futile tasks, where the successful completion of one just yields another popping up elsewhere. (Wikipedia)

Rust's intricate type system, control over memory, and lifetime rules induce sessions of whack-a-mole:

  • You write some code or try to adjust some existing code.

  • Rust complains you can't do that.

  • You change things.

  • Rust complains you can't do that either!

  • Repeat for quite a while, until you finally get it to compile.

  • Once it compiles, it often works.

Often the outcome is quite reasonable code, but I also frequently ended up with code that was less satisfying: overly complex, too much repetition, or a feeling some more further efficiency could be gained.

Now the "further efficiency" feeling is one I think you should usually ignore; avoiding the use of dynamically allocated memory for instance is not really a thing you should worry about too much unless you know the code you're writing needs maximum performance. You just get that feeling because it's explicit in Rust.

And in part the reason the code isn't always as pretty as it could be is definitely on me -- I'm still learning how to structure my Rust code.

But in part, I fear, this friction I feel is on Rust. It's a price you pay for the efficiency and control. Whether you should pay that price gladly depends on who you are and what you're working on.

Type Whack-A-Mole can be strangely addictive. It think this is because there's a quick feedback loop from the tooling; in this it's quite similar to test driven development. I've also experienced type whack-a-mole with TypeScript, but less frequently: in TS it's more about getting the ergonomics of the types right, deriving types from each other to make the user of the API have the best feedback possible. With Rust it's more about getting stuff to work, though the rules of Rust definitely push you into certain design patterns.

Tooling and Ecosystem

The Rust tooling is great: Cargo does a lot, Rust analyzer works in complex setups, autoformat works, and Clippy gives good suggestions and sometimes can even refactor automatically. The defaults are good!

In fact I think the Rust tooling is the best of any programming language I've used so far. The JS and Python ecosystem has good tools available too, but you always find yourself assembling a set of tools to use for each project and waste a lot of time tweaking things. If you install Rust with rustup a lot is just there from the start: the compiler, autoformatter, Rust analyzer, Clippy.

Rust has a lot of libraries and frameworks available. Rust packages (called crates) are easy to find online, though sometimes I find myself going between docs.io, crates.io and a project's github page a bit often. Sometimes you can find things in the github page that you can't find in the docs or vice versa. But it's typically easy to find the documentation. You also get some decent clues to evaluate Rust libraries for popularity and maintenance; I really struggled with that when evaluating Haskell libraries.

The documentation quality is mixed. Some Rust libraries have great narrative documentation. Sometimes you have to make do with examples in a directory (which does have Cargo support so is in a standard place). Too often I find myself staring at API docs and wishing for more examples and narrative docs. Surprisingly often I actually figure it out even with such minimal documentation: the type system helps a lot in figuring out how pieces can go together.

There are a lot of high quality Rust libraries available. There is also quite a wide breadth in what the library ecosystem offers, something I appreciate about the Python ecosystem, which really excels at this.

Compile-time Magic: Retuning my Intuitions

I've been programming for a long time. I think that experience gives me a good set of intuitions I use day to day during development. I know when it's time to refactor, and I have a storehouse of patterns for organizing code to make it more testable, configurable, performant, and so on.

Rust requires me to retune those intuitions. The aforementioned static type checking is one important reason why my intuitions don't always work out of the box, but the other reason is what you can do at compile-time.

In dynamic programming language compile-time features are typically almost absent. With Rust, there is a lot. Here's an incomplete grab-bag:

  • constants are evaluated at compile-time. This means you can't use dynamic memory allocation to construct a constant without special measures.

  • compile-time features allow you to conditionally compile code.

  • macros. These come in a lot of different varieties; you have things that look like function calls followed by a ! (like println!) that actually transform and generate code during compile-time. You can also can annotate any struct, function and such in code with special attributes to mark them for transformation.

A good example of where my intuitions weren't quite right for Rust were my struggles with test doubles.

Test driven development helps you design code with clear and minimal contracts between subsystems that you compose together. The Polly wants a Message talk by Sandi Metz is a great introduction to some of the ideas involved. If you want to test a subsystem that depends on something else, you compose it together with a fake implementation that provides just enough behavior to let you test your own code.

This is why in a dynamic language like Python and JavaScript I usually don't use mocking libraries when I write tests. Mocking libraries helps you create a fake implementation of something for testing purposes. I typically don't find it cumbersome to create the fake implementation by hand, and doing so makes the dependencies really clear.

Moreover some mocking libraries have the ability to override code in existing libraries with mocked versions during runtime. This is really powerful. It allows you to create tests for more tightly and implicitly coupled code. This is the opposite of making dependencies in code explicit and minimal so I actively avoid doing using that feature when I can.

But Rust isn't a dynamically typed language. You can't just create another fake implementation for a Rust struct and its methods. If you want an explicit interface in Rust you can have one with its trait feature, but traits have drawbacks too, and creating a trait just for testing purposes is overkill.

So in Rust you do more often want to use a mocking library to create your fake implementations. I've used mockall. This uses fancy macros to let you create a mocked implementation of a trait, and even a struct even though Rust's struct was not designed for replaceability.

By itself mockall doesn't get you out of introducing traits in the code that consumes the mocked code. So another insight that took a long time coming is that you can then use conditional compilation to make the mocked code available during tests, and the real implementation during production.

Rust uses macros and conditional compilation to solve problems dynamically typed languages solve during runtime. The application of compile-time magic can be powerful, like the amazing Serde framework for serialization and deserialization.

Even with macros available, often, but not always, you end up with more verbose code in Rust than you do in a dynamically typed language, but Rust, as a systems programming language, has different constraints.

Sync Rust ergonomics

Rust has a bunch of mechanisms for high level abstraction, but it's still more verbose than a dynamic language. This is inevitable given its constraints.

That said, I've found writing plain synchronous Rust code to be a pleasant experience. You have to worry about a lot more details than you do when you write, say, Python, but it's not too taxing. The tooling really helps -- the messages emitted by the Rust compiler and the Rust analyzer often contain helpful hints about what your problem is and how to solve it.

Some retuning of intuitions is necessary though. Here are a few I encountered on my way:

  • If in doubt, use enum match over dynamic dispatch. Rust supports dynamic dispatch, but when you use it life can get complicated really fast because everything, including memory allocation, is explicit.

  • You also want to avoid references in a struct if you can. References are fine as function arguments, but once you use one in a struct you need to start worrying about lifetime annotations. And even if you get that to work, you still won't be able to make circular references unless you start worrying far more about memory allocation than you typically want to. For referential data structures you may be better off constructing references out of array indexes. This feels like a betrayal of the static type system, but it's a lot easier.

  • Another area where Rust is different from most other languages in its error handling. Rust wants you to be very explicit about returning errors, and it wants you to handle those errors. I believe that Rust's design makes error handling better in a variety of ways, especially in a systems programming language, but it does put extra cognitive load on the developer. The ? operator helps a lot. In most code, the best advice is to use ? and to avoid a lot of custom error types, but instead to use the anyhow crate. With that, error handling in Rust is fairly ergonomic.

Async Rust

Async Rust is another story. The basics of async in Rust are all right. async and await are built into the language. Tokio, a popular library for constructing async systems, works well and is well documented.

Here are a few stumbling blocks I ran into over time:

  • You communicate between async tasks using channels. It took me a while to realize that to have multiple producers in a multi producer single consumer channel I needed to clone the producer to stop the type system from complaining, but having such compile-time checks is actually pretty great.

  • Sometimes you do really want to share state between workers. It took me a while to figure out to use Arc<Mutex<T>> for this use case, but I figured it out. You now have a significantly increased risk of deadlocks, so you need to organize things carefully.

Then came async in traits. Traits are like interfaces in other languages. Traits can be used without dynamic dispatch, but you need then when you want it. Traits, for various complex reasons I don't fully understand yet, don't support async methods.

Macros to the rescue! There's this macro provided by the async-trait crate that does allow you to use them anyway. But that macro is a leaky abstraction and it can interact with other macros, like the one provided by mockall to help you mock a trait. You don't want to end up in that situation, but I did. Macros are powerful, but they can have significant downsides.

Then there are async streams. Async stream iteration isn't built into the language yet, but that's not a big problem because a while loop works well to consume a stream. Generating an async stream is a lot trickier. It would be made a lot easier by a yield statement, something that Rust doesn't support yet. Unless you use more macro magic: the async-stream crate allows you to use the yield statement anyway! If you want to use recursion when you generate an async stream you have to worry about memory allocation, or alternatively use a crate called async-recursion with another macro.

You can avoid the use of these macros by making everything explicit, but that requires understanding of more details, like how dynamic memory location comes in. And since these abstractions are leaky, you probably need that understanding anyway. I've found async-trait not to be very friendly towards async streams, for instance. So I needed to understand that I needed to use BoxStream; the Box bit means that you're using dynamic memory allocation, which also allows you to use dynamic dispatch which you may need to when you want to mock a fake stream in a test, for instance.

Rust isn't very ergonomic yet when you need to write complex async code. The various macros help and hurt at the same time; because their abstraction is inevitably leaky you need to understand the complex underpinnings anyway.

The people working on this are fully aware of this. I'm confident that life will incrementally get better. But for the time being, beware.

Conclusions

It took a while to get used to it, but I am now fairly proficient at writing plain synchronous Rust code. I wrote Apilar in a 48 hours programming contest, and it included a new stack-based programming language with assembler and disassembler, a CLI and text-based visualization.

And by now I bashed my head against async Rust enough to be dangerous too.

Rust is one of the biggest and hardest languages I have learned. Why did I persist in learning it? Because I didn't have a systems programming language in my toolbox anymore -- most of my C and C++ experience is decades in the past. Compared to these languages I think Rust offers better ergonomics, in the areas of security, abstraction, and tooling. It also has an ecosystem full of really smart developers doing clever stuff.

Sometimes I want systems programming; it's useful when I really care about performance. High performance can sometimes make a qualitative difference too -- if you can do something a lot faster or with a lot less memory you might start to use it in new, creative ways.

Moreover, computers are devices that use a lot of energy. It would be nice if we could write more efficient software, to use a bit less.

Rust is not a language for all jobs or all programmers. I have no regrets in learning it at all though. I'd like to do some more with it in the future.

Thanks for reading; I hope it was interesting and useful to you!

Apilar: An Alife System

A week ago I participated in the langjam weekend. This was a weekend full of coding dedicated to the creation of a programming language. I had various ideas before it started, but I threw them all overboard once the theme was announced: "Beautiful Assembly". I could only work on one thing; something I had been thinking about for many years. I'm glad I did. I created Apilar, an artificial life system.

Artificial Life

Let's first sketch the context: artificial life, or alife. This is a fairly obscure topic that has interested me for a long time.

Biological life is marvelously creative -- evolution has thrown up all sorts of fascinating creatures, and their behavior individually and in the aggregate leads to yet more interesting phenomena.

The idea of alife is to learn more about biological life by examining simulated life. Going beyond relatively simple simulations, alife is interested in topics like open ended evolution and emergent properties. Can we create artificial systems that display various interesting properties of life?

Tierra

So how might one do that? There are many approaches. Back in 1991, an ecologist named Thomas Ray had a brilliant idea: he created a computer program he named Tierra. This modeled a virtual computer, with memory and an imaginary assembly language. In this language he wrote a replicator: a program that would copy itself in memory. To this he added mutation: sometimes random memory locations would be modified to random different instructions.

This sets up the preconditions for evolution:

  • replication - the programs can copy themselves

  • inevitably, some replicator programs are more "fit" -- they replicate and survive in memory better than others, for instance in being able to replicate themselves faster. So there's a form of selection -- and this selection is inherent in the system, not an external algorithm.

  • mutation - a way to generate new versions of programs.

Fascinating stuff started to happen when Tierra runs. The original human designed replicator was optimized by evolution and faster replicators took over. A form of viral parasitism evolved where some replicators would use the copying procedure of others to make copies of themselves. This leads to predator/prey dynamics.

Cielo

All this fascinated me, so in the mid 90s I set out to create a simulation of my own inspired by it. I named it Cielo. I've long since lost the C++ source code.

In Cielo, replicators live on a 2d grid. Each location on the grid has its own memory. The assembly language I designed allowed programs to move in the grid, split into two programs (one goes into a neighboring location), and merge with a neighboring replicator. Multiple processors could exist in a single locations.

There was also a notion of resources where resources would need to be harvested from the environment in order to grow, and dying replicators would be turned back into resources.

I saw interesting things emerge: replicators would quickly evolve to move around to graze for resources, replicators became more efficient, and evolved new looping code to help with moving around and harvesting resources.

Avida

I never did anything with Cielo, but other people did take the idea of Tierra much further and implemented a system called Avida and published a lot of academic research on it, including in prestigious journals like Nature. I'm happy to be acquainted with one of the creators of Avida: Charles Ofria.

Langjam and Apilar

Over the years, the idea of Tierra-like systems never let go of me. I kept tinkering with half-baked simulations based on the idea, leaving them half-finished. I would get stuck in ambition usually, and then move on to another hobby project in my spare time.

So langjam was a great way to become less ambitious and actually finish something. I had 48 hours. Go!

Apilar ("stack" in Spanish) is a Tierra-like system that's based around the concept of a stack machine. There are no registers, and instead instructions operate on the stack. So, for instance this apilar program calculates (5 + 7) * 8:

N5
N7
ADD
N8
MUL

Besides the basics of arithmetic, logic and control flow, Apilar has a bunch of instructions to help with replication: reading from memory and writing to it, starting up new processors and splitting a computer into two.

And I did it. I got it all working in 48 hours (with sufficient sleep! sleep is important). At the end Apilar had simple text-based graphics, a command-line UI, it regularly dumped the whole simulation to disk for analysis.

An Apilar replicator

Before the langjam was over I wrote (and extensively debugged) my first replicator and let it go in the simulation:

# This is a full-on replicator that can be used
# to see the simulation. It replicates itself,
# tries to grow memory, splits into two

# startup delay, because the stack of the copy
# will have the wrong start address without it
NOOP
NOOP
NOOP
NOOP
NOOP
NOOP  # delay so we take the right address after SPLIT

# take the address of the start, adjusting it for the delay
ADDR  # s
N6
SUB   # adjust s for delay
DUP   # s c
DUP   # s c c
N8
N8
MUL
ADD   # s c t target is 64 positions below start
SWAP  # s t c

# start copy loop
ADDR  # s t c l
EAT   # do some eating and growing while we can
GROW
SWAP  # s t l c
ROT   # s l c t
DUP2
ADD   # s l c t c+t
ROT   # s l t c+t c
DUP   # s l t c+t c c
READ  # s l t c+t c inst
ROT   # s l t c inst c+t
SWAP  # s l t c c+t inst
WRITE # s l t c
N1
ADD   # s l t c+1
ROT   # s t c+1 l
SWAP  # s t l c+1
DUP   # s t l c+1 c+1
ADDR  # end
N7
N3
MUL   # 21
ADD   # s t l c+1 c+1 end
LT    # s t l c+1 b
ROT   # s t c+1 b l
SWAP  # s t c+1 l b
JMPIF # s t c+1

# done with copy loop
DROP  # s t
OVER  # s t s
ADD   # s s+t
DUP   # s s+t s+t
START # s s+t spawn processor into copy
# now split memory just before it
N2
SUB   # s s+t-2 split_addr
RND   # random direction
SPLIT # split from s+t-2
JMP   # jump to first addr

And in a text-mode view of the world, I saw this:

A replicator that has grown into a blob on a 2d map

It was working!

Every # is a computer, with one or more processors on it. The rest (., x) are indicators of free resources in the area without computers. What you see here is a replicator that has managed to spread to quite a few new locations, starting to fill the map.

After a while, due to mutations, new behavior would emerge and the evolutionary process would be off into an unknown direction.

Langjam Results

Apilar wasn't a langjam winner. Here's a video of who did win, a lot of neat stuff. My congratulations to the winners!

And I got Apilar, prize enough. So what happened next?

The introduction of death

I started to explore Apilar runs in more depth, letting it run for a while. I noticed that simualtions would tend to peter out and replication stopped. What happened?

I speculated that it was due to overcrowding: it was too hard for a computer to die. When the map had filled up, replicators would be locked in place unable to place copies of themselves in neighboring locations. After a while they would all be mutated out of functioning, and that would mean the end of the evolutionary process.

So paradoxically in order to have life, I needed to introduce death. I introduced a simple measure where randomly some computers would be wiped out. This would create enough space for replication to sometimes succeed. After this modification, simulations would continue to run indefinitely, so I think my hypothesis was correct.

Early optimization

It turns out my original replicator sucks. Those 6 NOOP instructions in the beginning for instance were quickly dropped. Substracting 2 in the end before split also turned out to be unnecessary.

Neat stuff: obligate multiple processors

An Apilar computer in the default configuration supports up to 10 processors. Each update of a computer would run all the processors at the same time, so there was strong selective pressure to use this CPU time for useful purposes.

Eventually a descendant evolved that would grow much more quickly than before. My 10 year old son Marcus watched the simulation and asked what would happen with it if I restricted the maximum amount of processors to 1. I knew that would fail -- 2 is the minimum for it to work. I tried with 2 processors, but it turns out the replicator did not grow. I then went to 3; still this descendant was a dud. 4 processors; still nothing. I had to go all the way up to 9 processors for it to be viable. So not only could it use all those processors, it had to. Cool! Thanks, Marcus!

I don't know why; the evolved code was clearly based off my original replicator and some regions were highly preserved, but somehow it had evolved some complex choreography that made use of it.

Read/write heads

My original simulation had a problem: addresses are stored on the stack. When a SPLIT instruction breaks a computer in two, all addresses on the stack that point below the split point would be incorrect. Similarly with MERGE a processor could find its stack suddenly corrupted if it was on the second half of the merge. Evolution could handle that, but I wanted SPLIT and MERGE to be more equitable.

So over this weekend, I went back to an idea I implemented in Cielo: read/write heads. Addresses are no more stored in the stack but in special variables. These variables are pointers to the stack that the processor could use to READ and WRITE as well as move them FORWARD and BACKWARD. When a SPLIT or MERGE occurs, the simulation now automatically adjusts all the read/write heads to remain correct. Here's the instruction set documentation.

The same logic was also very handy to introduce insert mutations where a new random instruction is inserted into the memory, instead of the simple overwrite mutations I had before. It turns out insert mutations are very beneficial to evolution.

Here's the new starting replicator I wrote:

ADDR    # h0 has start of replicator
N1
HEAD
N0
COPY    # h1 has start too, used to READ in replication loop
N2
HEAD
N0
COPY   # h2 also has start
N8
N8
MUL    # 64 on stack
DUP
FORWARD # move h2 down 64, used to WRITE in replication loop
DUP
ADD     # 128 on stack
N3
HEAD
N2
COPY   # h3 at same address as h2
N4
HEAD
ADDR   # h4 is at start of replication loop
N8
N8
MUL
EAT    # try to eat 64 resources
N1
GROW   # try to grow 1 resource
N1
HEAD
READ   # read instruction onto stack
N1
FORWARD # move read head forward
N2
HEAD
WRITE  # write instruction at h2
N1
FORWARD # move write head forward
DUP     # 128, 128 on stack
N3
DISTANCE # read distance between h2 and h3
SWAP
LT      # if distance is less than 128...
N4
HEAD
JMPIF   # jump back to start of replication loop
N3
HEAD
START   # start new processor on offspring
N2
BACKWARD # jump back 2
RND
SPLIT   # split offspring at this point
N0
HEAD
JMP     # jump back to start of replicator

It's 57 instructions long.

More evolution

The whole "copy 128 instructions" business was hopelessly inefficient and quickly evolved away due to a point mutation. The jump backward 2 before the split point also was not very useful and would evolve away.

I watched the map undergo fascinating transitions: at some point half of the map was sparsely populated and the other half densely populated. Disassembly of replicators on the sparsely populated area found use of the MERGE instruction, which suggests a form of predation where replicators exploit neighbors by merging with them first.

Here's one of those evolved replicators that uses MERGE, with speculative comments by me:

ADDR     # start of replicator
MERGE    # merge with neighbor - what's on the stack here?
N1
HEAD
N0
NOOP
NOOP
ADDR
MERGE   # try to merge again
N1
HEAD
N0
COPY
# any 0..255 value that doesn't have an instruction is interpreted as noop
NOOP # 89
N2
HEAD
N0
NOOP # 216
COPY
N8
N8
MUL
DUP
FORWARD # still move 64, highly preserved
NOOP # 60
HEAD
SPLIT   # try a split for good measure
NOOP # 170
NOOP # 170
NOOP
N4
HEAD
ADDR
ADDR  # start of replication loop
EAT   # eat - what's on the stack here as the amount to eat?
N1
GROW  # grow 1 instruction
N1     # read & write section here, highly preserved
HEAD
READ
N1
FORWARD
N2
HEAD
WRITE
N1
FORWARD
RND   # no more use of distance, just a random 0..255 number?
N4
HEAD
JMPIF # jump back if the number isn't 0, good chance
NOOP # 120
N8
NOOP # 140
NOOP # 140
HEAD  # head 8, which is really head 0 due to clamping
START # start a new processor at h0
NOOP # 241
NOOP # 134
NOOP # 173
RND
SPLIT # try to split
# don't jump back, the processor is either going to die here if
# no offspring emerged, or run on offspring. Does it jump back somewhere below?

Interestingly it's 63 instructions long, longer than my original replicator.

The area of different sparsely and densely populated areas of the map lasted for quite a while. But a while later when I came back to it, the simulation had filled up with replicators completely. I think the prey had evolved a too effective way to counter predation -- disassembly found evidence of the use of the SPLIT instruction in the middle of a tight copying loop.

A longer run

I took the MERGE replicator I showed before and restarted the map. It was viable. In time it evolved into a map with some sparse and some dense areas like I've seen before. I let it run overnight. I came back to a fairly sparsely populated fast moving world:

A more sparsely populated map.

I disassembled one of the evolved replicators. I checked -- it's viable and when I seed a world with it, it results in a similar sparsely populated dynamic map. It was very long as stuff was going on I don't understand, but I managed to extract a shorter piece that seems to work. Let's try to read it.

First a word about stack underflows in Apilar: I don't know whether the stack is always empty but it certainly starts empty. If the stack underflows, then unsigned 64 MAX is placed on the stack.

If we try to get a value from the stack with a certain maximum, it's clamped to the range it wants using the remainder operator. For instance, there are 4 directions. Stack underflow clamped to 4 happens to be 0, which is the direction north.

ADDR    # h0 to start
MERGE   # try to merge with direction west (stack underflow)
HEAD    # h7 due to stack underflow, starts empty
EAT     # try to eat 127 resources
NOOP
NOOP
NOOP
NOOP    # 244
NOOP
EAT     # eat 127
NOOP
EAT     # eat 127
N7
N8
COPY    # copy h0, start into h7, [7] on stack
NOOP
NOOP    # 182
N8
N7
N2      # [7 8 7 2] on stack
HEAD    # h2 [7 8 7] on stack
COPY    # copy h7 (start) into h2, [7 8] on stack
NOOP    # 83
MUL     # [56 on stack]
FORWARD # move h2 forward 56, so 56 below start
START   # start a new processor there
HEAD    # h7 (due to stack underflow)
SPLIT   # try to split at h7, before start
N4
HEAD    # h4
EAT     # eat stack underflow, 127 resources
ADDR    # h4 to this address
GROW    # grow stack underflow, 127 resources
HEAD    # h0
READ    # read
N1
FORWARD # move h0 forward 1
N2
HEAD
WRITE   # write again at h2
OR      # 1 due to stack overflow
FORWARD # forward h2 forward 1
RND     # random number 0..255 on stack
N4
HEAD    # h4
JMPIF   # jump to start of replication loop if random number is not 0
N0
DUP     # [0 0]
HEAD    # h0, [0]
SPLIT   # split at h0, direction 0 so north
START   # start new processor at h0
JMPIF   # jump with underflow, so jump to h7

This replicator has evolved a new startup section before the replication loop, and it uses it to try to merge with neighbors and then split from them again. It doesn't eat in the replication loop anymore, only grows. It's also shorter than my original replicator, 52 instructions long.

It's interesting that evolution has even modified the central copying loop quite significantly: no more use of DISTANCE or LT, no more EAT, and the use of head 4 which was never in my original replicator.

The amount of use of stack underflow is interesting. If a processor runs this code with something on the stack, something else may happen entirely. I don't see any code that makes use of this, but I may be misreading it. And mutation or other replicators could definitely cause such rogue processors to appear.

Apilar futures

I have a lot of ideas for Apilar. Here's a few.

I've been working on an improved web-based user interface for it so it's easier to explore.

I also want to introduce the notion of multiple islands -- instead of the world being a single grid, the world consists of loosely connected islands. Each island could have a different configuration, with different amounts of resources and more fundamental settings such as maximum of processors and update rules.

If you're interested in Apilar, please let me know! I'm on Twitter, email and there's also the Apilar issue tracker.

Side-effect: learning

Whenever I do a hobby project, I challenge myself to learn a lot of new things.

I wrote Apilar in Rust. I learned a lot of new things about Rust and its various libraries along the way. To support the new web UI I've used Axum and Tokio for instance. And for the web frontend (in TypeScript with SolidJS) I've used PixiJS.

So I now consider myself more proficient in Rust as a side-effect.

I Was a 1980s Teenage Programmer Part 2: Olivetti M24

This is Part 2 of a series.

The Olivetti M24

When I was about age 11, around 1984, my father's office got a new computer: an Olivetti M24. Olivetti, like Triumph-Adler, started out as a typewriter manufacturer, this time an Italian one instead of a German one. And like Triumph-Adler, and IBM long before it, computers were just more office equipment.

The Olivetti M24 was an IBM PC clone and it had a good reputation. It was IBM compatible but had a faster Intel 8086 8 MHz CPU instead of the 4.77 MHz in the IBM PC. I realized that back then; my father talked about it.

Here's an ad for the M24.

/images/olivetti-m24-poster.jpg

Ad for m24 with dog and bowler hat

I think the black bowler hat in the picture is a sly hint towards IBM PC compatibility, as the IBM ads at the time featured a Charlie Chaplin tramp lookalike.

/images/charlie-ibm.jpg

Charlie Chaplin ad for IBM PC

Unlike the ad, we didn't have a color screen though, it was light blue on black:

/images/olivetti-screen.jpg

Light blue on black screen

RANDOMIZE TIMER

I continued my BASIC programming sessions. I had seen computer games at my uncle, who at some point had upgraded to a Commodore C64. The Olivetti M24 also had a few simple games available, written in BASIC. My goal was to create a computer game. My goal throughout my teenage years remained to create computer games. I was never very successful, but on this quest I learned about programming.

One of the great problems I had was how to generate random numbers. In my mind, games had to involve randomness, otherwise the enemies would behave the same each time you played the game. I learned that I could use RANDOM in BASIC to get a random number, but it got me the same sequence each time I started the program, which frustrated me. I learned about the RANDOM SEED, but that required you to either hardcode a number (leading to the same problem) or asking the user for one each time the program started up, which for some reason was unacceptable to me.

One day I joined my father on a business trip to a software developer in another town in the Netherlands, and this resulted in a breakthrough. I presented my problem to a programmer there, and he suggested I try RANDOMIZE TIMER. This was the first professional programmer I ever talked to in my life.

This sounded absolutely baffling to me - I realized TIMER had something to do with time, but how in the world would that fix my problem? But back at the office again I tried it out and it did. The magic worked!

I was elated! I am not sure when I put together that computers have a built-in clock and that this time taken as a number was arbitrary enough that it could serve as a seed for a random number generator.

User Guide

It's interesting to reflect on the fact that I barely spoke any English at this point. I had been exposed to English-spoken television with subtitles in Dutch, but I couldn't really read English.

The Olivetti M24 came with manuals in English. I would browse through them to try to learn more about the possibilities of programming.

I knew books; I read lots of them from the local village library. The covers had a title and the name of the author. The manuals had, I thought, the name of the author on it as well. The M24 manuals had titles, like "MS GW-Basic Interpreter under MS-DOS". Since the Olivetti brand was Italian, I assumed "User Guide" was the name of the Italian author. "User Guide" kind of sounded Italian to me. My parents were amused when they found out about this creative interpretation. I'm still amused to this day.

/images/m24-user-guide.jpg

Manual by famous Italian author "User Guide".

Information Space

Computers were magic, but if I cracked the magic I might be able to create a game. Some of the incantations I understood, and some remained a mystery.

The information space I was in was very limited. Obviously no internet. No BBSes either; no network of any kind. At some point later in the 1980s I saw Wargames on TV, but the notion of being able to dial into a distant computer was still science fiction to me. My interformation space about computers the early days mostly consisted of my father and the User Guide.

More Olivetti Stories

As to the M24, my father bought more of them for the office. He also acquired a second hand "luggable" computer a few years later, the Olivetti M21, at some point. I played Space Quest on it. The hard drive was flaky -- sometimes it would audibly spin down, which would block scene transitions form loading in the game. A good slap on the top would cause the drive to spin up again, and the game would continue.

In the early 1990s I went off to college and I was gifted an old M24 for me to use. I remember running a LISP interpreter on it; by that time I was able to copy programs off the Internet to run on it.

Next time...

Thanks for reading part 2! Next time, we'll go into when I got an actual home computer.

I Was a 1980s Teenage Programmer: the Alphatronic

I have been programming computers for a long time; I started as a teenager at some point in the 1980s. I thought I might reminiscence a bit about it. That's fun for me, but it also may also be fun for others to see a small snapshot of what programming could be like back then. For some, of my generation or older, there may be recognition, but for others who got into programming later this might be an unknown world.

I had a long article sitting as a draft for a few years. Today I read The Home Computer Generation, and it reminded me of it. There's quite a lot of material, so I will publish it as a series. Here is part 1.

In a Dutch Village

I grew up in a village in the south of the Netherlands. My father was an accountant who ran his own firm from an office near my house. It was next to the village church. A park with a some goats and deer and ducks was next door, and primary school was right next to it. All of these were within a few minutes walking distance, without even the need to cross a road.

Here's a picture of the office building: ivy-clad, tower of the local town hall next door. You can see the top of the steeple of the village church just behind it.

/images/office.jpg

My father's ivy-clad office.

My perspective on programming was extremely limited compared to my perspective now. I was a child, and I didn't speak English very well at all, at first. There was no Internet. Information did not spread quickly, at least not to me. The world is so different now!

Let's go back to the year 1983. I was about 10 years old.

I'm not entirely sure what the first computer was that I actually touched. It could have been a Commodore PET, owned by my uncle.

/images/commodore-pet.jpg

A Commodore PET computer. My uncle had one.

The Computer

I do know what the first computer was that I actually programmed -- the Triumph Adler Alphatronic. I have tried to reconstruct the model: I think it was a P2, but may it was a P3. It was in my father's office. I suspect I first encountered it in my house during a Christmas break as my father took it home, but I'm not sure.

/images/triumph-adler-alphatronic.jpg

The computer in my father's office. A Triumph Adler Alphatronic P2

Since my father's office was near our house, sometimes he would take me there Friday evening, when he had to work late. I would play. If I was lucky his work didn't require the computer, and I could play with that. If it did, I was out of luck and I had to amuse myself with other, much more boring, office equipment. That wasn't a lot of fun; I was there for a computer.

Note how I say "the computer"; my father had employees and there were many desks, but there was only a single computer in the whole office.

The Triumph Adler Alphatronic P2 was a computer made by a German typewriter and office equipment manufacturer. The Alphatronic came equipped with an Intel 8085 CPU. This was an 8 bit CPU with a clock speed of few megahertz. It had a monochrome screen, with amber letters in the standard 80x24 grid. The operating system was CP/M. But back then I had no idea what any of this meant. It was just the computer.

BASIC

My father had a BASIC implementation for the computer on a floppy disk. He taught himself how to program to help him to automate office tasks, like registration of hours for invoicing. He continued automating office tasks for years, writing ever more sophisticated software. Some of his creations were also used by his clients. Doing this automation gave his office an edge, but he also enjoyed technology and learning, as he still does.

/images/amber-screen.jpg

BASIC on an amber screen. I think we had a disk BASIC, not ROM, but it must have looked much like this.

My father taught me how to write a few very simple programs in BASIC. I remember we wrote a calculation program together that could do addition, substraction, multiplication and the like. I thought it would be very convenient for school arithmetic exercises but that it would probably be considered cheating.

My next program that I can recall writing was an AI. You could type in a word, and depending on whether the word was "good" or "bad", the program would respond with an ASCII art happy or sad face. The dictionary it understood was probably 5 words. It could respond to whole words only, not a word in a sentence, as I didn't know how to make the program look inside a string yet.

Programming as Magic

Programming back then for me involved some statements that I could understand, like IF and FOR and GOTO and PRINT and A = 10. If you made a typo in your code you generally got a Syntax Error. As a Dutch speaking kid I did know enough English to know that "error" meant something went wrong. "Syntax" was just a magic word to me and would remain one for many years to come. But I did know syntax errors were generally easy to fix; it would just be a simple typo somewhere.

BASIC also had a bunch of magic incantations like INSTR. If you used them wrong you would get the much harder to understand Illegal function call or a Type mismatch. I had no idea what a function call was as this old BASIC did not support creating your own functions -- you used subroutines for that with GOSUB. In BASIC everything was a global variable and there was no argument passing. I did not know what a "type" was either, or what "mismatch" meant. These were just the mysteries of the machine and you learned to work with them.

Next time...

Thanks for reading so far! Next time, we'll delve into my programming adventures with the upgraded computer in my dad's office.

SolidJS fits my brain

In this article I'm going to talk about the SolidJS frontend framework, and why I think it's cool and fits my brain.

React

I got interested in React because of a talk by Pete Hunt at JSConf Europe back in 2013. React was a baby then, a controversial baby, and this talk convinced me and many others to give it a serious look. What did I learn?

React had a very nice approach to fine-grained declarative components, and the use of a virtual DOM, even though de-empathised now, was a revelation. I gave up my own efforts on a frontend framework and switched to React instead.

State management

State management on the client is something I was interested in already. With React I went through the pre-flux era, flux and Redux. I thought Redux was cool, but I did find state management with Redux to be rather verbose. Newer layers above it have since softened these drawbacks.

Redux heavily leans on the immutability paradigm: you never change state, but you create a new version of it instead with each change. This brings some benefits, and one is related to performance: you can skip updating a particular UI component in your program entirely if you know the data you sent to it hasn't changed. You don't have to regenerate its virtual DOM or have to do any updates in the live DOM in that case. Immutable data has the interesting property that it lets you determine it hasn't changed with a very cheap comparison operation.

In 2016 I learned about MobX. This takes a very different approach: it's Key Value Observable (KVO). Basically MobX just lets you create objects and classes, annotate them very lightly to mark which data is observable, and then it tracks state changes in a fine-grained manner. If you update a property on an observed object, MobX updates only those React components that use it. Any other components are left alone. You don't need to think about it: it just works. Updates are more fine-grained than with Redux with less effort. It seemed like magic to me. In fact I recall spending a day at a React Europe hackathon trying to understand its magic when I first encountered it.

In 2017 I started using MobX for a real project, and I've been happy with it ever since. On occasion I've looked at other state management systems for React (including the built-in hooks), but nothing could compare with the ease of use and power of MobX.

The Genesis of SolidJS

This capsule history is a vague impression more than an accurate historical reconstruction, but it appears Ryan Carniato, the creator of SolidJS, also was impressed by MobX, and KVO frameworks in general. As I mentioned, React uses a virtual DOM - your React component produce a state tree, and this is then used to update the real browser DOM, only changing what is actually different. Ryan had the brilliant insight you could use a KVO framework directly to update DOM. This way you get fine-grained updates of the UI entirely skipping the virtual DOM. This lead to the development of SolidJS.

SolidJS Performance

The SolidJS approach turned out to be fast: according to various benchmarks SolidJS performs near the speed of hand-optimized DOM manipulation.

Does performance matter a lot? It's complicated.

React is fast enough for most purposes, and frameworks tend to engage in performance matches as it's an easy benchmark to compare. Users of frameworks like to use performance to evaluate frameworks far more than they should. Then again, performance can also be reflective of good design, and it's important a framework doesn't lead you into performance pitfalls. And low-powered devices are common especially in the mobile phone world, and higher performance approaches use less energy.

Raw performance of a framework isn't everything: developer usability matters. The fastest framework is no framework at all. But we use frameworks for a reason, and the reason is not performance, it's developer usability.

So let's look at developer usability next.

Solid and React as frameworks

What's important in a framework is that it gives you the tools to build your application well, keeping a lot of tradeoffs in mind: how easy is it to reason about, speed of development, maintainability, evolvability, and, yes, performance.

What does React do for you as a framework? It lets you avoid thinking about DOM updates. You can just pretend it's like a server-side framework, and that all your declarative function components get re-run each time you change any data and that this describes the new state of the UI, even though that doesn't really happen underneath and it can skip a lot of those steps as an optimization. It's easy to reason about and it helps the code that uses it stay maintainable and performant, and that's what makes it valuable to use it as a framework.

Solid looks and feels a lot like React, even though it's very different underneath. It offers fine-grained components you define like functions that take props and return JSX, just like modern React. It has functions that look a lot like React hooks; useState becomes createSignal, useMemo becomes createMemo, and useEffect becomes createEffect. Solid was immediately familiar to me as a React user.

How Solid is different from React

But Solid does a few things differently. You don't need to declare any dependencies for hooks, unlike in React. It does dependency tracking, so that when a value changes, any hooks that use that value run automatically. This is a great feature, as suddenly you don't need to think about it anymore. The subtleties of hooks in React are tricky. On top of this, Solid offers an easy way to integrate with server APIs with createResource.

React with MobX goes a step above plain React in ease of use and optimization - you can treat state as mutable, and MobX tracking is so clever only those React components rererender that use the state you changed. Solid has that built in. Solid has createStore which lets you define observable objects with fine-grained reactivity, much like MobX does.

Solid then takes this a step further; the update is not a rerender of the component (rerunning the function) but of a much finer granularity: when you change a value that's used in an expression in JSX somewhere, only that expression gets rerun and results in a DOM update.

With Solid, just like React, you can also, sort of, pretend it's like a server-side framework, and that all your JSX re-runs whenever you change any data. But in fact the function components don't get rerun. They only run once, when the component is created in the tree. That has some differences with React code:

  • You can't unpack props like is common with React, as props is a reactive object. This means in JSX you need to write <div>{props.foo}</div>.

  • You can't put a console.log in a component body and expect it to re-run in each update. Instead you need to place it in a function that is used in a JSX expression somewhere, or use createEffect.

What you get from this, besides performance, is easy to reason about behavior. Create functions, the equivalent of hooks, are just functions. They run once per component, though the functions you pass them are reactive. So can use create functions anywhere you like, such as in conditional expressions and loops. You don't even have to put them in the function component.

The granularity of components can be a subtle aspect of React performance optimization -- more components can support more fine-grained updates. But in Solid, this matters a lot less - you can make your components as big or small as you like and it won't make much of a difference.

My experience with Solid

After playing with Solid for a couple of months I've used it for real projects for the first time over the last month. I found it easy to think in Solid. Only very rarely has Solid surprised me unpleasantly. Refactoring and extending code was easy and it let me do what I wanted. The TypeScript support is also good. It's a subjective experience, but it fits my brain!

I haven't even talked about bundle size yet, but it was definitely neat when my entire Solid project's bundle is half the size of the react-dom bundle by itself. And Solid manages to pack a powerful state management framework in along the way!

Solid has a few basic principles you need to be aware of, but its API has quite a few knobs and possibilities. Luckily I didn't need to understand all of them to be able to use Solid effectively. And once every while when I ran into a new problem I recalled reading something about it in the API manual that I only half understood at the time, and suddenly it clicked and helped me solve the problem.

Last week I built a little library solid-dexie that integrates Solid with Dexie, an API on top of IndexedDB that's built into the browser. Dexie offers live queries, a form of reactivity to the contents of the database - when you add or modify a record the query result, and the UI that depends on it, automatically updates. Dexie also has React integration in the form of a dexie-react-hooks package. In my library, I figured out how to integrate it with Solid with a few lines of code -- it took a bit of headscratching but it's pretty neat!

Are there any drawbacks to Solid compared to React? There's one big one: the ecosystem of SolidJS is much smaller. React has an enormous range of libraries on offer that do all sorts of things. For Solid it's more limited. What is out there for SolidJS in its ecosystem is pretty good, as it has the benefit of refining the best ideas from the React ecosystem, but the ecosystem needs more time to grow. For some projects this drawback will outweigh the benefits. I love the benefits though, so I am going to look for more opportunities to use Solid in the future.

Is premature optimization the root of all evil?

Among programmers there is a saying: premature optimization is the root of all evil.

Where did that come from? In what context was it used? Does it still apply?

Citation needed

Looking at sources is good. So let's look at the source for this one.

It's a quote by Donald Knuth from a paper named "Structured Programming with go to Statements" from 1974.

Let's examine the title first. What does "structured programming" mean? I use it. You use it. Everybody uses it. The irony is that "structured programming" is dominating programming so thoroughly we don't actually need a term to describe it anymore.

But back in the late 60s and early 70s, it was debated whether it really was a good idea to replace the goto statement with higher level structured statements such as if and for and while, as well as functions. Unstructured programming with goto, jumping directly to a location in a program, could be more efficient. This debate sounds quaint now, but Knuth's paper describes structured programming as a "revolution".

Knuth is a good writer and I'm going to quote bits from his paper that stick out to me.

He writes:

I write this article in the first person to emphasize the fact that what I'm saying is just one man's opinion; I don't expect to persuade everyone that my present views are correct.

That also applies to this blog entry! But concerning optimization and evil, Knuth was definitely able to persuade people. Everyone is repeating it.

This is still relevant as well:

In other words, it, seems that fanatical advocates of the New Programming are going overboard in their strict enforcement of morality and purity in programs.

Programmers in various tribes still are convinced they know the Truth and look for purity.

(...) people are now beginning to renounce every feature of programming that can be considered guilty by virtue of its association with difficulties. Not only go to statements are being questioned; we also hear complaints about floating-point calculations, global variables, semaphores, pointer variables, and even assignment statements.

Let's take stock of these features in modern programming languages and their usage, where I count as "modern" anything designed in the last 30 years or so (I acknowledge that there are many fine programming languages older than this):

goto statements

Gone in most modern programming languages. A decisive victory for structured programming! Why? Compilers got better, but more importantly, computers goso much faster that worrying about the overhead of structured programmer ceased to matter. Most developers don't miss goto.

Floating-point calculations

They're ubiquitous. I believe there was quite a bit of progress in solid hardware implementations in the decades after Knuth wrote about this.

Global variables

Many modern programming languages still have them, but the consensus is that they should be used with caution at best.

Semaphores

I'm not an expert about multi-threaded programming, but semaphores are a useful primitive. Modern programming environments then build on this to provide higher level abstractions. Does this mean they're gone like go to?

Pointer variables

Raw pointers are gone from most modern programming languages, though modern system level programming languages such as Rust still provide raw pointers in code explicitly declared unsafe.

Assignment statements

They're everywhere, but there is a pure functional programming movement which advocates against their usage. Experienced developers in any language worry about restricting mutability in a range of contexts, as immutable values have nice properties.

Back to Knuth

Knuth continues:

Soon we might be restricted to only a dozen or so programs that are sufficiently simple to be allowable; then we will be almost certain that these programs cannot lead us into any trouble, but of course we won't be able to solve many problems.

This is hyperbole of course, but more seriously I think Knuth underestimated how much programming can be still be done under a very serious set of restrictions.

It's interesting to see both how much changed and how much has stayed the same.

The premature optimization quote

Back to our quote:

Premature optimization is the root of all evil.

Let's look at the quote in its context:

There is no doubt that the grail of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Wise words. Let's read on:

Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. It is often a mistake to make a priori judgments about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that their intuitive guesses fail. After working with such tools for seven years, I've become convinced that all compilers written from now on should be designed to provide all programmers with feedback indicating what parts of their programs are costing the most; indeed, this feedback should be supplied automatically unless it has been specifically turned off.

So Knuth advocates that profilers are to be built into the development environment. Profiler support is ubiquitous, but it's not typically "in your face".

Let's read back:

Of course I wouldn't bother making such optimizations on a one-shot job, but when it's a question of preparing quality programs, I don't want to restrict myself to tools that deny me such efficiencies.

Context matters! The tools Knuth doesn't want to miss include the goto statement.

As this shows, Knuth was someone highly concerned about performance. He literally wrote the book on algorithm efficiency.

The changed context

In software development, context matters. Knuth talks about it in his paper -- if he's working on a one-shot program he won't bother about complicated optimizations.

The context has changed rather drastically since 1974. The computers of that time were enormously resource constrained compared to the devices we have today. Computer resources were so scarce that they were implementing multi user computing on machines with far less computing power than an old smartphone. Your average smartphone CPU is at a minimum a thousand times faster than what Knuth had to share with other people. For all I know, my dishwasher might have more computing power too. No wonder people were obsessed about performance. They were so obsessed with performance they worried the impact of giving up the goto statement.

Do most programmers these days indeed spend enormous amounts of time thinking about the speed of their programs?

I would argue most programmers do not.

You get the regular lament that modern software is too bloated.

I agree it would be nice if software were more light-weight, used less CPU resources, used less memory, had fewer layers, and was more power efficient. But as some of these articles acknowledge, there are reasons things are the way they are: making software more efficient can take a huge amount of effort. And even though we may feel progress has been uneven and slow, software can often do a lot more as well. People decide, correctly or not, that the effort to make it more efficient is not worth it.

One of the most popular programming languages in the world these days is Python. The dominant implementation, CPython, is also one of the slowest language implementations in the world. This is by design:

Python is about having the simplest, dumbest compiler imaginable, and the official runtime semantics actively discourage cleverness in the compiler like parallelizing loops or turning recursion into loops.

Now this quote is from 2009 and things have changed somewhat since then, but my point stands. Python is still a comparatively slow language today.

It doesn't matter for most problem domains. Computers are fast enough. You can use fast libraries that are written in another language, which is an important reason why Python is so popular for machine learning. The powerful primitives built into Python also make it easier to write high performance algorithms, and a good algorithm can count for a lot.

I remember that 30 years ago, it still was much more common to obsess over the performance of programming languages. Around 1990 I learned Z80 assembly language because BASIC wasn't fast enough. A few years after that, I learned C as it was the next best thing after assembler but a bit more high level. Part of the reason for this concern for performance was because I was an inexperienced developer. But another part was because the computers available at the time were enormously more resource constrained compared to the ones we have today.

Of course in some contexts it still makes sense to obsess about performance today. It makes sense to do so in many contexts -- the inner loop of a game engine, or a database engine, or when you're engineering a web browser, or a simulation, or when you're implementing a programming language. It still matters how fast a web page loads. A change in performance characteristics can even fundamentally change our way of working altogether, because qualitatively new things become possible.

But overall, I think we live in an era where developers complain about software bloat far more than they complain about developers unnecessarily obsessing about performance.

Performance as a marketing gimmick

But, you may say, I saw graphs comparing performance of various frameworks just yesterday! That must mean that developers do care about performance.

Perhaps.

I think people at times care so much about the performance of frameworks because it's easy. So performance is used as a marketing gimmick.

People say, for instance, their web Python framework is faster. I've done that myself. But let's not forget that this framework is in Python. If you really need insanely high performance at single core low level HTTP handling, I would suggest you take a look at other languages. In any case, when I last checked a few years ago, Flask, one of the most popular Python web frameworks, was one of the slowest at trivial request/response benchmarks. Almost nobody actually cares as it's not important for almost any use case. It's fast enough.

People market how fast their JavaScript frontend framework is at updating the DOM, and how small the produced code is. This matters for a section of use cases - web sites that need to load pages quickly. But for so many web applications this barely matters -- the code size is minimum overhead compared to loading a few images, and the overhead of DOM updates is often insignificant compared to other factors.

People do this marketing because it's easy to compare two performance numbers, and much harder to compare what codebases feel like after you've used a framework for a while. Because it's difficult to compare frameworks in more sophisticated ways, it feels like you're making a good decision if you pick a framework based on its performance. If all else were equal it would make sense to pick the framework that's faster, after all. MongoDB is web scale, so you can't go wrong, right? But all else is typically not equal.

Why can optimization be evil?

Why can optimization be evil in the first place? We go to Knuth:

  • it wastes programmer time

  • it has a strong negative impact on maintenance (including debuggability)

Thank you Knuth!

When is optimization premature?

When it leads to evil - time wasting or harder to maintain code, which then also wastes time.

Premature optimization in the small

Do people optimize their own code a lot these days?

It still happens, especially in the space of (open source) libraries. Many of us on occasion spend a bit of effort to make something really fast, for the joy of the craft. I think it happens mostly in library and framework code.

Does it waste our time? Certainly sometimes, but not necessarily -- we may be doing it in our free time, and even if not we may learn a few tricks that can help us when we're in a real performance bind. A major performance improvement could even lead to the aforementioned leap in quality.

Does it hurt maintenance? A library tends to be a more constrained context, often with good test coverage, and the impact on maintenance is constrained. Users of the library aren't generally hurt by it, and may in fact benefit.

Conclusion: some of these optimization efforts may be premature, but usually only a bit.

What about micro optimizations in application code? Do we see developers, say, replace one for loop construct in JavaScript with another faster but slightly more cumbersome version everywhere in a code base "because it's faster"? I'm sure it happens, but is it really a problem for most development teams? Maybe I've been working with exceptionally well balanced software developers, but I think it's not the issue of our time.

So I'd argue premature optimization is not really a big deal in the small these days.

Premature optimization in the large

You may engage in pemature optimization not for your own code, but in your selection of dependencies or your choice of architecture. You bought into the marketing a little bit too much, and you pick a harder to use but higher performance programming language even though your problem can readily be solved in a performant way in a language like Python or JavaScript. Or you decide to pick a harder to use web framework based on performance even though you only need to handle a few requests per second and the database dominates that by far. Or you use a harder to manage microservice architecture even though your organization is tiny and you don't expect a lot of users either.

This kind of premature optimization happens quite frequently: it wastes programmer time and makes code harder to maintain and therefore we can declare this practice evil. Knuth was talking about micro optimizations, the goto statement, but the wisdom still applies here.

Often optimization is not evil

But often optimization is just fine. We can have the best of both worlds! We can have our cake and eat it too!

Imagine I need to look up something in a hash table, let's say a Python dictionary. I could write this:

def lookup(d, lookup_key):
  for key in d.keys():
      if key == lookup_key:
          return d[key]
  return None

We're definitely not optimizing here. The performance may be okay too, if the dictionary is not too big.

The optimized version looks like this:

def lookup(d, lookup_key):
   return d.get(lookup_key)

Here I use Python's dictionaries get method which finds items in constant time, no matter how big the dictionary is.

The for loop is far more evil; it's much harder to follow and thus impacts maintenance.

And it's also slower. And it doesn't scale as well.

Optimization in this case turns out to be not only only faster but also good. It's the opposite of wasting programming time!

You may assert that this example is ridiculous!

I admit it helps that I don't need to maintain Python's dictionary implementation. But that's the point. Today's powerful computers can run dependencies like Python with ease. The context was definitely different in 1974.

So since this optimization is not evil, this optimization is not premature. It's in fact never premature -- even if this dictionary never grows beyond 10 items and the cost of the for loop version was negligible, it's still better code.

This type of tradeoff can also happen in the large. I once introduced the use of a pre-existing index component in a codebase, where previously there was a Python for loop to filter records. We knew this filter would be used frequently, and we knew there were potentially many records. I was told that I shouldn't introduce an index unless I had shown by profiling that the for loop was too slow.

But profiling, especially with real-world data, would have taken an effort, the effort of the introduction of the index was comparatively low, and the impact on maintenance was low as well. I thought the tradeoffs were clear.

The wisdom of the wisdom

Back to Knuth's paper. He writes about a communication to him by Dijkstra, of Go To Statement Considered Harmful fame.

He went on to say that he looks forward to the day when machines are so fast that we won't be under pressure to optimize our programs;

I think we have reached this day years ago, at least for optimization in the small; barring a breakthrough in infinite performance computing we will never reach the point where optimization in the large becomes unimportant.

In fact machines are now so fast the pressure may be little bit too low. People can write doubly nested loops that could be easily avoided by using a hash table, and get away with it for far too long.

So what of the wisdom of "Premature optimization is the root of all evil"?

I think it's still valid in some contexts, though these contexts are typically the opposite of the low-level optimizations Knuth was talking about.

But we should perhaps also work to restore a bit of the good old inclination to optimize in our community.

Framework Patterns: JavaScript edition

Software developers use software frameworks all the time, so it's good to think about them. You might even create one yourself, but even if you don't, understanding the design principles underlying them helps you evaluate and use frameworks better.

A few years ago I wrote a post about patterns I've seen in frameworks. In it, while I did discuss other languages, I mostly used examples from the Python world. This is a revised version that focuses on frameworks written in JavaScript or TypeScript.

Let's give some examples of frameworks: well-known frameworks in the JS world include React, Express, NextJS and Jest. Frameworks are not all about solving the same problem and do not have to cover all aspects of your application - Jest for instance is focused on letting you write tests, but doesn't care about how you compose web pages.

Framework versus library

So what distinguishes a framework from a normal software library? You install both from npm, right? They all have a package.json.

You can see a software framework as a library that calls your application code instead of the other way around. This is known as the "Hollywood Principle": "Don't call us, we'll call you".

So whereas you can make a HTTP GET request using axios by calling the get function it exposes, you give a framework like NextJS functions and components to call, and it calls them. React lets you define functions that it then lets you combine in a tree, and React itself takes care of re-rendering parts of the tree and updating the browser when state changes.

Many libraries have aspects of frameworks so there is a gray area.

It's often claimed about React that it's "just a library", not a framework, but given that it presents a declarative way to structure UI for your applications including a special approach to state management, with support for both web UIs as well as native UIs, I certainly see it as a (micro) framework.

Frameworks make applications more declarative

The framework defines the grammar: framework-provided functions, objects, classes, types. Then you use that grammar by bringing some of the words: application-defined functions, objects, classes, types and config.

The grammar provides an organizing principle for your application, or at least parts of your application. A framework helps structure the way you write code, making it more declarative. Declarative code is code that says what it wants, not how to do it. Declarative code has less noise and tends to be easier to understand and adjust. Especially as a codebase grows over time, declarative patterns become more important to keep it manageable. Frameworks help you do that.

NextJs for instance makes your application more declarative in how particular pages match routes: you declare these by placing files with the names of these pages in a directory structure. This means that a developer new to a project can quickly see what pages exist and what code is used to render them, and easily add new pages as well.

Frameworks restrict

As developers our intuition may be that we want to use the tool with the least restrictions, so we get ultimate power and flexibility. Frameworks however do the opposite of that: they restrict what you can do. They force you to use it in a certain way, and if you step out of that, expect pain or the framework breaking. In return for following these restrictions, the framework gives you access to its powers.

This is quite similar to why we use programming languages. If ultimate power and flexibility was all that we wanted in software development, we'd all be using assembly language - it lets you exactly control which instructions are executed, and you can use memory in whatever way you want. It turns out that is very difficult to manage and understand, so instead we use higher level languages to help us do that.

An example of a restriction in React is how state is managed. Here's some BROKEN code that breaks that restriction:

import React from "react";

const myState = { value: 0 };

const Foo = ({}) => {
  const handleClick = () => {
    myState.value = 5; // Don't do that!
  };
  return <button onClick={handleClick}>{myState.value}</button>;
};

Here we manage state as a global object. When we click on a button, we modify that global state. But that doesn't do what we want: React does not notice this change, and the UI doesn't update with the new value after we click the button.

Let's fix that:

import React, { useState } from "react";

const Foo = ({}) => {
  const [value, setValue] = useState(0);
  const handleClick = () => {
    setValue(5);
  };
  return <p>{value}</p>;
};

Here we follow the restrictions of React: we manage state using its built-in useState hook. Because React restricts you in this way, React can now automatically re-render the component whenever there is a state change. That's the power the framework gives you, but you do have to buy into its restrictions.

Mega frameworks, micro frameworks

Mega frameworks are frameworks that aim to solve a large range of problems during application development. Famous examples in the web space are Rails and Django: problems solved span from UI in template rendering to interacting with the database through an ORM. When you deal with an application written with that framework you can expect the organizing principles of the framework to reach far into its code base. A newcomer to such a framework benefits by having to just look at one integrated source for solutions.

In the JS world mega frameworks are less common. Vue goes further than React in what it covers as a frontend framework: it has an official router and state management solution whereas React does not. But Vue nonetheless restricts itself to the frontend. NextJS also offers integration and supports server-side use cases, but is still focused on the UI part of the story.

Micro frameworks aim to solve one problem well. Examples of these are plenty in the JavaScript world: Express for programming an HTTP server, and React for managing a UI. The benefit of such frameworks is that an application development team is not locked into the framework so much and can adopt a collection of high-quality frameworks from a whole ecosystem. That's also its drawback: it takes effort to collect and maintain these.

A mega framework can be constructed from scratch, like Django or Rails historically were. You can also assembly a mega framework out of a selection of micro frameworks.

Whatever the size and scope of the framework, you can find patterns in them.

Configuration

So in a framework, we give our code to it, so it can call us. In order for the framework to call our code, we need to tell the framework about it. Let's call this configuring the framework. Configuration can take the form of JS/TS code, or could be done through a separate DSL.

There are many ways to configure a framework. Each approach has its own trade-offs. I will describe some of these framework configuration patterns here, with brief examples and mention of some of the trade-offs. Many frameworks use more than a single pattern. I don't claim this list is comprehensive -- there are more patterns.

Callback patterns

In the next section I discuss a number of basic patterns that help you inform the framework what application code to call.

Pattern: Callback function

The framework lets you pass in a callback function to configure its behavior.

Fictional example

This is a createForm function the framework provides. You can use it to configure what the framework should do when you save the form by providing a callback function:

import { createForm, FormData } from "framework";

function mySave(data: FormData) {
  // application code to save data somewhere goes here
}

const myForm = createForm(mySave);

Real-world example

Array.map is a (nano)framework that takes a (pure) function:

You can go very far with this approach. Functional languages do. If you glance at React in a certain way, it's configured with a whole bunch of callback functions called React components, along with more callback functions called event handlers.

Trade-offs

I am a big fan of this approach as the trade-offs are favorable in many circumstances. In object-oriented languages this pattern is sometimes ignored because people feel they need something more complicated: pass in some fancy object or do inheritance. I think callback functions should in fact be your first consideration.

Functions are simple to understand and implement. The contract is about as simple as it can be for code: you get some arguments and need to give a return value. This limits the knowledge you need to use the framework.

Configuration of a callback function can be very dynamic in run-time -- you can dynamically assemble or create functions and pass them into the framework, based on some configuration stored in a database, for instance.

Configuration with callback functions doesn't really stand out, which can be a disadvantage -- it's easier to see when someone subclasses a base class or implements an interface, and language-integrated methods of configuration can stand out even more.

Sometimes you want to configure multiple related functions at once, in which case an object that implements an interface can make more sense -- I describe that pattern below.

Pattern: Subclassing (inheritance)

The framework provides a base-class which you as the application developer can subclass. You implement one or more methods that the framework will call.

Fictional example

 import { FormBase } from "framework";

 class MyForm extends FormBase {
   load(): FormData {
     // application code here
   }
   save(data: FormData) {
     // application code here
   }
 }

const myForm = new MyForm();

Real-world example

This pattern is less common in JavaScript, which I think is a good thing. But there are examples, such as class-based React (which React has been moving away from for years now):

class Welcome extends React.Component {
  render() {
    // application code here
  }
  componentDidMount() {
    // application code here
  }
}

Subclassing questions

When you subclass a class, this is what you might need to know:

  • What base class methods can you override?

  • Which methods should you not override?

  • When you override a method, can you call other methods on this?

  • Is the method intended to be supplemented (don't forget super then!) or overridden, or both?

  • Does the base class inherit from another class also provides methods for you to override?

  • When you implement a method, can it interact with other methods on these other classes?

Trade-offs

Many object-oriented languages support inheritance as a language feature. You can make the subclasser implement multiple related methods. It seems obvious to use inheritance as a way to let applications use and configure the framework.

It's less common in JavaScript-based frameworks, perhaps because JavaScript developers have learned the lessons from other languages, or perhaps simply because classes were standardized relatively recently.

React used classes but is moving away from it. It always came with the strong recommendation only to subclass from React.Component directly, and never to create any deeper inheritance. An ORM like Sequelize also can work with classes, but my impression is that there too the inheritance hierarchy is supposed to be only a single level deep. Flat inheritance hierarchies indeed have less problems than deeper ones, as the questions above are easier to answer.

TypeScript offers the framework implementer a way to give more guidance (private/protected/public). The framework designer can put hard limits on which methods you are allowed to override. This takes away some of these concerns too, as with sufficient effort on the part of the framework designer, the language tooling can enforce the contract. Even so, such an API can be complex for you to understand and difficult for the framework designer to maintain.

I think the disadvantages of subclassing outweigh the advantages for a framework's external API. I still sometimes use base classes internally in a library or framework -- base classes are a lightweight way to do reuse there. In this context many of the disadvantages go away: you are in control of the base class contract yourself and you presumably understand it. But those are internal, and not base classes that a framework user has to know anything about at all.

Pattern: interfaces

The framework provides an interface that you as the application developer can implement. You implement one or more methods that the framework calls.

Fictional example

import { createForm, FormBackend, FormData } from "framework";

// typescript checks that you're not lying
const myFormBackend: FormBackend = {
  load(): FormData {
    // application code here
  }
  save(data: FormData) {
  // application code here
  }
}

const myForm = createForm(myFormBackend);

And inside framework:

export interface FormBackend {
  load(): FormData;
  save(data: FormData);
}

I gave a TypeScript example here, as this example is an especially good use case for that language. It works just fine in JS as well, if you just remove the FormBackend type, but with TS you get a compile-time error if you break the contract, and in JS you get a runtime one.

Alternative: interfaces with classes

In the above example we implemented the interface as an object literal, and this works well. There's an alternative implementation that uses classes (without inheritance):

import { createForm, FormBackend, FormData } from "framework";

// typescript checks that you're not lying
class MyFormBackend implements FormBackend {
  load(): FormData {
    // application code here
  }
  save(data: FormData) {
    // application code here
  }
}

const myForm = createForm(new MyFormBackend());

If you remove the type declarations, including the implements FormData bit, it works in plain JS as well, but again you won't get the benefit of compile-time checks. An advantage of the class-based approach is if you need multiple implementations of the interface each configured differently; in this case you can add a constructor to your class and store information on it, and create multiple instances of it. Then again, if you create objects on the fly you can do the same.

Real-world example

I had to look for a little while to find an example of this pattern; and then I realized the very editor I was typing in has an extension system that works this way. This is an example from the VSCode extension API:

const provider: vscode.DocumentSemanticTokensProvider = {
  provideDocumentSemanticTokens(
    document: vscode.TextDocument
  ): vscode.ProviderResult<vscode.SemanticTokens> {
    // analyze the document and return semantic tokens
  },
};

vscode.languages.registerDocumentSemanticTokensProvider(
  selector,
  provider,
  legend
);

This example in fact registers an interface with only a single method, provideDocumentSemanticTokens so it's functionally the same as the callback pattern. But it supports a range of registration APIs, some of which take more complex interfaces.

Trade-offs

The trade-offs are quite similar to those of callback functions. This is a useful pattern to use if you want to define related functionality in a single bundle.

I go for interfaces if my framework offers a more extensive contract that an application needs to implement, especially if the application needs to maintain its own internal state.

The use of interfaces can lead to clean composition-oriented designs, where you adapt one object into another.

You can use run-time dynamism with functions where you dynamically assemble an object that implements an interface.

My recommendation is to use this pattern over class inheritance in framework design, as the boundary with the application is a lot more clean.

Registration patterns

Consider a framework like Express or NextJS: given a URL it needs to find a function or React component to handle that URL. We can say that the framework dispatches to application code based on the URL.

The framework is in charge of decoding the URL and dispatching, but how does it know where to dispatch? Internally it needs some form of registry; a collection like an Array or a Map.

The code that the application registers could be a callback function, an object that implements a certain interface, or even a class.

Frameworks use different ways to let applications do this registration: we can call this configuration.

Pattern: imperative registration API

You register your code with the framework directly by invoking a function or method to make the registration.

Fictional Example

import { register, dispatch } from "framework";

register("chicken", () => "Cluck!");
register("cow", () => "Moo!");

Here we have a framework that lets us register animals and a function that should be called to make that animal's sound.

Let's look inside this framework:

type Handler = () => string;

const registry = new Map<string, Handler>();
export function register(name: string, handler: Handler) {
  registry.set(name, handler);
}
export function dispatch(name: string): string {
  const handler = registry.get(name);
  if (handler == null) {
    return "Unknown!";
  }
  return handler();
}

Our registry here is a Map where the key is the name of the animal and the value is the Handler function. That's an implementation detail however: we instead export a register function that can be used by the application developer. The dispatch function is an example of how the framework uses the registry.

In this example the dispatch is so simple we could have just as well stored the "Cluck" and "Moo" strings directly in the registry, but you can imagine an example where the functions receive parameters from the framework and do real work .

Real-world example

The Express framework for implementing web backends uses the imperative registration pattern:

router.get("/caravans", async (req, res) => {
  // application code here
});

We make the registration by calling router.get. This adds a handler object to an Array (the registry) in router. When resolving a request, Express goes through this array one by one to match the path in the URL, until it finds a matching handler.

The handler is application code that handles the request and produces a HTTP response.

The VSCode example above for interfaces also uses an imperative registration API - configuration gets stored in a Map.

Trade-offs

I use this pattern a lot, as it's easy to implement and good enough for many use cases. It has a minor drawback: you can't easily see that configuration is taking place when you read code. You can expose a more sophisticated configuration API on top of this layer: a DSL or language integrated registration, which I discuss later. But this is foundational.

The registration order can matter. What happens if you make the same registration twice? Perhaps the registry rejects the second registration. Perhaps it allows it, silently overriding the previous one. There is no general system to handle this, unlike patterns which I describe later.

Registration can be done anywhere in the application which makes it possible to configure the framework dynamically. But this can also lead to complexity and the framework can offer fewer guarantees if its configuration can be updated at any moment.

The registrations can happen anywhere. This means you can do them at the top level of a module, which can be very convenient, but it does means you rely on a side-effect of importing this module. Doing a lot of work during import time in general can lead to hard to predict behavior and makes it difficult to do overrides in a structured manner. Bundling tools like webpack also cannot perform their tree shaking optimization, reducing bundle size by dead-code elimination, in the presence of side-effects (see the sideEffects: false setting in package.json).

For these reasons it's often better to restrict an application's registration code to a specific function that needs to invoked to perform them, and not do them anywhere else.

Pattern: convention over configuration

The framework configures itself automatically based on your use of conventions in application code. Common conventions include:

  • File name conventions (name Jest test files .test.js)

  • Default or specially named module exports.

This could get more sophisticated as well, such as introspecting objects (using the Reflect API) or even function signatures.

Fictional example

export function handleChicken() {
  return "Cluck!";
}
export function handleCow() {
  return "Moo!";
}

So, anything prefixed with handle that is exported gets registered.

We need to bootstrap the framework somewhere by loading the module with our handle functions in it and introspecting it:

import * as myModule from "myModule";
autoRegister(myModule);

Let's look inside this framework. It's layered over the previous imperative registration example:

import { register } from "imperative-framework";

export function autoRegister(module) {
  Object.keys(module).forEach((key) => {
    if (key.startsWith("handle")) {
      const name = key.slice(6).toLowerCase();
      register(name, module[key]);
    }
  });
}

Real-world example: NextJS

NextJS uses convention over configuration:

  • Routes are based on filenames of modules and subdirectories in the pages directory. So, a file pages/foo/bar.js handles the URL path /foo/bar. Dynamic routes are also supported using the [param] convention: pages/post/[postId].js, which matches any URL path such as /post/one and /post/whatever.

  • Convention: the default export in a page module is the React component used to render that page. NextJS also looks at specially named functions such as getServerSideProps and getStaticProps to obtain server data to pass into the page.

Trade-offs

Convention over configuration can be great. It allows the user make code work without any ceremony. It can enforce useful norms that makes code easier to read -- it makes sense to postfix your test files with .test.js anyway, as that allows the human reader to recognize them.

I like convention over configuration in moderation, for some use cases. For cases where you have more complicated registrations to make with a large combination of parameters, it makes more sense to allow registration with an explicit API. An alternative is to use a high-level DSL.

The more conventions a framework has, the more disadvantages show up. You have to learn the rules, their interactions, and remember them. You may sometimes accidentally invoke them even though you don't want to, just by using the wrong name. You may want to structure your application's code in a way doesn't really work with the conventions.

And what if you wanted your registrations to be dynamic, based on database state, for instance? Convention over configuration is a hindrance here, not a help. The developer may need to fall back to a different, imperative registration API, and this may be ill-defined and difficult to use.

It's harder for the framework to implement some patterns -- what if registrations need to be parameterized, for instance? The framework may need more special naming conventions to let you influence that. Or alternatively it leads the framework designer to use modules, objects or classes over functions, but those have the drawback that they are more unwieldy.

Static type checks are of little use with convention over configuration -- I don't know of a type system that can tell you to implement a particular function signature if you postfix it with View, for instance.

Pattern: DSL-based declaration

You use a DSL (domain specific language) to configure the framework. This DSL offers some way to hook in custom code. The DSL can be an entirely custom language, but you can also leverage JSON, YAML or (shudder) XML.

You can also combine multiple languages: I've helped implement a workflow engine that's configured with JSON, and expressions in it are a subset of Python expressions with a custom parser and interpreter.

Fictional example

{
  "entries": [
    "chicken": "Cluck!",
    "cow": "Moo!"
  ]
}

Here we express the declarations outside of JavaScript. In this case we've used JSON.

Real world example: package.json

A package.json file is a DSL that describes a JS package:

{
  "name": "framework-patterns-example",
  "version": "1.0.0",
  "scripts": {
    "dev": "next",
    "build": "next build",
    "start": "next start",
    "type-check": "tsc",
    "lint": "eslint . --ext .js,.jsx,.ts,.tsx",
    "test": "cross-env NODE_ENV=test jest --runInBand",
    "test-verbose": "cross-env NODE_ENV=test jest --runInBand --verbose",
    "test-watch": "cross-env NODE_ENV=test jest --runInBand --verbose --watch"
  }
}

We can see plain data entries, but also an example of an embedded language in the scripts section, in this case the shell commands to use to execute the scripts.

Trade-offs

Custom DSLs are a very powerful tool if you actually need them. But they are also a lot more heavyweight than the other methods discussed, and that's a drawback.

A custom DSL is thorough: a framework designer can build it with very clean boundaries, with a clear grammar and hard checks to see whether code conforms to this grammar. If you build your DSL on JSON or XML, you can implement such checks pretty easily using one of the various schema implementations.

A custom DSL can provide a declarative view into your application and how everything is wired up. A drawback of DSL-based configuration is that it is quite distant from the code that it configures. A DSL can cause mental overhead -- the application developer not only needs to read the application's code but also its configuration files in order to understand the behavior of an application. If the DSL is high-level, this can be very helpful, but for more low-level declarations it can be much nicer to co-locate configuration with code.

A custom DSL can be restrictive. This sounds like a drawback but is in fact an important advantage: the restrictions can be built on by the framework to guarantee important properties.

A DSL can offer certain security guarantees -- you can ensure that DSL code can only reach into a limited part of your application.

A custom DSL gives the potential for non-developers to configure application behavior. At some point in a DSL there is a need to interface with user code, but this may be abstracted away quite far. It lets non-developers reuse code implemented by developers.

A DSL can be extended with a GUI to make it even easier for non-developers to configure it.

Since code written in a DSL can be stored in a database, you can store complex configuration in a database.

A DSL can implement a declaration engine with sophisticated behavior -- for instance the general detection of configuration conflicts (you try to configure the same thing in conflicting ways in multiple places), and structured, safe overrides that are independent of code and import order. A DSL doesn't have to use such sophistication, but a framework designer that designs a DSL is naturally lead in such a direction.

A DSL also provides little flexibility during run-time. While you could generate configuration code dynamically, that's a level of meta that's quite expensive (lots of generate/parse cycles) and it can lead to headaches for the developers trying to understand what's going on.

DSL-based configuration is also quite heavy to implement compared to many other more lightweight configuration options described.

Pattern: imperative declaration

You use a configuration engine and you drive it from programming language code in an imperative way, like imperative registration. In fact, an imperative declaration system can be layered over a imperative registration system.

The difference from imperative registration is that the framework implements a deferred configuration engine, instead of making registrations immediately: configuration is transactional. Configuration commands are first collected in a separate configuration phase, and only after collection is complete are they preprocessed, then executed, resulting in actual registrations.

This pattern supports configuration introspection tooling, and pluggable, extensible applications.

Fictional example

register("chicken", () => "Cluck!");
register("cow", () => "Moo!");
commit();

This looks the same to the user as imperative registration. The difference here is that register is much more sophisticated. It actually can detect conflicts between registrations for the same thing, and allows a way to do structured overrides. Only when commit is called does the registration in fact get applied.

So, if you do this:

register("chicken", () => "Cluck!");
register("chicken", () => "Moo!");
commit();

then the configuration engine tells you upon commit that you can't register two things for "chicken". It doesn't matter if these register calls happen far away from each other.

The configuration engine also allows you to override its default behavior. Let's say we have a special application profile "rooster" where we want the chicken to do something else:

register("chicken", () => "Cock-a-doodle-do!", "rooster")
commit();

Real-world example?

This is an underutilized pattern. Do you know of an example in the JavaScript world?

Even in the Python world, where the Pyramid web framework used this (and I use a language-integrated version of it in Morepath) it isn't used very often.

Trade-offs

This looks very similar to language-integrated registration but the behavior is declarative.

This brings some of the benefits of a configuration DSL to code. Like a DSL, the configuration system can detect conflicts ("the route name 'hello' is registered twice!"), and it allows sophisticated override patterns that are not dependent on the vagaries of registration order or import order.

Another benefit is that configuration can be generated programmatically, so this allows for a certain amount of run-time dynamism without some the costs that a DSL would have. It is still good to avoid such dynamism as much as possible though, as it can make for very difficult to comprehend code.

You can try to co-locate registrations with code, or do all registration in a separate location. But if you do co-locate registration, you risk running into JavaScript's growing aversion to module side-effects unless you take special measures; see the discussion about bundle size above.

Declarative registration a lot more heavy-weight than just passing in a callback or object with an interface -- for many frameworks that is more than enough ceremony, and nothing beats how easy that is to implement and test.

Registration pattern layering

Framework designers often directly implement a DSL or a convention over configuration system without too much consideration of how things get registered.

That is unfortunate, as I think defining a clean imperative declaration API layer underneath leads to a cleaner, easier to maintain and understand framework implementation.

The bottom of the configuration layer is an imperative declaration API. You can then layer convention over configuration, a DSL or an imperative declaration API over it.

Type patterns

The following patterns are specific to TypeScript. The idea is to let the type checker support the developers that use a framework - it gives the developer clear error messages and code autocomplete in their editor.

Pattern: Type Checking

Establish clear boundaries in code by specifying function type or interface.

We can be brief about this as we saw it in the interface pattern example above.

Pattern: Generic Types

Normally we give the framework our application code. But with TypeScript, we can also give the framework an application level type, so that it can use it to typecheck your code elsewhere.

export function registerThing<T>(thing: T, validateThing: (thing: T) => boolean) {
  // something frameworky
}

This code is very generic: it works over any thing type `T. Let's write a concrete type as an example:

type SomeThing = {
   name: string
   value: number
}

And something that implements the type SomeThing:

const myThing: SomeThing = {
   name: "Some Thing",
   value: 3
}

Then we pass the type SomeThing explicitly as T:

registerThing<SomeThing>(myThing, thing => thing.value > 2)

This way the framework knows about this type and uses it to typecheck the validateThing function argument as well.

Pattern: Generic Type Inference

Using generic types explicitly is rather heavy. Instead, we can let the framework API infer the type of the generic type argument.

This already works in the example above: since we have a parameter thing of type T we can also omit the generic type as it can be inferred:

registerThing(myThing, thing => thing.value > 2)

Because of type inference, TypeScript still knows the thing argument of the validateThing function is of type SomeThing.

A good framework API in TypeScript wants to be easy to use while avoiding any and offer type checking where possible. Generic type inference can be used to enable this.

Pattern: Type Generation

Sometimes the type information we want to use with a framework are not available as typescript definitions: they are available in some specification, or perhaps a database schema. To support development a framework can generate the types for the developers from this other source.

This pattern can even make a library behave a bit more like a framework in that you fill in the gaps to make it work with your application - but the gaps you fill in are not in the form of callbacks but types, derived from another source.

Real world examples include:

  • You have an OpenAPI specification of your REST web service. But during the implementation using a framework, for instance using Express, you want to make use of types derived from this, instead of redefining them, so that you are more sure you are implementing the right specification. You can use a tool like openapi-typescript to help you do that.

  • You are using Slonik, a library to write SQL embedded in TypeScript code. But Slonik cannot derive the types of SQL queries. So you use @slonik/typegen to automatically generate these types during runtime, getting you the benefits of type checking.

  • You are using Contentful as a CMS. The types are maintained by Contentful. But you want typechecking for CMS contents you retrieve from the Contentful API, so you use contentful-typescript-codegen. You do these by passing them into the contentful API library as generic types, turning this library a bit more into an application-specific framework.

Conclusion

I hope this overview helped you understand the decisions made by frameworks a bit better.

And if you design a framework -- which you should do, as larger applications need frameworks to stay coherent -- you now hopefully have some more concepts to work with to help you make better design decisions.

Roll Your Own Frameworks

Introduction

When I build an application, I build frameworks along the way. I recently realized that not everybody thinks this is normal, so I thought I'd give a description of what I do and why I think it's a good idea.

But let's stop for a moment and briefly discuss what I understand to be a software development framework. Examples of frameworks are frontend web frameworks like React, backend web frameworks like Django, UI component frameworks like Ant Design, an ORM like SQLAlchemy, or a form library like mstform (which I helped create), and so on.

A framework can be large or small, but in the end it's code that fulfills some task that you can control by plugging in your own code and declarations. Frameworks are declarative in nature, and declarations tend to be easier to understand and maintain than code that has a lot of moving imperative parts. This way frameworks also help you structure your application. My article Framework Patterns discusses a bunch of ways frameworks let you do that.

I will fully admit my bias up front: I like creating frameworks. But besides enjoying it, I also think this activity has enormous benefits when you build large, long-lived, successful applications.

Examples are good! Over the last two decades, I've written or helped to write the following "roll your own" frameworks while building applications:

  • several form libraries: from ZFormulator in the late 1990s to mstform most recently.

  • two workflow engines.

  • a frontend routing system.

  • an entire frontend web framework (before it was cool!). See Obviel.

  • countless tiny frameworks. This is important so I will tell you more later.

Why a framework at all?

Frameworks are overhead. You need to learn them. They can be in the way. They may have performance implications.

Why not just avoid them altogether? Use the platform, whether that may be the web browser or the operating system or whatever else.

I think this is an illusion. A platform is already a framework. If it fits what you want to do, great. But it may just not be a great framework for your particular purpose.

Frameworks offer features and help you get things done.

A more subtle effect is that frameworks also help with maintenance -- they offer a structure to your application code that makes it easier to decide how to add new features to it and makes it easier to navigate your codebase.

Successful applications tend to grow in complexity over time. Frameworks can help you prevent your application from growing into a big ball of mud.

Why aren't existing frameworks enough?

Why should you roll your own frameworks at all and not just build on top of an existing one? After all, an existing popular framework has many benefits: it is documented, you can hire people that already know it, and perhaps most importantly, you don't have to build and maintain it yourself. Even if you pick a less popular framework it still means you don't have to build and maintain it, and there is so much out there.

All this is true, and I encourage people to use existing frameworks where possible. But as everyone who has used a framework knows, you tend to reach points where the only way to make a framework do what you want is an ugly way. This is not a surprise -- all applications and all developers are unique, and a framework tries to generalize concerns, so it's likely it doesn't fit perfectly all the time.

That's usually okay -- you often gladly pay the price of more work in exchange for the feature the framework offers. But sometimes it's not okay; sometimes the price you have to pay to write nice code is too high -- it's difficult to write, it's hard to test, or the maintenance burden is enormous.

Small versus large

Small frameworks that do one thing well tend to be less constraining than larger frameworks that arrange a whole set of things for you. A web framework like Django offers a whole bunch of features out-of-the-box: from templating to database integration, all in an integrated whole. Because Django makes these choices, a development team does not have to make them. Removing the burden of decision making alone can be valuable, so you can focus on what is important. But it also makes it more likely that some of the choices do not fit what your application needs.

A smaller web framework, like Flask (or Morepath, which I created) does less, but also gives the developers more room to make the right decisions for the application. It's a trade-off.

The choice to remain focused can have an impact on the ecosystem surrounding a framework. React chose to remain focused. As it became more popular, a lot of creative solutions to other problems emerged in its ecosystem: forms, state management, UI component libraries, routing, and much more. If React had made these choices for the developers, there might have been less room for this creative ferment. But it does load up developers with extra choices to make, such as whether they should use Redux, Mobx or something else.

Mind the gaps

No matter which frameworks you choose to use, there will be gaps. There will be important functionality of your application where your existing framework doesn't have an opinion and you can find no smaller framework to help you in a satisfactory way. The price you have to pay for just "powering through" by doing a repetitive ugly thing is too high -- the code becomes unmaintainable or even impossible to write correctly. This may be tolerable for minor features, but unfortunately it's most likely to happen in core features of your application, where you spend the most effort. What to do then?

Roll your own

That's when I start thinking about rolling my own framework. I focus my own framework on exactly the problems the application needs help with the most. The benefit is that I can decrease the maintenance cost of the application code and accomplish difficult goals. The cost is that I need to write and maintain the framework.

I think people often underestimate the benefits of doing this and overestimate the costs, so I will discuss both.

Benefits

The benefits are the same as you get from any framework. Your custom framework helps organize your code in structures that help with maintenance, and makes hard things easier. Your own framework is likely to fit your application's concerns pretty well. Another big benefit is that if it turns out the framework needs new features, you don't need to wait for anyone and can just add them.

Application code tends to be difficult to test automatically. This is because an application by its nature tends to integrate things -- servers, file systems, databases, and so on. It's a whole. This means that application tests tend to lean towards integration tests, and integration tests are harder to write, slower to run, and more difficult to maintain than subsystem tests.

But the code of your framework is not application code and does not suffer from these problems. It's a subsystem. Tests tend to be easier to write and maintain and they can run quickly. So by creating a framework for application functionality you have taken that functionality out of the difficult and frustrating to test realm and put it into the fun and easier to test realm.

Because you have separated the framework from the rest of the functionality, it becomes easier to ensure loose coupling between the framework and the application. Loose coupling and tests allows you to move very quickly in a framework codebase, and make changes that can have a big impact right away throughout your application.

It's also easier to document a framework. This is because you have something separate you can point at that is not enormous and therefore not overwhelming to document.

All of these things incidentally tend to become easier if you separate your framework into its own software package and maintain it separately from the application, though this also has drawbacks -- you need to manage these packages -- so decide whether you should do this on a case-by-case basis.

Costs

All this means that the maintenance burden of your framework is less than you might expect -- if you extract a framework from your application you can effectively convert a larger maintenance burden in your application to a smaller one in your framework.

But you still need to create the framework. Is this something super difficult that only elite genius programmers can do? It would be cool to think so as this would mean I'm an elite genius programmer, but I actually think framework creation should and can be part of the toolbox of any developer. It's something that you can learn.

Tiny frameworks

The act of creating a framework may seem daunting, but a framework can be tiny and still be worthwhile. Many frameworks fit in a single screen of code.

Here are some things that may well fit in one screen of code and are frameworks:

  • a reusable HTML template.

  • a base class.

  • a React component with an onClick event handler.

  • a function that takes another function as an argument.

Examples of tiny frameworks I've helped to create, just in the last few months, include:

  • a way to define how to export and import fields of a particular data model to and from Excel.

  • a small wrapper to make it easier to talk to particular SOAP endpoints.

  • a way to use this wrapper to make it easier to write testing mocks for SOAP endpoints.

  • a Python decorator to declare certain common authorization behaviors more easily.

  • integration between URLs and form state to make it easier to express complicated search parameters.

You will note that I have a harder time describing these tiny frameworks as I can't just say "backend web framework" or "form library", which immediately call up a whole bunch of associations for many people. That's because these frameworks were designed to serve very specific goals.

Here are two slightly bigger frameworks I've built to help serve application-specific goals:

  • a frontend store that integrates an existing React Table component with a backend REST service. It takes care of synchronizing pagination, searchability, sortability, and manages frontend URL parameters.

  • a customizable way to normalize frontend JSON payloads and backend data where some fields are read-only for security purposes.

Again you will note it is more difficult to express what I mean. As long as you can define what they do well for yourself and the people who use them, that may actually be a good thing. It means you're solving real problems for a specific application.

Incidentally I could not describe my front-end web framework Obviel very well in the beginning, as people weren't very familiar with those ideas yet -- it was before Backbone, Ember, Angular, React and Vue came along. Now it's easy.

How to grow your frameworks

I won't go into the technical details of how to create a framework here. Look at existing frameworks for guidance, and read my Framework Patterns article. Instead I want to discuss ways to incrementally create frameworks while you build an application.

Start small

Certainly do not try to build a grand unifying framework that will solve everything once it is done. This is a trap. It will result in analysis paralysis or over-engineering. You risk solving problems you don't actually have and blinding yourself to the problems you do need to solve. Do not make the construction of a framework a requirement for the construction of the application that needs it.

When I say create frameworks when you build an app I do mean multiple frameworks. By all means don't start from scratch. Build on existing frameworks. When you have a particular problem and you suspect someone else has solved it already, look around first.

I try to stop myself from building a larger framework if I already know existing frameworks are out there that solve a similar problem. Only after due consideration of these do I start thinking about rolling my own. For tiny frameworks this doesn't matter that much, as they're very fast to create anyway.

If you are very experienced in a particular problem domain you may be able to build a framework independently of constructing applications that need it. But that is the exception to the rule: in general you should not build a framework before you are building the part of the application that needs it.

Look for opportunities

You won't know about all the frameworks you need to build when you start building your application. Just iteratively build application features. But keep your eye out for framework opportunities: that bit of code that is repetitive and annoying. General rule: repetitive (mostly) declarative code is fine, but repetitive imperative code is a risk and thus an opportunity for a framework that can help make it more declarative.

Then build a modest framework to help you. Integrate it with the application early.

Controlled growth

When you integrate a framework into an application, first tackle a single case, and then spread it out to all the other places you can use it. So try your validation system with a single form first, tweak it where needed, and then spread it to all the other forms, tweaking it as you go.

Make sure to spend time to convert existing code to use the framework you created. This can give you insight about gaps in your framework you may want to fix. The consistency is important. Programmers look for example code in the application first. Make sure all existing code uses the new pattern so that the old way of doing things that doesn't use the framework yet doesn't spread inadvertently.

Look for opportunities for growing existing frameworks. Your form validation library could perhaps automatically clear invalid fields or set defaults in the same validation phase. And since you have spread it to all forms already, now it is easy to add this new functionality everywhere in your application all at once.

Pretend a little

Don't worry too much about whether your framework is useful in another context. It's already useful if it helps you in a single application. But do pretend a little to yourself that you will open-source it. Have good tests and write documentation and a changelog. The future will be grateful.

But because you didn't open-source it or since your open source project has 1.5 users (like most of mine!) don't be too afraid to break APIs in the early days if you need it. Mold them like the wet clay they are.

Conclusion

Use existing frameworks where you can, but don't be afraid to roll your own when you can't. It may seem daunting but it can be learned. By extracting a framework both your application and the framework can become easier to manage.

So next time you are working on an application, look for framework opportunities. Don't be too ambitious, but start small, then slowly grow your framework. It's great to give it the open source treatment with tests, documentation and a changelog, but it doesn't have to be in set in stone right away because of that. It's your own framework and you can make it do what you need, even if you change your mind along the way.

So plant a few framework seeds in the garden of your application, and have fun!

Thank you

Thank you to those who generously helped to proofread this article:

  • Jürgen Gmach

  • Wasim Lorgat

  • Russ Ferriday

Looking for new challenges

Passion flower on my balcony

I'm looking for new freelance challenges again! I've had an awesome couple of years working for a client that still keeps me very busy, but it's time to start thinking about moving on to something new.

What can I do?

  • Build (web) applications.

  • Build software frameworks.

  • Mentor and guide teams when they build web applications.

Who am I in a nutshell? I'm a web developer with more than 20 years of experience. I am analytical. I ask questions. I am outspoken and honest. I am creative. I like to learn. I can build, mentor, teach, and help accelerate a team working on complex software. I care a lot about what I'm doing and want to be good to the people around me. And I grow flowers, fruit and vegetables for fun in my garden to relax.

I think I can afford to be rather picky in my search for clients, as I believe I have a lot to offer. So I'm going to describe what I enjoy doing next. You can decide whether I might be an interesting fit for you.

Technical Interests

I like learning and inventing. I have been developing software professionally for a long time, and I still love to build new applications and frameworks, and helping teams to do so.

Over the years I've worked on a range of applications: business applications, enterprise software, CMSes, and lots more. I understand that not all applications are the same. An enterprise with a lot of customers that need customized experiences requires different technical approaches than an application that has a single deployment. If you have something new to build, I know how to get it started and how to keep it under control. I like building applications!

Tomatoes and blackberries from my garden

I also enjoy creating software that helps developers build applications more effectively. This, to me, is an important part of application development: as developers we need to look for opportunities to extract components and frameworks from large applications. This helps to separate concerns and forces a loose coupling between them. This way we can stay in the red queen's race that is application development and maintenance.

So as a side effect of building applications, and because it's fun, I have worked on a lot of projects: several backend web frameworks (latest: Morepath), several workflow engines, lxml (the most powerful XML Python library), Obviel (an obscure frontend web framework I created before it was cool).

Web forms are a big topic that has been with me for my entire career as a web developer. Web forms are interesting as there are a lot of challenges to building a complex form in typical business applications. A lot comes together in web forms: UI, validation for the purposes of UI as well as data integrity, various modes of interaction, customization, backend interaction and security, API design and developer experience. I have been creating web form libraries from 1999 to the present day. I have a lot of ideas about them!

To learn more about what interests me and how I think, you can also consult my blog highlights.

A recent post that I'm a bit proud of is framework patterns.

Technical Skills

I have more than 20 years experience with web development and Python, and almost as much with Javascript. I've been working with React for about 5 years now. I've used both Redux as well as Mobx. In the last few years I've picked up TypeScript and more recently I've been learning Rust in my spare time.

I am an expert in doing test-driven development. I also am experienced in mentoring developers in these techniques.

I love Python and I have very deep experience with it, but I don't want to be pigeonholed as a Python-only developer -- I have a lot of experience with JavaScript as well, and believe a lot of my experience is transferable and I'm willing to try new things.

Project Circumstances

Sweet pea flowers growing in my garden

I'm a freelancer and have been self-employed for almost my entire career. I prefer to work with an hourly rate. I prefer longer duration projects where I can focus on the work, though I realize I have to prove my worth to you first.

I like to work remotely. I have have years of experience doing that. For larger projects I've worked with remote sub-contractors of my own as well.

I also like to go to you on premise. When I do that, I work with a team to build cool new stuff. This involves mentoring and guiding developers, as well learning from them. There are many ways in which we can work together: pair programming, mob programming, code reviews, presentations and coding dojos. I've had positive experiences working as part of a Scrum team the last few years.

I'm based in the Netherlands (Tilburg) and I'm not willing to relocate. Ideally I divide my time between visits and remote work. How we arrange that depends on where you are and what you prefer; if you're nearby a visit is easy, but business trips are certainly possible as well if you're further away.

I take care of my work/life balance because that is better for me, but also because I can do better work for you that way -- a well rested, relaxed mind is a lot better at creative work.

In Conclusion

If you want to hire a very experienced developer who keeps up to date, likes to spread knowledge, and can think out of the box, I'm here!

Framework Patterns

A software framework is code that calls your (application) code. That's how we distinguish a framework from a library. Libraries have aspects of frameworks so there is a gray area.

My friend Christian Theune puts it like this: a framework is a text where you fill in the blanks. The framework defines the grammar, you bring some of the words. The words are the code you bring into it.

If you as a developer use a framework, you need to tell it about your code. You need to tell the framework what to call, when. Let's call this configuring the framework.

There are many ways to configure a framework. Each approach has its own trade-offs. I will describe some of these framework configuration patterns here, with brief examples and mention of some of the trade-offs. Many frameworks use more than a single pattern. I don't claim this list is exhaustive -- there are more patterns.

The patterns I describe are generally language agnostic, though some depend on specific language features. Some of these patterns make more sense in object oriented languages. Some are easier to accomplish in one language compared to another. Some languages have rich run-time introspection abilities, and that make certain patterns a lot easier to implement. A language with a powerful macro facility will make other patterns easier to implement.

Where I give example code, I will use Python. I give some abstract code examples, and try to supply a few real-world examples as well. The examples show the framework from the perspective of the application developer.

Pattern: Callback function

The framework lets you pass in a callback function to configure its behavior.

Fictional example

This is a Form class where you can pass in a function that implements what should happen when you save the form.

from framework import Form

def my_save(data):
    ... application code to save the data somewhere ...

my_form = Form(save=my_save)

Real-world example: Python map

A real-world example: map is a (nano)framework that takes a (pure) function:

>>> list(map(lambda x: x * x, [1, 2, 3]))
[1, 4, 9]

You can go very far with this approach. Functional languages do. If you glance at React in a certain way, it's configured with a whole bunch of callback functions called React components, along with more callback functions called event handlers.

Trade-offs

I am a big fan of this approach as the trade-offs are favorable in many circumstances. In object-oriented languages this pattern is sometimes ignored because people feel they need something more complicated like pass in some fancy object or do inheritance, but I think callback functions should in fact be your first consideration.

Functions are simple to understand and implement. The contract is about as simple as it can be for code. Anything you may need to implement your function is passed in as arguments by the framework, which limits how much knowledge you need to use the framework.

Configuration of a callback function can be very dynamic in run-time -- you can dynamically assemble or create functions and pass them into the framework, based on some configuration stored in a database, for instance.

Configuration with callback functions doesn't really stand out, which can be a disadvantage -- it's easier to see someone subclasses a base class or implements an interface, and language-integrated methods of configuration can stand out even more.

Sometimes you want to configure multiple related functions at once, in which case an object that implements an interface can make more sense -- I describe that pattern below.

It helps if your language has support for function closures. And of course your language needs to actually support first class functions that you can pass around -- Java for a long time did not.

Pattern: Subclassing

The framework provides a base-class which you as the application developer can subclass. You implement one or more methods that the framework will call.

Fictional example

from framework import FormBase

class MyForm(FormBase):
    def save(self, data):
        ... application code save the data somewhere ...

Real-world example: Django REST Framework

Many frameworks offer base classes - Django offers them, and Django REST Framework even more.

Here's an example from Django REST Framework:

class AccountViewSet(viewsets.ModelViewSet):
    """
    A simple ViewSet for viewing and editing accounts.
    """
    queryset = Account.objects.all()
    serializer_class = AccountSerializer
    permission_classes = [IsAccountAdminOrReadOnly]

A ModelViewSet does a lot: it implements a lot of URLs and request methods to interact with them. It integrates with Django's ORM so that you get a REST API that you can use to create and update database objects.

Subclassing questions

When you subclass a class, this is what you might need to know:

  • What base classes are there?

  • What methods can you override?

  • When you override a method, can you call other methods on self (this) or not? Is there is a particular order in which you are allowed to call these methods?

  • Does the base class provide an implementation of this method, or is it really empty?

  • If the base class provides an implementation already, you need to know whether it's intended to be supplemented, or overridden, or both.

  • If it's intended to be supplemented, you need to make sure to call this method on the superclass in your implementation.

  • If you can override a method entirely, you may need to know what methods to use to to play a part in the framework -- perhaps other methods that can be overridden.

  • Does the base class inherit from other classes that also let you override methods? when you implement a method, can it interact with other methods on these other classes?

Trade-offs

Many object-oriented languages support inheritance as a language feature. You can make the subclasser implement multiple related methods. It seems obvious to use inheritance as a way to let applications use and configure the framework.

It's not surprising then that this design is very common for frameworks. But I try to avoid it in my own frameworks, and I often am frustrated when a framework forces me to subclass.

The reason for this is that you as the application developer have to start worrying about many of the questions above. If you're lucky they are answered by documentation, though it can still take a bit of effort to understand it. But all too often you have to guess or read the code yourself.

And then even with a well designed base class with plausible overridable methods, it can still be surprisingly hard for you to do what you actually need because the contract of the base class is just not right for your use case.

Languages like Java and TypeScript offer the framework implementer a way to give you guidance (private/protected/public, final). The framework designer can put hard limits on which methods you are allowed to override. This takes away some of these concerns, as with sufficient effort on the part of the framework designer, the language tooling can enforce the contract. Even so such an API can be complex for you to understand and difficult for the framework designer to maintain.

Many languages, such as Python, Ruby and JavaScript, don't have the tools to offer such guidance. You can subclass any base class. You can override any method. The only guidance is documentation. You may feel a bit lost as a result.

A framework tends to evolve over time to let you override more methods in more classes, and thus grows in complexity. This complexity doesn't grow just linearly as methods get added, as you have to worry about their interaction as well. A framework that has to deal with a variety of subclasses that override a wide range of methods can expect less from them. Too much flexibility can make it harder for the framework to offer useful features.

Base classes also don't lend themselves very well to run-time dynamism - some languages (like Python) do let you generate a subclass dynamically with custom methods, but that kind of code is difficult to understand.

I think the disadvantages of subclassing outweigh the advantages for a framework's external API. I still sometimes use base classes internally in a library or framework -- base classes are a lightweight way to do reuse there. In this context many of the disadvantages go away: you are in control of the base class contract yourself and you presumably understand it.

I also sometimes use an otherwise empty base class to define an interface, but that's really another pattern which I discuss next.

Pattern: interfaces

The framework provides an interface that you as the application developer can implement. You implement one or more methods that the framework calls.

Fictional example

from framework import Form, IFormBackend

class MyFormBackend(IFormBackend):
    def load(self):
        ... application code to load the data here ...

    def save(self, data):
        ... application code save the data somewhere ...

my_form = Form(MyFormBackend())

Real-world example: Python iterable/iterator

The iterable/iterator protocol in Python is an example of an interface. If you implement it, the framework (in this case the Python language) will be able to do all sorts of things with it -- print out its contents, turn it into a list, reverse it, etc.

class RandomIterable:
    def __iter__(self):
         return self
    def next(self):
        if random.choice(["go", "stop"]) == "stop":
            raise StopIteration
        return 1

Faking interfaces

Many typed languages offer native support for interfaces. But what if your language doesn't do that?

In a dynamically typed language you don't really need to do anything: any object can implement any interface. It's just you don't really get a lot of guidance from the language. What if you want a bit more?

In Python you can use the standard library abc module, or zope.interface. You can also use the typing module and implement base classes and in Python 3.8, PEP-544 protocols.

But let's say you don't have all of that or don't want to bother yet as you're just prototyping. You can use a simple Python base class to describe an interface:

class IFormBackend:
    def load(self):
        "Load the data from the backend. Should return a dict with the data."
        raise NotImplementedError()

    def save(self, data):
        "Save the data dict to the backend."
        raise NotImplementedError()

It doesn't do anything, which is the point - it just describes the methods that the application developer should implement. You could supply one or two with a simple default implementation, but that's it. You may be tempted to implement framework behavior on it, but that brings you into base class land.

Trade-offs

The trade-offs are quite similar to those of callback functions. This is a useful pattern to use if you want to define related functionality in a single bundle.

I go for interfaces if my framework offers a more extensive contract that an application needs to implement, especially if the application needs to maintain its own internal state.

The use of interfaces can lead to clean composition-oriented designs, where you adapt one object into another.

You can use run-time dynamism like with functions where you assemble an object that implements an interface dynamically.

Many languages offer interfaces as a language feature, and any object-oriented language can fake them. Or have too many ways to do it, like Python.

Pattern: imperative registration API

You register your code with the framework in a registry object.

When you have a framework that dispatches on a wide range of inputs, and you need to plug in application specific code that handles it, you are going to need some type of registry.

What gets registered can be a callback or an object that implements an interface -- it therefore builds on those patterns.

The application developer needs to call a registration method explicitly.

Frameworks can have specific ways to configure their registries that build on top of this basic pattern -- I will elaborate on that later.

Fictional Example

from framework import form_save_registry

def save(data):
   ... application code to save the data somewhere ...

# we configure what save function to use for the form named 'my_form'
form_save_registry.register('my_form', save)

Real-world example: Falcon web framework

A URL router such as in a web framework uses some type of registry. Here is an example from the Falcon web framework:

class QuoteResource:
    def on_get(self, req, resp):
        ... user code ...

api = falcon.API()
api.add_route('/quote', QuoteResource())

In this example you can see two patterns go together: QuoteResource implements an (implicit) interface, and you register it with a particular route.

Application code can register handlers for a variety of routes, and the framework then uses the registry to match a request's URL with a route, and then can all into user code to generate a response.

Trade-offs

I use this pattern a lot, as it's easy to implement and good enough for many use cases. It has a minor drawback: you can't easily see that configuration is taking place when you read code. Sometimes I expose a more sophisticated configuration API on top of it: a DSL or language integrated registration or declaration, which I discuss later. But this is foundational.

Calling a method on a registry is the most simple and direct form to register things. It's easy to implement, typically based on a hash map, though you can also use other data structures, such as trees.

The registration order can matter. What happens if you make the same registration twice? Perhaps the registry rejects the second registration. Perhaps it allows it, silently overriding the previous one. There is no general system to handle this, unlike patterns which I describe later.

Registration can be done anywhere in the application which makes it possible to configure the framework dynamically. But this can also lead to complexity and the framework can offer fewer guarantees if its configuration can be updated at any moment.

In a language that supports import-time side effects, you can do your registrations during import time. That makes the declarations stand out more. This is simple to implement, but it's also difficult to control and understand the order of imports. This makes it difficult for the application developer to do overrides. Doing a lot of work during import time in general can lead to hard to predict behavior.

Pattern: convention over configuration

The framework configures itself automatically based on your use of conventions in application code. Configuration is typically driven by particular names, prefixes, and postfixes, but a framework can also inspect other aspects of the code, such as function signatures.

This is typically layered over the procedural registration pattern.

Ruby on Rails made this famous. Rails will automatically configure the database models, views and controllers by matching up names.

Fictional example

# the framework looks for things prefixed form_save_. It hooks this
# up with `myform` which is defined elsewhere in a module named `forms`
def form_save_myform(data):
   ... application code to save the data somewhere ...

Real-world example: pytest

pytest uses convention over configuration to find tests. It looks for modules and functions prefixed by test_.

pytest also goes further and inspects the arguments to functions to figure out more things.

def test_ehlo(smtp_connection):
    response, msg = smtp_connection.ehlo()
    assert response == 250
    assert 0  # for demo purposes

In this example, pytest knows that test_ehlo is a test, because it is prefixed with test_. It also knows that the argument smtp_connection is a fixture and looks for one in the same module (or in its package).

Django uses convention over configuration in places, for instance when it looks for the variable urlpatterns in a specially named module to figure out what URL routes an application provides.

Trade-offs

Convention over configuration can be great. It allows the user to type code and have it work without any ceremony. It can enforce useful norms that makes code easier to read -- it makes sense to prefix tests with test_ anyway, as that allows the human reader to recognize them.

I like convention over configuration in moderation, for some use cases. For more complex use cases I prefer other patterns that allow registration with minimal ceremony by using features integrated into the language, such as annotation or decorator syntax.

The more conventions a framework has, the more disadvantages show up. You have to learn the rules, their interactions, and remember them. You may sometimes accidentally invoke them even though you don't want to, just by using the wrong name. You may want to structure your application's code in a way that would be very useful, but doesn't really work with the conventions.

And what if you wanted your registrations to be dynamic, based on database state, for instance? Convention over configuration is a hindrance here, not a help. The developer may need to fall back to a different, imperative registration API, and this may be ill-defined and difficult to use.

It's harder for the framework to implement some patterns -- what if registrations need to be parameterized, for instance? That's easy with functions and objects, but here the framework may need more special naming conventions to let you influence that. That may lead the framework designer to use classes over functions, as in many languages these can have attributes with particular names.

Static type checks are of little use with convention over configuration -- I don't know of a type system that can enforce you implement various methods if you postfix your class with the name View, for instance.

If you have a language with enough run-time introspection capabilities such as Ruby, Python or JavaScript, it's pretty easy to implement convention over configuration. It's a lot harder for languages that don't offer those features, but it may still be possible with sufficient compiler magic. But those same languages are often big on being explicit, and convention over configuration's magic doesn't really fit well with that.

Pattern: metaclass based registration

When you subclass a framework-provided baseclass, it gets registered with the framework.

Some languages such as Python and Ruby offer meta-classes. These let you do two things: change the behavior of classes in fundamental ways, and do side-effects when the class is imported. You can do things during class declaration that you normally only can do during instantiation.

A framework can exploit these side-effects to do some registration.

Fictional example

from framework import FormBase

class MyForm(FormBase):
    def save(self, data):
        ... application code save the data somewhere ...

# the framework now knows about MyForm without further action from you

Real-world example: Django

When you declare a Django model by subclassing from its Model base class, Django automatically creates a new relational database table for it.

from django.db import models

class Person(models.Model):
    first_name = models.CharField(max_length=30)
    last_name = models.CharField(max_length=30)

Trade-offs

I rarely use these because they are so hard to reason about and because it's so easy to break assumptions for the person who subclasses them.

Meta-classes are notoriously hard to implement. If they're not implemented correctly, they can also lead to surprising behavior that you may need to deal with when you use the framework. Basic assumptions that you may have about the way a class behaves can go out of the door.

Import-time side-effects are difficult to control -- in what order does this happen?

Python has a simpler way to do side-effects for class declarations using decorators.

A base-class driven design for configuration may lead the framework designer towards meta-classes, further complicating the way the framework uses.

Many languages don't support this pattern. It can be seen as a special case of language integrated registration, discussed next.

Pattern: language integrated registration

You configure the application by using framework-provided annotations for code. Registrations happen immediately.

Many programming languages offer some syntax aid for annotating functions, classes and more with metadata. Java has annotations. Rust has attributes. Python has decorators which can be used for this purpose as well.

These annotations can be used as a way to drive configuration in a registry.

Fictional example

from framework import form_save_registry

# we define and configure the function at the same time
@form_save_registry.register('my_form')
def save(data):
   ... application code to save the data somewhere ...

Real-world example: Flask web framework

A real-world example is the @app.route decorator of the Flask web framework.

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

Trade-offs

I use this method of configuring software sometimes, but I'm also aware of its limitations -- I tend to go for language integrated declaration, discussed below, which looks identical to the end user but is more predictable.

I'm warier than most about exposing this as an API to application developers, but am happy to use it inside a library or codebase, much like base classes. The ad-hoc nature of import-time side effects make me reach for more sophisticated patterns of configuration when I have to build a solid API.

This pattern is lightweight to implement at least in Python -- it's not much harder than a registry. Your mileage will vary dependent on language. Unlike convention over configuration, configuration is explicit and stands out in code, but the amount of ceremony is kept to a minimum. The configuration information is co-located with the code that is being registered.

Unlike convention over configuration, there is a natural way to parameterize registration with metadata.

In languages like Python this is implemented as a possibly significant import-time side-effect, and may have surprising import order dependencies. In a language like Rust this is done by compiler macro magic -- I think the Rocket web framework is an example, but I'm still trying to understand how it works.

Pattern: DSL-based declaration

You use a DSL (domain specific language) to configure the framework. This DSL offers some way to hook in custom code. The DSL can be an entirely custom language, but you can also leverage JSON, YAML or (shudder) XML.

You can also combine these: I've helped implement a workflow engine that's configured with JSON, and expressions in it are a subset of Python expressions with a custom parser and interpreter.

It is typically layered over some kind of imperative registration system.

Fictional example

{
   "form": {
     "name": "my_form",
     "save": "my_module.save"
   }
}

We have a custom language (in this case done with JSON) that lets us configure the way our system works. Here we plug in the save behavior for my_form by referring to the function save in some Python module my_module.

Real-world example: Plone CMS framework

Pyramid and Plone both are descendants of Zope, and you can use ZCML, a XML-derived configuration language with them both.

Here is some ZCML from Plone:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:browser="http://namespaces.zope.org/browser"
    i18n_domain="my.package">

  <!-- override folder_contents -->
  <configure package="plone.app.content.browser">
      <browser:page
          for="Products.CMFCore.interfaces._content.IFolderish"
          class="my.package.browser.foldercontents.MyFolderContentsView"
          name="folder_contents"
          template="folder_contents.pt"
          layer="my.package.interfaces.IMyPackageLayer"
          permission="cmf.ListFolderContents"
      />
  </configure>
</configure>

This demonstrates a feature offered by a well-designed DSL: a way to do a structured override of behavior in the framework.

Trade-offs

Custom DSLs are a very powerful tool if you actually need them, and you do need them at times. But they are also a lot more heavyweight than the other methods discussed, and that's a drawback.

A custom DSL is thorough: a framework designer can build it with very clean boundaries, with a clear grammar and hard checks to see whether code conforms to this grammar. If you build your DSL on JSON or XML, you can implement such checks pretty easily using one of the various schema implementations.

A custom DSL gives the potential for non-developers to configure application behavior. At some point in a DSL there is a need to interface with user code, but this may be abstracted away quite far. It lets non-developers reuse code implemented by developers.

A DSL can be extended with a GUI to make it even easier for non-developers to configure it.

Since code written in a DSL can be stored in a database, you can store complex configuration in a database.

A DSL can offer certain security guarantees -- you can ensure that DSL code can only reach into a limited part of your application.

A DSL can implement a declaration engine with sophisticated behavior -- for instance the general detection of configuration conflicts (you try to configure the same thing in conflicting ways in multiple places), and structured, safe overrides that are independent of code and import order. A DSL doesn't have to use such sophistication, but a framework designer that designs a DSL is naturally lead in such a direction.

A drawback of DSL-based configuration is that it is quite distant from the code that it configures. That is fine for some use cases, but overkill for others. A DSL can cause mental overhead -- the applciation developer not only needs to read the application's code but also its configuration files in order to understand the behavior of an application. For many frameworks it can be much nicer to co-locate configuration with code.

A DSL also provides little flexibility during run-time. While you could generate configuration code dynamically, that's a level of meta that's quite expensive (lots of generate/parse cycles) and it can lead to headaches for the developers trying to understand what's going on.

DSL-based configuration is also quite heavy to implement compared to many other more lightweight configuration options described.

Pattern: imperative declaration

You use a declaration engine like in a DSL, but you drive it from programming language code in an imperative way, like imperative registration. In fact, an imperative declaration system can be layered over a imperative registration system.

The difference from imperative registration is that the framework implements a deferred configuration engine, instead of making registrations immediately. Configuration commands are first collected in a separate configuration phase, and only after collection is complete are they executed, resulting in actual registrations.

Fictional example

from framework import Config

def save(data):
   ... application code to save the data somewhere ...

config = Config()
config.form_save('my_form', save)
config.commit()

The idea here is that configuration registries are only modified when config.commit() happens, and only after the configuration has been validated.

Real-world example: Pyramid web framework

From the Pyramid web framework:

def hello_world(request):
    return Response('Hello World!')

with Configurator() as config:
    config.add_route('hello', '/')
    config.add_view(hello_world, route_name='hello')

This looks very similar to a plain registry, but inside something else is going on: it first collects all registrations, and then generically detects whether there are conflicts, and generically applies overrides. Once the code exits the with statement, config is complete and committed.

Trade-offs

This brings some of the benefits of a configuration DSL to code. Like a DSL, the configuration system can detect conflicts (the route name 'hello' is registered twice), and it allows sophisticated override patterns that are not dependent on the vagaries of registration order or import order.

Another benefit is that configuration can be generated programmatically, so this allows for a certain amount of run-time dynamism without some the costs that a DSL would have. It is still good to avoid such dynamism as much as possible though, as it can make for very difficult to comprehend code.

The code that is configured may still not be not co-located with the configuration, but at least it's all code, instead of a whole new language.

Pattern: language integrated declaration

You configure the application by using framework-provided annotations for code. This configuration is declarative and does not immediately take place.

Language integration declaration looks like language integrated registration, but uses a configuration engine like with imperative declaration.

Fictional example

from framework import Config

config = Config()

# we define and configure the function at the same time
@config.form_save('my_form')
def save(data):
   ... application code to save the data somewhere ...

# elsewhere before application starts
config.commit()

Real-world example: Morepath web framework

My own Morepath web framework is configured this way.

import morepath

class App(morepath.App):
    pass

@App.path(path='/hello')
class Hello(object):
    pass

@App.view(model=Hello)
def view_get(self, request):
    return "Hello world!"

Here two things happen: an instance of Hello is registered for the route /hello, and a GET view is registered for such instances. You can supply these decorators in any order in any module -- the framework will figure it out. If you subclass App, and re-register the /hello path, you have a new application with new behavior for that path, but the same view.

Trade-offs

I like this way of configuring code very much, so I built a framework for it.

This looks very similar to language-integrated registration but the behavior is declarative.

It's more explicit than convention over configuration, but still low on ceremony, like language-integrated registration. It co-locates configuration with code.

It eliminates many of the issues with the more lightweight language-integrated registration while retaining many of its benefits. It imposes a lot of structure on how configuration works, and this can lead to useful properties: conflict detection and overrides, for instance.

It's a lot more heavy-weight than just passing in a callback or object with an interface -- for many frameworks this is more than enough ceremony, and nothing beats how easy that is to implement and test.

You can't store it in a database or give it to a non-programmer: for that, use a DSL.

But if want a configuration language that's powerful and friendly, this is a good way to go.

It's a lot more difficult to implement though, which is a drawback. If you use Python, you're in luck: I've implemented a framework to help you build this, called Dectate. My Morepath web framework is built on it.

In Dectate, import-time side-effects are minimized: when the decorator is executed the parameters are stored, but registration only happens when commit() is executed. This means there is no dependence on run-time import order, and conflict detection and overrides are supported in a general way.

Conclusion

I hope this helps developers who have to deal with frameworks to understand the decisions made by these frameworks better. If you have a problem with a framework, perhaps I gave you some arguments that lets you express it better as well.

And if you design a framework -- which you should do, as larger applications need frameworks to stay coherent -- you now hopefully have some more concepts to work with to help you make better design decisions.