A system that grows unceasingly becomes gnarled by past decisions; it must be allowed to succumb to a winter frost so that it may flourish anew in the spring.
I have a new metaphor: a well-worked codebase is like a tangled summer garden. My company's Rules Engine, in particular, has come to feel wild. It is a useful system, and we've taken a responsible, refactoring approach to extending it -- but the constricting vines of past decisions feel tight around us. To allow the garden perpetual growth may feel safer, but it is better to reap our harvest and to tend something new in the spring.
There are as many personal metaphors about writing software as there are people who write software. To say that programming lends itself to metaphor isn't quite right; software demands metaphor. I type with my hands, but everything I create is intangible and imaginary. The only way to talk about it, or to reason about it, is in a language-by-proxy. Otherwise it doesn't really exist1. So writing software is a recipe, or a blueprint, or a polymorphic mammal that quacks or barks.
It might be that the creation of a metaphor is one of those cobbles on the path from novice to less-novice, and so I'm offering my own2. It has to do with gardening, although I'm really no great gardener: my partner is generally the one to maintain our planters of parsley and cilantro, and I still get confused between the two. So this will be as maladaptive as any metaphor.
It's not about cultivating code, or about pruning technical debt. It's not features as flowers3. I'm concerned with the later seasons: as we find ourselves in the autumn and eventually the winter. What does it mean to harvest, to care for the soil, and to spend time reflecting while the ground is hard and little grows. What does it mean to reap knowledge of our domain, to maintain clear and coherent systems, and to use that understanding to continue growing ever more complex products.
How do we allow the ground to lie fallow, how do we plan to grow again in the spring.
I work at Hatch Loyalty, building a platform with which other, much larger, businesses run their various loyalty programs. At the heart of the platform is our Rules Engine, a production rule system that allows abstract marketing strategies to be translated into functional sets of actions and effects. The Engine was newly sprouted when I joined the company in 2016. It has continued to bloom.
In the beginning, the Rules Engine was a (relatively) simple system that examined events and emitted effects: someone purchases "Muscle Milk" and is issued points4. It's grown to support a variety of preconditions, to enable demographic- and behavior-based targeting, and to enforce various limitations on how frequently someone engages with a particular promotion. With each addition we've done our best to keep the Engine modular and extensible; I work with an astoundingly thoughtful group of folks. This has resulted in something that is coherent, but that has begun to feel like a bramble of blackberries. There is fruit in that thicket; it's mixed with thorns and dead wood.
We've refactored and rearchitected. We've rewritten subsections. We've renamed modules to better fit their responsibilities. The Engine has been cultivated as well as we know how: part by part as we've come to better understand the product that we're tending5. But we are still broadly constrained by a system that is under constant use and frequent development. Our past decisions are vines that wind throughout our code. The Engine, today, feels wild.
Two years ago my partner and I planted basil in a black plastic pot in our first balcony garden. It grew, but was stunted and withered before the summer was over. We moved, tried again in the following spring, and killed a second plant. It was late in the second summer when we realized that basil may not appreciate having its roots burned by the dark pot and the hot sun. But, our other balcony boxes were flourishing with flowers and thyme and mint and oregano. We could have tried a mid-season transplant, but the other herbs might have been shocked at being uprooted. And anyway it would have made a mess on the deck.
We have fundamental improvements to make to our Rules Engine. The Engine executes asynchronously as a background job, and we'd like it to execute synchronously: this means that performance is a greater concern. We'd like its processing to be more observable, and we'd like it to be idempotent6. These improvements require foundational changes to the Rules Engine. Our foundation is a creeping briar of past decisions. I don't believe that there is a better way to build software7, and yet we feel constricted. It is difficult to encourage systemic growth when surveying the state of what is introduces a myopic filter on what can be. Every choice we've made, every module we've built, has lead us to a product that is subtly, intractably, entangled.
I think about crop rotation, about letting a field lay fallow and allowing dirt to again become soil. I think about this past spring, and about how easy it was to find a new home for our basil after the Chicago winter had wiped away our porch garden. I want to harvest what we've learned about loyalty programs, about software, and about our Rules Engine, and to write it again. To grow it from seed in good soil.
There are risks, and there are preconditions. Our Engine is working well, and there are no time-sensitive new requirements on the horizon. We must be able to bet that work on this piece of our platform might remain dormant for a while; that it will continue to work and to yield value. I have some thoughts on tactics and strategy; I'm looking forward to sharpening those tools with the folks that I work with. And I'm looking forward to writing about that process in the future.
I do not get to choose when the Chicago winter sets in, and I am given no say in my basil's last leaves. But we have to make a deliberate decision on when to gather the fruits of our work and replant. It's often easier to sell -- to ourselves and to the business we work in -- the idea of an eternal growing season. I do not believe that it's an honest idea: our soil eventually turns to dirt if untended, and our foundation eventually erodes away.
This is not about starting from scratch; it's about beginning again with more perspective. I do not think it is so daunting. And there is something in the clear, brisk air that forces me to become alive. And there is something exciting in the coming spring.
Thank you for your time and for your attention.
Tactically it's incredibly important that we have integration tests outside of the Engine that we propose to renew. It's important to be able to draw thick lines around the interfaces in to and out of this part of our system. It's important to take a good few leaps into the sky, to look down at our existing garden, and to use all of the understanding we've developed to lay it out afresh.
I think that Basecamp's ideas on new products fit with this process. I have a feeling that they've done something similar to this when creating Basecamp 3, and might again as they write Basecamp 4 -- but I don't have any real details on that. I'm hopeful about a winter for our Rules Engine, but we have not yet embarked.
But I can only ever see what I've done through a screen. I can only feel connected to what I do by talking about it. Maybe the feeling is different for folks writing software in robotics. They program "jump." Their creation asks its magnetic brain: "how many bits high?"
It's important to track all promotions, but only a portion require limitation. This meant that we had to explicitly attach a "Tracker" to every promotion regardless of the desire to limit, which began to needlessly complicate our system. We refactored the orchestration of the Engine to remove the need to add an explicit "Tracker." We allowed the Engine to track engagement implicitly. This meant the addition of an optional model, the "Limiter", the root-and-stem removal of the "Tracker," and some data-backfills to put everything back in order. The Limiter's responsibilities, and name, now made sense. Our system was a little bit clearer.