Tea Leaves

Previously on this channel we discussed the role of abstraction in the context of constructing software. Abstraction enables laziness by hiding details that are unimportant or irrelevant to the problem at hand. This is a powerful and beautiful idea because it gives the appearance of providing programmers with a great deal of leverage against complexity. Of course, nothing is ever that easy. The problem is that at some point if you are going to ship anything someone has to worry about all those hidden details. And this is typically when people get into trouble.

In software, details are the enemy of simplicity and calm. In my experience, the place where details do the most damage is in the controller layers of the application. Armies of engineers have worked for decades to provide reusable tools for building the controller layers of custom applications. The temptation is to say “Hey, we can just make a generic rule engine to control these things and then everyone can define their own custom flow on top of the some data model and UI abstractions”. But that never works, because when you get down to the critical details, the control requirements for most applications are all different. Thus, while you can gain a lot of leverage in sharing the generic parts of the UI presentation and the data model, it’s almost impossible to actually make customizing the control easier.

So you end up writing everything from scratch every time or encoding every possible option into the generic rule engine. The short way to characterize this syndrome is: “you can’t share the controller”.

Anyone who has worked in software for any amount of time will recognize a whole class of systems for which this statement holds. These tend to be “workflow” applications that try to encode the existing technical or business processes of an organization into a shiny new computer application to make everything more efficient. Some consultant will come in and tell you to spend hundreds of thousands of dollars and buy an application engine from Oracle or SAP or Peoplesoft which has “modules” that encode “business logic” that let you specify exactly what it is you need. Except it only works if you then also hire a dozen or two dozen engineers from Oracle or SAP or Peoplesoft to come in and collect your specifications and write the all the control from scratch.

Of course, this process is tedious and error prone so more often than not the result is a buggy and bloated mess that completely fails. Hooray for abstraction!

There are similar problems in desktop software, which is why it is so much harder to build a framework for building applications than it is to build and ship a few individual applications by themselves. In both cases you have to deal with all of the annoying control details, but making an SDK means you then add the extra overhead of trying to anticipate and deal with everyone else’s details as well. Remember: it’s always better to avoid vague generalities when you could work on concrete requirements instead. Avoid premature reuse.

Unfortunately, engineers find it difficult to follow this rule. It is in their nature to think about everything at a generic abstract level instead of at a level of detail that is useful for the end user. Abstractions are an enjoyable distraction from the every day reality of dealing with messy details. Left on their own they would go and build a cool new way to structure an operating system kernel (say), rather than something that makes editing photos easier for end users. The reason is because if they are building their own abstraction, then they have control over the details of the requirements. This makes their code fun to work on because they are only solving problems that they find interesting. It also makes the end product completely worthless because no one wants to use an abstraction to get work done. Users want to use applications.

There is also a tendency to allow abstraction to distract us from all the details rather than just the unimportant ones. We then talk about a system or a feature as a black box that is already implemented, rather than as a placeholder for the real work that we’ll eventually have to do. One sign that you’ve entered into this danger zone is when you ask the question “how hard could it be?” For example, “That feature seems straightforward, how hard could it be?” No matter how tempted you are, you should never ask this question about a system you do not understand intimately. The answer is almost always “harder and more expensive than you think.” Unless you understand the details, you really have no way to know. If nothing else, in the crunch of a tight schedule, every new detail that you decide to implement adds complexity, risk, testing costs and unexpected bugs. Good software comes from understanding which details are the critical ones and building the right abstractions to deal with the those problems. In other words, you need to solve the right problems in the right way at the right level of abstraction. Don’t let yourself get distracted.