In defence of complexity - part 1

By Andrew Clifford

Although we aspire to build simple IT solutions, many arguments suggest that this may not be possible.

We have been exploring a hypothesis about making IT much simpler. We started with the view that, for good engineering reasons, we have designed systems based on shared layers of capability, and that these are the root cause of major management problems in IT. IT management techniques do not solve the problems and, by increasing our tolerance to them, dig us deeper into the problems.

We can address the underlying architectural problems by encapsulating technology layers inside systems and by aligning IT systems with business owners. This fixes the major problems. In the long run it also dismantles our view of IT as a specialist activity, and replaces it by IT subsumed into everyday business management.

This is heresy because it challenges the technical, managerial and organisational orthodoxies of IT.

There are many counter arguments to this hypothesis. Some of the counter arguments are good, and some are subtle and clever. But I will start with some of the counter arguments that I personally think do not hold water.

• It is not technically feasible, because it does not perform well or does not scale.
• The IT structures we have represent real business choices and not technology choices.
• There is no problem, IT delivers good value.

I think these arguments are wrong because:

• In the past shared layers did provide economies of scale, but improved capacity and falling prices make those economies now largely unnecessary. Where performance or scalability problems remain, these relate to intense processing requirements within a single system. Building shared layers would not help.
• We may have got our business colleagues to agree to the structures, for example to agree to a large shared Oracle database, but this does not constitute real choice. What other options did we offer? Taking a recommendation is not the same as making a real choice.
• IT does deliver good value at times, but in general the costs and constraints of IT are so large that, even if there is no obvious problem, the opportunities for improvement are huge.

Having covered what I believe are spurious arguments, I want to cover some good arguments. These all ask, "Is it feasible and worthwhile?"

• A lot of business data and processing is fundamentally shared. Splitting it out might make some parts of IT simpler, but it would not be better aligned and would create more problems than it solves.
• The problems are caused by the excessive variety of technical solutions and application software. Committing to one large package vendor is technically simpler, and the potential for misalignment is less of a problem than the challenges of managing multiple technical solutions.
• IT is fundamentally hard. In the same way that we rely on doctors and lawyers, we have to rely on IT professionals to bridge the gap between business need and technical implementation. The best designs are the ones that then make it easiest for the IT professionals, which means the shared, layered architectures that we have.

Next week I will continue this argument by covering some of the more subtle forces that encourage complexity.

Subscription