Covering up complexity

Over half of the jobs I'€™ve held during my IT career have been with public service providers '€” whether providing internet or web hosting services, email hosting, or now, cloud services. I have always found these jobs the most fun and rewarding due to the creativity that can be applied. For example, as a junior systems administrator for a small ISP, rather than sitting on the phone with customer after customer and manually walking them through setting up an email account, I was able to build a small web application for them instead. It allowed customers to create and manage their emails, as well as download dynamically generated scripts that would configure their desktop for those addresses.

In order for that small web app to be successful, I had to understand a large stack of domain knowledge. Maybe '€œstack'€ isn'€™t the right word '€” it was more like a parabola, with one end being the system and the other end being the user. On the system end, I had to understand how the internal email service worked so those accounts could be managed. I then had to configure the web app to work with the email system.

At the vertex was the actual face of the web app, where communication between the system and user took place. Going down the user side involved ensuring that only authenticated customers could access the system, and make it so that they could easily manage their email accounts and download the configuration script to would automatically add their email addresses to their desktop computer.

Each of those components, though described in only a few words, are large topics on their own. In order to have a working email system, I needed to install and configure qmail. A working Linux server was required for that, which needed working hardware. And on and on. I recently read a great blog post detailing the inherent complexity that comes with modern technology.

What happens when a complex system becomes so stable that the intricate inner workings can safely be ignored by the operator? Usually, the system is then marketed as a package.

Take cPanel for example. cPanel is a package that is installed on a clean server. When the installation is done, your server has all of the components it needs to host websites and email accounts on behalf of customers. In a nutshell: it'€™s web and email hosting in a box.

cPanel is a great product. I worked with it for many years and, over time, I began to see how much thought goes into making sure the complexities of web hosting are hidden. Someone with little knowledge of the web hosting stack/parabola could easily use cPanel to create their own web hosting business. And, in fact, this is a very common practice. This forum is a large and busy central meeting place for people involved with web hosting. The amount of people running small web hosting companies, largely due to the quality of cPanel, is extraordinary.

I find this subject interesting. The process of simplifying complex systems for greater adoption is a staple of civilisation. I do find it a little sad, though, to see the work that went into developing the initial complex system be tucked away and forgotten. Someone could end up being highly successful by using a simplified system, and leave the original innovators with nothing. The simplified system could also be used in ways the original innovators did not intend. Additionally, support of the simplified system can become more and more costly due to the lack of attention given to the hidden complexity.

In a lot of ways, '€œThe Cloud'€ is all about covering up complexity '€” but from an end-user point of view. It'€™s only a matter of time before we see cloud products covering up the complexity of the system'€™s side. What kind of startups will be possible because of these products? What kind of poor marketing tactics will we see when the cloud is commoditized? How much time will be spent reinventing best practices because of the hidden complexity?