software crisis

In 2006 I was ruminating on the complexity of software development, also known as the software crisis. This term was coined almost 40 years earlier and in 1972 Edsger Dijkstra spoke about it:

The major cause of the software crisis is that the machines have become several orders of magnitude more powerful! To put it quite bluntly: as long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming has become an equally gigantic problem.

— Edsger Dijkstra, Communications of the ACM (Wikipedia)

Complexity marches on

Perhaps my brain was already beginning to slow down in the face of the rapid pace of change, since by this time I was no stranger to complexity. What surprised me as I thought about it was that things didn’t seem to be improving. Despite huge advances in the tools and libraries at our disposal, it felt like we were losing ground. The tools vastly simplified the work compared to even just a few years earlier, but the expected functionality was becoming more complex faster than the tool releases or knowledge updates.

The next time I paused to think about this was as multi-core processors were becoming the desktop standard. Some industry articles were bemoaning the poor state of reliable language support for parallel algorithms. Others pointed out the more serious fact that was obvious to me as well: most developers had (have!) no experience in analysing or creating parallel solutions. Furthermore, these developers depend on their programming language to do the right thing, as they should be able to, but they won’t be aware of its limitations and will be at a loss when their assumptions break down without warning.

The sheer multitude of applications at our disposal and the ease with which we can connect with data and services shows that things are generally working well. However, it seems to me that the software we use today should already be capable of much more and it should certainly be much easier to produce.

Software crisis 2.0

This term was recently proposed by Brian Fitzgerald, IEEE Computer, April 2012. He points out two factors that are combining to greatly increase the expectations we have of software (and by extension of the software development community). Big data, driven by scientific and financial research, social networks, and the coming Internet of Things (100 billion connected devices, trillions of sensors), creates a huge data push. The growing population of digital natives, users who have grown up with technology, are creating a growing demand, or pull for increasingly powerful applications.

Fitzgerald points out that all the advances we have made in the our practice will be inadequate for the software development capacity required to service the combination of these factors.

The elephant in the room

There is potentially an even bigger problem, a third factor, that needs to be taken into account in any discussion of the ongoing software crisis. It is still unknown whether or not there will be a labour shortage as the baby boomers begin to leave the work force. We do know that the elderly are living longer and are more active so we will certainly have to rely more heavily on technology to support all of the population. Combined with year over year declines in computer science enrolments, the software development capacity could actually be declining for the near future.

As Fitzgerald points out, there is yet to be a silver bullet for increased software productivity. However, we have a social responsibility to improve software productivity wherever and however possible. This means that we have to create better software (applicable, maintainable, extensible) with the means we have at hand. We must place a greater emphasis on the upfront investment required to repeatedly release quality products. We have to commit ourselves to continuous improvement by constantly reflecting on the efficacy of our processes and adjusting to accommodate our insights and the changing development environment.