The Problems with Software
Everyone who deals with software knows there is something wrong. Software is riddled with bugs, software is hard to use, and software development projects are frequently over budget, slip schedules, and sacrifice function and quality. In order to find solutions to these problems, software is often likened to construction. Unfortunately this analogy is flawed and lessons learned from the construction industry are suspect.
Many intelligent people have examined the problems with software and tried to find ways to fix them. Over the few decades that we have been writing software we have improved in many ways: our hardware is unimaginably capable compared to that of even one decade ago, our programming languages are easier to learn and offer far more features every year, our software development tools care for more complexity with every version. In some ways we have even made software that is easier to use. Nevertheless, the problems remain.
The Construction Analogy
Software would be much better for everyone if these problems were solved. Improving the development process is believed to be the key to solving these problems. Professionals working in the software development field often think "wouldn't it be nice if the process of building software was like the process of constructing a building?" We could be software architects and designers who conceive of the structure of the software. We could be software project managers who control software projects. We could be software engineers and developers (construction workers) who actually figure out the details and build the software. And finally, we could be software inspectors who verify the quality and standards of a software product.
This analogy is attractive because the beneficial effects of the construction industry are ubiquitous and integral to our lives. We live in houses and we work in commercial or industrial buildings. We see the amazing edifices such as the Egyptian pyramids, the European cathedrals, or the Petronas Towers. We all imagine that the construction industry, having grown out of millennia of practice, has much wisdom and advice to offer.
If only we could copy the construction industry with its architecture, project management, design, engineering, tools, rules, guidelines, and even its pre-fabricated parts. If we could do all this, so it is asserted, then we could solve all the problems we have with software.
Those who look to the construction analogy believe the following things can be learned and applied to software creation:
- The Architect is the creative center of the endeavor.
- The design of a system is finished before construction begins.
- The people doing the actual construction work can be treated as interchangeable human resources.
- The flow of information can be made one-way from client to architect to builder.
- The construction work can be completed and the result handed off to the client.
Over the next several paragraphs we will examine how the analogy is broken (disfunctional) and why it is dangerous to use the analogy to guide our efforts to make software better.
Distribution of Buildings... Isn't
A building is constructed in the place it will be used. Duplication of buildings is a major effort. Buildings are not distributed from a construction location to the place or places where they will be used. In fact, a building is considered exceptionally well designed if it aesthetically fits with its environment.
On the other hand, software is almost always created in a different physical location (a development workstation) than the place it will be used or deployed. Software must be distributed (if it is commercial software such as an office suite or computer game) or deployed (if it is enterprise software). Unlike buildings, software is considered well designed if it can be made to work with little effort on many different hardware platforms in many different environments.
This difference has very significant consequences for the process used in construction versus what might be used in software. The construction industry has no advice to offer to the software industry about distribution questions. In fact the processes and techniques used by construction are heavily influenced by the requirement to build in place - to make it right the first time.
This difference also has economic factors. The physical manifestation of software is incredibly inexpensive compared to the cost of creating the definition of the software. Of course in construction, the physical cost (property and materials) can be a very substantial portion of the total cost of a building. The construction industry has no advice to offer on this economic difference.
Finally, this difference has an artistic component. Practitioners of building architecture are inspired to reach a completely different aesthetic goal than practitioners of software creation. The construction industry cannot help the software industry make beautiful software.
Public vs. Published Interfaces
In software, published interfaces (interfaces used by parties outside of the development team) are the most stable aspect of software. Nevertheless, published interfaces are only a specification and the actual implementation of an interface can change radically. For example, a communication interface specification such as TCP/IP can be implemented over many different physical layers: ethernet, 802.11b wireless, or even SneakerNet. The choice of implementation can be changed throughout the lifetime of a software system. Not only that, but it really is possible to change even a published interface. Can the analogous changes be made in a physical building? Are the requirements the interface? For example, square footage in a building.
"Project Management" is the Process for Construction
In a construction project there is one shot at making an essentially immutable physical structure. This immutability (which we will discuss more in the section "Buildings are Completely Rigid") is the central motivator for the process used by the construction industry. The only process or methodology that makes sense for building construction is the "waterfall": conceptualize, architect, engineer, construct, finish and hand over - period. It is true that these phases overlap, and that certain construction projects will emphasize different phases, but the essential process is just as immutable as the final product. This waterfall methodology more commonly goes by the name "project management." The Project Management Institute (http://www.pmi.org/) and other authorities on project management claim to be generic authorities, but in reality, they all have their roots deep in the construction and engineering disciplines. Software project management therefore is the inbred cousin of construction project management. In the world of software creation, project managers are a frustrated group.
In the software industry it is possible to have many different development methodologies that seem rational. The intellectual nature of creating software means that the process of creating it is not inherent in the product. The plethora of software creation methodologies and their practitioners is ample proof of this. The waterfall, spiral, iterative, ad-hoc, user driven, interaction driven, requirements driven, risk driven, budget driven, quality driven, calendar driven, market driven, technology driven, high ceremony and low ceremony variants all address different philosophies and practical considerations.
Anyone who claims the "One True Way" for software creation based on the construction analogy should be looked at with extreme skepticism. Even re-usable process frameworks such as the Rational Unified Process, should be looked at with some distrust.
Construction is Constrained
A construction project has many pre-determined constraints no matter what is being constructed. A building has a constraint of scale: it must be comfortable to humans. A building must respect physical laws otherwise it fails, potentially catastrophically. A building must often be made to tolerate a huge variety of environmental conditions. A building must be made of certain materials due to physical properties and availability. Construction must follow rules, regulations and codes imposed by parties well divorced from the actual requirements of a particular construction project. Actual construction work must occur in a specific physical location.
Software creation is not subject to the same constraints as construction. The intellectual nature of software means that it allows far more creative freedom. Complexity theory imposes very real mathematical constraints on software, but in practice, these constraints are rarely applicable or can be avoided using approximate rather than exact methods. Constraints of scale, physical laws, resource availability, external regulations and location do not apply or are much less critical for software than for construction.
These different constraints mean that very different factors are considered when doing software creation versus constructing a building. These individual constraints and the differences between them are rarely if ever addressed when the analogy is made between software creation and construction.
Buildings are Completely Rigid
A building must have a design before you begin constructing it. You cannot move an elevator shaft after the fact. You cannot double the square footage after a foundation has been laid (even building higher would require changes in the foundation). Yet "analogous" changes can be made in software at almost any time. Building architecture and function are closely intertwined, software architecture and function are nearly independent. Undoing or changing physical construction, while possible, is hard and expensive. That's why when you want to renovate, you look seriously at moving to a new (better) home or office first. Undoing or changing software is easy and cheap, otherwise we wouldn't have a new version of Microsoft Office (tm) every couple years.
Software is fluid. It is possible to build software before even any formal requirements have been gathered. And that same ad-hoc software can then gradually be built upon, modified, broken up, incorporated into other software systems, fixed, broken, respond to changes in business, discarded, re-found, re-used, translated into a new programming language, wrapped up in layers, abstracted, productized and eventually sold to someone else. Software can be created, used and discarded in a matter of minutes or have a lifetime that lasts decades. Software projects build upon each other structurally in ways which construction projects rarely do.
The analogy between software creation and construction implies that software can be thought of very rigidly. However, in practice the fluidity of software is far to valuable to be ignored. Iterative development methodologies attempt to address this fluidity with various degrees of success but the very fact of iteration breaks the analogy with construction.
Pre-Fabricated Building Materials are Simple to Use
Human beings grow up in a physical universe where physical cause and effect are relatively obvious in many instances. In the case of structures this is particularly true. The ancients, without any complex math or physics, were able to create astounding physical structures from the pyramids to the cathedrals. This obviousness means that creating standard and re-usable pre-fabricated building materials is easy, and it is easy to use them. A steel girder has a fairly simple API: fasten both ends to a structure and put load bearing stuff on it. Nails, screws, cement walls (waffle-crete), joists, and cut lumber are all standardized and easy to use.
Software is incredibly abstract and often very arbitrary. Cause and effect are not obvious or intuitive despite programmer's best efforts. Examining another programmer's code with a mind to using it (let alone modifying it) is not a simple exercise. Even function or class libraries and toolkits such as the Java Development Kit are incredibly complex.
In software, there seem to be only certain things that lend themselves to re-use: simple data structures, simple algorithms, and sometimes simple abstraction layers and software tools. Other things, such as processes, components, third party "solutions", and frameworks can all be incredibly difficult to learn to use and then integrate into a software project. Re-use in software is hard.
It should be noted that software developers and engineers have a very generous standard for "easy-to-use". If it takes less than a couple weeks to become proficient with a new software library (e.g. JDBC or XML-DOM) then a software engineer considers that "easy".
It often seems that the effort to re-use software components is just as difficult as building needed functionality from scratch. Most components are generalized so that they can be used in a wide variety of situations - with the result that any given piece of software made from pre-fab components has a whole bunch of unused functionality left over - pipes that lead nowhere, half-walls and stairways to dead ends. That extra functionality often makes it difficult to learn about and use a component and adds substantially to the integration work. It also means that the cost of developing the component's functionality has to be recovered somehow. Creating the needed functionality from scratch is often more efficient. All the extra functionality of a generalized component is left out making it easier to learn and use, and it is perfectly integrated with the other parts of the system immediately.
Those who promote the construction analogy in the form of component based development overlook the simplicity and intuitiveness of "components" in construction. Any analogy or methodology which deals with software components must explicitly recognize and address the incredible complexity inherent in the concept of software components.
The Humans Involved do Different Things
The human talents used in construction are different from those used in software creation. Software developers (architects, designers, engineers, testers, whatever) must work in an extremely abstract environment and at the same time deal with many more options. A press on a computer key can do literally anything it is possible to do with a computer; cause and effect are completely arbitrary. Yet pounding a nail always does the same thing. Software developers have the opportunity to be much more creative than their construction counterparts. Programmers are required to hold far more detailed and arbitrarily abstract information in their heads on a day-to-day basis than any construction worker ever does. Thus, trying to make a one-to-one correspondence between construction roles and software creation roles leaves many real and potential differences ignored. Just think about the actual differences between a software "architect" and a building architect or between a construction worker and a programmer. And is there any role in construction similar to a software tester? One might say that a building inspector is analogous, but the actual differences are huge!
The differences in talents required mean that management of software creation must be very different from management of construction.
Software is Fundamentally Different From Buildings
It seems there is something rigorous or logical about both construction and software development. The most obvious difference is that software is essentially intellectual whereas a building is essentially physical. This difference is often glossed over or given lip service when the analogy is made between the two. Even when the analogy is used deliberately and "scientifically" this difference between software and building is ignored or explained away as irrelevant.
Occasionally someone will point out this difference and even seem to discard the construction analogy because of this difference, but then go on to describe things that really are just new words for the same old analogy. Sometimes a few things are thrown in (like "iterative development") to try to distinguish the "new" approach from the construction analogy. Walker Royce of Rational Software wrote recently in Information Week an article that points out this basic difference between software and buildings (http://www.informationweek.com/story/IWK20020517S0002). However, in the article he discusses "architecture first", "component-based development", "design with rigorous ... notation", and "progress assessment" among other ideas from the analogy. As we can see, these are all problematic holdovers from the analogy.
The Construction Industry Has Problems Too!
Given all the fundamental and vast differences previously mentioned, there are still some software projects that might look like they are similar to construction projects and so some people will be tempted to say that the analogy is sometimes useful. There is one last issue that removes the desire for even that weak association between construction and software creation.
The construction industry has problems that are clear to anyone who cares to look. Construction projects go over budget, sometimes horrendously. Construction projects frequently deliver an imperfect building that fails safety, environmental or user requirements. If you have had your own home built, you will know what I mean! The construction industry has re-usable patterns, idioms, and parts, but there is a very clear dividing line between the intellectual constructs and the physical constructs. The as-built plans are always different from the architectural plans. And the reality of the construction site is that compromises are constantly being made even up to putting construction trash as filler inside walls! This is like deliberately putting bugs in software. It starts to sound like maybe construction projects might be just as bad off as software projects.
These problems in the construction industry mean we should question the wisdom of trying to learn from the analogy between software creation and construction. Do we really want to learn how to construct software from an industry that has exactly the same problems?
So What Are We To Do?
Is all lost?
Yes, and no. The first thing to do is to once and for all discard the analogy with construction. Our decades of trying to use it have not taken us very far considering the incredibly bright minds that have been working on it. Instead we have to explicitly recognize the true attributes of software creation: its incredible flexibility, its odd economics, and the remarkable human talents required to create it.
Analogies are very powerful tools for understanding new concepts. If we can find a better analogy between software and some other realm, then we can improve our understanding of software. If our analogy is extremely deep and strong, me might go beyond improvement and uncover completely unexpected applications and techniques for software.
Extreme Programming
One approach that seems very promising is that of Extreme Programming. Extreme Programming or XP as it is known, is introduced in the book "Extreme Programming Explained" by Kent Beck. The web site www.extremeprogramming.org is another very informative introductory resource. XP is famous or infamous for its "pair programming" but in fact this radical practice is only one small aspect of many utilized by XP. XP explicitly looks at some of the underlying assumptions of software creation and comes up with some new things. Notably, the dogmic belief in the concept of "exponentially increasing cost of change" is challenged. The results of this challenge are some sweeping ramifications that affect every aspect of software creation. Unfortunately XP is missing a coherent analogy to some other realm of human experience. Occasionally mention is made of the creativity and skill of the programmer. This might lead us to an analogy in the world of media.
A Few Thoughts on a Better Analogy
The amazing thing about software is that each new software program or system is a completely new medium of expression. The difference between a game such as Quake III Arena where users can create their own worlds to fight each other in, and a simple text editing tool such as Microsoft's Wordpad is just as large as the difference between sculpting and writing with a pen and paper. No one would debate that sculpture and handwriting are two different media. Therefore software creation is the process of creating new medias.
This analogy of media creation is anticipated by Marshall McLuhan's famous quote: "The Medium is the Message." And so software creation is really communication as well. Alistair Cockburn discusses software as communication extensively in his book "Agile Software Development".
Further Reading and References
Extreme Programming Explained by Kent Beck
Extreme Programming Examined Giancarlo Succi and Michele Marchesi, ed.
Object Solutions by Grady Booch
The Mythical Man-Month by Fred Brooks
Software Craftsmanship by Pete McBreen
Agile Software Development by Alistair Cockburn
No Silver Bullet - Essence and Accidents of Software Engineering (http://www.virtualschool.edu/mon/SoftwareEngineering/BrooksNoSilverBullet.html)
by Fred Brooks
The Inmates are Running the Asylum by Alan Cooper
No Silver Bullet Revisited (http://www.virtualschool.edu/cox/AmProTTEF.html) by Brad Cox
The Cathedral and the Bazaar by Eric S. Raymond
No Silver Bullet II
(http://www.muq.org/~cynbe/rants/correctness2.html) by Jeff Prothero
Other Software Resources compiled by yours truly, Mishkin Berteig.
![]](images/rt_edge3.gif){kind=small}
![[XML]](images/rssblue.gif){kind=small}
![[P]](images/print3.gif){kind=small}
