Is software design different?
Agile Bridge Building
A gently satirical look at agile software & extreme programming (XP) development philosophies
This paper presents a lightly satirical look at the latest fads in management of software design and development projects. It provides a high level overview of some of the problems with pattern based software philosophies and tries to familiarize you with much of the terminology currently being used to describe them. The latter is a fast moving target though. My aim is to show you that your hard learned understanding about what it takes to do a job well, on time, and within budget, is just as applicable to software development projects as it is to other types of projects. You'll see that everything you've ever learned about bringing projects to completion has not suddenly become obsolete. Let's start with what you already know: It doesn't matter if you're building software, a bridge, a space shuttle, or even something as seemingly trivial as a world class meal. The simple principles at the heart of producing any "job well done" are constant. They need not be a mystery to anyone who has ever performed a difficult task from inception to completion, and performed it well. So this article attempts to have a little fun while exploring what might happen when you apply my vocation's pop project management philosophies to another type of task; bridge building. Perhaps as software developers, we can laugh at ourselves a little along the way. What better way to learn? Presented for your approval...
The New Way to Build a Bridge
In this article the author describes a bridge building project to demonstrate the superiority of agile bridge building methodologies to old, outdated concepts that rely on Big Up Front Design. The initial goal for the project will be to produce a river crossing allowing traffic and commerce to cross over the bridge while (ideally) allowing river traffic to pass under the bridge.
It should be noted that the author's choice of the term "initial goal" is appropriate here, for it is a foundational principle of the agile bridge builder's manifesto that the client should always have the power, freedom, and flexibility to change his or her mind at any stage in a project's ongoing development, right up to completion. Put another way, everything is tentative, no decision should be written in stone.
Rapid Span Development
The bridge-building masters on the rapid bridge development team begin by floating a log out on the water, half on land and half into the water. This does not (yet) accomplish our project's goal, which is to build a river crossing that ships can pass under, but it puts a working span out on the water quickly. Allowing bridge investors and executives to see working wood beginning to make the river crossing is an extremely important concept in extreme bridge building (XB). It shows the project has immediate success. This in turn helps to mitigate the hurtful effects that nervous or less confident executives may have regarding our ultimate success. Having placed a working span of bridge part way out toward our goal, we may now refer to our efforts with undeniable correctness, as a successful bridge design and development project. This rapid approach combines design and construction into one easily comprehended step. It completely avoids all of the Big Up Front costs associated with the lengthy and COSTLY bridge design phases that were once used, back in the (old) days when people designed bridges before they built them.
The Self-Realizing Project
We have now embarked on Iterative Incremental Bridge Design (IIBD). A cornerstone of IIBD is how new design elements will suggest themselves the instant you begin building. For example, it is only by building this first section of the bridge that we are able to see that the log is not totally stable and could easily be washed away by changing water levels. So a team, perhaps a pair of bridge builders, may thus be intuitively assigned to rework, reinforce and better secure the first log to the embankment. These opportunities to "re-factor" elements of our design will present themselves often within the course of our bridge project and should be pursued relentlessly. Similarly the simple act of starting the bridge's construction exposes other design opportunities that seem to magically suggest themselves. For example, one will realize, with no need to engage in any wasteful Up Front Design oriented thinking, that the single log does not span the entire width of the water. So, almost as if by second nature the next design decision becomes apparent: Have another pair of bridge builders travel to the end of the first log and lash a second log to the end.
BUFD Is a Bad Thing
This effortless ability on the part of the builders to produce viable design steps at the exact same time the building effort commences is partly due to the synergy from the mystical chi of the bridge project itself. But be careful! If the team's Yin and Yang are not in nearly perfect balance your bridge building pair will get wet. This of course is not the result of the project or methodologies used, but the fault of the individual team members. They simply aren't disciplined enough to infer the patterns that emerge from the bridge construction effort. Simple attitude adjustment can frequently address this. Sometimes mocha latte is enough. Or an agile bridge building guru can come in to assist in inferring the pattern and facilitating the acceptance of chi. Developers who don't exhibit proper chi may be better off doing projects with the old-fashioned Big Up Front Design dinosaurs (or "BUFD's" ). But if you need a bridge we recommend you steer clear of BUFD developers. They are unfamiliar with the Zen and pattern of agile bridge development. They will invariably force clients to wait until a design is in place before allowing them to see their projects taking shape. The design phase can often take longer than the building phase. Worse, some of these BUFD bridge builders will charge for the cost of design. If you've been paying attention, you can now see that design is something freely performed by the project itself when you apply the principles of extreme, guru, master, wizard, bridge building to the task at hand. Today, thanks to the agile methods of IIBD, those touchy-feely psychobabble concepts like trust and delayed gratification need no longer have any place in the serious contractual business of bridge building.
IIBD Is a Realistic Approach
Of course, many other problems will arise as the bridge is being built and designed at the same time. But minor issues will likely be surmounted over the course of the construction effort. It is more important that clients are able to see working spans over the river almost from the very first day of the project. Agile bridge developers do not force clients to wait around for costly and burdensome design stages. Nor are clients asked to sit in endless meetings as designers attempt to determine what is really needed. In agile bridge development, after all, the circumstances of the ongoing construction automatically show clients what they really need. Meetings, if they are needed at all, will be stand-up meetings.
No Decision Should Be Final
Which brings us to our final undeniable point about the benefits of IIBD - total flexibility. The client is always free to change his or her mind at any time. Because the design is taking shape simultaneously with the actual construction of the bridge, there is no disconnect between the designer's vision and the customer's understanding of that vision. As the bridge takes shape, the customer sees the design trade-offs played out in real time, and can make his or her own decisions on each one. In this project for example, it quickly becomes clear that river currents are pulling the far end of the bridge downstream. The client can see for himself --without being told-- that more time and resources will be needed to complete the span. Likewise, the client in this case will be able to see that the original design goal of allowing boats to pass under the span was excessive. The new bridge will allow cargo to be transported by land vehicles, doing away with the need for boats altogether.
This saves much of the COSTLY bickering and bad feelings (not to mention lost business) that may be associated with having a designer explain this to the client Up Front during one of those long unnecessary design meetings. Also, no valuable developer time will be wasted designing this feature that --it turns out-- wasn't really required after all. In short, we were able to say to the client at the most opportune time: "You're Not Going to Need It".
The Old Way to Build a Bridge
Bridge builders who have not kept their skills current continue to build bridges in the old way. Recommended Viewing
Back to Reality
I hope you have enjoyed this little tale of two bridges, and that it might help you to recognize when you are being asked to excuse bad development practices simply because "software is different". We in the trade have lots of excuses and reasons but none of them really ring true. We will try to tell clients that software is different because you can't touch it, and therefore there is no real cost to make changes in software constructs. We know nothing could be further from the truth. . . . . .
Pliability Is Everywhere You don't have to be an economics wizard to understand that ultimately almost all of the cost of constructing a bridge (or a meal, or a skyscraper) is the cost of the labor and organizational efforts that go into converting raw materials into finished products. The value of the touchable materials that come out of the ground or forest is minuscule compared with the costs of handling them all the way onto and into the bridge. Intangible costs begin with the costs involved in digging (or growing) the tangible things. Intangible costs continue to be added at every step along the way to the bridge.
Cost issue 1 - It is all labor.
. . . . .
There are other costs associated with not planning a project before starting. One problem is, if you don't know what the finished construction will look like before you start building, you must wait until it's completed before you can see that a particular component doesn't make the grade. Maybe that first plank should have been painted before it was put in the river.
"So what?" we will say, "In software development it is easy to replace a component, even after the entire project is complete. All that is spent is the cost of developer time to re-write the code". As explained above, labor is everything and developer time is often the single biggest expense item in software projects.
This added cost, due only to lack of planning, would be unacceptable by itself, but it is not the only cost added. This is because the one component you replace is related to every other component in the project, and changing it will affect all those related components.
Cost issue two -
In software, as in bridge building, there is a domino effect. Each successive component often depends on the capabilities of the components below it. Reworking one low-level component will likely require reworking many of the dependent components that have been built above.
. . . . .
And the hemorrhaging caused by lack of an easily consulted plan goes on. Consider testing! I will be the first to tell you, testing is a good practice, and the tests themselves might be well-designed and carried out with great precision. A component may have tested fine for what testers thought was needed at the time of the test. But no matter how thorough the process, without the ability to reference a detailed and shared understanding of how the final project will use each component, the tests can not be relied on. They will be tests performed in total blindness.
But testing will give everyone a secure feeling, which will make it less likely that anyone will go back with any conviction to re-check these fully "unit tested" components. For one thing, retesting components that have already been thoroughly tested would correctly be flagged as a redundant expense by management.
The tendency not to return to fully tested components is further exacerbated among team members by the specter and costs associated with tipping over that "first domino" discussed above. That is, finding a deficiency in a low-level component that leads to the rework of hundreds of higher levels.
Third and final cost issue -
Many factors combine to greatly increase the odds that a project will be placed into service with components that have been fully tested and found compliant, but that are wholly inadequate. This risk can be effectively avoided simply by gaining a precise, easily consulted understanding of the completed project before component construction and testing begins.
. . . . .
Perhaps the most transparent excuse we programmers clutch to is that software is different from other design disciplines because humans haven't been doing it as long.
In the case of a satire about bridge building, for example, we will say that people have been building bridges for thousands of years. But people have only been building modern steal-cable suspension bridges since the mid 19th century. Discounting the fosil record, that would be right around the same time Ada Byron wrote the first modern machine instruction program. Whether we are talking about Rosetta Stones, or trees felled across a river, we are splitting hairs at best.
So to bring the point home without need for hair-splitting on either side, consider that we've only been designing winged flying-machines since around 1900, and that the above satire would work just as well if it were about "Extreme Jumbo-jet Building".
We who have been foisting so much change on others lately seem to be having a little trouble with the concept ourselves. Suspension bridges, jumbo-jets, integrated circuit design, and so much more, are all life's way of telling programmers: it's time to change now.
. . . . .
The documentary video recommended above describes an actual case where implementers had to change concrete to meet the plan. In one section, the concrete poured did not meet the requirements documented in those big up front specifications. Observe that without the big, detailed, up front specification, they could not have known what capabilities would ultimately be required of the concrete, and so would have had no way to know that it did not meet those capabilities.
In other words, without a big up front design this team could not have learned of the concrete's shortcomings until after the bridge was built; possibly not until it fell down. It is for this reason, and not because concrete is hard to change, that bridge builders put big up front effort into their bridge designs.
Is software really different? In the real world, the costs in resources and risk associated with software projects that fail after they have begun are remarkably similar to those associated with other types of projects. That is to say, they are stunningly high.
We Know How to Do This
The best chefs will always experiment and improvise to come up with new and better ways to build a world-class meal, but not while serving diners who have come for a world-class meal. The best building and bridge designers will continually experiment with new construction techniques and new materials combinations to improve their craft and designs, but not while building a project counted on to be safe and reliable. Even comedians (can you imagine anything more pliable than a comedy act?) will do improvisation and try out their new material, but only in the hard-core dives that are the purview of the most serious comedy enthusiast. And then, only when the customers understand that their purpose is to be guinea pigs (e.g. open mic or improv night). Trial and error is good. It improves us as individual practitioners, and it improves the profession as a whole. But it should not be sold as a product to people who expect to use it in practical and sometimes critical areas of their lives and businesses. Engaging in such folly may tarnish the credibility of our profession.
Left Foot First
In our early years, we in the field of software development learned a great deal about software design and structure. It is time, and we are ready now, to make and keep schedule commitments, well-planned "big up front designed" specifications, and project budgets. How can we be sure it's time now? By observing how silly our excuses for not doing these things have become. . . . . .
Three Simple Questions The following three questions should be asked by anyone requesting development work of any kind and should be answered to the requester's satisfaction by anyone performing that work.
This article is © Copyright, Dominic John Repici 2003 - 2011 ALL RIGHTS RESERVED. It is used here with permission. Links to this article are always welcome. However, you may not copy, modify, or distribute this work or any part of it without first obtaining express written permission from Dominic John Repici Production and distribution of derivative products, such as displaying this content along with directly related content in a common browser view are expressly forbidden! Those wishing to obtain permission to distribute copies of this article or derivatives in any form should contact D. John Repici at Creativyst. Permissions printed over any code, DTD, or schema files are supported as our permission statement for those constructs.
As a software guy, I'll be the first to admit my human language writing skills need all the help they can get. I'm thankful for people who have responded to my calls for help in that department. This article owes a lot to suggestions and support from them. Joo KNOW who joo ah! I'm in your debt. Extra special thanks to:
Links to selected resources will be added here.