A critical concept that motivates full-lifecycle testing is the cost of change.
Figure 1 depicts the traditional cost of change curve for the single release of a solution following a serial
(waterfall) process. It shows the relative cost of addressing a changed requirement, either because it was
missed or misunderstood, throughout the lifecycle. As you can see the cost of fixing errors increases
exponentially the later they are detected in the development lifecycle because the artifacts within a serial
process build on each other.
For example, if you make an requirements error and find it during the requirements phase it is inexpensive to
fix. You merely change a portion of your requirements model. A change of this scope is on the order of $1 (you
do a little bit of retyping/remodeling). If you do not find it until the design stage, it is more expensive to
fix. Not only do you have to change your analysis, you also have to reevaluate and potentially modify the
sections of your design based on the faulty analysis. This change is on the order of $10 (you do a little more
retyping/remodeling). If you do not find the problem until programming, you need to update your analysis,
design, and potentially scrap portions of your code, all because of a missed or misunderstood user requirement.
This error is on the order of $100, because of all the wasted development time based on the faulty requirement.
Furthermore, if you find the error during the traditional testing stage, it is on the order of $1,000 to fix
(you need to update your documentation and scrap/rewrite large portions of code). Finally, if the error gets
past you into production, you are looking at a repair cost on the order of $10,000+ to fix (you need to send out
update disks, fix the database, restore old data, and rewrite/reprint manuals).
Figure 1. Traditional cost of change curve.
It is clear from Figure 1 that you want to test often and test early. By reducing the
feedback loop, the time between creating something and validating it, you will clearly reduce the cost of
change. In fact,
argues that in
eXtreme Programming (XP) the cost of change
curve is flat, more along the lines of what is presented in Figure 2. Heresy you say! Not at
all, Beck's curve reflects the exact same fundamental rules that
Figure 1 does. Once again, heresy you say! Not at all, the difference is that the feedback loop is
dramatically reduced in XP. One way that you do so is to take a
test-driven design (TDD) approach (
Beck 2003; Astels 2003). With a
TDD approach the feedback loop is effectively reduced to minutes - instead of the days, weeks, or even months
which is typical in serial processes - and as a result there isn't on opportunity for the cost of change to get
out of hand.
Figure 2. Kent Beck's cost of change curve.
|Although many people have questioned Beck's claim, which was based on his own anecdotal evidence and
initially presented as a metaphor to help people to rethink the things that they have believed for years.
Frankly all he's done IMHO is found a way to do what software engineering has recommended for a long time
now, to test as early as possible - testing first is about as early are you're going to get. To be fair,
there's more to this than simply TDD. With XP you reduce the feedback loop through
pair programming as well as by
working closely with their customers (stakeholders). One advantage of working closely with stakeholders is
that they are available to explain their requirements to you, increasing the chance that you don't
misunderstand them, and you can show them your work to get feedback from them which enables you to quickly
determine if you've built the right thing. The cost of change is also reduced by an explicit focus on
writing high-quality code and by keeping it good through
refactoring, a technique where
you improve the design of your code without adding functionality to it. By traveling light, in other words
by retaining the minimum amount of artifacts required to support the initiative, there is less to update when a
change does occur.
Teams that pair together stay together.
shows, the reason why the agile cost of change curve has flattened is because we follow techniques which
reduce the feedback cycle. Agile techniques, shown in green, have short feedback cycles and therefore are at
the flat end of the curve. Traditional techniques, shown in red, have longer feedback cycles and therefore are
at the higher-cost end of the curve.
Figure 3. Comparing the feedback cycle of various development techniques.
presents a cost of change curve that I think you can safely expect for agile software development. As
you can see the curve doesn't completely flatten but in fact rises gently over time. There are several reasons
- You travel heavier over time. Minimally your business code and your test code bases will grow
over time, increasing the chance that any change that does occur will touch more things later in the
- Non-code artifacts aren't as flexible as code. Not only will your code base increase over time,
so will your non-code base. There will be documents such as user manuals, operations manuals, and system
overview documents that you will need to update. There are models, perhaps a requirements or
architectural model, that you'll also need to update over time. Taking an
Agile Modeling Driven Development (AMDD) approach
will help to reduce the cost of this but will not fully eliminate it.
- Deployment issues may increase your costs. When it becomes expensive to release software, perhaps
you choose to distribute CDs instead of releasing software electronically to shared servers, your cost
of change increases because you begin to follow more conservative, and more expensive, procedures (for
example you may be more inclined to hold reviews).
- You may not be perfectly agile. Many agile software development teams find themselves in very
non-agile environments and as a result are forced to follow procedures, such as additional paperwork or
technical reviews, that increase their overall costs. These procedures not only increase the feedback
loop they are very often not conducive to supporting change.
- We travel lighter. Agilists create high-value documentation, or in other words far less
documentation than traditionalists do, and we don't tolerate unnecessary bureaucracy. As a result we can
act on changes quicker because we have less work to do -- in other words, we maximize the amount of work
Figure 4. A realistic cost of change curve for agile software development.
An important thing to understand about all three cost curves is that they represent the costs of change for a
single, production release of software. Over time, as your system is released over and over you should expect
your cost of change to rise over time. This is due to the fact that as the system grows you simply have more to
code, models, documents, and so on to work with, increasing that chance that your team will need to work with
artifacts that they haven't touched for awhile. Although unfamiliarity will make it harder to work with and
change an artifact, if you actively keep your artifacts of high quality they will be easier to change.
Another important concept concerning all three curves is that their scope is the development of major release
of a system. Following the traditional approach some systems are released once and then bug fixes are applied
over time via patches. Other times an incremental approach is taken where major releases are developed and
deployed every year or two. With an agile approach an incremental approach is typically taken although the
release timeframe is often shorter - for example releases once a quarter or once every six months aren't
uncommon, the important thing is that your release schedule reflects the needs of your users. Once the release
of your system is in production the cost of change curve can change. Fixing errors in production is often
expensive because the cost of change can become dominated by different issues. First, the costs to recover from
a problem can be substantial if the error, which could very well be the result of a misunderstood requirement,
corrupts a large amount of data. Or in the case of commercial software, or at least "customer facing" software
that is used by the customers of your organization, the public humiliation of faulty software could be
substantial (customers no longer trust you for example). Second, the cost to redeploy your system, as noted
above, can be very large in some situations. Third, your strategy for dealing with errors affects the costs. If
you decide to simply treat the change as a new requirement for a future release of the system, then the cost of
change curve remains the same because you're now within the scope of a new release. However, some production
defects need to be addressed right away, forcing you to do an interim patch, which can clearly be expensive.
When you include the cost of interim patches into the curves my expectation is that Figure 1
will flatten out at the high level that it has reached and that both Figure 2 and Figure 3
with potentially have jumps in them depending on your situation.
What is the implication for modeling? Although it may not be possible to reduce the feedback loop for
non-code artifacts so dramatically, it seems clear that it is worth your while to find techniques that allow you
to validate your development artifacts as early as possible.
Note: Portions of this article has been excerpted from
The Object Primer 3/e.