THE HISTORY OF WATERFALL DEVELOPMENT

In 1970, Winston W.Royce wrote an article in which he described a sequential model for developing software wherein development flows, waterfall-like, through phases of requirements, analysis, design, implementation, testing, and maintenance. He then went on to explain why this model couldn't work and described an iterative process as a much better alternative. Inexplicably, not only did the software community latch on to the concept of the "Waterfall" approach, but they even credited Royce with having proposed it. A common feature of waterfall projects is that they tend to fail - 18% cancelled, 53% late, over budget or descoped - even worse for large projects (Standish Group report).

The approach, which subsequently became universally used and known as the Waterfall method, is predicated on the principle of completing, in detail, each stage before moving on to the next; the optimistic assumption is that the customer knows and can express the detailed requirements at the outset, that the software requirements as specified can be implemented predictably and that the requirements don't change during development.

Royce points out that the Testing stage, which occurs towards the end of the development cycle, is the first time that the developed system is experienced, as distinct from analysed. In the event that the delivered system is not what was expected or some requirements have been missed, then a major redesign is necessary. At the very least, the system design must be revised in response to weaknesses or errors exposed in testing, but if external criteria have not been satisfied then either the requirements must be changed (descoped) or the whole system must be revisited. He comments "In effect, the development process has returned to the origin and one can expect up to a 100% overrun in schedule and/or costs".

Royce accepts that the sequence of events is logically sound but that the principle of finalising each stage before the next commences is flawed. His outline alternative proposal comprises 5 steps:

1) Insert a preliminary Program Design phase between Software Requirements and Analysis; whilst this must be based on incomplete information (as there is no analysis), it will be able to deal with fundamental non-functional requirements such as performance and volumes, thus setting some constraints for subsequent analysis.

2) Document the Design so that subsequent phases are working towards a common, well-defined specification.

3) Do it Twice. Construct a pilot version of the system, the contents of which will depend on the extent and nature of critical problem areas. It is the second version which is delivered to the customer.

4) Plan, Control and Monitor Testing.

5) Involve the Customer so that he commits himself to the direction the development is taking at the earliest stage.

In modern-day parlance, this proposed approach would be described as risk-driven and iterative.

The majority of software projects are today still based on the waterfall approach, perhaps with some fine-tuning, and are often accompanied by intimidating methods and standards whose sheer complexity implies sophistication - this can be misleading. Characteristics of such projects typically include:

• Major problems occur only during the Integration stage, usually too late to be accommodated within the project calendar. The effect tends to be either a descoping of the project to deliver it on time, a squeeze on the time allowed for testing to try to hit the end date, an overrun or a combination of any or all of these.

• The tendency to build detailed project plans, right up to the end of the project, at as early a stage as possible. These soon become obsolete and are typically subject to frequent and time-consuming revision. This has a contingent effect in terms of disruption to the teams work schedules and often morale.

• Too much optimism in the earlier stages in terms of the amount of rework that will ultimately be necessary and the lack of adequate planning for rework.

• Potential disconnect with the users between requirements gathering and delivery because of the time lapse.

  and last, but certainly not least -

• The insistence by management that an accurate estimate of cost, timescales and resource requirements is submitted before the work even starts

Royce set the scene for iterative development, although it must be recognised that it was not a new concept: other scientific and engineering disciplines had been following an iterative approach for generations, if often informally. The simple rationale is that if you're doing something new, first explore the technical boundaries and constraints, by experimentation or some other means, then, when you know what is technically viable, review the design with the customer and confirm that both the technical and functional content are acceptable and justifiable. There can be few among us who cannot recall a number of major public sector projects which would have been abandoned at an early stage had the true ultimate costs and timescales been known.

Of course, much has changed since Royce wrote his article in 1970. Software development languages and tools, architectural concepts, reusable components, hardware cost and performance, sophisticated modelling techniques - and the maturing of iterative development techniques by a relatively small (and relatively successful) proportion of the development community.

The iterative process adopts the logical sequence of events of the Waterfall model but it follows the full lifecycle several times within a single project. Each iteration goes through the activities of Requirements, Analysis, Design, Coding, Integration and Testing and produces production-quality software at the end of the iteration. The content of each iteration is determined according to risk - the new, the most challenging, the most critical elements are built first, usually by a core team of the most skilled practitioners, so that the true nature of the project and its implications in terms of cost and time are constantly under review. Thus, certainty increases as the project proceeds - or, if the development is no longer viable, it can be halted at an early stage and at minimum cost.

The actual number of iterations in a given project will depend on the nature of the project, in particular the proportion of the content that is considered risky, but it is likely to be between 3 and 10. An important aspect of the approach is that each iteration is "time-boxed" - if it is planned to take 4 weeks, then it will cease at the end of that time. The subsequent iteration will incorporate any unfinished work but with the benefit of knowledge gained during the execution of its predecessor.

ITERATIVE DEVELOPMENT SUMMARY

Iterative developments display these characteristics:

• Highest risk items are built first
• Change is expected and accommodated, but controlled
• Testing occurs early and often
• Iterations are time-boxed not function-boxed
• Deliver early, deliver often
• Deliver production quality software each time

• Early risk discovery and mitigation
• Accommodates change and provokes earlier identification of change
• Manageable complexity
• Confidence from early, repeated success
• Early partial product
• Better progress tracking and predictability
• Higher quality, fewer defects
• Software better matches user needs
• Early and regular process improvement
• Communication and engagement demanded
• Prototyping and feedback encouraged

Winston Royce's original article can be downloaded in pdf format from the University of Maryland's website:

http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf.

Acknowledgments
"Managing the Development of Large Software Systems", Dr. Winston W.Royce, University of Maryland.
CHAOS Demographics and Project Resolution 2004, Standish Group

Copyright FCSL 2006. All rights reserved.
http://www.fcslTech.com

Reprint rights
This article may be published or distributed freely in its entirety only and must include the author's name, copyright notice and copyright owner's URL shown above.