The traditional predictive approach to project management is being rejected in favour an adaptive or Agile approach.
This is not a matter of buzz-words or faddish management technologies, instead it is a genuine commitment to help clients get the software they actually want - on time and within budget.
The Problem
There is a problem with the delivery of software. The more complex a project the greater the chance the project will be delivered over budget and behind schedule. As a project grows in complexity there comes a point where this potential for failure becomes almost a guarantee. Most experienced project managers understand this and strain their sinews to prevent it from happening and most experienced programmers have lived through the intense disappointment of seeing their work fail to achieve its initial promise. Yet time and again, despite the best efforts of genuinely talented and motivated people, software projects are delivered late, cost too much and do not function as the client expected - Why is this?
For each failed software project the problem typically turns out to be the plan. Now that may seem trivially obvious. Looking back over a failed project it is easy to suggest that if only the plan had been more precise then the project could have been more controlled and so more successful.
This is not correct.
The problem does not lie in the quality of the planning, the problem lies in the type of plan, specifically the attempt to create an up-front plan that covers the entire project life-cycle. This is not so obvious - how can you run a project without deciding what you need up-front?
To understand why up-front planning impedes the successful delivery of quality software it is first necessary to understand what is meant by a plan in this traditional sense, and then see how this concept can be dispensed with and replaced with a new type of planning mechanism.
What's in a Plan
At the start of a traditional project there is the familiar requirements-capture phase. This typically involves the writing of various specifications, a user specification that outlines the requirements in the language of the client, a functional specification that outlines the requirements in the language of the programmer and then perhaps a fully detailed technical specification that describes the requirements in a pseudo programming language.
Once complete, these detailed specifications provide the basis for all future work. They allow predictions to be made about the project's costs as well as its anticipated schedule. Specification documents also serve a secondary function. They give both the client and the engineers a form of 'contract' that, upon project delivery, allows everybody to compare what was promised with what was actually delivered.
This up-front planning process is often called the 'waterfall' model, it is a highly structured methodology that steps through requirements-capture, analysis, design, coding, and testing in a strict, pre-planned sequence. Progress is generally measured in terms of deliverable artifacts: requirement specifications, design documents, test plans and code reviews.
The Waterfall is Broken
There are good reasons why traditional, up-front planning fails. Unfortunately these reasons tend to make both clients and engineers feel uncomfortable so they are rarely spoken out loud.
Firstly, up-front planning means that the specification documents are written before any software is built. Experts, using all their intellectual powers and experience, attempt to imagine the software and in doing so mentally traverse all of its myriad details. Since no software has yet been built, the hypothetical assertions contained within these documents cannot be tested experimentally. In science an hypothesis that cannot be tested is called pseudo-science and by the same token a specification whose assumptions cannot be tested should be considered pseudo-planning.
Secondly, at the start of any reasonably complex project there is always an inescapable knowledge gap. This gap exists between:
- The business knowledge brought by the client
- The technical knowledge brought by the engineers
To begin with these two bodies of knowledge do not mix well as the clients do not really understand the language of software engineering and the engineers do not really understand the language of the client's specific business. This will change as time goes on and eventually the distinct bodies of information will mix and become one shared information landscape. However, at the start of a project when traditional up-front planning occurs, this inevitable knowledge gap leads to two critical and incorrect assumptions:
1. The client knows what they want their new software to do
Many clients come to a project with good idea of what they want, perhaps they have spent time and effort working this out, perhaps they have a legacy system that shows them much of they want and what they do not want. However at the start of a project the client cannot know what they want in sufficient detail to create a complete and precise plan. They can provide a business vision and they can provide business constraints but they cannot state in detail the processes required to deliver their vision because they have not yet absorbed the necessary details of the engineering environment. A superficial understanding can be gleaned during the initial planning meetings but this will not produce a sufficient understanding of the software they are commissioning.
2. The engineers know how to implement the client's business vision
Many engineers come to a project with a good idea of how to build business systems. They will have spent considerable time and effort building other, perhaps similar systems. However at the start of a project engineers cannot know how to implement the precise details of a specific business application because they have not yet absorbed the detailed business knowledge brought by the client. A superficial understanding can be gleaned during the initial planning meetings but this will not produce a sufficient understanding of the software they are being asked to deliver.
Predictive planning fails because an accurate plan requires a genuine, non-superficial understanding of both the client's business knowledge and the engineer's technical knowledge. Traditional specifications are created at the start of a project when both parties have not had enough time to come to such an understanding. It takes much effort to synthesize the two bodies of knowledge into a coherent whole, far more than can reasonably be assigned during the requirements-capture phase.
This means that plans created at the start of the project cannot be more than partially informed guesswork. Given that the nature of complex systems make them particularly sensitive to changes in small details, a plan for a complex system created with incomplete knowledge must perforce be a recipe for failure by degrees.
Does this really make up-front planning redundant? Is there a way to make the synthesis of the client and technical knowledge more efficient, perhaps by using advanced planning software? If this could be achieved then perhaps the planners could write effective up-front specifications that lead to accurate long-term costings and schedules.
Unfortunately there is another, more fundamental reason why detailed specifications must fail - regardless of their precision.
A specification is a description that attempts to outline features and functions in a natural language such as English. Yet software is actually written in the very precise syntax of a machine language. Engineers know that only computer code can truly express the details of a software vision, a natural language specification cannot be logically accurate enough. This means that natural language specifications must leave many implementation details open to interpretation forcing the engineer to skilfully choose from a set of implied options. Yet complex systems are sensitive to precisely these sorts of technical details, different choices will lead to different systems and, as often as not, unfulfilled client expectations.
Therefore, even where a specification guesses correctly, the natural language descriptions will contain subtle choices and hidden contradictions. It is only when the fuzzy language of the specification is transformed into the precise reality of the code that these choices and contradictions become apparent.
This leads to a profound truth about the nature of specifications: Greater precision does not lead to greater control. Instead the greater the precision the more varied and subtle the choices and contradictions become.
Planning For Success
Understanding these fundamental flaws at the heart of traditional software delivery, many forward looking managers and engineers are now moving towards a new project control methodology. In contrast to up-front or predictive planning this new methodology uses repeated bursts of short-term adaptive planning.
Agile Software Development throws out long-term planning and with it the traditional concept of a specification. Instead agile projects start with everybody discussing and sharing a simple vision of the end product. The vision is really no more than a mission statement that, at this early stage, explicitly removes the need for engineers to fully understand the business and for the client to fully understand the technology.
This means that an agile project can get started almost straight away, with the absolute minimum of requirements-capture. Instead of a long, costly and ultimately self-defeating planning phase, the engineers get to work building the first version (iteration) of what will become a rolling beta. Armed with a very short term plan covering just one or two weeks of work, the engineers build the first iteration and deliver it to the client for discussion and criticism. The rolling-beta is still only a sketch, an outline of the most important functions and how they might fit together. Mistakes and incorrect assumptions will have been made, indeed given the knowledge gap they cannot be avoided, but the mistakes are identified and quickly eliminated as the rolling-beta is regularly assessed by the client and engineers in close collaboration.
Once the first iteration is signed-off then the process begins again, a new short term plan is created and work begins on the second iteration. This iterative development continues and as the knowledge gap closes so the requirements and hence the software become ever more detailed and coherent.
Embracing Function Creep
As this hands-on process continues the client comes to properly understand the technical environment, what is expensive and what is possible, and as their knowledge grows so they begin to see new possibilities.
Clients changing their minds or adding new features during development is traditionally called function creep and remains the enemy of traditional planners. Yet to suppress this is to deny that clients can learn and modify their expectations as they see their software progressing. Rather than trying to ignore the client's input, the agile iterative process welcomes it as new and valuable knowledge.
Thus the client is encouraged to re-specify their product as it is being written. This is the ultimate guarantee that, in the end, the client will be satisfied. It is hard for a client to be surprised or disappointed with their software if they have played an active part in designing and deciding the goals at each iteration.
Equally, as the iterative process progresses the engineers will also come to a genuine understanding of the business. This allows the engineers to discuss the business processes with the client in a manner that allows a useful exchange of knowledge to take place. Questions to the client can be appropriately framed using the business terminology both the client and the engineers now share. Since the frequent iterations and short-term planning means that any incorrect business assumptions are quickly discovered, such mistakes can be corrected with the minimum of effort.
Engineers too, once they come to a genuine understanding of the business, can start to usefully contribute to the re-specification of the rolling-beta. New ideas and inspirations, whatever their source, can be welcomed, discussed and possibly incorporated as the software adapts over time.
Job Satisfaction
In summary, an agile software system evolves under the twin constraints of the client's business vision and the engineering environment's technical limitations. As the client and engineers come to a mutual understanding so new ideas bubble up and are incorporated as bad old ideas are identified and discarded. Before starting each iteration everybody discusses, negotiates and quickly reaches an understanding of what is actually required to fulfil the next set of short-term goals.
Thus an agile system organically grows its natural complexity out of a fundamental simplicity. As a result there are fewer surprises, the project risks are minimised and the client is more likely to get software that works.
