Question: Why do we test software?
Answer: Because history tells us that there will be defects. If software always worked, we wouldn’t verify that it did.
Question: Why then, are there defects in software?
Answer: The current (known-to-man) process of developing software is defective.
We can pretend the process isn’t defective, but reality tells us otherwise. Its end result; software, is defective.
The agile manifesto
Individuals and interactions over processes and tools.
(processes and tools aren’t good enough)
Working software over comprehensive documentation.
(software can’t be accurately documented)
Customer collaboration over contract negotiation.
(we need to constantly query the customer for accurate information)
Responding to change over following a plan.
(the plan doesn’t work when the environment changes)
These rules are not the perfect way of developing software, though they could very well be the perfect answer(s) to dealing with the current reality of software development and customer requirements.
Dealing with uncertainty
An agile team knows that they don’t know. An agile team knows that no one knows. They are agile because they need to deal with not knowing.
An agile team is telling you:
“We know what to do this month, and will evaluate what to do next month depending on what next month looks like. Its weird, we know, but there you have it.”
“Until you figure out what you really need, we will remain agile so we can react to your changes.”
Sounds reactive? It is.
It is short-term thinking. It is the kind of short-term thinking that remains a total necessity in a changing environment. It is also expensive (wasteful) thinking.
In those statements you also find the a big reason why agile can be a hard sell. How far up in an organization can you sell the truth that the project really is out of control?
Iteration length and efficiency
The most efficient team is one developer writing code for herself. The second most efficient teamis a customer and a developer/designer pair-programming-requesting. Some development work is well suited for this, for example prototyping and GUI development.
This is about as short an iteration cycle you can get:
Customer says: Looks bad.
Customer says: Looks good.
What about really long iteration lengths? A six month iteration would need very detailed specs (description) to complete. It seems there exists some kind of correlation between iteration length and specification detail.
The manifesto tells us, “customer collaboration”. In the iteration cycle example above you have a 50% shared time investment between figuring out what to do, and doing it. So, in an iteration where you have four coders doing 40h/week each, say 640 hours a month, and given an iteration (sprint) length of four weeks, how much customer collaboration do you need?
We don’t know the answer to that question, but its significantly more than 4 hours of monthly sprint planning work. Given an iteration cycle time, there is an optimum ratio between figuring out what to do and doing it.
Most of the times it is grossly understated in favor of the doing. On average, teams need to spend more time discussing what to do with the customer, and spend less time writing code.
Waterfall is a more efficient way of developing software, provided there is accuracy in describing what should be done in the long-term. However, experience tells us otherwise; that we can’t predict the long-term.
Why? Why do we have to develop software in short increments?
Designing the logic circuits of a complete system is a near impossible task. Agile admits this, and prefers to develop empirically in short-term increments.
If you design for the long-term and then need to change something, the system becomes subject to the butterfly effect; in Chaos theory “sensitive dependence on initial conditions”. That means a small change in a complex system can have large effects elsewhere in the system. A change in requirements can have devastating effects on overall design.
Customer: “The print function doesn’t work”
Developer (shrugging): “You wanted us to change the date format in the product import, and that change seems to have broken the whole print function. Sorry.”
Customer: “Its okay. I got more $$$.”
No one can accurately predict the effect of any change. We can minimize the risk, but we can’t remove it.
|An agile team is in effect telling the customer: “If you’re gonna keep changing stuff, we’re gonna deal with it by developing in short-term steps.”
An agile team is also admitting to itself: “We don’t know how to design complex systems except by short empirical steps.”
If we remove complexity and change from the equation, there is no need for iterative development. We would be dealing with a closed system that could be designed, coded, and be done with. Unfortunately, complexity and change exists. What we want to do is minimize them, and use a method that minimizes their effects.
The waterfall model had a few faulty assumptions:
- The team can deal with any level of complexity
- Requirements never change
Since both statements are false we remain “stuck” with Agile as the so-far best (but yet imperfect) way of doing it.
Approximated cost in hours to find a software defect:
Requirement Review 10 Minutes
Code Review 20 Minutes
Unit Level Testing 1 Hour
Automated Tests 10 Hours
Manual Tests 15 Hours
User Testing 20 Hours
Customer Finds 30+ Hours
(from “Quality through Change-Based Test Management” — IBM)
Most software projects think they are doing well with a process for user testing in place. That is on the hour cost of 20 hours per defect. Sounds good to you?
Spending money in the maintenance ball is spending it wrong. Spending it right would be to focus on the little requirements ball. There is where you find real leverage (10 minutes to find a defect vs. 30+ hours if an end user finds it).
Cost to user
The cost to user is difficult to estimate, but very important. Sometimes the cost is visible, for example a software defect that prevents 1000 users from working would be quite noticeable.
Sometimes the cost is hidden and more insidious. A design flaw that makes something difficult to accomplish carries a cost to user as well, but it in the form of stress. Imagine a bad design flaw staying with the application for five years, stressing out thousands of users that has to deal with it on a daily basis.
- Aim for user friendliness
- Thorough testing before release
- Easy way to report bugs
- Knowledgeable help-desk
Avoid maintenance cost through increased reliability
Software should be designed with maintenance goals in mind. Most of the time we design for unneeded complexity. Use a simple infrastructure, and make it complex only if you need to. Every item you add to the chain of items needed for operation increases the risk of failure of said chain.
- 24h service, or office hours?
- Automated surveillance reduces downtime
- Avoid complexity in server infrastructure!
- Surveillance tools for server admins
Minimize developer hours spent on data repair
Sometimes we need to alter the data (or metadata) of the database. The need can come from a requirements change, from a database crash, from badly designed user input validation, or from external data-load. If you are using very complex data structures in the database, these changes will be very costly to carry out.
- Easier to salvage database designed for enough normalization
- Easier to repair non-dynamic data structures
- Establish data contracts for integration early
- Use strict database validation. Never allow input or import of erroneous data
The developer team will experience turnover sooner or later. These costs can be reduced by relevant documentation and use of standards and standard technology. If the technology is special-special, and no one knows how to do anything except for the guy who is leaving, replacing him will obviously be very costly.
- Good specifications makes team turnover and knowledge transfer less costly
- Standard technology makes it easier replace team members.
- Standard guidelines makes it easier to replace team members.
Software design can increase maintainability
It is no surprise that a good design increases maintainability. What does that really mean? It means making changes to the application will be cheaper. Changes can be both developing something new, and repairing something that is broken. The challenge here is to keep the design working while adding complexity. Shortcuts introduce software rot and must be avoided.
- Domain driven design will reduce complexity
- If two things can be separate they should be separate. DRY (don’t repeat yourself) is a double edged sword.
- Good design is not an awesome ball of yarn that can do anything
- Good design is as simple as possible, not simpler.
Be aware of software entropy
The development team is in a constant fight against software entropy. Every change to the codebase carries with it the chance to mess something up. It can be by introducing new bugs, or simply by destroying the design. Alot of team turnover coupled with feature creep will guarantee software rot.
The team must be made aware of this entropy, because the application is constantly moving towards it. If it sets in too far, you will have an unmaintainable application where changes will be so costly and risky that its not worth doing them.
When you get the feeling that developers are very hesitant to add any new features, that they prefer to poke at the application from afar (with a tall rod), you are probably looking at the putrid pile of an unmaintainable application.
That is why every new feature must not only be weighed against the direct cost to code it, but also against the hidden cost of adding complexity to the application as a whole, making it more expensive to maintain.
- Allocate time for redesign and the fight of software rot
- Be extremely wary of feature creep
- Keep conceptual integrity in mind when adding new features
The broken windows theory is a criminological theory of the normsetting and signalling effects of urban disorder and vandalism on additional crime and anti-social behavior. The theory states that monitoring and maintaining urban environments in a well-ordered condition may prevent further vandalism as well as an escalation into more serious crime.
Wikipedia, Broken windows theory
The theory is that fixing broken windows could be a method to combat further vandalism. Instead of letting a broken window turn into a broken door turn into a broken wall turn into a wrecked house, you would simply repair the windows as they break.
The basic idea can apply to many different areas of life, for example litter on the streets, neglecting to pay your bills, and so on. This is entropy at work; what requires effort to maintain will break down without a steady input of effort. If you would chart the phenomenon it would look something like an exponential function bottoming out due to diminishing returns (it requires less effort to maintain a low entropy state).
The maintenance of software is also subject to entropy. Software needs a constant input of effort to remain maintainable.
The team agrees on rules regarding documentation and specifications. They set up rules as to what type of code goes into which parts, what functions goes into which domains. The team agrees on rules as to testing and qc, a promise to refactor early and often. All the good stuff, yea.
The application is being developed, and its quality is according to the rules set up by the team. If the rules are good and the team follows them, good software will result.
After development the application goes into production and the team scales down, sometimes leaving a group of junior developers to maintain the application.
After release a host of bugs and design shortcomings appear, often within the first few months. Some will require little effort to fix, some will require more. Suddenly the team is dealing with fewer hours yet harsher deadlines. Users are phoning in demanding new releases.
At this point the senior developer (or architect) might have left the team and is now allocated to other projects.
Keeping software working
Without the understanding of the design, and the understanding of discipline, will the junior team continue refactoring? Updating specifications? Separation of code? Documentation? Testing? Qc?
All good habits that were agreed upon as critical during development are soon forgotten in the maintenance phase. Entropy and software rot sets in, shortcuts are being made, and the codebase goes beyond repair.
The truth is that it takes a good deal of effort to keep a codebase workable.
Graphing the effort in maintaining a codebase might look like a jigsaw. For every change maintainability goes down. You need effort to push it back up again; effort to keep the design working, keep the specs updated, test the changes, qc, and so on.
If the team neglects pushing it back into “good as new” for too long, it will degenerate past an event horizon where the only way out is do it all over again. There are too many broken windows and too much effort is required to make it “whole” again.
In the figure above, the red line describes changes to code. For every change quality and maintainability goes down as complexity increases. That is why for every new feature, there is an associated effort cost to make sure the design is working. That means reevaluating object models, refactoring code, rewriting automated tests, and so on.
- Team lacking understanding of overall design
- Inexperienced coders
- No specifications
- Time pressure
- No method in place to deal with code maintenance
In the figure above there is a sufficient input of effort to bring the codebase back to quality after each change. The codebase remains maintainable, and there are fewer bugs.
- Coders understand value of refactoring
- Coders understand overall design
- Structure is in place to deal with redesign and refactoring
- Unit testing is in place
- Team will question feature creep
Fighting software rot
Software rot will set in sooner or later. If it goes too far, the application becomes unmaintainable. We want to keep the quality high for as long as possible, and there are a few things that can be done prolong the active life of software. Many of these items come through keeping a good practice culture in the project where you don’t allow for broken windows.
- Understand that redesign takes time
- Be extremely wary of feature creep
- Keep conceptual integrity in mind when adding new features
- Don’t add new features when stability is an existing problem
- Team members knowledgeable of overall design will do the big changes
- Separation of code
- Unit testing in place will encourage refactoring
- Good specifications will encourage refactoring
You could argue that these items aren’t practical, and yes some applications are just too far gone to help. What is important is to understand that some applications are vastly more maintainable than others. With good maintainability, adding new and complex features actually takes very little time. On the other hand, if the team chooses to ignore software rot you will quickly find yourself with an unmaintainable application where even the smaller features take a long time to implement.
The most critical period in an applications life cycle is not so much its infancy as its adolescence.
Before release the customer accepts bugs. Milestone deadlines are MS Project phenomena.
After release the nice customer is replaced by angry users. Deadlines? Yesterday.
Before release you have whiteboard meetings, cookies and pleasantries.
After release you get nasty phone calls and late nights at work.
The truth is: after release is where the design will be tested and where the difficult redesign will occur.
Is this, the period between release and maturity, where you need team discipline more than ever? Is this where you need experienced people who understand the applications original design and goals?
I think so.
At release the application isn’t “done”, it just got started.
Sounds like newspeak dunnit? Well, there is alot of uncertatinty when it comes to documentation and building software. Should you ask a team, mostly everyone would agree that documentation is good. If everyone thinks it is good, then why isn’t it happening? People know what they are supposed to answer, but they don’t know why. In fact, many secretly believe that documentation is a time-waster.
Documentation is useless because:
- Software is in a constant state of flux; documenting the changes is too hard
- No one reads documentation–program code is the real truth
My team doesn’t use documentation because:
- The software is too old–no one knows how things work anyway
- No one wants to pay for documentation
- We don’t have the time to document everything we do
If people really knew why specifications are good, they would be doing it all the time. My belief is that specifications are easily worth the time invested.
The use case
Here is something very much simplified, could be taken from an online system where customers purchase tickets to soccer games.
Examining this more closely, some questions turn up:
What if the game has already been played? Should the customer be able to buy tickets for past games?
What if the credit card data is rejected? Should we return the customer to the list of games?
What if the ticket server isn't available at the time of purchase? Should we undo our transaction from the customer credit card?
Unless we assume that the developer will find these items, and fill in the blanks, patching things up as he goes, the specification needs to be more detailed. I didn’t say better–it is fine for its abstraction level. But it needs more detail before it turns into working software.
If questions like these fall all the way through to the software coding level unattended, the application will turn out no better than the prescience of its developers.
Dealing with Lucky Luke
Again, why? Do developers really need specs? Can’t they just do it right? Well, obviously “right” is a matter of opinion. That is why we try to specify things, and alot of people, like Lucky Luke, they don’t like it.
The Cowboy coder
The genius Cowboy dislikes specifications because they hamper his freedom to do his thing–which is coming up with new genius solutions! Publicly he follows the company line. Privately he knows best and does as he pleases. The Cowboy coder is often covertly supported by the Pragmatist.
The pragmatist is a “genius” on the customer side who “gets things done”. Publicly dislikes specification and any kind of administration. Uses the “Yes yes, you keep on typing those docs. Just don’t bother us doers.” Privately fears specification because he understands it reduces his power and makes him accountable.
The Paper Shuffler
The Paper Shuffler believes that documentation exists in its own right. Documenting is simply something people in offices do, preferably with as much irrelevancy as possible. The Shuffler is often a failed doer (and humorously used as a straw man by The Pragmatist). Will slowly suffocate any good project with impossible hoops and hurdles if sufficiently empowered by higher-ups.
The three actually work quite well together; the Paper Shuffler keeps an outside apparency of order while the Pragmatist and the Cowboy coder “get things done”. It becomes a very private project that is difficult to change or understand.
This is the team that secretly uses a slackware linux desktop running apache and some obscure java beta framework as their production server, while at the same time renting an empty high-powered MS 2008 Server as the official server. They would, of course, be using that one if the server IT guys were competent enough to run it properly (and if C# was a real language).
With these teams, project documentation is impeccable while actual product documentation is sorely missing.
Specification and Agile
As a collection of ideas, agile (or Iterative) is awesome, because it openly distinguishes from waterfall which really was a horrible way of developing software. That doesn’t mean iterative development is unnecessary loose and haphazard; in fact one of its most popular methodologies, Scrum, is really quite formal.
The main thing about agile is its incremental and iterative nature. That is why I argue that agile is an enabler of strong documentation, because the method to write good specs is iterative as well.
Different life-cycles of specification (note, waterfall is not iterative development, only iterative maintenance).
In the figure above, I’ve tried to highlight the differences. The mixed method would be a team that sees the problems with waterfall, but doesn’t understand why specification is important in maintenance as well.
Iterative documentation process
- Specification is not mature until evaluated by implementation
- Specification changes with new requirements
- Specification lives alongside application during application life-cycle
Waterfall documentation process
- Specification is dead out of the water
- New requirements don’t change original specification
- Application is born out of specification, then discards it like a shell
Specification and implementation
Specification will very seldom be implementation quality from the get go. That is the faulty assumption waterfall makes. It is simply too difficult for the human mind to comprehend all detail before implementation is attempted. What you initially want to aim for is rough specification, with the intent to detail it later.
That is why specification refinement is an iterative process, and that will usually begin in the development phase. As code is written, the specification is continuously refined, continuously questioned, continuously improved upon.
Imagine a chemist piecing together a new theory late night at home, then going down to the lab the next day to see if it works.
“Huh uh, it blew up! Back to the drawing board!“, true enough for chemists and it should be true for building software as well.
Details? I’m a visionary!
Could you build a forest without knowing about leaves?
Description takes something to a higher detail level. It is a creative work where you add information that wasn’t there to begin with.
“If you have to squint to see them, you too can be a visionary!” — Yours truly
This is true all the way down to the software specification. As the programmer writes code, he is adding information. This information should not be guesswork. Don’t let it be guesswork. Specification in the form of guidelines and frameworks (which might be application or enterprise-wide) can help cut down on detail, but only if the team is aware of them. In fact, the team must not only be able to use them properly, but also know how to communicate their consequences.
When the team starts reading and writing specifications real knowledge sharing begins to happen. What used to be hidden is now out in the open and people will start imitating good practices. If you have a really good programmer, let the rest of the team learn from him by observing the way he writes specification and performs problem solving.
I first learned about the relationship between specifications and proper code from a very good senior developer. I was very much opposed to the idea of implementing changes in specification prior to joining that project, but reading his specifications was simply an awesome learning experience.
Specifications and testing
To test something, you need to know what to test. The detail of your testing, is directly related to the detail of your specification. If your spec is unclear, expect your test results to be unclear as well.“Well, this thing is supposed to send a mail if I do this and this. It seems it doesn’t anymore, did we change this?“ “This thing used to disable if this and that happened, but it is still enabled. Anyone know if it should be enabled in this new version?” “This thing now updates these things, but that doesn’t seem to work well with this other thing. Is it supposed to be working this way?”
See where this is going? We accept this because so many great minds tried, and failed before. We accept this because we know no alternative. It is of course unacceptable. The 1600’s had many great minds too, but not until the late 1800’s when methods of construction were finally formalized into a science, when design and construction was finally separated, did we manage to build skyscrapers. Exit the master builder stage left.
People still do that? As a developer, going through a folder of class prints before you dive into the code is very funny. Funny as in useless. If your code isn’t readable, chances are it sucks and should rewritten in an understandable way. You’re gonna be refactoring anything if you have to enter Visio to edit the class diagram? Crazy stuff.
Weak vs. Strong
- Implementation is guesswork
- No place for customer and developer to meet
- System knowledge is continuously forgotten, or lost when developer leaves project
- Understanding bugs is done by reverse engineering code
- Strong developer is decision maker. Team dutifully nods as oracle speaks.
- Implementation is translation of spec into code
- Customer and developer communicate through specification
- System knowledge is persistent and available to everyone
- Understanding bugs is done by reviewing specification
- Decisions are informed and often by team consensus.
How much is enough?
“How many angels can dance on the head of a pin?”
The answer to “How much?” is obviously, “Enough”.
The slightly longer answer is, “As much as you can stomach”.
The philosophical answer might be something along the lines of, “Document the information most subject to change”. Statics are usually well known and not interesting to document. If you would document a process, the biggest “obvious” steps change very seldom while the smaller higher detail steps get rearranged all the time.
Unfortunately, most documentation takes the opposite approach–we tend to document generalities; the big obvious boxes, while deftly leaving out the hidden details that describe how things work.
What we produce is absolutely correct yet perfectly useless.
Moving from generalities to detail implies accountability. It is arduous and time consuming but it needs to be done.
If you’ve read this far, and still disagree, let me tell you that alot of what I’ve written about above might eventually turn obsolete. Future software designers will snicker at the thought of documenting and programming (they will call it configuring) in parallel.
Looking ahead, more and more man-hours are being moved from the actual programming to figuring out what to program. This is because modern languages are making programming easier; plumbing is turning into mere configuration, data structures are plentiful and easy to use. What used to take a week to get working now takes a day.
Alongside these bigger building blocks will be an increase in visual tools. Programming by typing in code will decrease and visual designing (which is really closer to configuration) will increase.
As architects and developers (now software designers), generate interacting blocks of software out of visual models, these models instantly become approachable to team members outside the implementation group.
Models will decrease the gap between customer and developer, and remove the need for detailed specification.
That, finally, is self documenting software. What you see is kinda what you get.
Sure, implementation decisions remain, but these designing tools will be the new meeting ground for discussion to take place, for both developer and customer.
Until we get there, I suggest you do it by hand.