This post was originally written as a guest post on the MagRails Conference blog. You can find it here.

Agile practices (pair programming, TDD, continuous integration, etc) and methodologies (Scrum, XP etc) are intended to increase the productivity of teams that adopt them. In one sense you can think of it as “team + practice = improved team”. The bit that gets the most focus is “practice”; what we will explore here is the nature of “+”.

Filling up

If you want to take a bath, initially the bath will be empty. You make an intervention to fix this undesirable state: put the bath plug in the plughole and turn the taps on. Is the bath ready now? I mean, now? Unless you’re reading this over a modem from 1983 and it took you an hour to get this far, the answer is no: the bath hasn’t had time to fill yet. There’s a delay between turning the taps on and the bath being full, somewhere in the region of 10 minutes. This is significant: it means you can go and make a cup of tea without fear of flooding the house. It also means that if the bath isn’t full in half an hour, something is wrong. (Perhaps the plug is leaking slightly, or the flow through the taps is too slow.) Whatever happens, we intuitively know how to use this delay to manage “bath + running taps = full bath”.

The same idea is equally – if not more – important in software teams. Without a sense of the delay between the original state and the desired state given a certain intervention, it is easy to over- or under-manage.

Say your team currently integrates and releases on a 6-month cycle, at the end of which usually follows a month of mad scrambling and tail-chasing to deploy the code. You want to improve this, so you intervene by introducing fortnightly iterations, where code is supposed to be fully integrated at the end of each iteration. If after a month of this (2 iterations), the team’s productivity is down, changes are not being fully integrated, and not all integrated features are fully working, has the intervention failed? At this stage, there isn’t really enough evidence to suggest this. An understanding of the nature of the intervention and its inherent delays helps here.

Moving to frequent iterations reduces the batch size of work being integrated. A manual integration process twice a year may be fine; the same process every 2 weeks will show enormous waste. Therefore the team will need to automate its integration process, but this will take time. This effort will take capacity away from existing development, so in the short term you would expect measured productivity to go down. (I’ll ignore any debate on actual productivity for the purpose of this example.) Also, with more rapid integration, team members will also be forced to communicate more frequently, and meetings will need to be made more focused and resolve conflict faster.

If 3 months later, the same pain is still being felt, has the intervention failed? Now the answer looks more like yes: it should not take most teams three months to achieve significant improvements in a deployment process. However, the the real answer is context-specific, or “it depends” as it’s known in the consulting trade.

There is no need to pick just one or the other out of the two (arbitrary) time-points: reacting at 1 month or 3 months with no information in between is still less than ideal. When filling a bath, you don’t need to wait until the end to see if it’s ever going to fill: by checking occasionally, you can see if it filling as expected. You may be able to see water leaking as it becomes half-full, or you may see it filling slowly and realise you only half-opened the taps. This extra feedback lets you manage more effectively. But if the bath takes 15 minutes to fill under ideal conditions, and you’re only 10 minutes in, don’t expect it to be full yet.

You wouldn’t blame a bath for not filling quickly enough, so don’t blame a team for not improving fast enough, unless they are actually goofing off. Slow progress may be due to a new practice coming into conflict with existing policies or mindsets (which you should watch for), but even under ideal conditions will take time.

Draining out

The reverse situation is also important. When you take the plug out of a bath, it takes time to empty. Imagine you filled the bath and got in, but half an hour later – to your bemusement – you realise the water level has dropped significantly. What gives? It may be that there’s a tiny crack you didn’t notice when filling it, which has allowed water to seep out. Again, the inherent delay and your own perception means it took a while for the problem to become apparent.

Software development teams can have leaks too. Any time a process which maintains long-term productivity of is dropped or reduced, the team’s performance will also start to fall – but it will take time for the problem to appear. Reducing TDD will not cause the defect rate to rise sharply, nor the ease of adding features to fall sharply. Instead, what will happen is that 6 months down the line, customers are complaining about the increasing number of bugs, and developers are complaining about the increasing difficulty of fixing them. The delays between different people’s responses will not be uniform, either, so watch for the early complainers – far from being a nuisance, they’re the system giving you a hint of what the future holds.

There’s a particularly insidious situation that can emerge when practices are dropped, known in systems thinking as “shifting the burden”. Here, some team members will pick up the slack, increasing the delay before the fundamental problem becomes apparent – which it will do so dramatically.

Again, you wouldn’t pull the bath plug out of a bath and exclaim “The bath isn’t empty! We didn’t need the bath plug after all!”, so don’t let a team reduce or drop practices it knows to maintain productivity, only to act surprised when things fall apart later.

A few delays

Start looking for delays in your team and software development process. As a starter, here are some examples:

• learning a new language or framework
• changing the practices in use (including learning a new one)
• changing the members of a team
• changing suppliers
• taking on or dropping clients
• learning to look for delays

If you’re too busy in the day to think about the delays around you, think about it next time you’re running a bath. What are your thoughts? Feel free to leave your comments below. I’m happy to wait for them.

My name is Ash Moran. I’m a software developer and agile coach, and owner of PatchSpace Ltd (Twitter). If you have any feedback, questions, or would like to know more about my services, feel free to contact me at ash.moran@patchspace.co.uk.

There's an important concept I use - the origin of which I will talk about later - which applies pretty much everywhere I go. Namely: at any point in time, in any organisation (or process, or system) I'm involved in, there will be a vanishingly small number of bottlenecks - that is, the people, teams, machines or otherwise that limit the output of the whole organisation. In fact, there's usually only one.

Child's play

This idea is very easy to show in a simple, linear system. Imagine you're getting a group of children to make paper aeroplanes (let's call them Andy, Bob and Claire). The steps are as follows:

• the first child, Andy, folds the paper in half
• the second child, Bob, folds the wings
• the third child, Claire, tests that they can fly (by throwing them)

Let's assume that the children can do their activity at the following rate:

• Andy can fold paper at 10 planes/minute
• Bob can fold wings at 5 planes/minute
• Claire can test-fly planes at 20 planes/minute

What will happen over time if all the children work as fast as they can? If the answer doesn't leap out immediately, watch this short cartoon animation of people at work. (Really! I guarantee you'll be hooked.)

[As an aside: If you're having trouble working out why Andy and Claire aren't helping Bob with the wing-folding, you haven't spent enough time in corporate environments. Instead, imagine that the task is not making paper planes, but completing a piece of complex, tedious ISO9001 documentation. Alternatively, imagine that the children are being paid piecework for their activities, or are routinely whipped for not meeting a production quota. All the preceding situations should produce the relevant bottleneck suitable for our discussion.]

So how much is the group of children capable of producing overall? Because the slowest child (Bob) is only working through 5 planes/minute, the group as a whole can only make 5 planes/minute. Any excess planes Andy makes will pile up (as "inventory"), while Claire will be spending a lot of time twiddling her thumbs. This, in the simplest sense, is a bottleneck.

What happens if the bottleneck stops working? If you watched the video, you know the answer. Andy has enough extra capacity to build up a few spare planes, so he can take a break now and again. Claire can sprint to catch up. But the Bob is different: he can't sprint to catch up with Andy, and he can't produce any excess to keep a buffer for Claire. In short, there's a lot of pressure on Bob to work as efficiently as possible.

Bottlenecks in a software organisation

I should point out that software teams are different from groups of children - although given the way some are managed, not everyone appears to believe so. And software organisations are have more complex workflows than this - although people attempting strict adherence to the mythical Waterfall methodology might lead you to believe otherwise. If the bottleneck in a development team is not under pressure from others, they will be more than likely under pressure from themselves to do well. Also, software development is a specialised skill, and even moving roles in the same company has a long lead time due to the amount of tacit knowledge involved.

Now, people are strange things. In some sense, we can be abstracted as "resources" - I don't advise it though, if you want to make many friends. But as Tom de Marco points out in Slack [amzn]: we are not fungible. That is, you can't easily swap one person out for another, in all but the most simplistic situations - making paper planes, perhaps. Our lack of fungibility stems from a more fundamental issue of dealing with people: we are not mechanical parts, we are complex parts of a complex system. We have an influence back out on the people giving us work that, unlike machinery, goes way beyond failing due to regular maintenance, or topping out at a capacity limit. Both can be useful analogies, though - as most importantly, we experience stress.

Having described bottlenecks, the pressure on them, the way to look at them in a more complex organisation (and a few caveats to the model) - it's now time to answer an important question: who should be responsible for monitoring the health of the bottleneck?

The Golden Rule

You must never let the bottleneck monitor itself

Why is this?

You may have already seen the pieces. Let's recap. A bottleneck:

• is responsible for the throughput of the whole organisation (or team)
• is under intense pressure, either from itself, or from elsewhere in the organisation
• can't (immediately) get outside assistance, either because the skills are too specialised, or the organisation doesn't have the capability to accommodate high demand on it

To understand what happens when the bottleneck tries to monitor itself, let's take a simplified system. Let's pick a single individual, and let's call him Bob, Bob senior.

Bob could be a freelancer, a single worker in a department (perhaps with a specialised skill, left to his own devices), the lone manager of a department, or any other similar situation. Fundamentally, he is on his own in some way. And there's an important condition: Bob is a bottleneck. If Bob slacks, goofs off, or makes a hash of things, everybody loses. Quite possibly, nobody cares about our Bob unless he screws up.

Bob has two responsibilities: he must get his work done, and he must somehow ensure that his own work is managed. How should Bob manage his time? He has two conflicting requirements. Like everybody, he's in an evolving world, so he either has to sink or swim. (Although, like Deming said: It is not necessary to change. Survival is not mandatory.) Assuming he wants to swim, he must:

• do his work at maximum efficiency
• manage himself to ensure he spends his time effectively, and improve the way he works

(Aside: most organisations don't attempt to measure effectiveness - they measure efficiency at best, in terms of staff utilisation. As we've seen, keeping everyone 100% busy is not a condition for maximum throughput. Eli Goldratt's The Goal [amzn] has a lot more to say about this.)

The core conflict of a bottleneck

So our protagonist Bob is burdened with this conflict, and he must make the best of it. What happens if he chooses the route of doing? Then he will do more, but it will not necessarily be his best. He may make errors of commission - doing things that didn't need to be done, because he didn't stop to think if they could be eliminated. Or he may make errors of omission - missing things he could have seen had he stopped to think about the bigger picture. Both of these cry for more effective management. But Bob is effectively his own manager, so what happens if he chooses the route of management? Now something more insidious takes place. For every moment he spends managing himself, he is not doing his own work. Remember: Bob is a bottleneck, so he is under intense pressure to work efficiently, and hopefully effectively. Every moment he spends managing himself, improving himself, he is not doing.

What happens when a bottleneck is not doing? This is easy: as we've seen, the whole organisation loses. It is as true in a complex software organisation as it is with children making paper planes. And when Bob is making the whole organisation lose, he is putting himself under more pressure: either from someone else, or from himself, or possibly both. (Although he will probably just see it as work piling up faster.)

What happens when people are put under pressure? The become stressed. And what happens when people are stressed? This is complex, but a simple yet plausible explanation lies again in Slack: that pressure up to a point increases productivity, but beyond that, causes it to degrade.

This is an interesting situation. It means that if Bob does not attempt to manage himself, he will cause his own productivity to degrade in the long term - relative to the rest of the world around him - by not improving. But if he does attempt to manage himself, he will cause his productivity to degrade in the short term, by spending his capacity on management, not action. So our poor friend Bob is in a bind: he loses capacity either way.

Well, Bob is not one to accept defeat. He wants to be the best he can, bottleneck or not. So he makes a resolution (in his spare time, possibly a random thought from his subconscious) to manage himself as effectively as possible. But what is this activity? It's meta-management - the management of management, a form of management in itself. So in order to manage himself better (perhaps to spend more time doing), he must increase the amount of time he spends managing. And decrease the amount of time he spends doing. And so, increase the pressure he has on himself. And how does he resolve this? Well, he must choose to spend more time either:

• doing - to achieve more; or
• managing - to be more effective

Does this look familiar? We have a circle! And it has teeth. They often do, because the virtuous ones usually take careful planning, while the vicious ones lie waiting for us everywhere.

Hope for the future

So what can we do about this? Well, this is where I welcome creativity is needed, so the following are only a few suggestions.

• Greater appreciation of bottlenecks: Bottlenecks are a concept we all understand intuitively, but don't always consciously look out for. Seeing a few extreme cases (and they occur in day-to-day life) of what can go wrong when a bottleneck's time is wasted can help us focus on the importance of the situation.
• Protecting retrospectives: Allocating a percentage of a bottleneck's time to regular, continual improvement is essential. Nobody should be allowed to consider themselves "too busy" to review their progress, for therein lies a death spiral. Equally, they must trust their managers to know when their reflection and improvement is sufficient.
• Peer review: I've tried this personally, and it can be highly effective. Gathering opinions from impartial people on a regular basis can stop you veering off on a tangent. It is also naturally timeboxed, as it can be done in down-time, and you must return the favour.
• Making work more visual: It's easy for anyone to get overworked, but without a simple way to see what people are working on, bottlenecks may end up taking on tasks that are wasteful, redundant, or could be done better by someone else. Someone else needs to be involved in this, though.
• Creating a culture of slack: If we can accept there are only so many hours a week that any individual can work on one thing, we'll be less inclined to push for more and more work when what is really needed is rest and review.

These are mostly steps to exploit the bottleneck. ("Exploit" in this sense does not mean to depersonalise them in the same way "resource" often does, merely to not waste their time.) Most of the suggestions above are about time- and energy-management.

I've concentrated on what happens when the bottleneck is a single person. There's a lot more to be said about the bottleneck team, as opposed to the bottleneck individual, but most of the same concepts apply - they just have a more complex human element.

In all cases, though, the situation can be addressed without command-and-control attitude, or hostile criticism. The bottlenecks in most organisations are people, and the most effective way to waste time at a bottleneck person is to treat them as anything but. In fact, I suspect a lot of waste of bottleneck individuals' time stems precisely from the fact that we don't take into account the human element of pressure and stress.

Further reading

If you'd like to learn more about bottlenecks, the ideas I use come from the Theory of Constraints. There's a wealth of information out there, but I recommend reading the original business novel The Goal [amzn] (already mentioned), which explains how to identify and exploit bottlenecks. Its sequel It's Not Luck [amzn] explains why even complex organisations have few real bottlenecks, and explores different types. I've referenced Slack [amzn] twice in this post - it's not about bottlenecks, but a lot of the ideas about trying to run over-capacity are highly relevant. If you're interested in more references or further explanation, please ask in the comments.

A few interesting points came up on a mailing list thread I was involved in. Here are a few of them. The original comments are presented as sub-headers / quoted blocks, with my response below. This isn't a thorough look at the issues involved, but what I thought were relevant responses. Note: I've done some editing to improve the flow and to clarify a few things.

Why can't developers estimate time?

We can't estimate the time for any individual task in software development because the nature of the work is creating new knowledge.

The goal of software development is to automate processes. Once a process is automated, it can be run repeatedly, and in most cases, in a predictable time. Source code is like a manufacturing blueprint, the computer is like a manufacturing plant, the inputs (data) are like raw materials, and the outputs (data) are like finished goods. To use another analogy, the reason Starbucks makes drinks so quickly and repeatably is because they invested a lot of time in the design of the process, which was (and is, ongoing) a complex and expensive task. Individual Starbucks franchises don't have to re-discover this process, they just buy the blueprint. I'll leave it as an exercise to the reader to infer my opinion of the Costa coffee-making process.

It's not actually always a problem that development time is unpredictable, because the flipside is that so is the value returned. A successful piece of software can make or save vastly more than its cost. Tom DeMarco argues for focussing on the high value projects for exactly this reason. Note that this does require a value-generation mindset, rather than the currently-prevalent cost-control mindset. This is a non-trivial problem.

By far the best explanation I've read of variability and how to exploit it for value is Don Reinertsen's Principles of Product Development Flow, which is pretty much the adopted "PatchSpace Bible" for day-to-day process management. And when I say "by far the best", I mean by an order of magnitude above pretty much everything else I've read, apart from the Theory of Constraints literature.

Here is the data from my last development project. (Histogram generated in R with 5-hour buckets: the horizontal axis shows the duration in hours for the user stories - 0-5 hours, 5-10 hours, etc; the vertical axis is the number of stories that took that duration). We worked in 90 minute intervals and journaled the work on Wave, so we knew task durations to a pretty fine resolution. (We did this for both client communication and billing purposes.) The result: our development times were about as predictable as radioactive decay, but they were very consistently radioactive. Correlation with estimates was so poor I refused to estimate individual tasks, as it would have been wilfully misleading, but we had enough data to make sensible aggregates.

Rule of thumb: take the estimates of a developer, double it and add a bit

The double-and-add-a-bit rule is interesting. When managers do this, how often are tasks completed early? We generally pay much more attention to overruns than underruns. If a team is not completing half of its tasks early, it is padding the estimates, and that means trading development cycle time for project schedule. Cycle time is usually much more valuable than predictability, as it means getting to market sooner. Again, see Reinertsen's work, the numbers can come out an order of magnitude apart.

Also, this is the basis for Critical Chain project management, which halves the "safe" estimates to condense the timescale, and puts the remaining time (padding on individual tasks) at the end, as a "project buffer". This means that Parkinson's Law doesn't cause individual tasks to expand unduly. I'm unconvinced that Critical Chain is an appropriate method for software though, as the actual content of development work can change significantly, as feedback and learning improves the plan.

People in general just make shit up

It's not just developers that are bad with estimates either. Everyone at some point is just winging it because it's something they've never done before and won't be able to successfully make a judgement until they have.

As a community we need to get away from this. If we don't know, we don't know, and we need to say it. Clients who see regular progress on tasks they were made aware were risky (and chose to invest in) have much more trust in their team than clients whose teams make shit up. It's true! Srsly. Don't just take my word for it, though - read David Anderson's Kanban.

Estimating is a very important skill and should be taught more in junior dev roles

I propose an alternative: what we need to do is teach to junior devs the meaning of done. If estimation problems are bad enough, finding out at some indeterminate point in the future that something went out unfinished (possibly in a rush to meet a commitment … I mean - estimate!) blows not only that estimate out of the water, but the schedule of all the current work in process too. This is very common, and can cause a significant loss of a development team's capacity.

Having a rare skill means that a small increase in demand may translate to a large increase in the the rate your work is valued at, the rate you can charge. If so, focussing on that specialism should lead to greater profits, right? Maybe it can, but there are other forces at play. To demonstrate this, here is the story of a (fictional) thatcher and his business. Disclaimer: I've never been involved in the thatching business, but I've seen a roof thatched slowly, day by day. And believe me, it takes a long time.

The Story of the Thatcher and the Cottage-Owner

The thatcher lives in the countryside. He doesn't spend all his time thatching, in fact, most of his work is joinery for a local farm. But his own house is thatched, and he has kept his thatching skills fresh.

The houses nearby now mainly have tiled rooves, the mark of industrialisation increasingly stamped over the houses. But one cottage at least still has a thatched roof, and it inevitably needs repair from time to time. The cottage-owner has not lived there all his life, and is certaintly no expert on thatching.

One day, a storm damages the roof of the cottage: part of the surface is blow off, and some parts look susceptible to rain. The cottage-owner realises he needs his roof fixed, and goes out looking for someone to help. The first people he finds are tilers: tilers are plentiful and local in this landscape.

The first tiler replies: "We can't fix a thatched roof. We'd have to tile this for you. We'd have to start from scratch, and it will cost a lot of money." The cottage-owner goes to find another.

The second tiler replies: "Sorry, this roof is thatched. We can remove the thatching and tile it, but it will cost a lot of money." The cottage-owner goes off again.

The next person the cottage-owner stumbles across is our thatcher. The thatcher looks at the roof and says, "Yes, I see the surface damage." He investigates a bit further: "Let me look inside … ah, some damage from rodents." At this point he uses his own judgment to conclude that he has investigated the situation to an appropriate level, and presents his findings: "It will not be easy to fix, but I have the skills, and this is my day rate. It is moderate, if not cheap." The cottage-owner decides that the thatcher is the best person to help.

The thatcher makes progress on the roof, and  for a while, everything is fine. Then, as he tears up the damaged thatching, he realised why some part of the roof got so badly damaged in the storm: the beams underneath were rotten. He describes this situation to the cottage owner, and that he'll need to hire help from the farm to fix it. "I know how to do this, but your roof will collapse soon if it is not taken care of."

"But… I can't afford to pay more, I was hoping to be able to patch up the damaged part of the roof." the distressed cottage-owner explains.

And so, the thatcher is caught in a dilemma: does he continue to apply his specialist skill, which is now revealed to be suddenly less valuable than before (the amount of work has increased, but not the money paying for it); or does he tell the cottage-owner that he would be better to have his roof removed and rebuilt (possibly tiled), and lose this business forever? There is not even the possibility of finding another thatcher, because there are no more in the area. The cottage-owner, it turns out, does not want to "waste" the money invested so far, and so the thatcher continues.

The thatcher hires two farm workers, without whose help he could not hope to fix damage of this scale. Slowly but surely, he rebuilds and rethatches the roof, encountering more underlying damage along the way. Eventually it is complete, finished to the best of the thatcher's ability, but at great expense to the him, and at great sacrifice to his joinery work.

These are the slides for the presentation and interactive session at Systems Thinking Sheffield 2, held in February 2011 at the GIST Lab.

Note that the slides were prepared quite quickly, which means some of the examples are not as tight as they could be. Also, the output of the "story of the hosed monkeys" interactive tree-drawing session isn't included. I need to write a separate post about that one, as it raises interesting points both about behaviour in organisations and how to model it. (If you'd like to know more about this, please request it in the comments.)

This is the first time I've tried to present these ideas in this format, so I learned a lot. A few key points:

• Many people's instinctive reaction to figuring out why a situation plays out the way it does is by gathering facts, rather than by asking "why do we see this?", and challenging assumptions. That, I suspect, is becase we think primarily by pattern matching, rather than analysis.
• It's easier to introduce logic trees by presenting a partially-complete one (and they're all partially complete) and having people raise informal objections, than to teach by building one from scratch.
• People value the emphasis on externalising and de-personalising problems, and questioning, rather than directly criticising, logic. I included a reference to the Agile Retrospective Prime Directive, which went down well even with a largely non-software audience.

If you have any questions, please feel free to comment. I want to refine my presentation of logic trees over time. Many people are put off them at first, but everyone who has humoured me long enough to draw one said afterwards that they found the activity valuable.

At the Agile 2009 Conference in August, J B Rainsberger gave a talk called Integration Tests are a Scam, which you can see in video. The session is well worth watching. While it's long, and takes a while to get to the core issues, it's a very thorough analysis of the costs of slow test runs and (failed) attempts to enumerate all application behaviour from too high a level.

J.B. wrote an example of how focused tests can be used to detect integration issues in the blog post Surely we need integration tests for the Mars rover!. The example is worked through in pseudo-code. I find it hard to read extensive pieces of code, so I turned it into a coding exercise. Here is the pseudo-code translated into Ruby, with comments about the order of how it was built up (more interesting than replaying the SCM patches).

J.B.'s methodical approach to collaboration/contract tests is simple and powerful. The Mars Lander example makes a good concrete example. I highly recommend working through it in your language of choice; I learned a lot.

A common criticism of (aka excuse for not doing) test-driven-development is that it's too expensive in terms of developer time. Critics who take this position usually point to the time developers spend writing test cases, which at first seems like a sensible observation. There are (at least) two problems with this.

First - the same people that label TDD as waste are often people who will happily spend - or allow their staff to spend - hours or days at a time in a debugger. Testing to find defects is waste.

Second - and more importantly - writing test cases is not the same as running them to test the software.

At some point, somebody has an idea. They say, I have this problem (for our purposes here, we'll assume they know this exactly), and if I can write a program to do these things, then my problem will be solved. That person has the ultimate test case for the as-yet-unwritten software: if it behaves how they want it to, their problem will go away.

Now, a developer takes over, and turns this conversation into a set of ideas about what the code should do to implement this behaviour. At the very least, having written something, he should run it and inspect what it does, to verify that it behaves as they expect. (Some don't even do that much...) This is simple manual testing. But note two things:

• if he doesn't think about what it must do, he has zero chance of designing the right solution
• if he doesn't test his assumptions about the software's behaviour, he pushes errors downstream, where they become slower and more expensive to correct

• writing test cases is the process of formalising a spec so that a computer can be employed for testing
• running tests is effectively free

But, how free?

Information

The inspiration for this post came from chapter 4 of Don Reinertsen's Managing the Design Factory (It's All About Information). The purpose of this chapter is to explain ways to efficiently generate valuable information. The examples in the chapter are largely from circuit engineering, but even there, there exists a continuum. From page 76:

[Testing costs] could be twice as high with four iterations instead of two. This means that when testing costs dominate the economics we should concentrate on quality per iteration. We do not want to incur extra, expensive trials when the cost of a trial is high. In contrast, when testing costs are lower, we will get to higher quality faster by using multiple iterations.

So, if testing software is essentially free, how many iterations should we have? The answer is hinted to on page 74: this is an economic order quantity (Wikipedia) problem in disguise[1]. Out of sheer laziness to get an equation editor working, I'll reuse the slightly arcane, CC-licensed Wikipedia equation:

Where:

• Q* is the optimal order quantity - how many tests you should batch before you start a test run
• C is the order cost - the cost of a test run
• D is the rate at which the product is demanded - arguably requests for features (this is not explained in MtDF, presumably because you can demand features arbitrarily fast) 
• H is the holding cost - the cost of running tests late in development, when change is more expensive

The key, though, is that if C, the cost of running tests, is at or near zero, and H, the cost of making changes late is high (and every developer's experience is that tracking down bugs in old code is much harder than in freshly-written code) the optimal batch size of tests to hold is also at or near 0. Which in reality means:

You should strive to keep the cost of testing software at effectively 0,

and to run all your tests every time you make a change

If you've done TDD for a while, you'll know this intuitively. But expressing it in terms of existing economic models, already in use in other forms of engineering, puts it on solid ground.

I'll leave it open to interpretation exactly what I include in the scope of a "test", but that will be touched on in my next post. And if you doubt just how free software testing can be, take inspiration from IMVU's continuous deployment: Doing the impossible fifty times a day.

[1] You mean you didn't spot it either? :)

Here are the slides from the GeekUp Sheffield 20 presentation: Elephants in the Meeting Room

These are the slides for the NWRUG presentation on mocks, from July 2009.

Note that most of the slides were written in the middle of the night, and I didn't have much time to trim them down. And I didn't get to beta test them on a real live human being. So the presentation goes on a bit long, and some things look a bit strange without me there explaining them. I've corrected the slide that I noticed was spectacularly wrong (ie, the spec didn't even pass), but otherwise it's as presented

Also my opinions on some things may have changed since, so consider this an archive…