Last night, while awake in bed, I got to thinking – why is it that while most software professionals I meet agree that Agile is the best way to create software, most businesses remain unconvinced? Personally, I’ve yet to work for a business that’s fully embraced agile. Most of my recent employers have kind of reluctantly accepted that it’s ok for the software department to work in an Agile way (possibly after discovering how difficult it is to find software professionals prepared to work in a fully Waterfall environment these days) but outside the software department they still want to behave as if we were still doing Waterfall – with project phases, 5 year plans and endless status update meetings. This creates an unbearable friction where the software department meets the wider business and means that the many advantages of Agile – flexibility, time-to-market, delivering genuine business value – are lost.
Around 1am it hit me – it’s because building software is nothing like building a house.
If someone outside the software development world wanted to create a metaphor for creating a complex system from scratch, building a house would be perfect. It’s a large ‘thing’ with lots of different parts that must be designed, created and installed by various specialists and all come together in the right way to produce something of value. What’s more, we all understand what a house is. We’ve all experienced houses. We know what it’s like to be inside a house, we know what it’s like to be outside a house. Many of us will have performed some kind of DIY at some point and realise that there are hidden things going on inside the walls and above the ceilings that are necessary for the operation of the house. We’re even, perhaps in a more abstract way, aware that a house is connected to the wider world through utilities – electricity, phone lines, sewerage, etc. and that all these must be in place for the house to operate as expected (ok, not all houses are connected to sewerage, or electricity, I’m going to make a number of statements in this post that are only largely true rather than universally correct but lets not let pedantry ruin an early-morning epiphany)
So houses are clearly the ideal ‘thing’ to think about when one considers the process of building a complex system. The problem is, that building a house is necessarily a Waterfall process. You can’t build the walls until you’ve built the foundations, you cant build the roof until you’ve built the walls, you can’t fit out the interior until the house is weatherproof and unless you’re prepared to waste a lot of money you’re almost certainly not going to buy a single brick until you’ve got a detailed plan outlining exactly where the house should go and what it should look like inside and out.
When building a house, change is expensive. Even a small change, like moving the front door requires demolishing a perfectly good section of wall, wasting bricks and adding to our labour costs. Adding an extra storey late in the day might be all-but-impossible without tearing down and starting again.
Software doesn’t have this problem. Change still isn’t free, but it’s a lot cheaper. Add an extra storey? No problem – just give us the time to construct the storey and a few extra hours to integrate it with the rest of the house. Moving windows and doors can be done quicker than it takes to describe where to move them to. We can even move rooms between floors, turn bedrooms into bathrooms or relocate the entire house 2 streets further down with speed that would make Taylor-Wimpey green with envy.
If we were to stretch the metaphor to its absolute limit, the Agile approach to building a house would be to ask the client what the most important room is, whether it’s a bedroom, bathroom or kitchen and build that in its entirety. Then, get the client into the room to give feedback and incorporate their changes, keep getting the client into the room at regular intervals until they’re happy then start on their second most important room. At Agile Home Builders Limited we don’t even need the client to tell us up front whether they want a bungalow or a high-rise block of flats. Even if they change direction entirely half way through the project and want to turn it into a factory or restaurant we can accommodate that, and as soon as we’ve built enough rooms for the client to find it useful they can move in (and start paying us!) while we finish the rest of the building. Great, right?
OK, says our business person, I kind of like the idea of having clients paying us as soon as we have something barely useful to them, but the rest is just silly. If I’m going to ask you to build a house, it’s because I know what I want to build – I know up front whether I want a bungalow or a block of flats and I’d be pretty poor at business if, half way through a project, I changed my mind about building a family home and started building a restaurant. I know on day one what I want you to build, so why can’t you work in a way that works for me? I’ll tell you up front what I need you to build, then you tell me when it’ll be done so I know when to start marketing it, and then go off and build it.
Why? Because building software is nothing like building a house.
If we want to build a house to our specifications, we actually don’t need to tell the builder all that much – location, size, number of floors, number of bedrooms rooms. It would probably take less than half a side of A4 to write a spec for a house that a builder could take away and ultimately deliver something that did 100% of what I needed and maybe around 90% of what I wanted. OK, in the real world I’d probably want to have a bit of additional input on the size and location of rooms and perhaps get quick nitpicky over things like the kitchen fittings and floor coverings in the latter stages but ultimately it still wouldn’t take that many words to describe the house of my dreams.
Software is the opposite. We could take pages and pages just to describe the front door/login system – is username/password ok? Do we want to support face or fingerprint recognition? Do we want to integrate with any of the big identity providers? What about 2FA? Are there multiple levels of access? Can unauthenticated users do anything? How do we configure new users? What happens if a user fails authentication? What happens if a user forgets their credentials?
Is this because building houses is so much easier than building software? Of course not – it’s because we’ve been building houses for thousands of years but only building software for a few decades.
If you gave me the address of any house in the country, I could tell you a lot about it without visiting it or doing any research at all. Things like, most of the walls are vertical. Most of the floors are horizontal. There are internal doors to allow people to move between rooms. There will be some kind of heating system. If there’s more than one floor then there will be at least one staircase to allow people to move between them. Clear windows will allow light into most rooms. There will be some kind of roof. The roof and external walls will be weatherproof. I could keep going. Ok, if you’re trying to trip me up you can probably find a house that doesn’t match some of my statements but it’ll be the exception rather than the rule. What’s more, I can use many of the same statements for any house in the world and expect to be mostly accurate, and if I steer clear of statements around electricity and indoor plumbing I can even expect to get reasonable accuracy on any house at any time in history. What does this tell us? That there are a number of statements that can be made that apply to almost every house that has ever been built, or to put it another way, a large part of the specifications for a house are so universal you don’t even need to mention them.
Houses are a mostly solved problem, their purpose is very well understood and the necessary features that allow the house to fulfil its purpose are well known. Software is completely different. On a historical scale, we’ve barely got started with it, and the pace of progress so far has been so fast that we’re doing things now that would have been impossible, or at least very impractical, just a few years ago. What this means is that it’s pretty much impossible to get an up-front description of what the software needs to do and then just build it – so much would need to be explicitly described and recorded. If we were to attempt this the requirement gathering process would take far too long, people would get bored and important things would still get lost in translation, missed or forgotten entirely. Our estimates and timescales will only include the things we’ve written down and so will be too optimistic and when we come to build it we’ll end up with whole lot of gaps where the software team will have to make assumptions which may not match what the customer actually wants, and we’ll end up with a lot of areas where we think we’ve built what the customer asked for, but there’s actually been a misunderstanding. If the customer then never gets to see the finished product until late in the project there is likely to be a huge amount of rework required, which increases costs and pushes out the (now imminent) launch date even further.
This is why Agile advocates the iterative approach – build something small but complete, show it to the customer, get feedback, use that feedback to inform design for the rest of the system. Yes, there’s still a likelihood of rework, but reworking part of the system is a lot quicker and cheaper than re-working a whole system, and we’ll know up front when developing the rest of system that e.g. when the customer said all UIs must be ‘secure’ they meant using https rather than requiring a successful retina identification scan every 30 seconds to prevent lock-out.
Anyway, that’s what I think, it might even make sense. Either way, I think it’s time for a nap.
