3 May 2016
The Messy Middle: When Your Software Project Feels Like a Failure
by Doug Alcorn
I’ve been thinking about a few different ideas that are similar but contrary. The first is that I think we often gloss over how risky it is to build custom software. Joel Spolsky famously wrote just a few years ago what we’ve paraphrased to be “never do a big rewrite.” Yet my last three projects have been just that: Rewrite an existing system.
Software development is almost always a form of R&D. It’s right in the name! We know how risky research is, and we don’t really expect it to produce viable business results all the time. But I think we often forget that development might go sideways, too.
The trouble is knowing when to pull the plug and when to soldier on. The sunk cost fallacy makes this decision harder. But there’s something else that makes all this risk hard to manage, too. It’s the messy middle. The point when a software project feels like a failure. Things don’t make sense. Everyone is discouraged.
It’s something I’ve experienced on my last three software projects. There’s a natural rhythm to an app build. The biggest challenge on all new projects is establishing a good working relationship between the design and development team and the client. In the beginning, we’re all excited to work together. It’s like a honeymoon.
Next there’s a setup period as we begin to build the app. Then, ideally, we’re releasing to production early and often. But in big projects it’s hard to have enough software written initially that the client feels comfortable releasing to actual production. We may call the software “released to production,” but unless the client (or their clients) are actually using it for real world needs, it’s not really in production.
This causes stress. On big projects it may take a while before the system is used for production purposes. The client may not be seeing real progress. Value is measured by what’s in production, but nothing is really in production, yet the cost continues to accrue. In particular, my last two projects have both bottomed out at about the same time: somewhere around 15 weeks.
The client has paid us a significant amount of money, but they aren’t seeing the progress they expected to see. The tunnel is dark. Budgets are nearing the end, and it’s hard to see that the project will end successfully.
It’s the dreaded messy middle. The client really wants to know how much farther. How much more money is it going to take to finish? We don’t really know because it’s hard for us to really know when we’ll have enough functionality to actually go to production. Only the client knows that and may only know it when he or she sees it. The client is frustrated, and the team is discouraged.
Don’t Give Up Hope
We see this on almost every project we do. It’s normal and part of the emotional roller coaster of custom software development. At some point, the project has to become real. In my experience what makes a project real is when the client starts to feel the cost of the project. In almost every case, it’s only a few weeks after a project hits this low point that we all experience a major breakthrough.
This breakthrough point is usually when the client and the team really connect. They really figure out exactly what’s needed. The foundation is built and now that clarity is found delivery quickly follows.
Recently, I was working on a project where we were having some really hard conversations near the end of the budget about what features were going to make it and what weren’t. When I say hard conversations, I mean conversations between executives about liability.
We finally decided on some scope and priority to get us through the next two weeks. As it turns out, one of our hardest, scariest features only took us a single day. After that, we literally finished all the “have to have” and “nice to have” features in the next week and went live. It was kind of amazing how quickly everything fell together.
At the time, everyone was scared about even being able to get to production. We weren’t sandbagging. We didn’t have any idea that feature was going to be that easy to finish. In some ways, it was luck of engineering we were able to stumble upon a solution that solved the problem so easily. On the other hand, that solution was a product of the foundation we had built into the application and the deep domain knowledge we had gained.
That’s the point. We don’t know when things are magically going to get easier. Our experience has been that it will.
So how can we solve this recurring problem? The easy answer is to recognize that you’re in the messy middle. Accept that it’s normal and keep your spirits up as you make your way through the toughest part of your project. Focus on the most valuable thing to build next and keep going.
The tough answer is go to production sooner. There really is some period of time where the team needs to lay the foundation. But if we can find the right focus, we should be able to get something to production and in front of real users sooner than 15 weeks. It’s hard. In the beginning, there’s lots of vision of what the future can be and little focus and clarity on what should be right now.
Just know that most projects hit the messy middle, and typically, you’ll feel like your project is failing right before you turn the corner to success.
