The real reason software sucks and how to fix it
Everyone has a plan 'till they get punched in the mouth.
—Mike TysonIt was at one of my first gigs as a product manager -- for a telecom startup in Oakland CA called MaestroConference -- that I learned the importance of planning for surprises.
Every few days someone would come in with an urgent need for the product -- often a request from a customer or investor. I’d take the idea, add them to our requirements document, and notify the developers of the change.
And each time they’d push back because, of course, I’d forgotten some important detail, asked for something technically impossible, or explained something poorly.
While I was busy clarifying these current changes, I’d be fielding new requests from the CEO and management team. Each wave of changes got more complex and changes piled on changes until I was confusing myself. Often the changes were “documented” in email threads and private conversations, which meant our requirements document was not trusted and rarely referred to.
Eventually, with the help of a coach and a willing management team, we were able to craft a planning system that allowed us to respond to the changing and uncertain environment, while still managing to ship a good features. I’m proud of the product and the system we created, and both are still going strong over five years later.
Through all this I couldn’t shake the feeling that I was ignorant and had somehow managed to stumble upon something real product managers already knew. But as a consultant I know this isn’t true -- I find similar problems in almost every organization I work with.
Across the software industry, effective planning is a rarity and few handle requirements very well. With so much chaos in the process, is it any wonder that so much of the software out there is so bad?
After years of working in product development, I’ve come to believe that the human psyche is so risk-averse that we would rather do almost anything than sit for even a moment in the discomfort of uncertainty. But the reality is that we are working on large, complex projects in dynamic markets with new technology, and uncertainty just comes with the job.
The secret is to plan for change and plan to learn as we plan our products -- to dynamically steer our organizations rather than just aim them and hope for the best. If we build our planning systems well and work in a disciplined way, it becomes relatively easy to answer the three most common questions of our business partners: What am I going to get? When will I get it? How much will it cost? And, we can answer these questions well and still build products that delight our customers and are documented for our auditors, users, and future development teams.
Unfortunately, most businesses prefer to write detailed project plans that mask what they don’t know rather than take the time to build flexible and transparent steering processes that encourage collaborative work on creative solutions.
The five levels of planning
In all organizations, planning happens in stages: The company vision gets broken down into a product strategy and roadmap, which in turn gets divided into individual releases. As people coordinate their work, they may break things down even further into iteration goals -- assuming they are working in iterations. Each individual or team in turn sits down on a daily basis (at least somewhat deliberately, we hope) to plan what they’ll accomplish each day. If this process happens efficiently, each person’s daily work is directly linked to the company’s overall success. But how often does this actually happen in an efficient manner?
When transmitting electricity on high-voltage lines there is a “line loss,” meaning some of the electricity is lost in transmission from power plant to home or business. The same thing often occurs with the vision of a business. There is a loss of focus and purpose as we proceed from vision to daily action. A planning cycle provides that clarity and helps your company run more smoothly, with happier employees and more value for your customers.
The five levels of planning -- vision, roadmap, release, iteration, and daily -- are not presented as a rigid format. Each organization is different and your planning strategy may have more or fewer distinct phases. But if you develop a set of disciplined practices at each level, you’ll begin to build the kind of organization that can create great products quickly and cost effectively.
Improving your digestion
If you want to improve how your organization plans, the first thing to do is build a system that offers a high degree of visibility and transparency. This allows you to harness the intelligence of your entire organization by crowd-sourcing solutions and enabling everyone to make better day-to-day individual decisions.
While transparency may seem like a nice thing to have, secrecy in your organization can erode your coworkers’ enthusiasm and intelligence. While producing this book at Rally, I regularly shared the content -- even the stuff that wasn’t “ready” -- with the entire organization, and each time I learned something that helped me produce a better product. Was it scary? Absolutely! But it was incredibly helpful in making this book the best it could be.
In order to develop this visibility, you’ll need a regular cadence of planning meetings at each level of planning. These should be built into your culture and done at regular intervals -- even if there’s not much to talk about. If you plan only when there’s a crisis, you’ll have more crises. If you plan habitually and with discipline, you’ll only rarely get surprised. This is a good thing.
And the information that flows out of these regular meetings needs to flow in feedback loops that go in both directions. Sure, your release planning needs to inform your iteration planning and daily meetings, but information from daily planning and iteration planning is also vital to your release planning efforts. And the people managing the roadmap need clear information about what’s going on at the release level if they’re to make the best decisions possible. Feedback circuits are as valuable in organizational design as they are in electrical engineering.
It really all comes down to building a culture and set of systems that allow your people to collaborate at all levels to create the most solid plans they can.
I know you’re working in the real world with funding, accounting, and phase gate systems that are based on a very different mindset from what I’ve presented here. And about now you’re probably thinking “this won’t work here.” If that’s the case, I encourage you to review the pieces in this section carefully. We’ve done our best to address the most common concerns and objections to a more Agile way of doing things. As you read, you’ll get a deeper understanding of the topic as well as see a clearer path to applying these principles in your organization.
You may also get a very real idea of the cost associated with not adopting a more dynamic and Agile set of practices. As psychologist Theodore Rubin said, “The problem is not that there are problems. The problem is expecting otherwise and thinking that having problems is a problem.”