This article is by Adam Pisoni, co-founder and former CTO of Yammer. He is also a founder of Responsive.org, a new movement dedicated to helping companies become more adaptive and empowering.
If you don’t regularly read Steve Denning’s work, you should. He's one of the better writers on the future of work. In an article he wrote for Forbes last week, he starts by stating that the Agile product development method has gone mainstream, but poses the question as to whether it has now become a fad like so many other management theories. His conclusion is that Agile is not merely a fad because, “Agile and Scrum deal directly with current business issues in a way that the 20th century initiatives sidestepped. They give a direct voice of the customer through the product owner and give a voice to competence over authority.”
While there is a lot of great substance in his article that I agree with, I have to disagree with his conclusion. Agile did indeed represent a positive step forward, but I do not believe it went far enough. Before Agile, it was all too common for teams to spend a tremendous amount of time and resources building something only to find that their customer didn’t want it by the time they were finished.
Agile was fundamentally trying to solve for how to build successful products in a world that changes too quickly to predict what people want and how best to build it for them.
While Agile did educate a generation of software developers on the importance of experimentation and customer feedback, it failed to change the old, centralized, command-and-control system of management which remains a large part of the problem. Even with Agile, disempowered engineers working with too little context still ended up taking too long to create products customers don’t even want, resulting in dissatisfied customers, disengaged engineers, and frustrated managers.
For context, I’ve either been a full-time software developer or leading teams of developers from 1995 to 2014, except for a brief hiatus after the first dot-com bubble burst. In 1997, I was leading a team of engineers using the Waterfall methodology, a predecessor to Agile. With Waterfall, you did all design and planning upfront assuming you could then break up the work and build until completion.
In 2004, I worked for a company that relied heavily on analytics, A/B testing, and even released weekly, all of which was ahead of its time. I remember the first time people started talking about SCRUM, and I remember having a clear sense of the problem it was trying to solve: Bad management.
To be fair, as I re-read the Agile manifesto and principles, I agree with every one of them. They're as good and true today as they were in 2001 when they were written. Agile stated that you should be building and iterating to learn as opposed to following some grand plan. Furthermore, your customer should be an intimate part in the whole process to make sure you’re learning from them as you build. Since your goal is to be constantly absorbing new information, you should expect to change directions, constantly.
Before Agile, management would attempt to create a plan which perfectly documented (i.e. predicted) what needed to be built, how it would be built, and how long it would take to build – the latter being perhaps the most important piece. As we now know, these plans rarely succeeded at predicting any of those things with any degree of accuracy. This usually led to highly unrealistic expectations and timelines from the start.
To make matters worse, given the lag time between when the plan was finished and the product shipped, it was highly likely management would change their mind about what to build with no sense of the cost. This made engineers, who thought their job was executing on “the plan” very angry. It meant that unrealistic deadlines were made continuously worse forcing many teams to try and do even more planning to reduce later changes. Ironically, the lack of transparency in the building process ended up giving engineering teams tremendous autonomy to choose the order in which to build things, how to build them, who would do what, etc. Of course, this type of autonomy and lack of transparency usually end up resulting in mistrust between management and engineering. In the end, no one was happy.
Enter SCRUM, which was designed to introduce more transparency to the process and drive engineering efficiency. Each week, managers could decide what stories (a.k.a. features) they wanted next, and engineers would then estimate their cost in time and resources. The team could then tally the cost of all prioritized stories to figure out how much was reasonable to commit to in the next sprint.
SCRUM dramatically increased transparency and made it harder for managers to ask for more than could be accomplished. It also acknowledged the need to update plans iteratively. But the increased transparency actually gave engineers less autonomy than before. Engineers’ authority was reduced to determining how long something would take, and picking the next story from a queue determined largely by management.
An engineer building a story might have no idea of the larger context as to why that story was important or what larger initiatives it was a part of.
While SCRUM did manage to rein in impulsive managers, it ended up being used more to exert tighter control over engineers' work.
Management still decides and prioritizes features, as well as who is on what team working on what more generally. With its emphasis on cost estimation of predefined chunks of work, given very little context, SCRUM harkens back to the detailed instruction cards Frederick Taylor would give factory workers at the turn of the last century.
Taylor’s Scientific Management method taught that managers were the ones who had the expertise and context to figure out what should be done. It was thus their job to give detailed enough plans to employees to ensure they needed to make as few decisions on their own as possible.
Near the end of the list of Agile principles sits an oddly out of place tenet that SCRUM ignores:
The best architectures, requirements, and designs emerge from self-organizing teams.
Theoretically, it is possible to implement SCRUM as a way of tracking the work in self organizing empowered teams, but it doesn't prescribe that and it's rare to see. Most books on SCRUM come at it from the perspective of top-down management, not self-organization.
Agile was an important step forward in acknowledging the importance of testing and iterating based on customer feedback, but the vast majority of its implementations are targeted at increasing efficiency and control vs empowerment or self organizing teams. One of the founders of Agile agrees.
You can argue that the problem isn’t Agile itself but merely implementations like SCRUM. That nuance would be lost on most people, as SCRUM and Agile have nearly become synonymous. Unfortunately, as Agile became popular, it also became co-opted by traditional command-and-control mental models resulting in methodologies like SCRUM.
What’s the alternative? Check out Spotify or Yammer's engineering methodologies. Or Holacracy if you want to get really crazy. What they have in common is this: instead of pushing work to individuals, they empower teams who are given more autonomy to decide what and how to build. Also, these teams are composed of people from different disciplines to ensure they're able to do whatever is necessary without waiting for others to give permission or provide assistance.
Whether your organization is using SCRUM today or not, if you're still feeling like you are moving too slowly and not building what your customers want, there are ways of experimenting with new models without trying to change everything at once.
Pick a project and put together a small dedicated team of people with all the skills required to do it, even if those people have different managers. Give them the problem and empower them to solve it however they see fit.
Ideally, the project would be short enough to quickly know if the experiment is working. If you're uncomfortable giving them that power, you either didn’t pick the right people, or have to admit the problem may be with you. Calling it an “experiment” will make it easier to sell your ideas. You'll likely find that a small, dedicated, cross-functional team of experts can accomplish a tremendous amount in a small time.
The next challenge is figuring out how to scale it when it works. This usually requires new solutions, which will cause new problems, which will require new solutions. It’s not easy, but the rewards are worth it.
You’ll know it’s working because your team will be coming up with more new ideas and getting those ideas in front of your customers faster than before. That’s the goal.
Of course, autonomy requires trust which requires shared context. That's why these newer models depend on full transparency to keep everyone in the loop. At Yammer, we made project teams short-lived so that people have more opportunity to work and learn from a diverse set of people, but that's just one way to go about it.
Finally, it's not enough for management to be the voice of the customer. Engineers and designers themselves must understand and be able to empathize with the pains and goals of the people they are building for.
Agile represented an important step our shift away from industrial era organizations and practices. But if you want to create an organization built for the 21st century you will need to look beyond Agile towards newer models that distribute authority, are self-organizing and truly invite your customers, partners and employees to own the process.