Why Is “Agile” not Working?
84% of agile transformations are not successful yet.
In the 14th State of agile report, the vast majority of respondents (84%) said their organisations were below a high level of competency with agile practices. What would happen if companies would say this about other equally relevant business activities like accounting, sales or operations? Would we find that acceptable?
The key challenges organisations face when adopting agile have remained essentially unchanged for the past years. Challenges with organisational culture, resistance to change, and lack of support and skills remain problems. We believe these are symptoms of the underlying problems rather than their root cause.
The main challenges experienced in agile transformations include:
- Inconsistencies in processes and practices 46%
- Cultural clashes 43%
- General organisational resistance to change 46%
Let's zoom in on these three challenges and see why in our experience, you should not start with Culture or resistance to change.
Culture is the wrong end of the right stick
We believe culture is not a factor that we can impact directly. You can’t ‘install’ a culture unless maybe like Mao’s Cultural revolution with unethical means, and we all know how that turned out. The attempts that we have seen to explicitly influence culture consistently fail — and often with demotivating consequences for the people involved.
Attempting to change an organization’s culture is a folly, it always fails. People’s behavior (the culture) is a product of the system; when you change the system, peoples’ behavior changes. Systems-thinking thought-leader John Seddon
Agile Culture
When people are introduced to Agile, they are often attracted by the promise of a much better corporate world. Examples like Google, Spotify or Start-ups are used to paint a perfect agile picture. However, what is perceived as simple enough behaviours that we could easily copy to our own organisation are often the results of drastically different fundamentals than are present in most traditional companies.
Manage the System, Not the People
Studies show that people resist change if it’s forced upon them. What we can do, however, is manage our practices and processes. It might not sound too agile; after all, the agile manifesto, a document containing the basic agile values and principles, states “individuals and interactions over processes and tools”, but even removing processes and tools almost always needs some process and management. We have actually seen the lack of management involvement hinder agility. When current processes are simply removed without catering to the underlying needs of the people who were part of such processes, it is no surprise that we are met with resistance.
What we also can do is manage the composition of the teams. Here, management is usually heavily involved, but often with the old perspective of creating efficient teams that can take care of old objectives and paradigms as opposed to teams that have the potential to develop an agile culture. When these teams are plunged into something like Scrum, no wonder they experience cultural clashes and inconsistencies in processes and practices.
Of course, most managers already know this. So why is it that we still have not managed to achive the benefits of agility?
A complex system that works is invariably found to have evolved from a simple system that worked. John Gall: Systemantics: How Systems Really Work and How They Fail
- All of the Big 5 business consultants we worked with had to admit they had never personally seen a successful agile transformation.
- Not a single agile coach we know has run a successful large business.
- Almost all of the managers we worked with were not knowledgeable enough about agile business.
With all due respect, it’s like the blind leading the blind.
Business consultants will help you with a business transition, which will result in dealing with your business in a different way. Agile coaches can give you an agile transition, making you work differently than before. Managers must bring the two together if we want a successful agile business transformation. Transformation is not only about doing stuff differently; it’s about becoming something different — reaching a stable new state of the organisation that gives different outcomes and does so sustainably. Unfortunately, this rarely happens, and it’s even more unfortunate that not much is done about it.
Utopia
The agile coaches community seems to be drifting to an evermore utopian view of agile. The No Estimates movement and the bashing of Scrum or Agile are good examples of this, as they are propagating more liberation of the teams by having less management and structure. Although they might be worthy goals in themselves, we can not simply abandon all structures and get rid of management. In theory, this might sound appealing, but this is almost never done in practice. Some governance and some structure always remain within the organisation, so we might as well actively manage both. Paradoxically, the journey to self-management needs management and definitely quite some structure.
Classical reorg
On the other hand, the business consulting community seems to be drifting the dystopian way by propagating fixed agile scaling frameworks that are often easier to map to a current org chart and the previous ways of working. This regularly results in some initial improvements but never fulfils the true promises of agile. Due to a lack of agile experience within their own organisation, major business consultancy firms are often hired to implement the agile transition. These consultancy firms often deploy a team of consultants with astronomical daily rates. Clear dates and targets are often negotiated in the contract to limit the duration and, therefore, the cost of this operation, not rarely with hefty fines if these targets are not met. As agile is something we can not easily measure directly, these targets are often related to tangible changes that are not necessarily agile. Classical reorgs are what consultants are hired for, which is often the bigger part of the implemented change, traditional turf wars included. New roles are derived from some out-of-the-box scaling framework, and off you go, the agile transition is done.
Big bang
These contracts are often tendered to several parties. As the organisation is new to agile, it’s difficult for them to decide based on quality or track record, so one of the main selection criteria is the total price of the contract. The consultancy firms try to be competitive not by reducing their rates but by limiting the transition duration. This is often where Big-bang transitions originate from: not the need to become agile as soon as possible, but the need to complete an inherently non-agile fixed reorg project.
Like contractors, who make most of their income from the “extra” work needed to get the home you want, consultancy firms are banking on follow-up assignments when people find out that things aren’t working yet. There is actually a perverse dynamic at work here. Responsible managers have committed themselves by appointing a consultancy firm and working together with them. In traditional organisations, this leads to connecting their faith to the result of the delivery of the contract rather than the agile transformation. It’s difficult not to lose objectivity when you have so much skin in the game. We now face the situation that both managers and consultants are looking to deliver a transition that is just a reorg within an often impossible timeframe if you also want to introduce agility. I think we can all guess what goes overboard first!
We have repeatedly seen that management does not get the actionable support they need from either business or agile consultants to create agility within their organisations. We have rarely witnessed focused support on actually achieving progress towards positive business outcomes through agility.
More the factory, less the sausage. Martijn
Management is often so heavily involved in the daily business that they lose sight of the organisational system they are managing. All sorts of things are reported to the top layer, and we have been genuinely amazed by some of the discussions we were faced with in C-level meetings. The more intricate the info, the more respect often gained by fellow participants. Not seldom do we see management teams try to outsmart each other with the level of detail they know about some minor crisis that has escalated all the way up the food chain. Where as more often than not, the teams can work things out themselves. It seems to be in the nature of management that we value escalation of problems more than individuals and interactions, and this is especially true when it comes to any issues with other departments, where we see this game being played to its fullest extent. Agile is, for a significant part, the reverse of this dynamic: solving problems where they arise with and by the people that have them over solving problems in some boardroom far away from the actual problem.
Why
If the business need is clear, the Why for the change, we do not need to use ambiguous container concepts like Agile, Transformation or Mindset. We can make things very concrete and solve problems where they happen with the people who have these problems. These are real business problems, not problems that agile coaches consider bad because they diverge from some ideal state of agility. They are real business objectives, not some compliance that a consultant is hired to deliver towards a massive scaling framework. When solving these small or big problems locally in an agile way, we experience very little resistance, if any, towards the changes. By doing this throughout the company with a local focus, we might lose track of the bigger business agility picture. We believe this is where agile management connects all the small changes to the bigger agile ambition and vice versa. Evaluate the progress towards our agile business ambition and hunt for problems that might impede this progress.
Three Vital Pillars: People, Structure and Content
Throughout our agile consulting years, we have identified three vital pillars in our agile organisational system: People, Structure and Content. We need to focus on these core elements for any successful agile transformation. We use these as perspectives to detect problems that might hinder the business outcomes we desire. We will try to improve by initiating changes along these axes, creating almost always a combination of balanced actions in these three domains.
Here’s an example: People complain that they are stuck in endless meetings. As managers, we would traditionally maybe instruct people to limit the number of meetings or arrange meeting-free blocks in agendas. In our experience, this just spreads the problem and might have side effects. On the other hand, if we change the structure of the meeting to a stand-up, a more active meeting form where people remain standing so they don’t settle in comfortably for endless debates, we see shorter and more productive meetings. If we change the people attending the meeting to just the team members, we avoid having to explain things to outsiders and keep the meeting focused. If we change the content by setting the agenda to a fixed format with standard questions requiring just a short response, we can switch from a boring, endless status meeting into a vibrant affair. Not a one-off fix but embedded permanently.
The Three dimensions: People, Structure, Content
People
The heart of the system is our people. Respect for people as individuals and as a group forms the foundation of modern management. We have seen that, in the right environment, people can achieve amazing results. A framework or management method has not produced a single significant improvement, breakthrough or innovation. It is always the interaction of thoughts and acts among people that drive organisations and cultures forward. Facilitating these interactions is ultimately the aim of any agile manager. Important to emphasise that we are not looking for a fixed idea of the ultimate agile employee but rather for the continuous development of people and teams.
Structure
How we structure our organisations. How we organise the people. It could be teams, departments, or hierarchies, but also processes, meetings, practises, communication, frameworks and incentives. As most modern companies are digital-centric, the architecture of your business, along with the architecture of your software, are two of the most important structures that should enable business agility. Structures form the backbone of our system. Without it, people feel lost, a feeling of disorder sets in, and we inevitably roll back to the previous state of the system. This is almost always not agile. It’s not rare that when there is a lack of structure, people would actually rather be told what to do than explore the unknown further to create their own structures. Structures might sound a bit static and not so agile, but in a modern organisation, they could be as fluid as needed.
Content
What we create. What we produce. Not just the output but everything that flows through our system. Documents, research, e-mails, briefs, analysis, backlogs and our products as final output. Content is the fuel of the agile engine. We can create the most agile structures with super agile people, but the content will finally determine the outcome of our endeavours. As we no longer have overarching project plans and content is created by every team individually, we will need to manage the evolution of this distributed stream of content. Just telling people to create a backlog and get stuff Done is usually not enough. Unless you have unlimited time and resources, you probably want to expedite the creation of truly valuable agile content, as this is key to achieving any positive business outcome ASAP.
The scope of an agile business transformation is all-encompassing. Often, the Why, How, and What of the business will change to benefit from agility truly. The previous Why of the organisation and its purpose, if at all known and expressed explicitly, often needs adjustment given the current reality change that triggered the transformation in the first place. The How, a company’s DNA, and our way of working will change fundamentally, and this is the core of the transformation. As soon as the How becomes more clear, the What will inevitably have to follow if we truly want to transform. Leading to the need for continuous interaction and reflection on these three fundamental elements. To detect this need and actively act upon it is what agile management is all about.
The same applies to interaction between the three vital pillars: People, Structures and Content. We often see that Agile is not working because there is an unbalance between these three elements and the Agile ambitions. You can have the most agile people, but if the structures they operate within aren’t agile, the business outcomes will not be what you had envisioned that agile would bring. On the contrary, these situations often end up in frustration. You can have the ultimate agile framework in place, but if your content is still waterfall projects, you will get the same problem. It’s only when you balance People, Structures and Content and align them with your agile ambitions that things start to work and deliver positive outcomes.
What NOT to do
Our quickest advice to avoid failure in your agile transformation is to NOT do certain things.
- Don’t copy-paste someone else’s agile model.
- Don’t do a big-bang transformation.
- Don’t do a bottom-up transformation only.
- Don’t do only a top-down transformation only.
- Don’t “install” a culture.
More on this in the following articles and, of course, also on what you should do. But for now, know that warning signs should start flashing when coaches or consultants mention these approaches.
But, but, but… I can’t just do nothing!
You actually can do nothing. It’s better to do nothing than do the wrong thing. Although it might not sound very agile with our bias towards action and experimentation, we know some things will not work, even though they might make perfect sense at the time if explained in some elaborate shiny consultant PowerPoint. We can save you a lot of trouble, frustration and time by avoiding these pitfalls. What you can do is ask yourself and your organisation the following question: What problem are we trying to solve with the transformation? If you don’t get a fairly common and satisfactory answer across your organisation, don’t transform anything yet, but first, find the Why of the change. You will most definitely need it later to align the people. Please don't use someone else's why or copy it from a book. These might be great inspiration but you must find your own compelling why. Good agile coaches can help you find your problems and start from there.
This!
Problems have one major benefit over solutions: they exist. Solutions, on the contrary, could be numerous, and we don’t yet know what would solve our problem. We might have an idea, but in a complex world, no one can predict the future. Problems, on the other hand, can generally be fairly well analysed and quantified. It might take some effort and the learning of new skills, but it has definitely been done successfully and repeatedly before. So, the first thing that we suggest you do is not something specifically agile at all: gather data on your problems. If you don’t know your problems, start collecting data, and you will soon bump into some problems. The next article will dive into how to do this more concretely.
Summary: Manage the system, not the people
- Get the Why of the transformation crystal clear.
- Let the How evolve with the What.
- Let go of hands-on micro-managing your day-to-day.
- Three dimensions need to be addressed: People, Structures and Content.
- Don’ts: Culture implementation, Copy-paste a model, Big-bang, Bottom-up or top-down transformation.
- What problem are we trying to solve?
- Share your problems and ask your questions.
In the next articles, we will dive into these in more detail following our structure:
- Why, with root causes and research.
- How, with principles and theory.
- What, with hands-on actions to take.
We will start with how to determine what problem we should be solving with Agile.
We will gradually build up this series one article at a time. Let us know your burning questions or most pressing problems, and then we can expedite them in our series. Use the comments below or find us on social media.
Previous: Why a Series on Agile Management?
Next: What is our problem? (Comming Soon)
