The Goal Is Not Agile

You hear about and read about agile transformations, like this article about ANZ

“We need to break with some of the traditional 20th century approaches to organising and working to ensure we are more responsive to 21st century customer expectations.

“Moving to implement the agile approach at scale in our business is an important evolution in how we run ANZ which will allow us to respond much more quickly to customer needs, create higher staff engagement and make further improvements in efficiency” – ANZ CEO, Shayne Elliott

But what does agile transformation really mean? From the article, the goal of ANZ’s agile transformation is to specifically target customers which is a hopeful sign. If we look at the agile principles we see customer focus is essential:

  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  • Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
  • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

ANZ seems to be looking to improve employee engagement. This can be achieved by recognizing the agile team as equal business team members. They bring a different talent set to the table but they’ll be closer to the customer problem and potential solution than anyone else.

And lastly, ANZ describes their goal to be more efficient. This doesn’t just mean doubling output but doubling the value of each outcome.

ANZ has announced they will use agile as the tool to achieve the objectives. One can interpret from the article that the goal is not to be agile but to use an agile approach to improve and be better. Easy, right?

Although there are several aspects of an agile transformation, I’ll mainly look at it from the agile team’s perspective and how they can effectively change their working environment to embrace the agile culture.

Agile culture

Agile is a mindset or cultural phenomenon that is hard to quickly adopt when the culture is Hierarchical and Command & Control. The ANZ article says, “The transformation program will … shift to a less hierarchical organisation, relying on small, autonomous teams.” I’ve seen companies who take a different road in adopting agile practices to improve. They adopt agile without having a ‘agile transformation’, instead they test and experiment (see this blog post) with new ways of working and discover agile ways worked better than most. The common activity whether doing large-scale ‘agile transformation’ or using a gradual adoption is experimentation.

In the book, “Blue Ocean Strategy,” W. Chan Kim and Renee Mauborgne cite four hurdles a company will face trying to institute broad change:

  1. Cognitive – people must understand why the change in strategy and culture is needed
  2. Resources – changing an organization will require shifting resources
  3. Motivation – workers have to want to make the change
  4. Institutionalized politics – overcoming the way we do things around here

So where does agile fit into this picture? If your organization is doing waterfall or is rigidly hierarchical in nature or is Command & Control, all four hurdles will be a challenge. Agile as a tool won’t solve any of these challenges, in fact, it’s the opposite, these four hurdles will need to be overcome for agile to work best.

How agile transformation happens

Let’s assume for the moment that the organization is a typical C&C waterfall environment. Businesses can be successful with waterfall but that success can have limitations. These limitations primarily center around taking advantage of opportunities requiring speed and the ability to fine tune the product as you go through continuous customer engagement. When we choose to do agile it isn’t because we want agile, it is because we know the waterfall methodology we’re using won’t always work for us.

Why the need for change

Change can be driven for strategic or tactical reasons. The ANZ initiative seems strategic by drawing the customers and the business closer together. Another way agile finds its way into an organization is tactical, like when an important business opportunity comes along requiring very quick results with specific functionality and quality.

For either reason, agile and scrum may seemed the only way forward. There are rumors and anecdotal stories about how great scrum and agile are and for some businesses, the decision to try agile re-enforces one hard truth, “Necessity is the mother of invention” or simply:

Necessity drives innovation

The first hurdle in cultural change, Cognitive, is met: we understand the need to do something different to be more successful. In other words, can the waterfall business cite an instance where a new product is created and delivered in weeks, not months (or years)? With a win here, the business, will be competitively better positioned. The agile team, product, and the executives are convinced a change in strategies and culture are necessary but this understanding is not enough to cause change. Trialing any new idea or new practices is a good way to build confidence that a change will result in success.

Get the right people

For a trial project the business doesn’t engage all developers but only a select group. This trial project has a clear goal and a definite release date, neither can be changed. What can change is the scope (as long as the goal can still be met).

The Resources chosen for the first scrum team are special in their experience and talents. Although there are whole product teams that might be used for this new business opportunity, the business decides to build a new development team by cherry picking people from existing teams. The team might be hand-picked for a number of reasons which may include:

  • Years of experience – with experience comes confidence they’ll be successful.
  • People who are self-motivated – these are not people who sit around waiting to be told what to do.
  • People who are innovative – these people are known for their new ideas and abilities to think around problems.

Selecting people for the first agile initiative is often driven by technical expertise and their ability to solve problems. These people are asked to do something very different from their waterfall paradigm but they are more goal driven than most and change is less stressful to them. After a minimum amount of agile training, 45-minutes feels about right, they’re off.

The idea of shifting resources to get lean will happen without much thought. The initiative to be agile in this trial means traditional reporting lines are cut and the agile team reports directly to the agile sponsor.

The only real casualty will be middle managers who traditionally told the developers what they needed to do. With agile, we push this responsibility onto the team. But agile needs managers to clear away obstacles that slow down or prevent the teams from being successful. It’s only when managers can’t let go of their control over people that they become the obstacle.

Motivation for change

One ingredient used as a motivator is the challenge to be more successful. Using your business’s own track record might be the best evidence that change is required. It’s probably already clear that current processes and practices are not given to quick wins. A long decision chain slows things down. If the decision maker doesn’t like the choices available there could be lots of back and forth negotiations. All of this results in delays. What might not be clear is agile cannot change these circumstances. Agile is more mindset than physical and therefore it takes people to push for change. Agile becomes a guiding set of principles that you aspire to achieve. Arguably the most important of the agile principles after satisfying customers will be getting the development team and business in sync and working together as equals.

One very successful agile transformation I was involved in started with the business leaders meeting with the development team and restating the business problem not as leader-to-subordinate but peer-to-peer; the business has this problem and we need to work with you to solve it. We’re not intending to create an elite team but a team with strong mission understanding. This team wants to be successful and given the business or customer problem, business leaders are asking the team to solve these as best they can. In my experience, this will super-motivate the team to try new ways and explore new ideas. The best part is when the team completes the mission, they become the seeds in other newly formed agile teams. They’ll bring with them stories of both successful and failed adventures, but more importantly, they are the seeds to start institutional change.

Institutional change

“One of the things that limits our learning is our belief that we already know something.” ― L. David Marquet, Turn the Ship Around!: A True Story of Turning Followers into Leaders

Changing the way ‘things are done around here’ is hard. If being successful is a motivator for change then it’s clear that the business needs to understand what processes, practices, and decision paths are slowing us down. It’s likely that processes created for a specific reason have bloated or the original problem has itself changed or disappeared. One way to make this work is to drop everything, then analyse each process or practice to get the essential value of it. For example, if a previous practice had the chief architect review the detailed design once completed, you might change this to the team does the detailed design together with the chief architect. Or maybe an architect needs to be a member of the team. Either way can eliminate a hand-off and a potential delay. The point is to streamline existing processes to capture the essence and intended value, not necessarily throw them away. Of special note are those practices and requirements which are generally non-negotiable such as security, quality, and regulatory. Adopting agile doesn’t mean putting the company or customers at risk. Be cautious and review & streamline these with the right people.

There may also be processes that can be made part of the team’s definition of done (DoD) checklist. For example, we had a governance requirement for high-level designs to be reviewed by a ‘design steering committee’. We made updating the high-level design part of the DoD. We didn’t have a written process for this but did have a checklist that served the development team. High-level designs were decided upon during grooming with the chief architect. Once these were documented the chief architect worked with the steering committee, allowing the team to get on with the business of solving customer problems.

Changing institutionalized practices will be a challenge. Changing institutionalized management structures and practices can be impossible. Changing management often means changing the way human beings see themselves. If you’re the manager of a product team and the team is now empowered to make some decisions you once made, or owning practices you once controlled, you might feel a loss, even if you know:

  • Decisions are best made by people who have the knowledge to make that decision.
  • Keeping practices current, efficient, and relevant is best done by the people who use them the most.

By empowering the agile teams to make some decisions and giving teams the responsibility to continuously improve their work practices doesn’t mean managers are obsolete. Managers are critical in setting the boundaries from which the teams operate. Management are the enablers for the agile teams. It’s management’s responsibility in an agile world to ensure the agile teams understand the boundaries, what decisions they’re empowered to make, what processes and practices they own, and, most importantly, management ensures the agile team has the knowledge and support they’ll need to be successful.

Success is about being better, not being agile

My first scrum team was successful beyond all expectations. So successful in fact that the CEO asked afterwards, why aren’t we doing this “agile thing” with all our teams? With that comment from the CEO, comes the first truth about agile transformations: it’s not about agile, it’s about being better. I would wager most CEO’s are less concerned with how or what you do to be successful than they are about being successful. The article about ANZ says “implement the agile approach at scale” which can mean many things.

In my situation, what agile meant to the CEO and the rest of the business was a different approach that included:

  1. quick feedback loops to make decisions,
  2. high visibility of progress and setbacks,
  3. highly motivated teams,
  4. the agile team’s focus on business goals,
  5. the agile team’s continuous engagement with customers, and
  6. the teams desire to solve customer problems with the customer.

All these things made the teams better, made the business better, and made the customers happy. These agile things could not have happen at the scale it did without change.

Agile Product Backlog Grooming Reduces Risk

Agile product backlog grooming is a development team and product manager/owner activity that removes the necessity of massive up-front planning. Backlog grooming is also a key risk reduction activity for an Agile project. The risks reduced or eliminated can include feature dependencies, costs, schedule, building the wrong product, knowledge, feasibility, and capability.

With Waterfall projects, there’s a great deal of up-front analysis of requirements and locking down scope from which cost and schedule are derived. There’s no need to ‘groom’ these requirements throughout the project because they’re not meant to change. Product backlog grooming is a frequently occurring process in Agile projects for the simple reason that Agile projects do not do full up-front requirements analysis. With Agile, the assumption is we focus on the value delivered to the customer and anticipate the customer changing requirements as their business world changes and evolves. Agile caters to the dynamic customer and business while Waterfall works really well for customers whose world changes very slowly if at all during the lifetime of a project.

In anticipation that some of the up-front analysis of requirements will be wrong, the Waterfall project manager puts in a great deal of effort to define and manage the risks associated with locking down scope so early, often times before people in the project know what they’re getting themselves into. This is a good thing because no one can know everything up-front.

Backlog grooming

With Agile, there is no pretense that we know everything. In fact, the Agile project team goes into the project knowing that they know very little of the details. For example, a typical Agile project might know enough to plan for the next 2-3 sprints (6-weeks if doing two-week sprints). Scrum defines an activity call ‘Product Backlog Refinement’, grooming by any other name, which is an ongoing process conducted by the Scrum team to add details, review, refine, estimate, and order the user stories in the backlog. While the Waterfall project has a risk registry and an ongoing risk management plan, Scrum involves the Scrum team in backlog grooming as a key risk reduction activity. These risks include:

  • feature dependencies – Scrum teams will try to make stories independent but this is a goal, not the reality. Implementation dependencies will exist at times and through the efforts of backlog grooming, these dependencies are identified. Once identified, the development team and product manager can order the backlog taking these dependencies into account
    costs – grooming can reduce wasteful work being prioritized too high. The scrum team combs through the backlog looking to keep those stories customers are willing to pay the most for near the top. Grooming keeps those stories that make achieving business goals and customer goals at the top while placing stories about, lets say, place settings on the titanic, at the bottom or removed entirely.
  • schedule – most projects work to some schedule and grooming helps keep focus on those stories that will make your project a success. Schedules might be tied to sales cycles, business cycles, customer deadlines, or events. The goals for these scheduled dates are usually agreed to by the teams but based on the available knowledge when the commitment was made. Backlog grooming can help keep the biggest bang stories, those stories that provide the greatest contribution toward the goal, at the top even as more knowledge is gained.
  • building the wrong product – backlog grooming asks if the stories in the backlog are what the customer needs so business goals can be achieved. As the project progress so does the likelihood that customer wants and needs change. Backlog grooming sessions check the staleness of information used to prioritize stories. If it’s been awhile since the team spoke with customers or customers have been absent from sprint reviews, then an action from a grooming session could be to re-validate a customer problem or solution with customers.
  • knowledge – product backlogs usually have the structure where large stories sit near the bottom and progressively smaller stories populate the backlog until the top 2-3 sprints worth of stories are small, ‘sprint ready’ stories. The process of taking larger stories and breaking these down to sprint sized stories never ends and will transcend projects. The process of breaking down stories involves digging out more details. These details can reveal a knowledge gap exists that needs to be closed before a story can be worked. Unless the story came up from the depths unexpectedly, there will always be time to recognize the knowledge gaps and find the knowledge either through training or getting a temp specialist.
  • feasibility – like the knowledge risk, feasibility can be handled before actual time to implement the story. When following a routine groom paradigm where large stories are broken down over time, it will be easier for the development team to provide earlier analysis of feasibility. The first analysis of feasibility would normally occur during the ideation phase, when a feature is added to the road map. However, the difference between “we should be able to do this” and “we know we can do this” can potentially put a company out of business.
  • capability – this ties in closely with the knowledge risk but is broader than team skills and capabilities. This includes things like test environment, development environment, build environment, customer access, and stakeholder involvement. Again, as larger stories are broken down during grooming, the team assesses the these needs to make implementation and delivery of the story successful.

Things that impede grooming sessions

Development team doesn’t feel responsible for the user stories.

“writing and rewriting user stories ain’t my job.”

I think this will be the impediment most often seen slowing down grooming sessions. I found that once development got used to their traditional roles, it was hard to get the developers on board with writing and rewriting user stories. Although the product owner remains accountable for the order of the backlog, they are not necessarily solely responsible for the stories.

All I can advise is setting the expectation that as members of the Scrum team, they all equally share the responsibility for the product backlog and its contents. This is best if re-enforced by management. The other factor will be time, it may take several months for the full team to accept this new responsibility. The Scrum Master can use the retrospectives as a vehicle for improving the grooming sessions. For example, can the bigger stories be broken down by anyone else other than the product manager? Someone in the team will suggest they can help the product manager. This is a very positive step forward. A note of caution: developers might try writing user stories that are very technical in nature.

Product manager not giving up control of user stories

I found that this will happen but goes away rather quickly as it’s the development team that needs good stories so they can attain a good understanding of the work required. This follows these three steps:

  1. Product manager controlling all aspects of user story writing
  2. Development team is in a struggle with the product manager over meaning and acceptance of user stories
  3. Product manager makes a business decision to allow the development team in on writing stories and reducing the amount of time explaining what the stories mean.

Backlog grooming sessions not a team event 

How much fun is backlog grooming for your Scrum team? Are you part of a team or scrum master of a team that goes to grooming only to agree with whatever the senior developer or team lead says? “Does everyone agree?” “Yes.” [Audible ‘TICK’] Maybe is was better when we did waterfall!?

It’s important that the first backlog grooming sessions be well facilitated so the team can be drawn in to the backlog and backlog grooming process. Going around and asking each individual what their opinion is, to explain their understanding of the story, and to describe what the high level design is will help. Some people will take to this quickly while others will take several grooming sessions to draw them out. Don’t give up but continue the facilitation until participation becomes natural for the team.

I’ve also seen where having the team lead or team manager as part of the team can stifle participation at grooming sessions. Providing coaching to the manager to encourage a ‘servant leadership’ approach is of inestimable value to creating a self-organizing team. This has worked great in the past and was the subject of a talk I gave at Scrum Australia 2016.

Being Agile When The Business Isn’t – Transparency

In the last two posts, ‘Being Agile When The Business Isn’t – Requirements‘, and ‘Being Agile When The Business Isn’t – Team‘, I talked about how to be more Agile in an environment where the business is strictly Waterfall. I’d like to continue this theme with some ideas and practices to make the teams more ‘transparent’.

Inspect, Adapt, & Transparency

Most will recognize Inspect, Adapt, & Transparency as the three pillars of Scrum but they’re ideas that even the most Waterfall of businesses can use and take advantage of. These three pillars are the core to continuous improvement, building what customers need to solve their problems, and helping bring the greatest valued work to the forefront. I’ll focus on transparency as I believe once you have transparency, inspection and adaptation will follow naturally.

Perhaps the big difference between Agile Vs traditional project management is how these three items are acted upon. When I was doing big Waterfall projects our alleged transparency came from the project’s communication plan. This communication plan defined the reports the project team was responsible for generating. At or near the end of the project, we wrote project closure reports which often included lessons learned. While writing the closure report, we were being formally transparent but only after everything was done. The real problem with the closure report was the timeliness of it. Because this is traditionally done at the end of the project, nothing learned here can help the just completed project. We also suffered from fading memories. We remembered the specific events but often needed to speculate and debate over the causes. Interestingly, we also seemed to forget these lessons for the next project or determine the next project is different enough so the lessons weren’t considered applicable. I’m not suggesting nothing was learned but any learnings carried forward tended to be the exception.

The result is some learning is done but the changes haven’t been trialed or proven to be effective in solving whatever the problem was. To improve requires openness and transparency so you can inspect what has happened and adapt to new and better ways throughout the project not just at the end. To maximize the value of improving, the project team needs to be continuously inspecting outcomes and results including any new ways of doing things. Building in transparency from the start will help.

Transparency

Transparency is arguably the first step for improvement. Nothing will help a business improve better than seeing undeniable evidence that some work practices, processes, and communications are less than optimal or just plain bad. In the Waterfall environment, the project plan includes what practices and processes the teams will use and the order they’ll use them. Most project plans also have a communication section where and how the various project statuses and metrics are reported. However, it’s only an illusion that transparency exists by executing a communications plan. The reason is, communication plans dictate specific content and intervals, usually within categorizes like any of the following:

  • Current project status
  • Current risk assessment
  • Resourcing status
  • Requirements management status
  • Stakeholder management status
  • Change management status

This doesn’t provide any insights or linkages to bigger or widespread problems. There’s also the problem that the reporting is done by the project manager. This can lead to information being filtered and potentially skewed due to conflict of interest. Project managers are generally held accountable for the project plan which means they’re accountable for perceived progress or the perceived lack  of progress. And finally, after a while, the project manager is most often simple ticking the box on his or her reporting requirements.

Daily standup & transparency

Scrum Trainer Charles Bradley wrote in his article, “The Tao of Scrum – Transparency“, “the best transparency leads to the best collaboration loops”. To maximize the value of collaborative feedback loops requires transparency and to me, transparency implies anyone can quickly gauge what’s happening in a project with a minimum of effort. Although you might be stuck with an old-fashioned communication plan, there’s an easy ceremony you can implement with your Waterfall team that gives extra clarity to what’s happening on the project: a daily standup. Each day the team talks about how their plans for the previous 24 hours went and what their plans are for the next 24 hours. It also provides an opportunity for others with their own unique perspectives to make observations and comment.

In my Waterfall past, as the project was approaching the final stages of test, we had all the project managers, line managers, and the program office people sit together daily to talk about specific tasks the teams were doing and when we thought they’d be done. Although this worked quite well for the management team, the nuance of individual planning was lost. Had there been a team standup, the confidence or uncertainty was more likely to come through. As it was, what was reported to management was an interpretation of plans and status.

In a Waterfall environment, the full project team might be large since the team is built from specialist who usually don’t take on tasks outside their specialty. This can mean the daily standup is large if everyone goes so careful monitoring is essential. The daily standup must remain valuable and true planning must occur otherwise there’s no point to it. In Agile, one valuable component of the daily standup is the collaborative nature of getting the most important work done first. In Waterfall, the project manager and product manager might need to be a little flexible in their detailed project plans to allow the collaborative nature of the implementation team to emerge. Better still is for the project plan be aligned with getting functionality done in order of importance rather than based on components and skills of the team.

Plans & transparency

Another aspect of transparency will be the visibility of the plans. If the project is using MS Project or Jira for Agile teams, it is a huge mistake to think this by itself constitutes transparency. Both these tools track progress when properly maintained but in both cases, people often don’t take the extra steps necessary to open a file or log on to Jira or even learn how to use the tool. The real problem isn’t progress wasn’t tracked, it is, but the real problem is the difficulty of getting to this tracking data and properly interpreting it. These tools have their advantages but ease of use is not always one of them. It’s not that these managers and program office people are lazy but rather their time’s at a premium. Visualizing the project and sprint status on a wall is easy. Bits of sticky paper with notes, in columns is easy. I have used a big wall as a Gantt chart or User Story board to great success. It really doesn’t matter if you’re doing Waterfall or Agile, the advantages of this high visibility is the same: huge.

If your team and stakeholders are global then it’s a wise idea to set up a camera on the planning board. I had used a camera while we did standups to allow global stakeholders to see & hear what was going on. I had a second camera on our ‘story board’ 24/7. This was intended to add more transparency and to give our global stakeholders a bigger slice of the picture, allowing for more feedback and collaboration.

Transparency is a conscious effort

Agile isn’t immune to poor transparency and communications. As Scrum Master, I’ve been cornered by the product owner/manager and asked what the ‘real’ state of progress was in the sprint. Two things needed to happen: the product owner needed to attend the daily standup and the team (with some advice from the Scrum Master) needed to better present state sprint progress. In Waterfall, the project manager has their hands full too. While Agile implies the implementation team is involved and responsible for managing expectations and communicating value completed, Waterfall usually relies on the project manager to do these. It’s up to the project manager to move beyond the project plan and establish a collaborative atmosphere where visibility and transparency become natural within the project team.

Transparency can be scary as it will expose people, their work practices, and their outcomes to all. It’s not the natural state for developers who learned their craft in a Waterfall environment. The first time I had transformed a Waterfall team into a Scrum team, one of the team members told me that they were completely stressed out by the high visibility of their work. In his case, a technical decision had proved wrong in a most open sort of way. In the past, management would probably not know that a technical glitch had happened but Agile made it visible. In hindsight, the team member realized the focus was to identify and correct a technical decision quickly and not to allocate blame or ridicule, (he was unconvinced of this in the first days). Transparency became a tool to do better and improve but not without some pain on the journey. Making previously hidden work habits visible will be stressful for some but this can be countered by avoiding the culture of blame and ridicule.

Summary

By using transparency as an aid to see what’s working well and what can be improved, the opportunities for improvement are more likely to be seen. Transparency has two key accomplices: daily standups and the wall project planner. Both these can be used in Waterfall environments without too much effort beyond simply doing it. Having the implementation team being transparent and talking directly to their plans each day is a key element of Inspection & Adaptation. This daily transparency reduces the risk of going off on a tangent as illustrated in the earlier story about an unsuccessful technical decision.

Transparency also extends beyond traditional tracking tools. Although these tools help track and document progress, it requires both effort and training to use these to their full potential. Training all stakeholders and customers with these tools is hard and may be impractical. Using a simple system of sticky notes and a big wall can convey the same information in a visual way that transcends most communication roadblocks. If there’s no wall, then use a big whiteboard. No whiteboard, then use the windows. You can even put a camera in front to broadcast the planning board across the internet if necessary.