Agile Product Backlog Refinement (Grooming)

Feature image by Photo by Crystalline Radical.

 

In my last post, Agile Product Backlog Grooming Reduces Risk, I talked about using Product Backlog Grooming sessions as a project risk mitigation tool. In this post, we’ll further explore the details of backlog grooming with some tips to increase the value of grooming. These tips are for the Product Manager, Product Owner, and Development team to help make the grooming experience valuable and easy.

Development and Product teams begin the process of collaboration before the Product Manager has pitched the new/modified product idea to the business or before anything is actually in the product backlog. At this stage, most if not all information is based on previous experiences and assumptions. However, depending on your business structure, the Product Manager will probably validate most Problem & Solution assumptions prior to presenting the project to the business. The Development person here is the product architect or designated representative of the team(s) familiar with the product.

  1. Product Manager gets an idea (from any source) on how to achieve business goals with his or her product
  2. Product Owner and Development discuss feasibility of the idea, reworking the idea until thought feasible, based on the limited knowledge at hand.
  3. Product Manager/Owner develops business rational including the risks based on limited knowledge.
  4. Product Owner and Development collaborate to create the Business Canvas. You might consider using Ash Maurya’s Lean Canvas or Roman Pichler’s Product Canvas and Vision Board.
  5. Product Owner and Development collaborate to validate the Customer Problem and Solutions with customers through interviews and user testing. (Remove any Canvas entries that failed validation. Might need to go back to step 1.)
  6. Product Owner and Development provide an initial estimate. We’re looking for the ballpark, order of magnitude type estimate here e.g., 6 weeks or 6 months?
  7. Present Business rational and Canvas to business stakeholders for approval and funding.

Establishing preliminary understanding

The Product Manager’s product feature idea is now on the product roadmap (or equivalent). Depending on your business’s average lead time for work all the following steps could occur over the course of months or in rapid succession. My own experiences had roadmap items identified up to a year before the project, not very timely for customers but there are other factors at play including priorities, competition, and ‘hunches’ from the executive team.

These few steps are for illustration purposes and may not be what your team wants or even needs so feel free to modify for your specific needs. These steps will probably be done once. However, depending on the timing, may need to be revisited if the Product Owner and Development architect determine the data gathered has gone stale.

  1. Create one epic in Jira for one idea or feature. Most people use a tool rather than a big wall these days but if you have the wall space and an enthusiastic Scrum Master, by all means use a wall to define the high level projects.
  2. Development architects do initial technical scoping and high level design or impact mapping on current architecture. Unless you’re contemplating a new product, this will be based on the product’s current technical foundations and platforms.
  3. Full development team discusses the problem and possible solutions (non-technical). This is with a larger group of people using validated feedback from customers about how they envisioned their problem being removed from the user perspective. This serves as the baseline for UX designs. The discussion stays at the ‘what’ level but the full team might have something to say about feasibility and sizing.
  4. Full team provides updated estimates for Product Owner.
  5. Create the paper or wireframe UX designs & validate with customers.
  6. Development team (developers, UXD, and testers) along with Product Owner to define or confirm their collective understanding of what a ‘sprint ready user story’ is i.e. they define the Definition of Ready.
  7. Full team provides an updated estimate for the Product Owner.

The value to the business achieved during the above steps is to develop a good high-level understanding of all the work necessary for the initial epic user story. The team does a couple of estimates so the Product Owner can determine if the feature is still economically feasible. You’ll note customer involvement, looking at display mockups. The ideal situation is the customer finds that only a portion of the proposed work is necessary. The Product Owner and Development team keep focus on satisfying the customer in order to achieve a business goal.

On-going backlog grooming

This section details the steps for mainstream backlog grooming that gets repeated until a story is actually in a sprint. Most of these steps will be familiar. The full development team is ideally one Agile Scrum team but if there are several teams working a product, the grooming session should include those teams likely to work the stories. With a large number of teams (> 3), the LeSS framework, by Craig Larman & Bas Vodde,  suggests doing the grooming sessions in stages:

  • Overall Product Backlog Refinement (shared among all teams)
  • Team-level Product Backlog Refinement
  • Multi-team Product Backlog Refinement

 

  1. Full development team, using UX and architectural designs, begins breakdown of Epics into smaller stories using the Definition of Ready (DoR) guide the team established in the previous section.
  2. Product Owner and Development team continue to prioritize stories based on customer/business value but taking into account any implementation concerns or risks.
  3. Each user story is reviewed starting with those at the top of the backlog. Every user story is accompanied by notes, acceptance criteria, and estimates using the DOR as guide. If the story can be associated with a step in the ‘customer journey’, that would be a great help for stakeholders toward understanding the work.
  4. Product Owner and Development team review & discuss each story including those that are marked ‘sprint ready’. For stories not ‘sprint ready’, discussions on how the story could be implemented, could be tested, the customer value, and discuss compromises as necessary. ‘Sprint ready’ stories are checked that nothing has change that could affect the team’s understanding of the story or its priority.
  5. Again prioritize stories based on costs and customer / business value
  6. Check that all Epics and UX designs are covered with stories and have been prioritized.
  7. Make stories sprint ready and keep anyone who’s interested informed. My experience is most managers will not be looking through tools on a daily basis to determine progress. But being transparent is critical and using a big wall in a conspicuous location will provide the necessary details to management. If you have a ‘story wall’, you can use colored stickers to indicate status of stories i.e. sprint ready.
  8. The last step of every grooming session will be to again review the estimates and priority of the stories. Always check  the most valuable work is top-most on the product backlog.

The desired outcome of these sessions is:

  1. Big stories are split
  2. Stories are estimated
  3. Stories are well understood by the team(s) likely to implement them
  4. Stories are made ‘sprint ready’ based on the team’s Definition of Ready

Summary

In my experience, the amount of time needed between roadmap item to having a couple sprints worth of ‘sprint ready user stories’ is about 4 weeks calendar time, assuming development teams are busy doing sprint work as their primary activity. This provides time for any proto-typing, UX design, additional customer/user interviews, and getting input on marketing, sales, support, or other business concerns that could affect implementation choices. This could take a few days if done by an experienced project team in a full-time workshop.

Agile Product Backlog Grooming Reduces Risk

Agile product backlog grooming is a development team and product manager/owner activity that removes the necessity of massive up-front planning. Backlog grooming is also a key risk reduction activity for an Agile project. The risks reduced or eliminated can include feature dependencies, costs, schedule, building the wrong product, knowledge, feasibility, and capability.

With Waterfall projects, there’s a great deal of up-front analysis of requirements and locking down scope from which cost and schedule are derived. There’s no need to ‘groom’ these requirements throughout the project because they’re not meant to change. Product backlog grooming is a frequently occurring process in Agile projects for the simple reason that Agile projects do not do full up-front requirements analysis. With Agile, the assumption is we focus on the value delivered to the customer and anticipate the customer changing requirements as their business world changes and evolves. Agile caters to the dynamic customer and business while Waterfall works really well for customers whose world changes very slowly if at all during the lifetime of a project.

In anticipation that some of the up-front analysis of requirements will be wrong, the Waterfall project manager puts in a great deal of effort to define and manage the risks associated with locking down scope so early, often times before people in the project know what they’re getting themselves into. This is a good thing because no one can know everything up-front.

Backlog grooming

With Agile, there is no pretense that we know everything. In fact, the Agile project team goes into the project knowing that they know very little of the details. For example, a typical Agile project might know enough to plan for the next 2-3 sprints (6-weeks if doing two-week sprints). Scrum defines an activity call ‘Product Backlog Refinement’, grooming by any other name, which is an ongoing process conducted by the Scrum team to add details, review, refine, estimate, and order the user stories in the backlog. While the Waterfall project has a risk registry and an ongoing risk management plan, Scrum involves the Scrum team in backlog grooming as a key risk reduction activity. These risks include:

  • feature dependencies – Scrum teams will try to make stories independent but this is a goal, not the reality. Implementation dependencies will exist at times and through the efforts of backlog grooming, these dependencies are identified. Once identified, the development team and product manager can order the backlog taking these dependencies into account
    costs – grooming can reduce wasteful work being prioritized too high. The scrum team combs through the backlog looking to keep those stories customers are willing to pay the most for near the top. Grooming keeps those stories that make achieving business goals and customer goals at the top while placing stories about, lets say, place settings on the titanic, at the bottom or removed entirely.
  • schedule – most projects work to some schedule and grooming helps keep focus on those stories that will make your project a success. Schedules might be tied to sales cycles, business cycles, customer deadlines, or events. The goals for these scheduled dates are usually agreed to by the teams but based on the available knowledge when the commitment was made. Backlog grooming can help keep the biggest bang stories, those stories that provide the greatest contribution toward the goal, at the top even as more knowledge is gained.
  • building the wrong product – backlog grooming asks if the stories in the backlog are what the customer needs so business goals can be achieved. As the project progress so does the likelihood that customer wants and needs change. Backlog grooming sessions check the staleness of information used to prioritize stories. If it’s been awhile since the team spoke with customers or customers have been absent from sprint reviews, then an action from a grooming session could be to re-validate a customer problem or solution with customers.
  • knowledge – product backlogs usually have the structure where large stories sit near the bottom and progressively smaller stories populate the backlog until the top 2-3 sprints worth of stories are small, ‘sprint ready’ stories. The process of taking larger stories and breaking these down to sprint sized stories never ends and will transcend projects. The process of breaking down stories involves digging out more details. These details can reveal a knowledge gap exists that needs to be closed before a story can be worked. Unless the story came up from the depths unexpectedly, there will always be time to recognize the knowledge gaps and find the knowledge either through training or getting a temp specialist.
  • feasibility – like the knowledge risk, feasibility can be handled before actual time to implement the story. When following a routine groom paradigm where large stories are broken down over time, it will be easier for the development team to provide earlier analysis of feasibility. The first analysis of feasibility would normally occur during the ideation phase, when a feature is added to the road map. However, the difference between “we should be able to do this” and “we know we can do this” can potentially put a company out of business.
  • capability – this ties in closely with the knowledge risk but is broader than team skills and capabilities. This includes things like test environment, development environment, build environment, customer access, and stakeholder involvement. Again, as larger stories are broken down during grooming, the team assesses the these needs to make implementation and delivery of the story successful.

Things that impede grooming sessions

Development team doesn’t feel responsible for the user stories.

“writing and rewriting user stories ain’t my job.”

I think this will be the impediment most often seen slowing down grooming sessions. I found that once development got used to their traditional roles, it was hard to get the developers on board with writing and rewriting user stories. Although the product owner remains accountable for the order of the backlog, they are not necessarily solely responsible for the stories.

All I can advise is setting the expectation that as members of the Scrum team, they all equally share the responsibility for the product backlog and its contents. This is best if re-enforced by management. The other factor will be time, it may take several months for the full team to accept this new responsibility. The Scrum Master can use the retrospectives as a vehicle for improving the grooming sessions. For example, can the bigger stories be broken down by anyone else other than the product manager? Someone in the team will suggest they can help the product manager. This is a very positive step forward. A note of caution: developers might try writing user stories that are very technical in nature.

Product manager not giving up control of user stories

I found that this will happen but goes away rather quickly as it’s the development team that needs good stories so they can attain a good understanding of the work required. This follows these three steps:

  1. Product manager controlling all aspects of user story writing
  2. Development team is in a struggle with the product manager over meaning and acceptance of user stories
  3. Product manager makes a business decision to allow the development team in on writing stories and reducing the amount of time explaining what the stories mean.

Backlog grooming sessions not a team event 

How much fun is backlog grooming for your Scrum team? Are you part of a team or scrum master of a team that goes to grooming only to agree with whatever the senior developer or team lead says? “Does everyone agree?” “Yes.” [Audible ‘TICK’] Maybe is was better when we did waterfall!?

It’s important that the first backlog grooming sessions be well facilitated so the team can be drawn in to the backlog and backlog grooming process. Going around and asking each individual what their opinion is, to explain their understanding of the story, and to describe what the high level design is will help. Some people will take to this quickly while others will take several grooming sessions to draw them out. Don’t give up but continue the facilitation until participation becomes natural for the team.

I’ve also seen where having the team lead or team manager as part of the team can stifle participation at grooming sessions. Providing coaching to the manager to encourage a ‘servant leadership’ approach is of inestimable value to creating a self-organizing team. This has worked great in the past and was the subject of a talk I gave at Scrum Australia 2016.

Being Agile When The Business Isn’t – Transparency

In the last two posts, ‘Being Agile When The Business Isn’t – Requirements‘, and ‘Being Agile When The Business Isn’t – Team‘, I talked about how to be more Agile in an environment where the business is strictly Waterfall. I’d like to continue this theme with some ideas and practices to make the teams more ‘transparent’.

Inspect, Adapt, & Transparency

Most will recognize Inspect, Adapt, & Transparency as the three pillars of Scrum but they’re ideas that even the most Waterfall of businesses can use and take advantage of. These three pillars are the core to continuous improvement, building what customers need to solve their problems, and helping bring the greatest valued work to the forefront. I’ll focus on transparency as I believe once you have transparency, inspection and adaptation will follow naturally.

Perhaps the big difference between Agile Vs traditional project management is how these three items are acted upon. When I was doing big Waterfall projects our alleged transparency came from the project’s communication plan. This communication plan defined the reports the project team was responsible for generating. At or near the end of the project, we wrote project closure reports which often included lessons learned. While writing the closure report, we were being formally transparent but only after everything was done. The real problem with the closure report was the timeliness of it. Because this is traditionally done at the end of the project, nothing learned here can help the just completed project. We also suffered from fading memories. We remembered the specific events but often needed to speculate and debate over the causes. Interestingly, we also seemed to forget these lessons for the next project or determine the next project is different enough so the lessons weren’t considered applicable. I’m not suggesting nothing was learned but any learnings carried forward tended to be the exception.

The result is some learning is done but the changes haven’t been trialed or proven to be effective in solving whatever the problem was. To improve requires openness and transparency so you can inspect what has happened and adapt to new and better ways throughout the project not just at the end. To maximize the value of improving, the project team needs to be continuously inspecting outcomes and results including any new ways of doing things. Building in transparency from the start will help.

Transparency

Transparency is arguably the first step for improvement. Nothing will help a business improve better than seeing undeniable evidence that some work practices, processes, and communications are less than optimal or just plain bad. In the Waterfall environment, the project plan includes what practices and processes the teams will use and the order they’ll use them. Most project plans also have a communication section where and how the various project statuses and metrics are reported. However, it’s only an illusion that transparency exists by executing a communications plan. The reason is, communication plans dictate specific content and intervals, usually within categorizes like any of the following:

  • Current project status
  • Current risk assessment
  • Resourcing status
  • Requirements management status
  • Stakeholder management status
  • Change management status

This doesn’t provide any insights or linkages to bigger or widespread problems. There’s also the problem that the reporting is done by the project manager. This can lead to information being filtered and potentially skewed due to conflict of interest. Project managers are generally held accountable for the project plan which means they’re accountable for perceived progress or the perceived lack  of progress. And finally, after a while, the project manager is most often simple ticking the box on his or her reporting requirements.

Daily standup & transparency

Scrum Trainer Charles Bradley wrote in his article, “The Tao of Scrum – Transparency“, “the best transparency leads to the best collaboration loops”. To maximize the value of collaborative feedback loops requires transparency and to me, transparency implies anyone can quickly gauge what’s happening in a project with a minimum of effort. Although you might be stuck with an old-fashioned communication plan, there’s an easy ceremony you can implement with your Waterfall team that gives extra clarity to what’s happening on the project: a daily standup. Each day the team talks about how their plans for the previous 24 hours went and what their plans are for the next 24 hours. It also provides an opportunity for others with their own unique perspectives to make observations and comment.

In my Waterfall past, as the project was approaching the final stages of test, we had all the project managers, line managers, and the program office people sit together daily to talk about specific tasks the teams were doing and when we thought they’d be done. Although this worked quite well for the management team, the nuance of individual planning was lost. Had there been a team standup, the confidence or uncertainty was more likely to come through. As it was, what was reported to management was an interpretation of plans and status.

In a Waterfall environment, the full project team might be large since the team is built from specialist who usually don’t take on tasks outside their specialty. This can mean the daily standup is large if everyone goes so careful monitoring is essential. The daily standup must remain valuable and true planning must occur otherwise there’s no point to it. In Agile, one valuable component of the daily standup is the collaborative nature of getting the most important work done first. In Waterfall, the project manager and product manager might need to be a little flexible in their detailed project plans to allow the collaborative nature of the implementation team to emerge. Better still is for the project plan be aligned with getting functionality done in order of importance rather than based on components and skills of the team.

Plans & transparency

Another aspect of transparency will be the visibility of the plans. If the project is using MS Project or Jira for Agile teams, it is a huge mistake to think this by itself constitutes transparency. Both these tools track progress when properly maintained but in both cases, people often don’t take the extra steps necessary to open a file or log on to Jira or even learn how to use the tool. The real problem isn’t progress wasn’t tracked, it is, but the real problem is the difficulty of getting to this tracking data and properly interpreting it. These tools have their advantages but ease of use is not always one of them. It’s not that these managers and program office people are lazy but rather their time’s at a premium. Visualizing the project and sprint status on a wall is easy. Bits of sticky paper with notes, in columns is easy. I have used a big wall as a Gantt chart or User Story board to great success. It really doesn’t matter if you’re doing Waterfall or Agile, the advantages of this high visibility is the same: huge.

If your team and stakeholders are global then it’s a wise idea to set up a camera on the planning board. I had used a camera while we did standups to allow global stakeholders to see & hear what was going on. I had a second camera on our ‘story board’ 24/7. This was intended to add more transparency and to give our global stakeholders a bigger slice of the picture, allowing for more feedback and collaboration.

Transparency is a conscious effort

Agile isn’t immune to poor transparency and communications. As Scrum Master, I’ve been cornered by the product owner/manager and asked what the ‘real’ state of progress was in the sprint. Two things needed to happen: the product owner needed to attend the daily standup and the team (with some advice from the Scrum Master) needed to better present state sprint progress. In Waterfall, the project manager has their hands full too. While Agile implies the implementation team is involved and responsible for managing expectations and communicating value completed, Waterfall usually relies on the project manager to do these. It’s up to the project manager to move beyond the project plan and establish a collaborative atmosphere where visibility and transparency become natural within the project team.

Transparency can be scary as it will expose people, their work practices, and their outcomes to all. It’s not the natural state for developers who learned their craft in a Waterfall environment. The first time I had transformed a Waterfall team into a Scrum team, one of the team members told me that they were completely stressed out by the high visibility of their work. In his case, a technical decision had proved wrong in a most open sort of way. In the past, management would probably not know that a technical glitch had happened but Agile made it visible. In hindsight, the team member realized the focus was to identify and correct a technical decision quickly and not to allocate blame or ridicule, (he was unconvinced of this in the first days). Transparency became a tool to do better and improve but not without some pain on the journey. Making previously hidden work habits visible will be stressful for some but this can be countered by avoiding the culture of blame and ridicule.

Summary

By using transparency as an aid to see what’s working well and what can be improved, the opportunities for improvement are more likely to be seen. Transparency has two key accomplices: daily standups and the wall project planner. Both these can be used in Waterfall environments without too much effort beyond simply doing it. Having the implementation team being transparent and talking directly to their plans each day is a key element of Inspection & Adaptation. This daily transparency reduces the risk of going off on a tangent as illustrated in the earlier story about an unsuccessful technical decision.

Transparency also extends beyond traditional tracking tools. Although these tools help track and document progress, it requires both effort and training to use these to their full potential. Training all stakeholders and customers with these tools is hard and may be impractical. Using a simple system of sticky notes and a big wall can convey the same information in a visual way that transcends most communication roadblocks. If there’s no wall, then use a big whiteboard. No whiteboard, then use the windows. You can even put a camera in front to broadcast the planning board across the internet if necessary.