Customer Response Team
I was asked at breakfast this morning how I would handle a situation where resources were not committed to my feature teams because they have ongoing responsibilities for supporting and maintaining other products. My answer? I would probably choose not to play. Simply put, if there is ANY prerequisite to establishing rhythm in the market driven, planned release arena, it is to start by isolating those resources from the other less predictable (and likewise more difficult to control) sources of interruption. The good news is that once this is done and once the system or product is stabilized and control is established over the more reactive sources of work, these too can be corralled into something that resembles a predictable system.
Here’s a view of the model workflow as work for the product development team originates from the many sources that exist in a typical software company. The names of the departments/individuals that source the work may be different in an IT shop situation, but the important part is that you sort out these sources into three different categories based on the time horizon in which a response must be generated.
This kind of work is clearly the most time-critical; in a SAAS shop, this is “production down” for either some or all of your customers and for on-premise products this is a “customer down” that may involve some combination of immediate phone support combined with travel to the site to resolve the issue. Because of the severity of the situation, everyone needs to stop and react immediately. For this to be effective, someone needs to have the responsibility of triaging the situation and determine who can help. If the process for this to happen is not known, then everyone stops, and because many situations can be solved quickly with minimal effort, people get conditioned to ignore these as false alarms. You need to have clear accountability for the triage and resolution or the result is ineffective communication, long resolution times, and disappointed customers. With time and a clear focus on root cause analysis leading to robust solutions to the event, you can bring this type of work to near zero. Reaction times expected are measured in minutes to hours and product delivery of a hotfix is ASAP.
All healthy companies have customers! Some are customers you want to keep and others are new customers you want to close. In either case, the support or sales teams in your organization will be a constant source of requests that tie directly to the well being of a current or prospective customer. Time lines will be tight, specific commitments will need to be made that are inevitably outside of your current release plans, and without a method for handling this work through a dedicated team, you will instead find your product roadmap and release plans constantly hijacked by customer driven priorities. There are times when it is appropriate to interrupt your planned activities and to expand the resource pool to include more of the team (I’ve done this twice in the last seven years), but when this happens, I have always timeboxed the activity to a specific collection of issues and been very carefully in managing the stopping and starting of the usual rhythm. Reaction times expected are measured in days to weeks and a monthly product delivery in the form of a maintenance release is usually acceptable.
The final and most strategic category of work for your team are the new features on your product roadmap. If your company has a clear product vision and active product management driving it, then there will be no shortage of work in this area, and very frustrated product owners if you do not figure out how to allocate enough resources to satisfy a reasonable part of the need and provide a predictable outcome. Promises will be made here too, but these promises will be in the form of marketing campaigns and other commitments to your largest customers to future features on that roadmap. Reaction times expected are measured in months – I find the most effective rhythm for this type of work is a product release every 4-6 months.
Now that you understand the categories and expectations in each area (as well as the ship vehicle and rhythm that makes sense), we can focus on how you organize in order to deal with these differences. As you can see, I’m proposing two distinct teams to handle this work. There is overlap for customer driven work, but the primary responsibility for reacting to the request, prioritizing the work, and delivering the solution lies with the Customer Reponse team (or CRT). The reason I show the overlap is that there are times when it makes sense to have the work done initially by a feature team in your main branch, but the responsibility for backporting to your released branch and shipping the product stays with CRT. These situations require agreement from the product owner to buy the story in the current iteration and a timeline that allows for everyone to wait until the start of the next iteration for the work to happen, but as long as these conditions are true then it is not particularly disruptive to the rhythm of the feature teams – after all, we are Agile, right?
Let’s dwell on that for a moment – if we are Agile, then why do we need different processes for dealing with this work? Agility still requires that we contain the periods of change to the transitions between iterations – within the iteration itself, the individual features teams need the latitude to focus on their assigned stories without interruption. “Agility” is a relative term – if your goal is to ship every 4-6 months, then changing your priorities every 2 weeks is “agile”. If you are responsible for maintenance releases that ship monthly, then you are probably changing your priorities weekly. And if you are are “production down”, then all of your priorities change immediately.
The Customer Response team has a difficult job, but one of the best ways to bring order to the chaos is through a prioritized list. Given that CRT serves multiple constituents, there is a tendency to have multiple lists with one for each internal customer group. However, unless you have put a significant number of people on the team, you may not have the luxury of working on three “number one” priorities at the same time. Instead, our process is to meet weekly with all three constituencies involved and to merge the separate priorities into one list we call “The Dirty Dozen”. The merging of the priorities is done publicly with everyone involved, and we take 30 minutes a week to review the list and adjust as needed for changes or new issues.
The net effect of operating CRT is that the challenges of managing releases on multiple code branches and of performing forward and back port efforts requires configuration management expertise. You want to look for someone that has this knowledge, has the leadership presence to deal with multiple internal constituents, and who is calm under fire. Our current CRT lead has a couple of favorite sayings – “Go slower to go fast” (he’s a NASCAR fan) and “Act quickly, think slowly”. Both capture the fact that reaction time is important, but this team often operates without a safety net. You need experienced, careful engineers in the role that understand how to accomplish a goal in the most surgical way and yet can live with the pressure of knowing that their changes will likely be deployed into production on their blessing. As our automated regression capability has improved, we have begun to rely more heavily on automation to help us confirm that a fix has had no unexpected impacts, but until you get to that level of capability, you are completely dependent on the skills and clarity of thinking in the engineers that do the work, so choose them carefully.
What proportion of the team needs to be dedicated to CRT? I have seen it as high as 25% of my resources in the past, but with improvements in product quality and process capability, it can get lower than 10%. If you are using offshore resources, you usually can not depend on them for the immediate response work, so CRT tends to be an expensive team because it is usually all local resources. There are certainly exceptions to this, and it is possible to use offshore resources to fulfill the Customer Driven work.
One last note – if you are curious about how this fits into the organization chart for the the development team, check out: http://craigkn.wordpress.com/2009/09/10/scaling-the-agile-development-organization/