Whenever me and my teams are facing a period of turmoil, I like to motivate the troops with the following quote attributed to Army Chief of Staff General Eric Shinseki:
“If you don’t like change, you’re going to like irrelevance even less.”
Always seems to set the proper mood 
I have now needed to “introduce” myself to a few different organizations over the years – each time I do I pull out the same presentation that I gave years ago. What is surprising to me is that although some of the details change, I find my values and philosophies formed from early leadership experience and training dating back fifteen years now still hold true today. I’m sure there are many paths to success, so this is by no means a formula for others to follow – it’s more important that you be true to yourself.
My Values:
My Philosophies:
My Mission:
My Role:
I don’t think I’ve had this much pressure to choose and choose carefully since naming my kids so many years ago, and then we had months to consider the options and make the decision. Both of my partners had to make fast decisions and simply invented names for their small businesses off of the tops of their heads and both actual drew from their families for inspiration. Now that we are trying to more carefully choose something that conveys the brand image we want to create, this is a LOT more difficult. Of course, you also have to figure out if the domain names and trademarks are available – no wonder everyone just started going with madeup words.
Just in case any of you find yourselves inspired with a suggestion, there is still time to contribute. Here’s a short sample from our business plan that captures what we intend to do as well as a list of words that might get the creative juices flowing. Your reward? A free subscription to my blog for life
Excerpt from NewCo business plan:
“NewCo is a management consultant practice founded in the belief that we can help clients succeed at creating an Agile development organization, especially when applied to an offshore team. Our target clients are small to mid-market software organizations with technical teams ranging in size from 5 to 25 people and annual budgets ranging from $1 to $10 million. These teams will range from internal IT organizations for a much larger organization to product development shops responsible for commercial on-premise or SAAS solutions. Our clients are either trying Agile and/or offshore for the first time or they may have tried it before but with mixed or disappointing results. It’s also possible that they have a current project underway that is off track and they are looking for leadership to “fix” it – this is a more difficult situation in which to be successful as the expectations for feature set and schedule are already set and the environment is likely already politically charged, but given the right support and flexibility to make changes to these expectations, we would consider this as well.
NewCo is assembled with experienced engineers that have all shipped multiple successful projects using Agile development techniques and with practical experience doing so with remote teams. As a result, we expect to have an impact on our client that goes far beyond providing training or advice – we expect to make a lasting change to their operation, team structure, cost, and effectiveness that continues on long after the engagement ends. We do this by working with the senior management team, director level software managers, and prospective team leads to build internal champions that will sustain the use of Agile indefinitely. We will not compromise in this respect for the sake of growth – each team member must have a proven track record prior to joining. We intend to build and maintain a premiere brand based on all team members contributing to the state of the art and speaking or publishing regularly to support that brand.
In addition to our management consulting practice, NewCo also will provide access to offshore development through outsource arrangements in which NewCo can provide the technical team leadership but the resources would come from a third party offshore outsourcing firm. NewCo will foster relationships with these partners that are complementary – our goal is to work with the domestic organization to enable effective remote development while their goal is to plug their offshore resources into such an organization. At a minimum, there is definite potential for lead-sharing in these relationships, but beyond this we also expect that some clients will want us to also own and operate the offshore team in the form of fixed bid projects. In these situations, we will maintain control over the choice of partner.
<snip>NewCo has access to the offshore developers that built the reference application and can establish an offshore development capability for performing the customization based on the statement of work generated by NewCo. NewCo will use agile methodologies to run these projects, providing a virtuous cycle that will feed the development and improvement of our agile/offshore practice with real world experience providing solutions based on the methods that we are advocating.
In summary, our mission is to improve the speed, predictability, and quality of software development so that our clients can either deliver more or reduce cost. We use the appropriate combination of Agile, automated testing, and offshoring to accomplish these goals, and we expect to create a self sufficient team capable of continuing these methods without outside help.”
Also, here’s a list of words that are appealing to us (at least some of us):
craft
rhythm
metronome
meter
rhythmic
cadence
beat
pulse
tempo
flow
stream
balance
clockwork
fulcrum
Thoughts? A little effort and inspiration and you could be part of making history…
My second chance was actually as a management advocate rather than as a technician. During my first early contact with Highjump I was introduced to their test lead and the conversation turned to the fact that they had purchased an automation tool and trained on it but had made little progress in actually putting it to work. The organization had the intent but the people involved had not figured out how to allocate the resources to make it work.
When I later came on board as a director and had a chance to work with the team for a while, two things became evident – they were under-investing in testing and the person responsible for advancing automation still had made no progress. In fact, they had regressed – the licenses they purchased earlier had not been used and senior management now had the belief that automation was a waste of money.
What followed was a process I shall politely refer to as MM as we sought to justify the expense of investing in tools again, with the deadlock broken by me recruiting new team members with experience using automation. With these hires we were able to get the OK to renew the licenses we had and to restart our foray into automation. We started with a team-wide effort designed to show the broad interest and capability we had throughout the team and to drive the selection of the tool suite we would use going forward. The effort culminated in a presentation to the senior management team demonstrating what had been accomplished. I’ve scrubbed the presentation and provide it here (Automation – Blog) if you are interested in the specifics – the acronyms represent different applications that we sold at the time.
In this presentation you will see that we refer to three levels of maturity for automated scripts – I’ll repeat them here for illustration:
While at the time we were referring to the maturity of the scripts themselves, I realize now that this is also a model for how each tester matures in their understanding of automation. There are other approaches to testing being commercialized (such as model-based testing), but for the genre of technologies that I’ll simply refer to as “script based testing”, I’m now convinced that there is a best way to solve the problem: Data driven scripts are the best method for producing robust, agile automation.
Through this effort, one thing became clear – of the cross section of people in the team, some people “got it” and others did not. All were interested in learning, but it was easy to see that those that had already demonstrated interest in automation through past commitments were going to be successful, and those that had been manual testers were likely to remain that way. This brings me to my fourth rule of automation: Good automated testers are born, not made. They have to be driven by the desire to avoid manual testing (which is common), but also by the need to solve the problem not just because they don’t want to do it anymore but because it is the right way to do it (not common). They move quickly through the maturity levels mentioned above and they ultimately develop data driven frameworks that fit a fairly consistent pattern. I’m not saying it is impossible to convert an experienced manual tester to an effective automated tester, but I have yet to see it happen.
Back to our story. We were eventually successful at getting the funding to purchase a new tool in the next budget cycle. Our team chose the Compuware suite, but don’t consider this an endorsement. The list is more than 5 years old, and I think all the competitors in the space are good enough to do right by you. Our product line divided between applications and the platform on which these applications ran – we decided to focus our efforts on automating our flagship application to completion as this would give us the most leverage.
I want to dwell for a minute on the importance of choosing your projects correctly. We chose this product because it had a large feature set that rarely changed but it was also under constant active development and was releasing 3-4 times per year. This ensured that we would see the long-term benefit of many release cycles using the automation. My point is – don’t dabble at the fringes of your product line because it is easy and safe there – automate the most strategic part of your product line to ensure that you have the highest possible impact on your business.
How does this story end? At the time I would have considered it successful, but now I know differently. What we did worked, but we failed to transform our processes and our business around it. We didn’t apply it broadly enough, nor did we make the tougher decisions that would have really committed us to it; it affected the product where it was applied, but overall our testing methodology, staffing, and schedules were still dominated by manual testing. As a caveat, I need to point out that my story ended three years ago and the state of the art today could be much different, but I think that Goldilocks would agree with me that at the time, our commitment was “too small”.
Alright, I admit the title is a gimick and probably only confused you, but if I had written something like “Why Automation Is Crucial to Agile” you would probably have breezed right by it. Instead, you thought – now what the heck could that be about? Well, let me explain…
I’ve had three very different experiences using traditional “Automation” tools for provide functional regression – the first was too big, the second too soft, but the third was just right – get it? I wanted to tell all three stories and compare and contrast what worked, what didn’t and why I am so THRILLED with my most recent experience. Here goes.
Scene 1: Too Big
My first real chance to see Automation in use at serious scale was at Great Plains (which had just been purchased by Microsoft). I should probably start by defining what I mean, because I had certainly seen test harnesses built before this that were designed to help unit test a chunk of code, and I had worked with a couple of testers that had used desktop products to push buttons on thick client Windows applications, but this was the first time I had seen a system built around it.
Great Plains had taken the approach of creating a tool of their own that could execute a suite of Rational Robot test scripts. They had built a centralized lab of servers that they could use to run these tests on a variety of different operating systems (virtualization was still new at this stage so these were physical servers). This side of their investment was great; in fact, it went well beyond the resources I have ever seen committed to automation.
BUT … (you know that was coming, didn’t you) … I was quite surprised to learn that the end state of this investment was an accumulation of automated scripts so large (and unorganized) that the effort required to find and fix all of the broken scripts that resulted from changing any element of the user interface far outweighed the development cost of making the change. The net effect was that the test team was often telling the business that they could NOT change the product.
Consider for a moment what this means – when the goal is to create agility through iteration to ENABLE change, they had instead created efficiency in the execution of the testing itself. They could repeat the testing of the product quite frequently as long as it didn’t change, but then, who cares? This leads us to the first rule of automation – the goal of automation is to allow the product to change with confidence, not just to make the testing process more efficient and cost effective. The latter is a valid business benefit but with automation, bigger is not better.
Now that you know what happened, you may be interested in why. This happened as a direct result of who had built these tests. You see, the organization had for some time been hiring interns and fresh college grads to come in as lab technicians – their basic responsibilities were executing the test suite and collecting and reporting the results. Over time they learned how the product worked but most could not debug a script to determine if the failure was with the script or the product.
Despite the large investment in equipment and tools, this organization had under-invested in the people. A software automation test suite must be architected correctly to be agile, and this rarely means using “record and playback” to automate a use case. In fact, let’s make this my second rule of automation: “Record and playback” is not your friend and should only be used sparingly. A well engineered suite will be data driven, using common core routines to accomplish common tasks. It will port well across machines, environments, operating systems, and versions of the product – in other words, it will be robust rather than brittle. And if built correctly, it will allow one software test engineer to keep up with the changes made by a team of 10 developers.
Let me take a moment to describe an example of exactly what I mean – let’s say you are tasked with building automation to stop and start a Windows service. Since the only hammer you have is “record and playback”, you decide to automate the process of going to My Computer->Manage, expanding the “Services and Applications”, choosing the “Services” node, choosing the service from the list, and clicking the stop and start service buttons. Sweet! Five minutes of recording and I am done with my task. I run it again on the machine where I recorded it and it works like a charm, so I’m done!
But then other people start to use my script and start reporting problems. On other machines it takes longer to shut down, so the start service button is available when the script tries to press it. You add a delay or write some code that waits for the button to become available before pressing it. Somebody else tries to run your script on Windows Server 2000 where your recorded it on Windows XP, and the names of the nodes in the Computer Management console are different, so now you add code to detect the OS and have conditional logic that controls what to look for in the treeview. But it still doesn’t work, because there are different services running on that machine, so clicking on the twentieth item in the list stops and starts the wrong service.
After several iterations of completely rewriting the code that you recorded, it no longer resembles your original recording, and you certainly can’t start over with a new recording as it would take longer to recreate your added logic. The final conclusion you reach? It wasn’t even worth recording it in the first place EXCEPT to learn the names of the objects you need to click and what NOT to do.
And then, after all of this work, some annoying and smug software engineer asks you “Why didn’t you just do a ‘net stop w3svc’ and ‘net start w3svc’ from the console?” Where the heck were you when I needed you two weeks ago? It turns out that not only does this work, but it works across different machines, operating systems, etc. and it works every time. You see, for every task you want to accomplish, if your knowledge of automation is limited to record and playback then you will almost always find a way to create a complex yet brittle way to solve any problem. You have a hammer, so everything looks like a nail. An experienced software engineer knows that a hammer is often not the right tool when you are looking for finesse.
The insight I gained while working with this test team is that it takes and talented engineer to create such a product, and it should be viewed as a software product. Microsoft distinguishes between two different roles in their test organization – “Software Design Engineer/Test” and “Software Tester”. You need to former to build your automation suite and the latter so assist her or him in repeating regression runs or defining new datasets that will drive additional test cases. It’s also easy to fall into the trap of trying to use a business analyst with strong domain and product knowledge to build the automation – avoid this as well. Your BA should help define the important use cases to verify and maybe even help construct the input data that drives the scenario and the expected outputs, but don’t have them code the automation suite.
Great Plains was proud of their investment in automation (as they should be) but they demonstrated this commitment by building A LOT of scripts. In this case, more was not better. As Goldilocks would say, this automation was “too big”.
I was doing some more rooting around for tidbits from my past that might be interesting and I stumbled on the following document. The context for this is a running narrative of the state of our product development organization over what turned out to be a nearly 18 month period of time. It started with a request from my boss at the time to produce a “state of the state” after roughly 3 months with the organization, and as he came back to ask me to update my assessment, it actually turned into a running commentary on how we had evolved, the progress we had made, and how the issues had changed as our product organization matured. I’ve gone through and removed just a few of the specifics as this was originally intended for internal consumption only but the document is almost completely intact. The first section is my original assessment, followed by three more snapshots done 1 month, 6 months, and 15 months later.
In hindsight, I think the original assessment is pretty close to the state I would expect to find in most product development shops that have been shipping commercial product with some measure of success. The suggestions I made at the time became a prescription for how to correct some of the imbalances that existed and to get more from an already reasonably capable process. I can say that this pre-dates the popular terminology for “Agile” practices, but in it you will see a process mindset that sought to balance the different dependent steps in the process and increase overall throughput from the system. You will also notice that the initial assessment is inwardly focused on processes contained within the product development department, but over time I found that to improve the system as a whole I had to change the expectations of other departments as well. This is always more difficult, but figuring out how to do this through negotiation with other leaders in the organization is the hallmark of “executive material”.
This also reminded me of another process worthy of an essay of it’s own – the “Macro Planning” process. While useful to help us through the transition to Agile, I’m not sure I would recommend it as a long term program management process, but it served its purpose at the time.
Enjoy!
————————————————————–
Product Development “State of the State”
January 9, Year 1
Updated February 12, Year 1
Updated June 19, Year 1
Updated April 21, Year 2
Introduction
Please treat this assessment as confidential to the Product Development department.
I would like to preface this discussion by noting that my intent in this document is to focus on issues and resolutions rather than to identify strengths. It will be easy for you to feel that my conclusion of PD operations is negative overall, but that is not the case. What I have found is that we have a team of bright, highly engaged, and hard working team members. The processes within PD seem well defined, well understood, and consistently practiced. In general, I view the current operations as largely successful, and I think you will see that my proposed remedies represent mostly subtle shifts in role definitions, expectations, and priorities.
Although much of this analysis is based on first hand knowledge, there are some observations and conclusions that are simply subjective hunches. My goal in this report is to document my opinions as of the above date, but these may continue to change as I collect more information. Many of the observations turn out to be manifestations of the same root problem, but that becomes clear in the analysis.
I will be updating this document on a semi-regular basis to include any new observations that I would add and to update on progress we are making to address the issues mentioned. I will add this information in a separate “Update” section organized by date at the end of the document.
Initial Observations
Too much work in process
100% utilization of all resources does not equate to maximum throughput – in fact it is often less than optimum. Whenever you have dependent process steps (in our case, design, coding, test, and release), you have to balance the flow or work through the system or you have work in process piling up in front of process bottlenecks. I believe QA is the most visible bottleneck today, and I also think that we have been minimizing the appearance of this pileup by constantly redefining release content in an attempt to minimize the impact on test. The net effect is that we seem to miss opportunities to ship the products. This also means a lot of wasted planning effort trying to optimize the utilization of upstream resources when this does not actually mean that more product ships.
Insufficient emphasis on test staffing
Test automation could be used to provide more effective regressions
Test team not technical enough to initiate use of automation
My impression is that we have been consciously limiting QA staffing to control the scope of its efforts and to cap the salary expenditure. I believe that this is harming us in several ways:
“Big bang” releases causing more harm than good
Our collection of modules too complicated to plan for module wide convergence. There are too many variables and constraints to account for to ensure that they all converge to a point where we can ship the whole product line on a given day. Perhaps this was simply a weakness in planning, but my first observation was that we should also pursue a technical solution to this problem. Long term we need to use our own EAI (Enterprise Application Integration) technology to solve this problem, (and we eventually did – Ed.) but in the short term we need to figure out how to artificially force and support the separation of these releases. The net result has been that nothing is fixed in time or feature set and downstream functions like training, sales, etc are forced to react to PD.
Ineffective prioritization scheme
We have the entire infrastructure for communicating inter-module and intra-module priorities figured out but don’t seem to really be making the hard decisions. We still have two #1, two #2, and two #3 priorities and have a hard time saying “no” regardless of the priority level. This has been a Catch-22; the lack of a high level planning function prevents you from knowing when you are resource bound and need to make a decision. The lack of a clear priority decision makes it very difficult to do resource planning.
Lack of emphasis on releases and dates in favor of “project” orientation
One of the more difficult problems I had getting started was mapping the list of active projects to a release schedule that defined the name of the release, the expected date, and the expected scope. I soon realized this was a systematic issue – PD resources were tackling their projects without an upfront plan for how they would ship. As a result, when the end game approached, there was no obvious effort to stabilize the product so that it could ship and it appears that sometimes the QA process would stall out as resources were switched out to other priority projects. This behavior is a lot less likely when the release has a name, date, and expected scope defined.
Immature product management function
Technical leads need program management mentoring and support
In order for product development to function in a time-bound manner, we need to be able to pair the technical leads with strong product managers that can make tough scope decisions during crunch time. I have already gained a great deal of respect for the team and the work that has been done to organize and prioritize projects. However, we need to work on aligning their team behind the public priorities and on developing confidence in the individual product managers to make most of the decisions within the confines of their module. The MRD (Market Requirements Document – Ed.) process is the right solution, but we need to concentrate more on the business justification for the work so that the team can appreciate when investment tolerances are exceeded.
The same situation exists within product development. Many of our technical leads are inexperienced at or reluctant to accept the program management planning and supervision responsibilities and may find themselves over-committed on their technical contribution to the development effort. We will need to reinforce the importance of these functions and extend their planning to cut across functional boundaries.
Inefficient expectation of handoff between Development and Test (Test as QA)
My experience has been that if you set the expectation that no bugs should be found by “QA”, then the end result is to extend the elapsed calendar time required to complete the product. This approach also makes the preparation of automation impractical. Instead, if you design the schedule to involve test in the design phase with builds delivered to test on a regular basis throughout the implementation cycle, then not only are problems found sooner and outcomes more predictable, but I also believe that total elapsed calendar time is reduced.
Configuration management tools and processes do not currently support concurrent/parallel release development
We currently depend on keeping files checked out for long periods of time because we do not have a configuration management and build process that can ensure that these check-ins do not break builds for other teams. This tends to delay the integration process and increase schedule risk during the final stages of the project. It also means that the configuration management team does not get a chance to stabilize the build or to discover and resolve installation issues. As mentioned before, we need to release the products on staggered schedules and this will almost certainly force our hand in this regard.
Need better isolation between new product development and continuation engineering in order to improve predictability and accountability
I think we should consider dedicating resources to own the whole lifecycle for dealing with product issues that escalate to Product Development; this includes problem evaluation, defect fixing, build, testing, and deployment. New product resources could be used to review changes if necessary but should not have any other responsibility.
New Observations and Update (February 12, Year 1 - 1 month after original assessment)
New Observations – There are a few additional observations I would like to make regarding PD and its interaction with the rest of the organization.
We need to elevate expectations of Tech Support
This is related to the earlier continuation engineering observation, but the solution is different. I think that the standard for escalation to PD should be that Tech Support has contained it to a reproducible behavior that can not be fixed by modification to the customers’ configuration or process objects. Only issues in the Platform or base applications should escalate to PD. This will almost certainly require the involvement of Operations staff to determine if the problem was the result of their customization or due to a flaw in the base product. Far too many calls are escalating to PD without appropriate triage.
Disrupting plans to react to short term opportunity or to unplanned internal needs has a cost
I do not expect the behavior to change, and most of the time I understand how these changes are in the best interest of the company. Simply put, the issue is this: when I am told that a particular deliverable is truly important to the company, I will believe it and I will make sure that the team understands the importance of what they are doing. If we later decide, to adjust that priority in favor of something else before the team has had the chance to complete their delivery, then the message they receive is that it was not really as important as they were told, and after this happens a few times, they can rightfully conclude that it never is really that important. This undermines our credibility as leaders, and can eventually limit our effectiveness.
Technical lead role needs clearer emphasis on “program management” responsibilities
This is really an extension of my earlier point, but I am finding that the current technical leads are surprised by my emphasis on their program management performance over the other responsibilities in their role. I believe this may be a change from previous priorities. I will be emphasizing basic leadership/supervision skills in their professional development plans.
Should de-emphasize role of detailed task tracking in favor of more visible milestones
I find that I do not use our project tracking solution to track product status like has been done in the past. I think that there used to be more emphasis on this as the only visibility and predictability afforded to PD management. I do not want to see effort wasted in frequently updating these schedules unless there is clear merit in doing so. Please don’t interpret this as a recommendation to avoid planning – we should still produce plans to the level of detail that we do today, but allow teams and individuals more latitude in changing course as they work through the plan without the overhead of updating it every time this occurs.
Update – Below are updates to the original observations I made – there are a number of organizational and planning process changes that have occurred that I believe have had or will have an impact on our operation and I have made note of them where appropriate.
Too much work in process
We were bottlenecked at QA (as evidenced by a significant amount of work in process or completed but unable to be tested). We have since added 2 additional testers (one started in Feb, the second will start in March) and we have flattened the org structure so that Betty Boop (names are changed to protect the innocent – Ed.) could be assigned as test lead to one of the products. We have also changed our approach on planning to keep developers involved in the project through completion of the release rather than assuming that they can begin work on the next project. They are expected to perform testing activities when not involved in fixing defects. This avoids the perception that there is more capacity than is really there.
Insufficient emphasis on test staffing
See above – this is being resolved.
Test automation could be used to provide more effective regressions
Test team not technical enough to initiate use of automation
I have produced a justification and solution concept for the use of automation. One of the recent hires has automation experience with Rational Robot for UI regression testing. We intend to approach this in several steps, with the first and most significant milestone the creation of a set of objects that will allow us to automatically regression test all services without needing to manually run our applications.
“Big bang” releases causing more harm than good
We are well underway in transitioning from a “big bang” release to modular releases. There is still a lot of uncertainty within the team about the payback now that we are really beginning to see the implications, but once we have revised our configuration management, installation, and release processes, I remain convinced that the change will prove worthwhile, if not necessary. We have already seen situations where a change will need to occur for a single project, and if all other product releases had been contingent on that project finishing, then they would all have been held up. In addition, we seem to be smoothing out the test workload.
Ineffective prioritization scheme
Mickey Mouse’s efforts to extend our macro planning horizon through the end of the fiscal year have helped bring stability to the nearer term priorities. We are still reacting to the increased priority on one product feature set, and it is now obvious that it takes us 2-3 months to turn the ship unless we can summon the courage to cancel active efforts in favor of new priorities. I was pleased to see that we were able to reduce our priorities to a clear list with only one module at each priority level and that we have started to make some tough decisions about what we are not going to do in the coming year.
Lack of emphasis on releases and dates in favor of “project” orientation
This has probably been the most profound change – the macro plan has us focused entirely on releases instead of projects and everything we do is timeboxed to contain feature set.
Immature product management function
SAs need program management mentoring and support
The working relationship between product management and PD is stabilizing and the macro plan is truly the joint accomplishment of both teams. There are still occasional situations where we second guess each other in our roles, but this seems to occur less frequently. I will continue to emphasize.
Inefficient expectation of handoff between Development and Test (Test as QA)
For several current projects we have changed this expectation; testers are getting involved earlier for longer periods of time (although there are fewer testers assigned) and we will be trying to keep the same testers working on a product for multiple releases. We are encouraging earlier deliveries of incomplete product to test so that they can deliver feedback about form and function much earlier in the process. I think this has already helped us avoid a late flurry of ECNs (Engineering Change Notice – Ed.) to revise a product that was supposedly “code complete”. This change has been difficult due to some pretty strong conditioning that any defect found by test is cause for embarrassment and concern by QA that the product is not stable. It is still true that this should be the expectation at code complete, but not before.
Configuration management tools and processes do not currently support concurrent/parallel release development
We have started to use branching in VSS (Visual SourceSafe) to allow us to have parallel release efforts in progress (including multiple versions of our database schema running in parallel). This stretches the limits of VSS, and I remain convinced that someday we will need to purchase a solution that provides more sophisticated management of concurrent build branches. Finally, I don’t think we have made much progress towards a fully automated build process, but the frequency of builds is increasing due to the several concurrent projects that are active and I’m sure the CM (Configuration Management) team will soon realize that they need to fix this. We need to get to the point where we could build nightly if we chose.
Need better isolation between new product development and continuation engineering to improve predictability and accountability
No real improvements have occurred here since I have been holding off on formalizing a “continuation engineering” function until Daffy Duck has finished his efforts to modularize the product line. I expect to have the group in place and functioning effectively by the end of the calendar year. The mission of the team will be to act as a self contained group capable of the evaluation, fix, build, test, and delivery of hotfixes to the field to react to high priority customer situations. One measure of success would be to have their customers unaware that they are not working with the actual product development team.
New Observations and Update (June 19, Year 1 – 6 months after initial observation)
New PD product releases are becoming irrelevant due to increasing gap between what we are selling and the functionality that is actually present in the base application suite
I was disappointed to see how little interest there was around finishing two new products and I’m concerned about the inevitable impact on commitment and morale if people perceive that their work is not valued by others in the company. I believe that an active and more influential product management function will change this as we work with them to get to a leading position with respect to sales and marketing. We need a fundamental change in behavior from selling what we don’t have to selling what has been released in the base application or we risk overdriving the system with revenue sold that can’t be realized.
Future macro plans need to account for a lighter workload in the summer months
Our overall capacity is easily down at least 10% in June, July, and August due to vacations and it is also not safe to count on the usual amount of overtime effort. End of July/August releases should be light in functionality.
More productive toolset needed/need to end of life old toolsets
We need to invest in obsoleting old tools (we currently require 4 different compiler environments to build our products) as well as taking the technology/learning curve risks around basing newer releases on more productive tools such as .NET. It will be increasingly difficult to remain competitive with more modern codebases if we do not do this.
Technical leads and developers need to engage in making testing more effective
The test team needs help getting over the hump in building automated testing into each project team’s plans. We need to get the strong technical resources thinking about how to do this efficiently; I’m convinced our current test team can execute on what we devise, but in general they seem to struggle with taking the initiative to force the change. Cleopatra and Gertrude are exceptions to this rule, and I have also seen some promising ideas from Daisy Duck.
Agile methods need a truly representative pilot project
We risk giving “agile” methods a black eye by representing our current efforts as “agile” when what we really are doing is just undisciplined. I am worried that we will develop a negative opinion towards agile methods without really having tried them. Having said this, I do see this as a gradual transition occurring over many months, so I still think we can get there. Our odds of success would increase dramatically with a trained coach in a focused project.
Professional services team is trying to move and grow too fast
I don’t have the facts to substantiate my claim, but my suspicion is that the call volumes after a transition to Tech Support are too high and that this is due to us rushing this transition. With the increased visibility and pressure on margin in professional services as well as the dramatic growth that has and will continue to occur, I am concerned that both PD and Tech Support will be forced to clean up the mess; PD will have an ever increasing number of disruptive events due to poor planning for implementations and Tech Support will see call volumes that increase faster than the new customer rate due to poor testing and rushed transitions.
Update – Below are updates to the original observations I made – there are a number of organizational and planning process changes that have occurred that I believe have had or will have an impact on our operation and I have made note of them where appropriate.
We need to elevate expectations of Tech Support
This problem is even more urgent now due to a large number of transition/go live events planned over the next 3-4 months. The change in role for Goofy has helped prevent improper escalations, but they still occur. I now believe that there are two fundamental problems preventing us from attracting or developing the skill set that is needed in the tech support team: leadership and business model. The team needs a leader that can be deeply involved in teaching everyone there how the product works and how to diagnose problems. The current emphasis seems to be around improving the “systems” to fix the problem, but I think they have more than adequate structure to succeed. What they need are better trained, more competent staff. My concern regarding the business model is that current margin expectations will not allow that time to focus the time, headcount, and $$$ needed to address the skill gap.
Disrupting plans to react to short term opportunity or to unplanned internal needs has a cost
No change has occurred.
SA role needs clearer emphasis on “program management” responsibilities
We are making some headway in this area – the SAs all have training objectives around supervision, delegation, and project leadership and some have started taking courses. I have also seen that hands-on mentoring in the context of two products has helped mature Jack and Jill in this respect.
Should de-emphasize role of detailed task tracking in favor of more visible milestones
This has been happening, although I have occasionally needed to clarify or re-communicate our expectations around our project tracking tool given that detailed task tracking is not a “requirement”. People have interpreted this as “I don’t have to use project tracking”, but we still depend on it as a basic document repository and workflow engine.
Too much work in process
We worked our way through the series of Spring releases that represented the “bubble” in the system and we have nearly worked off the project backlog by releasing most of that content. With the release of Release 6.0, we should be all but caught up and we should then be able to determine if our current dev/test ratio is at a reasonable steady state level.
Insufficient emphasis on test staffing
I’m pleased with our progress in this area; the only concern I have is that we still may not have sufficient test resource on the Pluto team, but given that they are still working off backlog, I can’t yet tell if this will remain an issue. What is clear is that poor testing on Pluto will be our #1 source of future interruptions.
Test automation could be used to provide more effective regressions
Test team not technical enough to initiate use of automation
The team has been able to show some progress in this area; they have all learned enough about the Robot tool suite to be interested in it and to start actively applying it to current efforts. I am disappointed that we have not made any headway in being able to test the platform runtime services with a automated, reference application.
“Big bang” releases causing more harm than good
This change is finally starting to have the expected impact on the rest of the organization. As long as we maintain that the primary customer of new releases is professional services for new implementations, the modular release approach should continue to work and we can continue to enjoy its benefits in PD. I am starting to hear some pushback about releases happening too often and I think this
Ineffective prioritization scheme
I’m very pleased with the progress in this area – product management has been steadily improving in capacity and Tom, Mickey, Harry, and I have sorted out the rules of the game for changing the plan.
Lack of emphasis on releases and dates in favor of “project” orientation
I think this transition is complete – in fact, we are now guilty or assuming that everything we do will release. The only risk in this assumption is the overhead to the organization of adding a release to the macro plan, especially regarding test and documentation. For the most part, this is only a timing difference, so I’m not particularly concerned.
Immature product management function
SAs need program management mentoring and support
The only concern in this area is some recent issues around technical leads taking “too much” initiative in order to get the result they want. I am actually pleased to be hearing this because it is a welcome change to hearing complaints about lack of access to sign-off resources preventing progress in PD. Until product management has stabilized in headcount and role, we will need to continue to be creative in getting past obstacles.
Inefficient expectation of handoff between Development and Test (Test as QA)
Recent releases have effectively blurred this line, and feedback to date has stressed the benefits of early test involvement, so I think we have completed this change in expectations.
Configuration management tools and processes do not currently support concurrent/parallel release development
There has been little change on this due to Daffy’s departure – I expect an improvement now that the team is staffed up again.
Need better isolation between new product development and continuation engineering to improve predictability and accountability
We have isolated the test responsibilities for CRDs (Customer Request Documents – customer-specific features)/hotfixes to a single individual and that has streamlined the process. The number of upcoming transition events that are planned will almost certainly spill into PD, and my strategy for this will be to sacrifice Scrooge and the Earth project if this happens. I would rather miss one date by a long way than to miss all of them by a little.
New Observations (April 21, Year 2 – 15 months after initial observation)
Internal
There are a lot of things that I don’t have to think or worry about any more, suggesting that they have become fully integrated into the department and company.
The best example – modular product releases. We have completed our sweep of the products and they are all modularized. The last product release cycle occurred with almost no effort on my part (and I was the release coordinator). The next release will be more complex (more products involved) and there will be more for the coordinator to do, but Jack will cover that role.
I am rarely involved in managing customer support issues, and Mickey and Tom are even less so. Hotfixes are performed and shipped without my knowledge. Build processes are running smoothly (albeit with minor grumbling about response times). Both D and Jamie have already achieved independence in the management of their teams and require only occasional guidance.
Unified Product Development (UPD - the subject of an article all on its own) and the Steering Committee are self-sustaining and require only a few hours a week of my time. I believe that the Steering Committee has created the most functional director-director working relationships that we have.
Modular releases are here to stay
Beyond the expected benefits of smoother release cycles, I am also seeing that modular releases allow us to appropriately neglect modules that are not worthy of investment. Although there is the occasional question about specific release content or compatibility, I believe it is safe to say that the method has been successfully adopted and will sustain until we decide to change it.
The macro plan is less urgent but more important than ever
I attribute the loss of urgency around the macro plan to my belief that both Mickey and Tom have gained sufficient confidence in my alignment with their interests that they spend less time worrying about the allocation and shepherding of resources. Tom is now more engaged than ever in external efforts and I have no sense for his comfort level with our current activities. Finally, I no longer think that the macro plan is part of encouraging larger investment in PD – I now believe only “off the balance sheet” investment from NewCo will make that happen.
There are a couple of reasons that it is more important than ever:
The department reorganization is having the expected benefits
Segmenting between application, strategic, and support/maintenance has accomplished several things. We have two managers in training that are showing a lot of talent for the role. The isolation between new product development and maintenance is where we need it. UPD is serving its intended function of standing in the gap and narrowing the difference between what we have and what we sell.
I have noticed that I have to be more careful about involving my leadership team in priority and resource decisions, but the effect is certainly tolerable. I am learning to work through them to get priorities communicated and to ensure that objectives are clearly set with their team members.
We are dependent (perhaps overly) on test resources to release product
Now that we have institutionalized the early involvement of a test resource and changed expectations around the role that they play, I am having difficulty covering the increasing number of concurrent efforts in progress. It’s evident that UPD needs at least one full time test resource to match the current productivity rate (likely to be closer to two if it returns to full staffing), plus it appears we will have sustained investment in four other products. We may see more time from Lily as Jack assumes the CM role, but that is difficult to count on.
I believe that the optimal number of tests has more to do with the number of concurrent efforts in progress than ratio of developers to testers, but I also question if we are getting better functionality faster because of the investment in test. I am curious what would happen if we staffed the next effort (especially an app team – I’m not willing to compromise on Platform) without a tester assigned to see what happens.
The prospect of additional PD investment is a blessing and a curse
There are short term distractions as we build the technical and business cases for the additional investment. I am doing nothing that would count on these things happening, but I am definitely tempted, as several of them are large and I am afraid we are underestimating how long it will take to effectively ramp for any one of them, or worse yet, more than one. If one of them cracks and we need to quickly hydrate a team, then I think we need to be looking for a way to add multiple developers in one “acquisition” rather than growing the team one hire at a time.
CM and QA have not matured as hoped
More progress has happened since Billy Bob’s departure than occurred before – it was healthy to get people to stop thinking of it as somebody else’s problem. I believe that we will see evidence of real impact by end of June.
I remain interested in distributing the installation responsibility across the development teams. There are a couple of possibilities here – we can cross train installation skills across the whole technical services team and then spread the responsibility for particular projects across those people, or we can place the responsibility in the hands of the technical lead. Installation remains an afterthought and contributes to the chaos at release time, especially for strategic efforts.
With the advent of nightly builds, I will have the “smoking gun” I need to get the fundamental behavior change in QA. I believe we have been providing sufficient access to tools and training to prepare people but have lacked the compelling reason for change. I expect this to be institutionalized by end of year.
Offshore development has mixed results
I would continue to invest, and even increase the investment level, but I know that we are continually frustrated over working out the details of making this work. There is a risk that if we do not carefully monitor the people that are accepting the work from Transylvania, then we will end up negating any benefit by having them rewrite significant portions of the work. My feeling is that the work product needs to stand on its own – we should be doing light acceptance testing and pushing defects back for repair. We now believe it easier to do the high-level requirements and design here and to communicate detailed design to them for completion, so we are experimenting with that approach.
Pluto releases need to be tightened up
I have been frustrated by a number of things in the last release cycle. Scope and resource changes have been frequent, reflecting the lack of a passionate advocate for release (as well as a compelling feature set, perhaps). The separation of tool and runtime will help with this – we need to push this through ASAP. I have been surprised that there have not been more challenges around our use of resources in this area. We will always have to overcompensate by producing artificial intensity as these releases will rarely be customer/market driven.
External
Ala “The Goal”, PD is not the bottleneck in converting ideas to recognizable revenue.
We are finally seeing most of the effort required to drive absorption of domain and product skills throughout sales, services, and support. I am convinced that this is our biggest issue in getting to revenue.
The product silos are doing more harm than good
We need to encourage Sales, Product Strategy, and Operations to blend these businesses as the distinctions are becoming increasingly artificial. Current customers are suffering from ineffective support in professional services and customer support because accountability boundaries do not match skill set boundaries any more. We have people approaching customer support and PD for help in making implementation decisions when there are others within professional services better prepared (but not incented) to provide support.
We need a longer lasting commitment to a clearer product strategy
Less is more – I think we are at risk due to indecision and under investment. We are fighting a many front war and having difficulty making satisfactory progress on any of the fronts. The reason we can not make these decisions is because we lack a factual basis for making them; as a result, politics and personalities determine the ideas which receive sufficient momentum to get resources and yet there is no objective evaluation of the success of these efforts as clear goals were never established. Often the only thing measured and accounted for is the effort expended in PD related to the resulting feature set (because it is easy to count). We respond reactively to opportunities in the pipeline (which change to frequently to provide the consistency needed to mount an all-out attack on a market). I am not sure that we need a new overall strategy (the breadth of our product vision is not the issue) – just clearer direction on how we work towards realizing that vision.
If you have ever had the horror of seeing me dance, then you can appreciate that rhythm is NOT a natural talent of mine. In fact, I even had an alternative working title for this essay – “So You Think You CAN’T Dance” Nevertheless, I find that most of my energy seems to go into helping a team find its rhythm and to match it to the cadence that makes sense for the business. While I’ll concede that there is no single right answer for what that rhythm should be, I do believe that all organizations benefit from establishing a rhythm and investing time and energy in maintaining it. In fact, let’s split that statement into two parts and deal with each separately.
All Software Development Organizations Benefit From Establishing a Rhythm
Rhythm helps at many levels, but probably none more than at the company level as you try to unravel the answer to the great mystery – when will we have the next release? Time and again I have seen organizations deadlock on answering this question because of the inevitable circular dependency. When will it be? That depends on what is in it. What’s in it? That depends on when it will be. Organizational angst builds as you try to figure out how to break this deadlock and everyone waits to see what will happen that finally forces a decision.
In fact, the force behind this is so powerful that I’ve started applying another physical law to this phenomena: “nature abhors a vacuum”. If the development leadership does not take active control of the situation, someone or something else will and the result will be less than desireable. The most common thing that happens is that the sales team will just start making external commitments on their own; customers will only wait so long to get these questions answered, and you can’t really blame them or the sales team for doing what is needed to keep the business moving. But the implication of this is that people unqualified to know the extent of current commitments and the effort required to deliver on new commitments are making cost and date decisions. Not good.
It seems to me that there are really only two options for breaking the deadlock as there are only two victims from which to choose: you can fix the scope or you can fix the date. I’ve tried both over the years. Fixing the scope leads to two strong reactions; first, everyone panics and tries to make sure that their pet feature or project is part of that commitment (because they have no idea when they will next get the chance – remember, schedule is floating), and second, the team assumes that a lot of detailed planning needs to occur before the date commitment can be made and communicated. See the previous paragraph for the expected result – not good.
The answer is probably now obvious – fix the date. In fact, don’t just fix the next date, fix the one after that too. Here’s where the rhythm arises – every situation has a natural rhythm that fits the business and customers best; find that rhythm and commit to it. And yes, you can ship too often. It’s your customers that tell you this and they will tell you that you need to back off if this is the case. You can also ship to rarely; my experience in 5 different organizations is that anything longer than a 4 to 5 month ship cycle goes beyond the patience of ANY company or customer base. The fastest I have seen is weekly, but this was a company living on the raw adrenaline of “internet speed”.
Within that ship cycle exists the need for another rhythm – the iteration of the team as it builds features. Since the ship schedule is set but feature set floats, the first thing that the business needs to know is how often they get to steer the feature set. In general, more often is better, especially when the needs of the business are driven by the pursuit of sales opportunities versus a more deliberate product strategy. The other factor relates to the development team itself and the group dynamics of the feature teams. I have now had 3 different teams start with the assumption that their iteration time could be anywhere from 2 to 6 weeks, and while all started with 4, ALL self selected to reduce it to 2 weeks.
Having observed this process many times, I think there is actually a natural law at work here. The tradeoff is between the amount of work that can get done so that the team has made meaningful progress and the intensity felt within the team because of the pace set by the length of the iteration. There is also overhead to ending one iteration and starting another and even a very efficient team will lose hours to a day in doing this, so times less than 2 weeks are hard to sustain. If you look at how the time gets used within the iteration, it generally takes a couple of days to get through the iteration planning and design. This leaves about 6 days for real work, a day for final integration testing and preparation for the iteration review, and then you start all over again. If you make the iteration longer, it seems like the two things happen – first, the extra time just leads to more planning and design without really expanding the time to build the feature, and second, larger granularity stories are allowed into the iteration because capacity estimates allow. These longer stories are inherently more risky to implement and probably should have been broken down into a more digestible size. Finally, the whole team seems to like the pace – deadlines are real and immediate, there is some pressure but not too much, and the teams generally finish the stories that they buy. My final recommendation – start at 2 weeks and only increase it if you find this unsatisfying.
It Takes Time and Energy To Maintain A Rhythm
This is both counter-intuitive and disappointing, but definitely true. If a rhythm is natural, then why doesn’t every system or process seem to find one? The answer is easy – rhythm is not natural, but chaos is. The role of the leaders and managers is to create structure where it would not naturally exist, but once you do you have to care for it as well. The good news is that it takes MUCH less energy to sustain than it does to create, but that doesn’t mean it can be neglected.
I have been reminded time and again how quickly the habits can disappear. If there is ANY reason to doubt that the next iteration or release won’t follow the established pattern, then these rumors seem to spread like wild fire and all the old, bad habits around competing for attention now that the structure is gone seem to immediately erupt. As a result, there needs to always be someone that fulfills the basic project management goals of scheduling the estimation meetings, iteration reviews, and buying meetings needed to sustain the regular flow of work into and through the feature teams. Once your rhythm is established, nothing is more threatening than “starving the beast” and not being ready to start because priorities have not been determined and stories have not been estimated.
Don’t confuse this with the job of serving as scrum master for a particular feature team – this is instead a process-level responsibility that would usually be held by the VP or Director responsible for the orderly flow of work through the teams. In my experience, it takes about 10 hours over the 2 week period of time to sustain it – not a huge commitment, but at times it can be challenging to make sure that these things ALWAYS happen no matter what else is going on. It doesn’t matter if you have a budget or reviews do – the process comes first.
As a final note, while I realize that there are other systems that can be used to organize the flow of dependent work streams (such as kanban), these systems don’t address the other natural issues that arise for managing expectations of a software development organization. As a result, I’m firmly convinced that time-bound iteration is the best compromise out there. If you disagree, I’d love to here your thoughts.
I’m not sure there is such a thing as a “business” geek, but if there is, I was one of them. No kidding – in junior high I started checking out books on how to start your own business so I could learn about it, and of course I vowed that if I wasn’t a superstar CEO by the time I was 25, then I was clearly a failure. Oh well – I turn 43 on Saturday…
Here I am 30 years later, and I think it is finally time to begin. Last night I met for dinner with two old friends with the sole purpose of talking about what we wanted to be when we grew up and whether now was the time to finally do so. I realized this morning that I had already started down a path worth chronicling on its own – the genesis of a new business from conception to reality. I have long admired the work of Joel Spolsky, and one of the highlights of his work is that he draws from his experiences in building and shipping FogBugz at Fog Creek Software as context for communicating his ideas and as insurance that they are grounded in practical reality. I hope to do the same.
My partners to be – Scott Colestock and Adam Henry – are both past colleagues that made my short list of people whom I hoped to work with again. Although we have all taken different paths through our career, we share a common passion for agile development and we have spent recent years applying Agile practice in a variety of organizations. The conversation at JJ’s focused on a simple question – would a company combining our interest and talents be more than the some of the parts?
From there it ranged far and wide – how would we get scale and get leverage out of a simple consultancy model, especially since none of us are interested in having our immediate organization grow beyond a dozen or so carefully selected and highly skilled professionals? Could we build a “lifestyle” business that is rewarding, a client base that benefits from our experience, and most importantly, a brand around a proven methodology for introducing Agile in an organization and applying it successfully to managing local and offshore development? Do we stick with a pure consultancy, or like Joel, should we also run an agile offshore practice as our crucible for testing new ideas? Would this enhance the strategy or distract? Would it dilute the business or provide important diversification to improve our odds of long term survival? Only 1 in 10 new organizations still exist after 5 years, and those that do are rarely successful solely on the strength of one original idea.
There are some many little decisions to make in the midst of these larger strategy questions – what do we call ourselves? How do we organize? How do we share common costs and rewards while still encouraging individual effort to get and stay billable? How do we market ourselves? How do we build a pipeline and track the conversion into real engagements?
I can hardly wait until our 5 year anniversary when I can stand in front of the team and tell the story of how Adam, Scott, and spent an evening at JJ’s deciding to start a company, and I can hardly wait to see what it will become!
So, here it begins, and in addition to writing about all of these events, I also hope to also start capturing pictures of our humble origins. It’s only been 24 hours and it has already been a wild ride.
