Marcato

It’s been a while since I shared any thoughts on how Marcato is doing, and with the New Year upon us, it seems appropriate to look back on what we have accomplished so far and to look forward to goals for the coming year.

First, as Scott suggested when we started, the financial mechanics fell quickly into place (although not without some hard work from the Colestocks to make it happen) and I’ve successfully crossed the gap in cash flow that happens when you leave an employer and take on self employment.  It was good to have the guidance and confidence of others that had made that transition to help along the way and I now see that this is part of what Marcato needs to provide to all of its associates.

Business is brisk – there are a lot of businesses that are looking for help to improve the speed and value delivered by their development organizations, but we’re moving fairly carefully and working on them one at a time as we make sure we are engaging a qualified coach and building the organization with talent that matches the brand we are building in the local market.  We are in the final stages of placing a new associate with a new client and hope to be in a position to announce both soon.  We’ve also built a pipeline of both client and associate prospects and we’re working through matching them up as appropriate.

Winter Sunset - Coon Rapids, MN

We’ve also made some inroads on partnerships – we’re working with Coherent to provide offshore services to one client, and we’ve also focused part of our practice on Microsoft Team Foundation Server as a fully functional application lifecycle management (ALM) tool that supports Scrum and provides the ability to associate and track all of the elements you would need to provide a rigorous and certifiable quality process.  We’ve been successful at the same thing by combining multiple best of breed products together but I’ve never seen it all come together in so complete and comprehensive a fashion.  We’re using a local instance to manage the flow of work to and from the team in Belarus and so far I must say I’m impressed!  We’ll likely turn this experience into a deeper partnering relationship with Microsoft as we position ourselves as “the Agile process guys that also know TFS really well”.

With all of these activities going on AND a more than full time client engagement, the biggest challenges we face are balance and focus.  You can power your way through “too much to do” in bursts, but it’s just not sustainable, and since the client always comes first, whatever capacity you might have is usually drawn in by that client.  This is where focusing on your strategy is important, because you will have to make tough choices and tradeoffs and these are easier to make when you are guided by principles and a mission that help you make them. I’ve been through several years of strategic planning sessions in various organizations (including the client I’m working with today), but never one as natural as this.  In a recent bi-monthly coffee shop meeting, Scott and I grabbed a piece of scratch paper and took 15 minutes to write down the mission, objectives, and metrics that we have for Marcato:

2010 Strategy

Our mission:  To be recognized by the local market for our expertise in Agile coaching

To do this, we must believe that working with any one of our team members can make any software shop (IT or product) in town better.  That is the standard we set for all members of our team.

Our progress and results will be measured by:

• Active in presenting at Agile-related events under the Marcato brand
• Active blogging – at least one post per week, encourage other contributors
• “Go to” organization for TFS, especially with respect to Agile
• All clients are referenceable – willing to be listed on our web site or take calls from other prospective clients
• Know and engage all local competition as potential partners
• Identify size of local market and identify list of companies in the “sweet spot”
• Active and identified in all local Agile forums
• Grow total revenue over $1M and team count to six

May the New Year bring you balance and focus and with it happiness and success.

When you focused on an iterative process that will bring about “non-functional requirements” (like performance!) it is good to be as crisp as you can be in defining what success looks like.

I’ve found that it can be useful to express performance goals (or current operational characteristics) in three parts…

The first part of a performance goal is a transaction rate or volume as a function of time.  You might say something along the lines of: “On any given day, we expect message rates building up from 0 messages/sec at 7:00AM to 180 messages/sec at noon and then back down to 0 message/sec at 10:00PM, roughly in the form of a bell curve.”  Or you might say, “Every morning at 8:00AM, 5000 customer service representatives log on to the system and perform, on average, 3 inquiries per representative per minute until closing time at 5:00PM, 7 days per week.”

The second part of a performance goal should be a maximum acceptable latency, or in some cases, a fixed-length time constraint.  You might say,

• “95% of all requests must be satisfied in 20 seconds or less.”
• “All inquiries must return in less than three seconds.”
• “All messages that arrive between 9:00pm and 4:00am must be completely processed, with enough reserve capacity to process previous night’s batch in same window.”

The final part of the goal should speak to the distribution in the type of work (if it isn’t already homogenous).  You might say,

• “35% of all system interactions represent an open-ended search, 40% represent direct retrieval of a record, and the remaining interactions are new orders.”
• “10% of all interactions represent a new client, while all others represent an existing client.”
• “90% of all interactions are in the happy path, while 10% require alternate path handling.”

Once you have constrained your goals along these three axes (throughput, latency, and content distribution), you’ll find it is easier to formulate testing plans and to keep the team focused on the immediate problem at hand.  There will always be parts of your system that are sub-optimal and represent clear opportunity for tuning.  (I’ve got a system that is thrashing the garbage collector at the moment.)  But if that tuning doesn’t represent the current shortest path to achieving your goal, it needs to wait.  Don’t lose time to premature optimization.

Every new team starting Agile goes through an awkward phase where you are beginning to walk but the steps just are not natural yet.  During the first sprint, you are happy just to see the various Scrum practices put into play, but you also are building the tools and infrastructure as well.  Even if you had the luxury of having an existing Application Lifecycle Management (ALM) tool in place, you are still learning how to use it and to organize your information inside of it.  However, it is a reasonable goal for the team that you would have these mechanics in place by the end of the first two-week sprint.

In our current project, we decided that with the second sprint we would tackle getting enough information into the tool (in this case, Microsoft’s Team Foundation Server or TFS) that we would be able to see an actual burndown chart.  For this to work, you need to get the rigor in place to have the team breakdown stories into tasks (or in TFS, Product Backlog Items into Sprint Backlog Items), give the individual SBIs an estimate of total effort, and also update the time remaining on a regular basis throughout the sprint.

Of course, even if you do these things, you probably won’t get the sprint burndown chart that you see in the books (you know, the one that is a straight line going down and to the right and intersecting the x-axis on exactly the last day of the sprint).  In order for the chart to look like you want, you need to be able to foresee all the work and break it down into tasks on day Zero.  You also need to be able to keep the team focused only on work that related directly to the stories that were purchased, and you need to have all team members execute the work with the productivity they expected.  Gee, is that all…

Instead, what you are likely to see is something like this:

Burn Up!

Why is this happening?  For lots of reasons, and all of them good (in the long run), and the worst thing you can do at this stage is panic.  We did task breakdown and estimation, but not consistently across the team.  A few of the team members knew that their number one priority was going to be working on the build and deploy process, but we were slow to get a story created for that, and since it was really more of a discovery process, it really was more like a research spike that had to get finished before other work could progress in a meaningful way.

As you can see from the picture, we were mid way through the sprint when another round of task breakdown and estimation happened.  First, the sprint backlog improved as we closed out completed tasks, but then it climbed right back up again as we identified and estimated more work.

It’s still early as the sprint is not done for a couple more days, but it will take something of a miracle to get all the work that remains done in those couple of days.  Is this cause for concern?  Sure, but only in that there will be a lot to learn from the sprint and to refine during our next retrospective.  Some of the team members are new to the process and will need to see it in action again to really get it, but with this picture we now have a good example of what NOT to do, and with this kind of visibility, improvement is just around the corner.

Winter Sunset - Coon Rapids, MN

In general, I would say that you should expect it to take 3-4 sprints over 8 weeks of time before you see your velocity stabilize and your sprint burndown charts start to burn down and not up, so if you are still within that window, then gentle course correction should be enough.  However, if it has gone on longer than that, then there is a leadership issue and an intervention is needed to get the team to take this more seriously.  This is the process capability that leads to predictable outcomes, so you have to invest the time to make it work.  Remember – first measure, then improve.

As a product owner, I need the commitment between the product owner and the release manager to be documented prior to the sprint beginning.

In my previous post I described the genesis of the sprint contract idea based on a blog post I read. I also described why there was a need for such a document. This post will go over some of the mechanics of how my release manager would create such a document.

In order for this exercise to be useful, there are a few key parameters that must be considered.

Timely
In order for the contract to have any real impact on the outcome of the sprint it must be produced in advance of the sprint planning meeting. It will go through a couple of revision cycles as I negotiate priority user stories with the release manager.

Easy
The release manager spends a lot of time in our backlog of user stories, so the idea of creating a separate document at first seemed like an extra step. As the product owner, I felt that producing this artifact outside of the system used to manage the backlog provided some advantages to me.  Probably the most important advantage was that I did not need to sift through the details myself, which allowed me to easily focus on the stories at hand and quickly give feedback to drive priorities. From the release manager’s point of view, if the stories are written well in the backlog system, it should be a matter of cut and paste to create the document, followed by some minor edits.

Single Page
A key tenant of Agile is to produce just the right amount of documentation to support the software development process. An important part of the sprint contract is keeping it to one page in length. One page helps keep the content at the right level of detail, plus it makes for an easy-to-use scorecard I like to use at the sprint review to evaluate whether or not each of the commitments were met.

Stretch the Team
At the beginning of implementing this process, I was most concerned that the document would either have a far too aggressive scope. If they were unable to complete every commitment on the document, they would feel as though they had failed. As the product owner, the challenge during the editing process would be to get as much as possible in the scope of the sprint. I found the key was to go for a stretch of the scope to maybe a story or two beyond what I thought could be accomplished. In future posts, I will present some data to give an idea of how much to expand the scope. My goal was to never achieve 100% as then I knew that more could likely have been finished in the sprint.

Working Software Is The Priority
The stories in the sprint contract must represent working software to be demonstrated in the sprint review. Any stories that don’t should not be on the sprint contract document. Again, this document sets the stage for the sprint review, so other user stories (for example, we like to have process improvement stories) are not necessarily interesting for the attendees of the sprint review to discuss.

Measurable
I like to use this document as a scorecard, so the items listed on the document should be written in a way that makes it easy to assess success for each item on the document. This is important to the team so they know what the definition of success will be as the product owner follows the events of the sprint review.

In the next post, I will talk about the impact the Sprint Contract has made with my team, including the data. Hope this is useful, as it has really helped.

This series is also posted on my blog at http://www.thebigplaysblog.com.

As a Product Owner, I need the development team to deliver the functionality agreed upon during sprint planning.

My scrum team had developed the tendency to quickly defer user stories they did not complete into the next sprint.  The tool allowed them to game the burndown so that it always looked like performed well, but I was not satisfied with the amount of software presented at each sprint review.  It just didn’t add up.

I tried asking nicely, being impatient and demanding, begging and pleading among other motivational techniques from my project management past.  Still, no real change in results.  Then, I tried my good friend Google, and I ended up at this blog post from an Agile coach named Peter Stevens: http://bit.ly/85b6R.  At last!   The sprint contract concept.  It’s all about setting expectations up front in planning by focusing on what will be presented at the sprint review.  In my environment, sprint reviews include stakeholders from all over the world that are using our software, so it’s critical the “show” is a good one.

In future posts I’ll review how I adapted Mr. Stevens’ approach and what results I saw and why I think I saw what I saw.

I’ve mentioned before that we had a rude awakening regarding our test suites and test processes as we dealt with restructuring and the loss of some of our staff (see Automation and the Three Bears – Scene 3: Just Right), but I haven’t talked much about how we handled that situation or why we weathered those storms reasonably well.  To understand what we did and why it made sense, you probably also need some background on how we got there in the first place.

Regression suites can take on a life of their own.  First, many make the assumption that tests created to exercise new features are appropriate for use in the regression suite.  As the release of that feature approaches, confidence in it starts to wane due to its newness in the product. Testers find it easier to just promote every test for the feature into the suite (and maybe even write some more) instead of doing the due diligence to think about the minimum effort appropriate for a functional regression.  Second, testers are inherently conservative – it’s very easy to let more tests in, but very difficult (because of the risks they must take) to decide to remove something from the suite.  After all, when have you ever heard of a tester that was scolded for writing too many test cases?

Another source of growth for the test suite comes not from new feature development but from the reaction to issues encountered after a product has been released.  The bug gets reported, managers ask “Why didn’t our testers find this problem?”, and the natural response is “We’ll add a test case to make sure this never happens again.”  Over time this accumulates to quite a few tests, and the ironic part is that these particular bugs are actually less likely to happen again than other problems, yet you continue to test for them over and over again.

We had all of these phenomena happening, and over time we had accumulated thousands of tests cases with about 1800 or so selected as our stabilization “regression” suite.  Responsibility for the suite had passed through several hands with no one left even consciously aware of what was there.  We were stuck on top, dead center – the existing suite was too large to tackle and the individuals in the team were not clear on how they would decide what to add or remove.

When it became clear that we needed to tackle this problem, we decided to start by listing the kinds of different test suites that we had.  These included categories such as new device platform testing, device port testing, new server feature testing, server stabilization, and deployment testing among others.  From here we assigned each category of testing to a lead, and their first responsibility was to document the test strategy for that category.

Here’s an example we developed for stabilization (regression) testing:

Strategy

• We will rely on automation to identify functional issues in the web user interface
• We will use manual testing to exercise features that can‘t be automated
  • Need manual testing for features that are not exposed through UI
  • New features
    • Deeper testing around new features but looking for high level functionality and interactions between features
    • May only reuse 2-3 test cases that were created in new feature testing
  • Make sure that all teams in in all geographies can contribute
    • Physical device testing still happen in US; depends on access to carrier network
    • Use automated testing to perform functional validation of new mobile applications
• Use for device-related server stabilization
• Use first pass to identify all defects – should try to finish it before defect fixing starts
• Don’t start second pass until the defect fixing (for critical defects considered necessary for release) is done
• Goal is to avoid the need for a third full regression pass

Operating Constraints/Goals

• First pass – whole distributed team doing testing
• Second pass – some developers reassigned and fixing defects
• Target 5-10 days for each pass
• 4 weeks from code complete to release candidate

Winter Sunset - Coon Rapids, MN

With this framework in place, we were now able to use these criteria to take a pass through the test base and determine how best to accomplish these goals.  While we did this process for each type of test plan that we had, the impact was most profound for functional regression/stabilization.  With these objectives in place and a commitment born of necessity to accomplish them with automation, we not only achieved these goals but surpassed them.  As I mentioned before, now that the rules of the game and objectives were clear, our team responded very well to the challenge and compressed the full stabilization/regression cycle to under 2 weeks.  Automation made this possible, the setting the vision, aligning the team behind it, and getting the right resource on it made it real.

One of the hardest parts about getting started with any project, Agile or not, is selling the plan for the project.  This is a tough task, as you are surrounded with uncertainty – it may be a new product, new team, new tools, and you may also be faced with adopting a new process at the same time.  Yet somehow, in the midst of all of this uncertainty, you are supposed to produce a plan.

Any business decision maker has been trained to make these decisions in the framework of a cost-benefit analysis; they will project the future profit streams they expect from the product or improvement and weigh that against the up front investment they must make to get the functionality that will deliver that profit stream.  While complicated to model, especially if you are discounting the cash flows from the product in future periods, it can be done, and the end result is a projected return on investment (ROI) for the work.  In principle, you could do this for every major release – simply weigh the incremental increase in sales against the cost of developing the features and the ROI can be calculated.

Here’s an example of how this might be modeled to illustrate the point – first, notice the initial development investment which creates a cumulative hole.  Then, as the revenues and net profits start to flow, you see that we start to climb out of the hole.  This view of the lifecycle of the project also clearly shows the breakeven point at about the 18th month and the profitable ride through product maturity and decline.  Many companies compare projects using time to breakeven as the primary measure of success.

My Software Project

The picture I show here is incomplete, as time obviously continues until the product is no longer sold or supported, but if you extended the model to include this time, then you could also compute what is known as an Internal Rate or Return (or IRR).  If the cash flows are already discounted by the cost of capital, While hard to compute, the answer allows you to make a clear comparison between alternative product development investments and to choose those projects which have the largest positive IRR.

Given this framework, the job of product development is to provide estimates of the time and cost to achieve a given functional footprint.  Marketing and Sales then provide the expected sales levels, and you now have everything you need to complete the model and to make the funding decision.  You can also assess the risk inherent in various projects and plot this against the IRR; here’s an example I found just to illustrate the point:

Value vs Risk

This kind of bubble chart allow you to easily see how the entire project portfolio shapes up – note how they even used the size of the bubble to indicate the remaining development cost.  A company can choose it’s tolerance for risk and identify the group that best fits their risk/reward profile.

These are the tools by which senior management makes product development decisions, especially in larger organizations.  Every MBA student gets exposed to them at some level, especially if they study product development or entrepreneurship.  This is the kind of analysis they want to see, especially as they present to their board or investors that are providing the financing to fund the startup or next big software development endeavor.

I present this to you not because it is right, but because it is what anyone with an informal or formal business training has been taught to expect.  It’s the frame they have, it’s what they expect to see, and it confuses them when you talk about something different than this.  As a result, if you hope to map their expectations to your reality, then you need to learn both languages and how to translate.

“Let’s Get Real”

Let’s turn our attention to Agile.  If you take a look at the model in more detail (I’ve attached it so you can if you choose), you see that modeling the outcome of the project requires estimating the cost per month for development.  Since one of the principles of Agile is that you avoid big up-front design, then how are you supposed to figure out what these costs will be unless you produce a detailed work breakdown, estimate the effort on each task, and then roll it up into one big estimate again?  And how can you breakdown the work without a detailed design describing how you will build it and allowing you to carefully time when you will need specific resources to accomplish the type of work?  And of course, once you have that plan, all that remains is to execute it with efficiency and discipline, and 9 months later you have the product that you wanted at the cost that you expected and you are selling it to the market that you anticipated.

As an engineer, it’s a compelling picture free of risk – all I need to do is convince my business partners that they should fund my up front investment in understanding the problem and developing the design and schedule and we’ll be golden.  Of course, this also depends on capturing all of the requirements for the system before the design starts and in having an accurate understanding of what sales will be once the product is complete.  Of course, those aspects of the problem are not really my responsibility, so I can trust that they will be done with the same rigor which we will apply to the development portion of the overall project.

Um, yeah…right…

Let’s get real.  Projects fail because the organization that is sponsoring them loses the will to succeed.  This can happen for LOTS of reasons, but I’m going to go out on a limb and say that after many years of pursuing this dream, failure by the technical team is becoming the least of my worries.  Will there be missed ship dates?  Sure, but not by much, and if the overall plan which includes these dates is so fragile that one missed date renders the business strategy a failure, then it was simply a bad strategy.  If the “market window” for the opportunity was only a few months long and missing the delivery date by a few weeks means we missed the market window, then it doesn’t sound like you were ever really chasing a viable going concern.

If the funding runs out, if the business strategy changes, if you discover that you are building the wrong product, if the company gets sold, if you buy another company, if your sales are too low, if it takes too long to build the channels, if the product is too complex to sell through the existing channels, if the company decides it can make more money doing something else, if your sponsor leaves the company or gets promoted, if other projects gain favor and steal your support, if you don’t fund the marketing effort needed to generate demand, if you don’t commit the resources you need to document, install, and support the product…the list goes on and on.

Don’t misunderstand me – I’m sure that the official story will be that “due to schedule delays and cost overruns, the project was cancelled” but this is just because technical teams and individuals generally do not play the political game well and do not have the connections, persuasiveness, or political savvy to manage perception.  And if you find yourself in an environment where you have people waiting to pounce on your team at the first sign of weakness, then you really have to ask yourself – can we be so perfect that we can be successful?  And by successful, I don’t mean that you shipped a quality product on-time and and with the feature set that was requested – I know that you want to believe this is sufficient to be golden.  Instead, I mean the success that comes from real people using your product, recommending it to others, and coming back for more.  I mean success in the market place.

Given the sea of uncertainty that surrounds our little ship, do you really think that focusing inward on making it even more air tight is the best strategy?

Magnolia Gardens and Plantation - Charleston, SC

“or Let’s Not Play”

So, let’s turn this problem inside out.  You are the technical lead assigned to work with a product manager on a new concept that has enough political support to assemble this much of a team.  Your leaders ask for a project plan and challenge you to explain how you can estimate the costs of your project when you don’t have a detailed, upfront design and plan.

You are at a crossroads.  You can react one of two ways.  You can take the blue pill, tell them that you’ll need 4 months to collect the requirements and to produce a detailed shared fiction complete with 4′ x 6′ Gantt chart suitable for framing.  Or you can take the red pill and ask them to explain the market and sales projections that show exponential growth and sales increasing by a factor of 10 in the first year.  Ask them to show you how they can prevent all of the storms of change that might sink your ship from happening.

If you take the red pill, one of two things will happen.  They will be offended by your questions and decide to go find some other sucker that will tell them what they want to hear, or they will decide to form a partnership with you in which you all acknowledge that you are surrounded by risks and that the best way to proceed is with a working relationship in which you set aggressive goals to ship early and often, you expect and embrace change by using a process such as Agile that gives your business team the ability to react to what they learn as your collective understanding of the market matures, and you give them the transparency they need to see the real work is being accomplished and that you are steadily moving toward the minimum viable product you need.

I recently went through this process and was told that expectations had been set that it would take 12-18 months to get to the functionality that was really needed by the market.  Implicit in that statement is that no one really expected me to ship for at least a year – sweet, right?!?! This is not an opportunity – it’s a recipe for disaster.  My response:  I don’t know what is needed yet or why, but I know that our plan will be to ship in 6 months or less.  That’s the plan we took to the investors, that’s the goal we have, and in fact we currently expect to ship twice by then.

What’s the moral of this story?  If you have the ability to use processes, people, and tools to deliver software products, then you have something of value and you deserve to partner with business people that have similar abilities in developing markets, channels, and the ability to sell.  Even if your product is for internal consumption, you still need business people that can build the buy-in needed to exploit your product for maximum benefit and to create an internal business success.  If you offer them a transparent, capable process that reacts to their changing needs, then you are in fact offering them the best they can get from anyone and you are a worthy partner that should expect the same from them.  This isn’t about making life easier for engineering or avoiding accountability – it is instead about creating successful products in an uncertain environment.

It’s time for us to accept that Agile is the best we can do given these realities and to turn our attention from avoiding past failures to creating new successes.

Afterword

The title of this essay – “Let’s Get Real or Let’s Not Play” – comes from a book by sales guru Mahan Khalsa; early in my career while a consultant at Microsoft, I had a chance to hear him speak and to study his book.  While I have never really aspired to learn how to sell, my recollection of his message is as clear and as simple as the title – you can let others lead you to invest a great deal of time and money pursuing their business through blind sales processes, or you can avoid the wasted expense by focusing instead on those fewer customers that are willing to directly engage with you and to help you understand what the really need.  His argument is that the latter is a more successful sales strategy.

Lately, I’ve become fascinated with the power inherent in instrumenting just about anything you repeat within your development cycle.

Now…your current development lifecycle tools might provide the opportunity to capture and report on some data in an automated way.  But there are probably more touch points than those tools can cover – and therefore more opportunities to observe/improve. 

Think about the “Goal/Question/Metric Approach”, spotlighted in the forward to “Agile Adoption Patterns”:

• Develop goals
• Generate questions (that will make your goals quantifiable)
• Specify measures
• Develop mechanisms to collect data
• Collect/validate/analyze data in real time for feedback
• Analyze data post-mortem to assess conformance to goals

How many places can you apply this?  You might decide to measure:

• How long does a continuous integration build take to complete, including unit test execution?  (Speed of a CI build affects team rhythm quite a bit.)
• How long does it take to deploy to a given physical environment using automation?  (Again, this is a team rhythm factor.)
• How long does a database restore take at full production data volume?  (This will affect your disaster recovery planning.)
• How long does it take for your app server to respond to first request after a cold start?  (Recycles should be cheap…)
• How long does it take to reach the key business milestones within your automated performance/regression cycle?
• How large are the payloads served by your web tier?
• How long do your various web services take to execute?

There are countless things you might want to measure – but here is the thing: Most of the time people won’t – often because they don’t understand how useful the history and trend line will be.  They either don’t realize the value in continually “getting well/staying well” with a particular metric, or they believe they are engaged in a short-term (single pass) optimization exercise.  (But most things tend towards entropy…)

So I would advise creating a discipline of “pause and instrument” – allow your team the time required to instrument a process, to create a historical repository, and to radiate (publish) the trend lines in whatever way makes sense.  You will quickly find powerful feedback loops as people absorb and respond to the new information streams.

Some thoughts on technical implementation, just to get your own creative juices flowing:

• Have someone on your team become an expert with LogParser.  Really.  You can use it to gather structured data from a variety of sources, including from your unstructured log files already emitted by your app.
• Your instrumentation can start off as just simple CSV (comma separated value) files…nothing says low-tech-start like “echo foo,bar >> DataPoints.csv”.
• On Windows, check out the built-in “TypePerf” tool – it will generate CSV files for any standard performance counter based on a polling interval. 
• You can easily bring CSV files into a database when you have a table definition that matches your “CSV schema” – just use BCP (for Sql Server) or the appropriate bulk import utility for your database.  Centrally available long-term history is a very good thing.
• To report, use Excel to create pivot tables directly on CSV files, or on data in your database tables.  Don’t normalize your data – let data repeat so your pivots are drop-dead easy.  Think about stating time in a couple different horizons (hour, day of week, etc.) to make it easy to slice the data in Excel without having to get fancy.

As Brian H pointed out in a recent comment, one of the more difficult challenges for introducing or operating scrum is getting team members wholly committed to one team.  Rather than comment back, I though I would dedicate more time to the topic – this is an important one to get right.

My experience has been that effective, high functioning teams only form when team membership is stable and team members are dedicated to one team.  While the transition is hard to accomplish, the steps are relatively straightforward:

• Reorganize to separate reactive, maintenance work from feature development
• Create balanced feature teams that receive and process work assignments from a variety of projects

I cover the first step in detail in another post (see Customer Response Team), but I realized there is more to say about the second step.  We’ve all had experience with matrix organizations – the notion here is that you have project managers that draw together a collection of people that have an HR report to a functional manager.  The functional manager is responsible for making sure that everyone is fully allocated and is usually responsible for creating and sustaining the process definition, where the project manager is responsible for the outcome of a particular project and accomplishing it with the resources assigned to them.

The matrix system works, but it doesn’t serve the needs of software development very well.  First, it tends to group people by function, and since the manager of your function performs your review, it tends to emphasize role over project outcomes.  These dividing lines become entrenched, and functional managers end up overly involved in business of the team as they seek to expand their own influence and protect their team members.  There are, simply put, too many cooks in the kitchen.

Water Tower - Harmony, MN

What happens if you turn this inside out – make the cross functional feature team the primary unit of organization?  Role specialization still exists, but the team is self-directing in figuring out the best way to get the assigned work done.  Functional managers still exist, but only to advocate and evangelize the spread of good practices throughout the organization.  They still generate reviews, but the reviews are based largely on peer review from fellow team members and on the outcome of their projects.  Project managers still exist, but they are embedded in the team as scrummaster and they continue to represent the state of the project to the outside world, but they share accountability for the outcome equally with everyone on the team.

Projects are broken down into pieces (features) that can be assigned to an already constituted team.  Teams are expected to have enough diversity in skills that they can be assigned any kind of work yet still be capable of accomplishing the goal.  Work can be reassigned to a different team if progress is insufficient, but you don’t change team membership between iterations.

Of course, this only works well by having the buy-in of the manager responsible for how the teams are organized – in most organizations it is the VP of Product Development or IT that would have sufficient purview to create the structure.  As I mentioned earlier, that person also has to shunt work that requires immediate attention away to be handled by another team and process – you can’t mix dramatically different time tolerances together and get an acceptable result.

I have seen the end result of not doing this too many times – people with multiple project assignments shuffle weekly or daily between status meetings reporting that they have made little progress on their assignments for this team because they are too busy with other priorities.  Everybody looks busy, but macro level output suggests otherwise.  This is definitely a “smell” – a clear indication that something is rotten in the state of Denmark.  It’s unsatisfying for you as an individual because you feel that you have let one of your teams down, and it’s inefficient for the organization as the more cycles that are spent context switching and updating, the less that is actually being done.

I’ve seen this in my own consulting work as well – it’s difficult to manage the transitions and overlaps between multiple clients because you can have all the best intentions in the world yet still find it difficult to always be there when your client needs you.  This is what it is like in a high functioning team – everyone chips in, everyone works really hard, and everyone shares in the success knowing that they carried their fair share of the load.

What to do about it?  I think each individual needs to be accountable for their career and this decision.  You can’t blame your leaders for asking if you have the capacity to work on more things (OK, you can – they need to knock it off, too), but only you can say “Let me finish this first, and then I’ll start working on that”.  You may think this will put you in jeopardy with your leaders, but I can tell you the opposite is true.  I would much rather have someone that regularly completes what they start than someone that takes on too much and never finishes anything.  I don’t like excuses.

I think people are confused on this point – at an individual level, they think that success will come from showing a long list of projects as accomplishments, but a savvy leader will see right through that and recognize that being a “chicken” in many projects means being a “pig” in none. “Chickens” flap their wings and make a lot of noise; “pigs” ship stuff.  As for me, I’ll take bacon any day.

Mmm, bacon…