Marcato

I don’t have anything particularly profound to add to the theory or practice of refactoring, but I do have a few observations worth throwing out there for consideration.

First, I do think there are two kinds of engineers in the world – bottom-up (detailed) and top-down (conceptual) thinkers.  For the most part, early developers were bottom-up thinkers.  Top-down thinkers were not willing to master the magical incantations required  to make a computer do what you want when all you have are command lines and assembly language.  After all, they had big ideas and all those trivial engineering details only confused the picture.

But over time measured in decades, something magical occurred.  We learned from our mistakes and started inventing languages that were inherently easier to use and opened up the world of programming to many more people.  Computing power grew to the point that we could afford to use managed runtimes, portability improved, and IDE’s became very powerful.  Like it or not, Visual Basic was the hallmark of this new generation, and while we probably all agree that VB enabled people that shouldn’t program to think that they could, it also liberated many more top-down thinkers to begin to create solutions that started with the most obvious manifestation of the product – its user interface.

I believe that the many early attempts at mastering system development through up front design were invented by bottom-up thinkers.  To them, this is simply the most natural way to operate – you begin by building libraries and low level classes as you construct a hierarchy of inheritance and abstraction.  While not every bottom-up thinker has this issue, I have heard this process described as “building an edifice to the gods”.  Weeks and months go by with little demonstrable product available and the business starts to get frustrated wondering if engineering has lost touch.  In many cases, they have – significant amounts of effort go into solving the most generalized form of the problem because they still don’t really understand exactly how it will be used.  Architecture without purpose does not have intrinsic value – - it’s a beautiful work of art in a dark room.

Minneopa State Park - Mankato, MN

Compare this to how Agile would tell you to behave – build demonstrable features with the simplest possible design needed to deliver them.  Have confidence in your ability to rewrite and restructure as needed to fulfill the new requirement.  Introduce architecture when it is evident that it has business value – either through enabling scalable deployment models or to support the mass customization of your solutions in the market.  Build the solution from the top down and create the classes you need when it is clear you need them.  Introduce base classes and inheritance when it is obvious that the alternative is to have a significant amount of common code in multiple places.  Build interfaces when it is clear you need them to support specific extensibility goals or to enable good testing.

Thus my first conclusion – Agile is a methodology for top-down thinkers.  As a top down thinker, I like both that it allows me to engineer in a way that is natural and that it supports running projects in a way that leads to shipping products and business success.

Second, I wanted to mention my recent experiences building a mobile application using Agile technical practices.  Mobile apps can be a little challenging for doing Test Driven Development as the tools are less mature and it requires more effort on your part to implement mocking strategies to enable automated unit tests.  One approach is to try to use a JUnit-derivative such as JMUnit or it’s BlackBerry derivative called BUnit, but both of these require running the unit tests on either an actual device or a simulator.  As a result, the coupling of these tests to continuous integration (CI) tools is, shall we say, less than straightforward.  JUnit itself will not deploy and run on a BlackBerry as it depends on classes that do not exist on the J2ME platform.  An alternative (and the one we chose) is to build mocks for device services such as communication services and storage and to use these mocks from JUnit in a traditional J2SE project.  With this approach you also get the nice JUnit integration into Eclipse (or your IDE of choice) and easy integration to CI tools.  Yes, it takes extra effort and creativity, but it’s worth the result.  Abstracting your underlying services into interfaces on a mobile platform is a good idea anyway as you may find that you need to have carrier or phone model specific implementations of these services – your mock implementation becomes just another available plugin for your architecture.

This experience has really solidified my commitment to TDD and refactoring – I’m very happy with the state of the code produced and the speed with which we can react to changes and add functionality.  There have been a few instances where it made sense to consolidate functionality into base classes, but we waited until duplication of code made the need evident.  I’ve also gained a better appreciation of how inheritance and overrides tend to maintain a good balance between reuse of a third party API while still allowing you to extend the classes to get functionality different than intended by the designers.  Managed OO languages like Java and C# sure make this easy – I rarely feel like the language or tools are making it harder than needed to accomplish what we want.

I’m also happy with the presence of these unit tests as documentation for how the classes should be used – this is a contract engagement and the responsibility for supporting and extending this codebase will fall to others after I’m gone, so it feels good to leave behind good examples for how the objects should be used.  The classes that could be generalized enough to unit test are also the classes most likely to be re-used in other Java based mobile platforms.

One final note – I’ve been obsessing about how to best meet the goal of building with no warnings because there are instances where I have methods required to comply with other interfaces that are not used or mock classes that are not used in a normal production build, yet the presence of these unused methods throws warnings during the compile.  I know that the rules say I should rip these out, but somehow that just doesn’t seem like the right thing to do.  In some cases, I have settled for commenting them out for now knowing that makes it easier to put them back if I need them later – and that exact thing has already happened a couple of times.

A comment posted against my recent post on Agile Testing reminded me that I had ignored an important topic usually grouped with other testing activities – performance testing.  Our team at Gearworks (now Xora) made significant progress in this area, so I thought I would take some time to explain what we did and how it worked.  The proof, as they say, is in the pudding – by the end of our efforts we had improved the scalability of the product by at least a factor of 5 as well as effectively eliminated the most common sources of downtime in production.  We also succeeded at bridging the great divide between product operations and product development and earned the trust and confidence of some fairly jaded members of that team.

My perspective on performance is that it is a feature of the product just like any other piece of business functionality.  I have seen teams ignore it altogether at their peril, treat it as a constraint (“the product must be performant” – oh, really; thanks for making that clear), or worse yet relegate it to an already overburdened QA team.  We all know that doing performance testing at the end of a development effort is risky – the product architecture is committed, time is short, and only the most compelling evidence will prevent deployment of the product – and yet this is how it is most often handled.

Banning State Park - Sandstone, MN

I’m going to go out on a limb and state that the long term scalability of any complex business system is the database.  We’ve had to tackle both CPU and I/O limitations, but regardless of the type of performance bottleneck, the source of the problem is always the application itself.  A good DBA can do wonders in making sure that everything is indexed properly, but once you move beyond the obvious mistakes that can be made in configuring the database itself, the burden for finding further improvement really lies with the development team.  I have a confession to make:  at one point in my career I fell completely out of love with object-oriented programming as it was clearly the root of all database abuse.  Since then I have seen that the proper (if not expert) use of persistence layers such as XHibernate can alleviate much of that pain.  I’m willing to let OO back into the tent IF you also commit be becoming a guru in a persistence framework.  And no, don’t write your own or I’m putting you right back in the penalty box.

Enough on that front – my purpose for this post was not to get into the technical particulars of performance tuning (this is a black art in and of itself) but rather to step back one level from that and talk about how you organize and what process you follow in order to ensure that performance gets fair consideration.

When we tackled performance as a feature, we started by retaining the best DBA we could find with expertise dedicated to our database platform of choice.  We created a feature team dedicated to performance and assigned another engineer to work with him.  The first goal was to make the measurement of performance as cheap as possible, so we put effort into building out an environment where performance of a build could be characterized.  We also worked to ensure that we could rollback the environment into a known starting state (we actually say disk and table fragmentation affecting the performance profile in a way that was uncharacteristic of our production environment).  Finally, we put time into extracting metrics from product logs that profiled the important figures of merit, especially frequency and duration of database calls by operation performed.

Once we had this basic infrastructure in place we could now do several things:

• We could analyze our production logs and characterize the load profile and compare it to the analysis of our performance run – this helped eliminate the need to try to theoretically model the load profile.  Instead, we simply measured both used that to determine if our load test was over- or under-representing some aspect of system usage
• We couple repeat the tests as often as needed.  The goal was to run them at least once per 2-week iteration.  By comparing the profile to the previous run we could quickly identify when some change to the code or schema had introduced a new problem area.
• We could compare releases and use this information to assist in making the ship-ready decision
• We could use the results to determine where our next effort should be focused, which queries needed to be tuned, or even which parts of the system needed to be re-architected to expand overall throughput.

Now, if you are going to convince the powers that be in the need for this kind of investment, you need a business case.  Our’s was fairly simple – we ran a hosted, multi-tenant system in which there were both penalties for down time and clear costs to increase the production capacity of our system.  These costs came from either expanding the number of servers or increasing capacity in our database SAN.  At one point we were faced with a quote for $80,000 to rebuild our SAN on new hardware in order to ensure we had enough capacity to handle a drive failure in our SAN.  Our only alternative was to decrease the SAN I/O to the point where we had capacity to spare even during a drive rebuild.  With this kind of motivation we were successful in figuring out the latter and were able to avoid the expenditure.  In fact, over a 12 month period of time we increased the system’s capacity five-fold through progressive and occasionally dramatic improvements in the application’s use of the database and the database’s use of the SAN – as a result, although our number of users increased by 25% in that period of time we not only avoided additional capital expenditures but also managed to reduce our operating cost.  In fact, our goal was to continue this trend for the foreseeable future – we expected to be able to continue to consolidate services, reduce servers, and reduce overall operating costs while doubling load in the coming year or two.

Such compelling cases do depend on your business situation, but the point is clear – treat performance as a feature and organize a cross-functional team to tackle performance and you can make significant strides toward improved stability and scalability.  You do not have to spend a lot of money on duplicate hardware (we didn’t) or expensive load-generation tools (we used open source or home-grown tools), but you do have to invest the time to make performance testing pervasive and free.  If you do, then you will discover yet another example of how “quality is free”.

One of the Agile practices is that all bugs should be fixed within the iteration that they are produced.  The reasons are probably obvious – you don’t want to accumulate a lot of product and schedule risk by building an unwieldy pile of issues.  Worse yet, when you do decide to ship you find your ability to do so severely limited.  You frantically work to stabilize the product but find that your attempts to fix bugs are sometimes creating new ones.  It’s a frustrating time for everyone involved.  Better to avoid this and simply keep things clean as you go.

So, now that the scales have fallen from your eyes and you see the error of your ways, how do you make it right?

Most developers understand that the unstated social contract is that “you fix your own bugs”.  For that reason, few will argue is a bug recently introduced by them is assigned to them to be fixed.  Larger system issues that seem to span features or subsystems can be a little harder, but usually you can get it down to one or two people that will feel some accountability for the problem.  Start by declaring that from this day forward, new bugs will be fixed as they are introduced, and enforce it by making it clear that features are not done until all of the bugs related to that new feature are complete.  The teams might grumble, but the testers will thank you and the developers will fall in line.

BUT…be prepared for what happens when you realize that you also need to tackle the bug backlog.  Come on, admit it – you have one.  It might even have hundreds of low to medium bugs in it, the detritus of countless triage sessions in which you realized that fixing this bug was not worth the energy or risk, so the collective decision was to push them to the “next release”.  When the next release comes, you realize that the priority of the issue was low then and nothing has changed, so in all likelihood it is even less important now that they get fixed.  This is the slippery slide which sustains mediocre products in the marketplace.

However, now that you are living the Agile lifestyle, you have decided this is not good enough.  You are going to dig yourself out of that hole.  First, be practical.  You can’t fool or force your team into investing its energy in a pile of relatively worthless problems.  If you have issues that are over a year old and that have been pushed out for 3 or 4 releases, you may want to admit that you really don’t intend to fix it.  The best way to get started on this problem is to do a frank housekeeping sweep and clean out all the stale issues.

Waterfall - Mankato, MN

Now that you have a smaller set of more actionable items, what do you do?  If you are already working against story points and measuring team velocities, then one helpful technique is to actually assign these defects story points.  We tried this out and found that as long as a team is getting new “credit” for the fact that they are fixing old bugs introduced by someone long gone, the work becomes tolerable.  With each buying session for each iteration, you can now set aside a portion of the team’s capacity (say 10-20% of their estimated story points) to buy defects.  Don’t expect your product owner to be happy about this arrangement – they were likely part of the triage that created the pile in the first place.  Instead, the commitment to make this happen needs to come from the Director of VP level in the technical team.

Product owners are forced to balance a number of opposing forces – market demands, competitive moves, sales tactics to close near term business, etc – with all of these things to consider, the thing most likely to get short shrift is product quality.  You can’t even trust your customer to make that decision – I learned this valuable lesson years ago.

At the time, we had a relatively new machine in the market and a customer that was evaluating that machine with the intention of buying 40 more had run into an issue during the evaluation.  Our team was working into the evening to fix the problem and we were fortunate enough to find a solution.  The developer that was working on it had tested it, and it was my call to decide what to do.  I decided to contact the customer, let them know we thought we had a fix but that it would be another day before we would complete our testing.  I knew they were under some pressure to get the evaluation done right away, so I decided to offer them the chance to “beta test” our fix while our testing continued.  They of course said yes, so we bundled up the fix and sent it to them.

The next morning I got an irate call from the customer.  ”Why did we send them a fix that only introduced new problems?” they asked.  I replied, “But I thought you understood that we had not finished our testing and that there might be problems with it.”

That’s when it hit me – no matter what I did to set the expectations of my customer, THEY DON”T ACTUALLY EXPECT THERE TO BE ANY ISSUES.  Nothing you say in advance will diminish the effect on your reputation when those problems are found.  If you listen to what they tell you, you will ship a product before it is ready.  If you really pay attention not to what they say but how they act, you will realize that you should not skip important steps for the sake of expediency.

The moral of the story?  You are responsible for your own reputation, and only you can make the right decision when it matters most.  Don’t put your reputation for quality in the hands of your customer, your product owner, or your VP.  It’s your decision.

Let’s start with a simple (and likely accurate) assumption – you have a QA team.  This may not be true, in which case you can just skip ahead to the recommendations below on how to move forward.  However, for those that are still with me, I have a few other suspicions about your QA team:

• the group is led by a senior QA lead or manager
• the QA testers all report to the QA manager (both as HR report and for routine work direction)
• the QA team was added to the organization after some traumatic experience (may not be true, but if it is this is the source of great drama!)
• you have a 4:1 or 5:1 developer:tester ratio (or worse)
• the QA team is separate from development and may even report to the VP Product Development (or higher) as an independent organization
• the QA team pushes hard to get requirements and design documented up front so that they can build their test cases
• QA is most involved in the final stage of integration and regression testing as they seek to confirm that all features of the product are functioning as expected
• QA is repeatably frustrated that their suggestions for improving the product are ignored
• developers are not allowed or expected to be involved in testing
• QA is perceived as the bottleneck that prevents the product from shipping
• the QA manager has a say in making the ship decision, but readiness reviews with senior management are strained as the development organization is afraid it will be surprised by what is communicated by QA

The reason these suspicions are likely true is that this is the textbook canonical form for how to organize a software development organization.  It’s also common for the simple reason that far too many testers have had their integrity compromised by project managers or developers that try to bully them into shipping a product before it is ready.  I would even go so far as to say that most process management disciplines (such as CMM (Capability Maturity Model), ISO 9000, or DFSS (Design for Six Sigma)) intentionally set out to force such a structure and assert that the Quality organization is distinct from the rest of the operational entities and must have sponsorship within the executive team.

Well…”Mr. Gorbachev, tear down this wall!”  You may have the best of intentions behind this structure, but it is very likely wasting time, creating even more frustration and animosity between the teams, and delaying product releases without appreciable improvement in quality.

Cranberry Bog - Warrens, WI

In earlier posts I described my preferred structure (see Scaling the Agile Development Organization), but it’s worth some extra effort here to explain in more detail how all aspects of testing should be handled.  In particular, we also need to explain how we move from a structure and role expectations described above into this new model.  Since responsibilities shift and required skill sets change, this transition is not natural and will require explanation, training, and coaching.

In the end state, there are three distinct forms of testing in use:

• Unit testing – created by the developers and exercised on a regular basis by the continuous integration process
• Feature testing – created and executed as part of each iteration by the manual testers assigned to the feature teams
• Regression testing – created by your automation testers and executed at least weekly

I know what you are thinking – we already have this, so we’re done!  Possibly, but I doubt it.  I am guessing that while your teams might say they do these things, that once I describe them in more detail below you will have to admit that what they are doing is different both in structure and degree.

. . .

Unit Testing

Unit tests are written by another developer as part of pair programming or by the developer that built the objects under test.  The goal is to design an architecture that allows most of the objects and encapsulated business logic to be tested with each build.  You are measuring the code coverage delivered by these tests and you have stated goals to maintain or improve this coverage over time.  Tests are meaningful and involved and exercise edge conditions that are not likely to be tested or accessible through other means.

. . .

Feature Testing

Testers have been assigned to each feature team with at least a 4:1 ratio within the feature team and they sit with their associated team.  The tester is involved in the iteration kickoff and spends the first day or two of the iteration writing exit criteria for each of the stories.  These exit criteria drive the design of the test cases, most of which are written in parallel with the development of the feature as the tester is directly involved in the ongoing informal discussions and daily scrum.  The feature team develops its stories one or two at a time instead of having many in parallel so that the workload of the tester is smoothed out throughout the entire iteration and there is not an impossible workload on the last day of the iteration.  Developers routinely help with testing the features built by others if the tester is overloaded – this happens naturally because the team is self directed and they all want to get the feature done so they can move on to the next feature.

. . .

Regression Testing

There are one or more testers responsible for owning and updating the automated regression suite.  The regression suite has been carefully designed to ensure that it exercises all important interactions within the system; it is NOT just a repeat of the functional test cases done during feature testing but is likely a subset of that combined with use cases that span features.  The automation testers meet with the feature team testers on a regular basis to understand new features that have been built and how they should be tested.  Updates to the regression tests likely lag behind new feature development by one or two iterations, but in general most new features have been automated before the product needs to ship.  The automated regression tests are run at least once a week.  These tests represent the majority of the testing that will occur during “stabilization” or “hardening” after the code freeze has occurred, and stabilization is complete within a week or so of the end of the last new feature iteration in the release.

. . .

If your QA team is engaged in these test activities as described, then they split between one of the two activities described above.  It’s difficult to achieve the goals of the automated regression testing described here without using someone very skilled in the art (see Automation and the Three Bears – Scene 3: Just Right).  If you do have that person, then they may be able to teach one of your current team members how to develop automated tests within their framework or else use them to execute and interpret the automated test results.  This is an ongoing process with new defects being constantly logged, fixed, and confirmed.

The remainder of the team should be assigned out into each of the feature teams and will receive their day to day task management from the scrum master in that team.  Their activities include product design, test planning including writing exit criteria and test cases, and executing manual tests.  They are responsible for ensuring that their feature meets the functional requirements originally communicated in the story by the product owner.

Cranberry Bog - Warrens, WI

What does the QA manager do?  Well, if this is all working as expected, then the manager has little by way of day to day responsibilities outside of their HR role.  For this to work, they MUST let go of being involved in the details of the daily work within the teams.  They are there to help and coach, review work product, and make sure that the testers are delivering on their role as expected, but they are chickens, not pigs.  In the end, they make their decision on whether the product is ready for release based on the advice of the features teams and not because they have tested the product with their own hands.  I have had great success involving the QA manager in an operational role handling support escalations – each of these represents a quality or process breakdown where the defect needs to be fixed and the process also examined to learn from the mistake.  While it sounds like this involves them in tactical break-fix work, I believe it is the exact opposite – there is nothing more strategic than creating a “learning organization”.

Once you have achieved this structure and you have everyone working effectively in their new assignments, there are a few virtuous improvements you will notice within your QA team and across development as a whole:

• the workload on test has been smoothed – there are no dramatic peaks at the end of the iteration of release cycles
• stabilization happens much faster – there is less testing done and it is done more efficiently through automation
• unit testing forces design and test activities that result in higher quality product being created by the developers
• testers have the opportunity to influence the feature design as it happens and find that their ideas for improvement are being used
• developers understand their role in ensuring quality and are involved during iterations and help out with stabilization if needed (although this should be minimal due to automation)
• the quality of the product is transparent to all and ship decisions become routine
• no one worries about the independence of the testers because they are empowered within their team to do the right thing

Let me be the first to admit that these changes will be seen as threatening by all involved.  Not everyone will agree with the goals, and some may decide that they don’t like their new responsibilities.  However, I have seen several organizations make this shift successfully and I know that you can do it as well.

Last week I had the chance to travel with a client to a potential data center location near Houston, Texas.  I didn’t realize that the data center was not in Houston proper, so we ended up taking a car ride for about 90 minutes out of Houston to get to the location.  Much to my surprise, the trip ended as we rolled up in Bryan, Texas.

Bryan, TX

The picture I snapped with my BlackBerry hardly does it justice, but this sleepy little town is home to about 65,000 people and has grown dramatically over the last 30-40 years because of the rapid growth of nearby Texas A&M.  Also, because of the university and its convenient proximity to railroad right of ways that pass through the city, this town also happens to be  a crossroads for major telecom trunks for all the major carriers.

Due to a fortuitous combination of bandwidth, low cost power, and location inland far enough to be out of hurricane risk, this town also happens to be the home of Fibertown – a world-class, Tier 4 data center of impressive design.  Housed in an old Woolworth’s building and build literally on top of the local bomb shelter, this is one of the most impressive feats of engineering that I’ve ever seen.  From the street you have no idea what this plain white and unassuming building holds, although if you could see the multiple train diesel engines on the roof you might suspect that something was up.  Resident in that facility is a service provider with rack space to spare and a brand new installation of Dell, Cisco, and Compellent fully loaded with the newest VMWare virtualization products.

After the tour we walked across the street to Mr. G’s Pizzeria – an adorable Italian restaurant where we were served by Mr. G himself.  Mr. G is rumored to be a reformed opera singer, although he did not grace us with a performance.  His personal office occupies one corner of the place with papers piled in great stacks.  Absolutely adorable.

Definitely not what I expected when we got there, and I’m hoping to get back there again some day.

Marcato is hitting a landmark of sorts – we will soon have more work than the founding partners can do with their own six hands.  This is a good problem to have, but it moves the bottleneck for the business from selling to recruiting and presents a new set of challenges.  Foremost of these is that we need to learn how to sell others into an engagement and build out the monitoring practices that ensure that our brand is maintained throughout the delivery.  All of us have experience with working in consulting so I’m confident we will sort out the latter, but none of us have been involved in positioning other people to work with our clients.  Since its not a natural behavior, it will take some effort on our part to overcome the barriers.

This brings me to another topic – if you are following this discussion and have an interest in putting your Agile skills to work coaching others, please contact us and we’ll set up an interview.  You can reach us at info@marcatopartners.com or by calling me at 612-940-3003.  Also, we’ve been working on our draft of our website – if you want to learn more about who we are and what we do, you can get a sneak preview of our unofficial site at http://marcatopartners.wordpress.com/

Last topic – it looks like we will get a chance to get some small press through the Minneapolis/St. Paul Business Journal; keep an eye out there for us, and you can be sure I’ll let you all know if and when it happens through this forum.  Exciting!

I’m often amazed that Dilbert is popular enough to be on the front page of the comics – in a way, it’s sad that there are so many “pointy haired bosses” in the world that a comic like this would be tragically funny for so many people:

It’s no wonder that process is a four-letter word – if the motivation a business has for investing in process is to reduce the jobs involved to a lowest common denominator, then it deserves that reputation.  My early impressions of process advocates were that they were people that needed process to protect them – to be blunt, they were the lowest common denominator, so they had the most to gain by having process in place.

This opinion put me in a very awkward position as I started to tackle management roles.  For a leader, process is one of the tools you use to do your job.  If I couldn’t figure out how to reconcile this with my personal opinions about process as an individual contributor, then I would very likely fail as a manager.  You may find yourself in a similar situation, and it may even be why you are avoiding a leadership role.

What I wanted to find – my holy grail – was a process that made the best people on your teams even better.  In fact, if I could find a process that had popular support with the most talented on the team, then that was the process I wanted to understand and advocate.  What I wanted to find was a process that brought structure, discipline, and predictability to an activity that is naturally chaotic.  I also needed to better manage the expectations of others outside the product development organization and combat their desire to build a “shared fiction”.  Finally, I needed a process that I would be willing to follow myself as I am terrible at convincing others of something that I do not believe or practice.

North Shore - Lake Superior, MN

The answer for me were iterative and incremental development techniques now known as Agile.  Every organization I have worked with has met this challenge with skepticism and a certain amount of fear.  Marketing and product management have to learn to let go of their fear that product development won’t deliver unless a hard contract is extracted up front.  Development needs to work hard to master the practices, but more importantly, each person on the team has new obligations to step up their game as technologists and improve their design and coding skills.  Proper application of the practices brings rhythm to the operation, ensures that meaningful work is ready at all times, and drives the individuals involved to improve their skills and to deliver high quality product on-time.

I’ve just finished another great book to recommend:  “Practices of an Agile Developer” by Venkat Subramaniam and Andy Hunt.  This book is designed to teach the individual team member more about the practices expected of them to contribute to an Agile team.  It focuses mostly on the technical disciplines of test and development, but it does a great job of setting the bar high for these team members in furthering their development as effective engineers.  I’ve added it to the “must read” list for technologists involved in Agile and would suggest that it is probably the best primer I have read so far for explaining to that audience what they need to learn.

I’m currently working with a client on a project that includes a fair amount of development effort for me, so I’ve had the opportunity lately to “practice what I preach” and to apply the principles of unit testing and test driven development while also trying to “Program Intently and Expressively” and to “Attack Problems in Isolation”.  This hasn’t been easy as the project is for a mobile platform and the tools and architectural practices available there are not nearly as mature as for server development.  I’ve had to put in quite a few extra hours of time to invent patterns that make sense and allow for typical unit testing to occur, but it has all been worth it and I really like working in the code base after making these investments.

Even after all these years of development, there are still “Aha!” moments where I realize that I have really LEARNED something.  Here’s a great example – yesterday I was working on a class where I had used the usual rules of encapsulation to hide the implementation of internal data structures.  I do this out of habit and as a good practice, but I admit to being skeptical that it ever really pays off – after all, how often do you go into a class and re-plumb the internal implementation without affecting the interface?  Then I hit the point where my next feature to implement was to be able to store and retrieve a collection of these objects from storage.  The class had a couple of private members that were Date objects, and for some reason the Date object in the mobile class library I had available did not implement Persistable.

Argh…  However, dates can also be stored as longs, and it turns out that this intrinsic data type is Persistable.  A couple of quick changes to switch my private values from Date to long and a little work in the setters and getters to do the conversion and I now had a Persistable class with no impact to any of the code that used it.  Some of the code even shrank – this class also implements Comparator so that it can be used in sorted lists, and now instead of needing to convert the Date to a long so I could compare them, I could now just use the long values that were there.

Sweet!  The simple truth – this was possible because I had been focusing on unit testing this class and the practice of building those tests had helped make sure I had a clean interface into this class.  Also, one click later and all of the unit tests for the class passed and I knew that I had not broken any of the functionality of the class.  It’s was such a simple but powerful example that I know that if you ever experience something like it yourself, you’ll become a convert.

Ping!  Finished another one.

Time for another pellet…

The career of a software developer has many possible paths – some choose to remain “technical”, while others pursue leadership opportunities.  Regardless of the path you have in mind, I was reminded of an important economic truth while reading “Practices of an Agile Developer” this morning:

Bald Eagle - Potato Lake, WI

“Machines and CPU cycles used to be the expensive part; now they are commodity.  Developer time is now the scarce – and expensive – resource.”  (p. 36)

This simple fact has been true for at least 5 years now, and it has changed just about everything.  Technical expertise now requires a high degree of specialization in order to justify the cost of you as a resource.  You will have to work harder and train longer to achieve a level of proficiency that you can sustain over a long period of time.

It also leads to more mobility – companies may only need your particular specialization for a period of time.  If your contribution is “just” project management, product management, coding, or testing, then it is likely that those skills can be acquired somewhere else at a cheaper price.  There are, however, several very durable skills that companies will always value:  domain expertise, leadership skills, and process development to name a few.  Regardless of what you do, if you are able to position yourself as capable of delivering what a company needs in a cost efficient manner, then you’ll have all the job security in the world.  You just have to be prepared to listen to what they need and deliver it on their terms, not on your own.

In a recent meeting with a venture capitalist, I was surprised to learn that he considered the use of distributed teams in low cost countries (aka offshoring) a done deal.  In his words:  ”I think that there are now enough people that know how to make it work that if I were to start up a new venture, I can assume that you would choose to use offshore resources from the very beginning.”

That’s a huge change from when this phenomena first bumped into me about 10 years ago.  At that time, it was a board led dictate that we begin using cheaper offshore resources.  It’s a simple question of competition – when your competitors are able to spend the same percentage of revenue on development but can engage 2 to 3 times as many resources…I’ll let you do the math.  Yes, productivities differ, but not in the long run (6-12 months), and while you may be willing to work harder for a while to try to match pace, this also is not sustainable in the long run.

There are a few exceptions to this rule, but even those are starting to disappear.  We used to do all of our own mobile development locally because the reality was that the job of porting an application to a new device was mostly about dealing with the idiosyncrasies of the hardware and carrier network.  It’s hard to imagine how you can debug an issue on a Verizon phone in Bangalore or Minsk.  But guess what?  An enterprising company recognized that this was an opportunity for them to capture part of that value chain, and they now provide physical phones on network that you can control and view from ANY web browser ANYWHERE in the world.

Sorry for the harsh reality, but I needed to get your attention and to motivate the real point that I want to make – I have seen and helped MANY people make the successful transition to living the distributed lifestyle.  My goal is to help you understand what it is like and how to get there yourself.

The Typical Day

When I would show up at the office at around 7 am, there were usually one or two people already there.  These were the distributed team leads.  I would walk by their offices and wave but not interrupt, as this was their quality time.  We were working with teams in Eastern Europe and had 2-3 hours of overlap every morning.  The remote folks had also adjusted their schedules and would tend to arrive later in the morning and to work further into the evening (6 to 7 pm was fairly typical).

The leads had their headsets on and were most likely talking to their liaison in the remote team – we used Skype or ooVoo and cheap cameras and headsets.  Everyone was set up on IM as well and we always have it running when we are working.  Larger team meetings also included multiple cameras at different sites around the world and GoToMeeting or Webex for desktop sharing.  With all of these technologies available all of the time, impromptu questions or issues can be quickly dispatched as if the other person was in the cube next to yours.

Routine is also important – regular scrums at the same time of day or larger group meetings on the same time and day really help communication flow.

If today happens to be iteration review day, then everyone on all of the teams knows that we meet starting at 8:30 am in a virtual conference.  Since all of the infrastructure for participating is there and remote access is pervasive, I might decide to sleep in and attend via phone, webex, and camera.  If I’m travelling, then I can still manage my schedule to be somewhere where I can participate.

What do the local team members do?  As already mentioned, a significant part of their day is spent in communication.  They also review code, make architectural and design decisions, and often write code themselves.  Once their remote team is done for the day, they still have half a day for their own use, and depending on their particular interests, they can use it as they see fit.  Their job is to ensure that the team is successful – as leaders in a self-directed team, they are entitled to do this any way they see fit.

Rusk Township City Hall

This is the trade-off I hoped to communicate – companies have to trade off flexibility on their part for flexibility on yours.  If you need to be available to your remote team anytime and anywhere, then they need to give you the resources to do that (and they can afford to from the money they are saving).  Once you have that infrastructure in place, you can participate when and how you see fit.  You may realize that you only need to be in the office a few days a week- more power to you!  You may head home from the office after lunch because you have a school function to attend but then you’ll be back on the phone with your team that evening to wrap up a few project deliverables.

It’s been a while since I made this transition myself, but I still remember it well.  You may think at first that you have to try to do two jobs at once, but you’ll quickly figure out that this is not sustainable.  Invest your time instead in getting the infrastructure and flexibility you need, build a sustaining rhythm with the remote team, and then make sure you scale back your total hours to a reasonable amount of time.  You should end up working a similar number of hours in a day, maybe just not all in a row or at the same place.

I’d like to wrap up by reiterating one final thought – your value in this new world comes not from what you do with your own two hands but from the value you liberate and the leverage you create by enabling 10 hands to work on your behalf.  Once you get to see this in action, you will find that it is just as satisfying to see your ideas become reality through your team’s efforts AND the dirty secret is that it’s a lot less work!  You really can ask for something before you go home at the end of the day and find it waiting for you when you get to work the next day.  The people I have seen that grabbed onto this challenge and made it work for themselves have learned a skill that is valuable in any software organization in the country.  Now THAT is job security.

In recently ran into the following post – an impressive interview with Sue McKinney, the VP of Development Transformation at IBM.  In 18 months and about 200 workshops they trained roughly 7000 people on Agile practices.  Of their 26,000 developers, over 70% of the teams are using at least one Agile practice.

http://agile.dzone.com/videos/sue-mckinney-agile-2009

Some worthy quotables:

North Shore

“We basically made it come down to, ‘Short, time box iterations with stakeholder feedback –working software.’ The reason why we did that was we wanted to implicitly drive change without me telling people to change.”

“Things have to change. You have to now think about automated test suites. You have to think about continuous integration. You think about being more careful in your check-ins and so forth. So I didn’t have to tell the development teams what to do. Just by that one sentence they knew that they had to change in order to be successful. That’s the type of change we were trying to drive. That was very critical for our success, to get to a very crisp of what we were trying to do.”

“The one thing we didn’t want to do, or I didn’t want to see, is people fail and go back to the old style. I wanted to avoid the trough of disillusionment.”

Despite the great investment and accomplishment, this can’t have been easy for IBM.  After all, those 26,000 developers includes all of Rational, the inventors of RUP.  While it’s admirable that they focused their teams by reducing the Agile Manifesto to an even more focused statement, I can’t help but notice that this was done by eliminating the right hand list of practices that are to be eschewed.  Granted, the Agile founders were not throwing the practices out the door but instead chose to emphasize other things, but this still seems like a compromise by IBM.  The shortened version allowed them to emphasize the positive attributes of Agile without taking a politically unpalatable stance against practices that had been endorsed by IBM for years.  A delicate balance, that’s for sure.

I, for one, think that there is a difference, and I’m concerned that with each compromise made to spread “Agile” in name by weakening its message and  broadening its inclusiveness will only devalue the philosophy.  Agile is much more than the sum of one or two practices – it is a system that depends on capturing the energy, intelligence, initiative, and sheer joy of building that makes for great software engineers.  At its heart is the conviction that if we all work hard together we can raise our collective game to the highest possible numerator instead of building systems that expect the lowest common denominator.  It works because it pushes the decision making authority out into the teams where it belongs and makes them responsible for the outcomes; through the rigor of iterative rhythm it builds trust between engineers, managers, and customers.

You will see this theme again soon as I have another entry to write around CMMi.  In a similar vein, Microsoft has been “going Agile” as they morph MSF to encompass it (it was close to begin with – this is actually not a stretch).  However, they also have an obligation to be CMMi Level 3 certified.  This story will be about how they have been working to shoe-horn these two contrary philosophies together.

I’ll close with a great piece of advice from my last boss:  ”When you stand for nothing, you fall for everything”.

My pursuit of measuring code coverage for tests actually started more than 10 years ago after attending my first conference on software quality.  At the time we were developing in C, and the state of the art involved instrumenting and recompiling all of your source code with special headers provided that introduced interceptors into every function call.  Needless to say, this was a disaster, and after a few failed attempts at instrumenting the code base for our core product, we eventually gave up without getting the answer.

It seems like such a simple thing – all I want is to know that my test cases are exercising ZZ% of the product.  Why is this important?  In part, because this becomes a metric that I can use to control my level of investment in testing. but more importantly, because of the impact on how the team behaves and how the product is designed.  Sure, it’s nice in principle to be able to say “I have 64% coverage today and I’m getting a defect rate of 0.9 defects per whatever and that’s not good enough, so I think I’ll task the team with raising that to 75% coverage”.  Such goals are readily achievable and the effort required to achieve a specific goal like that can be taken in the context of other business goals and the cost-benefit made clear.  Without such tangible measures of business value, it’s hard to convince the business and the developers that the investment is worthwhile.

Coon Rapids, MN

The next benefit is in creating the behavior you want in the team.  You want developers to do two things:

1. choose architectures that enable testability through unit tests, and
2. choose to take the time to build the unit tests

Once you have determined your baseline percentage – let’s say when first measured you are at 29% – you can now set as an exit criteria for all future iterations that your percentage must be the same or higher for the iteration to be done.  This helps ensure that new or refactored code is being tested with at least the same rigor as your earlier product, so you have reason to believe that the investment in time is happening and that the quality level at the output is comparable to what you have seen in the past.

Architecture comes into play as a limiting factor to the percentage that is practical to achieve.  If most of your code is trapped in “code behind” in the UI, then you are forced to use UI test tools to get at it.  This kind of automation is fragile unless build correctly.  Assuming your developers are not willing to open up that can of worms, they will find that they soon bump up against diminishing returns – without a serious change in architecture, they can’t improve the code coverage.  Since its unlikely that you can rewrite your product at this stage, you may have to accept that the best you can do is to slowly refactor your way into a new design pattern that exposes much more of the code to being accessible from unit tests that are build into the product.

Now that the topic is introduced, let’s explore the three areas of complexity and benefit revealed by attempting to measure code coverage – they are:

• Getting the metrics right
• Sorting out the technology and impact on product design
• Improving process capability

Getting the Metrics Right

I started this discussion with the simple notion of a single test coverage percentage, but of course it is more complicated than that.  If you have looked into the topic, you will find that all code coverage tools actually report two percentages:  block and branch coverage.  Blocks of code are chunks of code with no branching into or out of the chunk.  Branches represent every possible conditional or looping pathway that connects blocks of code.  There is no simple way to blend the two concepts, so you should probably just plan to measure both and to have goals set for each.

In principle, you would include every block and every branch in your “denominator” for your coverage calculation, but in practice this is neither appropriate nor fair.  As soon as you have a measurement, you will see the team start to change its behavior to improve the measure.  If you have a team of overachievers, they will want to get to 100% regardless of the practicality or cost of doing so, and when they start to realize that there is a lot of code that is inappropriate to include in the score, you are going to find that you agree with them.  For example, you may be using a framework or library that comes from another vendor.  Perhaps the framework code is tool generated and always includes a bunch of template code that you should be able to assume just works without spending your time creating conditions to exercise each branch.  Or perhaps some of this code is locked up in Javascript that runs in the browser and can not really be instrumented or measured.

Regardless of the rationale, the first method you will use to improve your scores is by excluding inappropriate code.  My advice is to be firm but reasonable – you should challenge the team to find a way to get more of the code, especially anything that implements business rules, validation, etc into a place where it can be readily tested and measured.  Perhaps you decide to accept those limitations for now and to exclude those portions of the product from the measure until you see the team reach a reasonable steady state percentage.  Once you think you have hit the point of diminishing returns, now challenge the team to refactor with a new pattern so that a higher percentage of the total product code is included in the measurement.

Once you have baselines for both, I like to actually build the measurements directly into the continuous integration/build system.  My opinion is that having either your block or branch percentage drop below their baseline should be considered a “broken build” that is just as serious as a syntax or other error.  If you measure and report this way, this “information radiator” will help guarantee that unit testing stays in the forefront of the team’s minds.

Impact on Technology and Design

I mentioned earlier that my efforts in this are started in C; I’m happy to report that with the newer managed languages such as *.NET or Java, the job of measuring code coverage has become MUCH easier.  Because the code executes in a runtime, interception is now much easier and involves no impact to the client code (it’s done by the runtime instead).  This takes care of the first dealbreaker – enabling the measurement itself – and you will find good tools available for your managed code technology stack of choice.  If you are still working on unmanaged code, I would be hesitant to suggest that you even bother with trying to measure code coverage.  I have yet to have a good experience in C/C++, but perhaps there are newer tools that make this more feasible – let me know if you have had an good experiences with actual products.

Assuming that you do choose a managed code stack, the next important impact is on the architecture of the code, especially the UI.  Whether thick or thin client, the goal for maximum testability is to have as little code as possible in the presentation tier.  In fact, depending on the framework you use you can probably even afford to do little or no automated testing of the presentation tier if you are using client controls that come from the vendor or browser.  In general, you want any user action to call into one or more objects that handle the response to that action.  In this manner, you should be able to get full coverage on all of the code in your model and controller and you may be able to host your view in a mocked environment as well so that most of your presentation code can be tested too.  If you are unfamiliar with this kind of architecture, then I do recommend getting some outside help – we did and it made a big difference.

Let me concede up front that using Model-View-Controller (MVC, or MVP or MVVM or whatever the flavor of the month is) does tend to make very simple UIs more complex to understand and to change, but this is only until you become familiar with the pattern.  Once you are over that learning curve, I hazard to guess that you will prefer it and find it more productive to create and maintain.  It is intimidating at first – I’ve been to presentations where it takes an hour for someone to explain their Hello World example.  Stick with it and punch through to the other side, because while it seems overly burdensome in the small it really helps as the complexity of the application grows.

Improving Process Capability

One of my best experiences with using MVC involved a re-write we were doing of an existing web application runtime from VB 6 to C#/.NET as part of an overall effort to internationalize our product line.  While it took us 3 attempts and outside help to get past the learning curve, once we got rolling down the other side of the hill, we were able to achieve test coverage measurements in the low 90′s.  More importantly, this was without a doubt the highest quality Version 1 product I have ever helped launch.  True, we did have an existing reference model for comparison, but we wrote every single line of code from scratch and still achieved remarkable reliability and scalability.  I attribute the quality level directly to our choice of architecture and use of unit testing throughout the development cycle.

This wasn’t easy, as the investment up front in learning how to apply a new design pattern cost us a couple of months of time, but the product has been in the market for years now and nobody remembers that schedule impact any more.  Instead, what is memorable is the level of process capability achieved and the confidence that the team had in being able to refactor as needed, add new features, or fix bugs and to rely almost exclusively on unit tests to tell us the quality of the result.

Call To Action

Now that you have seen more of the iceberg below the waters, you can better appreciate the impact you can have on a team by offering the innocent suggestion that “maybe you should measure your test code coverage”.  I do recommend that you use these experiences to help both your management team and the development team understand what they are getting into and why it is worth it – this will help you through the crisis of confidence that will come later as both parties start to challenge the worthiness of the goal.  If you are in a leadership role in product development, then there is probably no other single measurement you can ask for that will have such a profound effect on your products, processes, and people.  If you are starting new green field development, then you MUST start this practice on day one and make sure that you get an architecture that enables it.  If you already have an existing design and codebase in place and it is not designed for testability, then you need to look for an incremental means to move to a new architecture – try reading “Refactoring to Patterns” by Joshua Kerievsky for advice.  Regardless, if you can’t tell me your unit test coverage for your project today, then you have a great deal to gain by figuring out how to support this simple metric!