Breakthrough Moments
<Reprinted from www.projectknighton.com/ckblog>
Over the years I’ve been involved in countless situations where one or more people are perplexed by a problem (as we were yesterday) – many are solved quickly, but some seem to take significantly more time and energy to be solved. So many of these problems are solved by workarounds and these tend to fade into obscurity. Yet some solutions stand out, primarily because they ended with that “aha!” moment that is so fascinating. If you are not a problem solver yourself, then the best comparison I can make is that sudden shock of clarity and chill you got down your spine when seeing a movie like “Sixth Sense”. You suddenly find yourself seeing the problem from an entirely different perspective (and the rush can be dizzying) and all of these accumulated mysteries suddenly snap into place. The longer you have been working on a problem, the more euphoria you experience – I think because the brain becomes locked on the puzzle and all of that energy is suddenly released.
I once had an intermittent problem (they always seem to be intermittent) that took me two weeks to solve. In the end it turned out that it was caused by an interrupt service routine that reused a macro that was also used in the main program. If the ISR happened to occur in the middle or running that macro, then my register variable got hammered and just about anything was possible when the ISR returned to the main program.
By the way, I’m also fascinated by what happens just before something catastrophically fails. One moment it is whole and in one piece, and the next it is shattered and unrecognizable.
What do these things have in common? They both have that moment where rapid change occurs after periods of apparent stability, and I’m intrigued by the unique mix of circumstances that actually cause the breakthrough to occur. Where does the sudden inspiration come from? Who figured it out? What triggered the breakthrough? And most importantly, would it have been possible to solve the problem sooner if those same circumstances were created earlier or was it necessary to struggle through to that point in order for the solution to be possible?
There is a fairly predictable recipe for creating breakthrough potential but no way to guarantee the outcome. It always starts by having one or more tenacious people study the problem, but often the first people on the scene are necessary but not sufficient for finding the answer. They become the reference point and experts on the issue and are probably the people that get the credit for solving the problem. But in my experience, if the problem is going to become legendary, then they are not sufficient to the task. Instead, after studying the problem they become anchored in a set of assumptions, very likely due to the fact that they somehow created the problem and therefore their minds are stuck believing that reality matches their mental model of what is happening.
Some secrets to success:
- Recognize when you have invested enough to be competent but are simply stuck, then go ask for help. Explaining the problem to someone else almost always shakes loose the assumptions you are making and allows you to suddenly see the fallacy in your assumptions.
- Avoid “Post hoc, ergo procter hoc” – after, therefore because of – so many delays are caused by faulty assumptions about causality. Coincidences do happen.
- Embrace “Post hoc, ergo procter hoc” – if you have tried everything else, try listing the things that have changed recently to see if you can identify the difference between a time when it worked correctly and now when it does not.
- Don’t assume it is just one problem – the best problems are often the result of a complex combination of factors and if you assume a single cause you just can’t get the puzzle pieces to fit together.
- Walk away from the problem for a while – if you feel like you are looping through the same facts and reaching the same conclusions, let your subconscious work on it. Many problems are solved by sleeping on it.
- Keep mixing it up – pull in different people, change the groupings, look for new expertise
- Don’t forget to Google! So many breakthroughs come from a simple piece of information easily found if you just look.
- See the problem for yourself – don’t listen to what they tell you is wrong, only get them to show you the issue first hand and then try to filter out their mindset. If you become anchored by their “facts” then you will likely get stuck in the same rut.
We had a recent example of this last one that still blows my mind – the customer had two phones of a new model that both blew out of the application unexpectedly whenever they tried to activate the phone. We struggled for days trying to reproduce the behavior – same phone, same firmware, same version of the application, same company configuration – and nothing seemed to work. In addition, because they could not get past activation, there was no way they could turn on debugging, so we eventually built them a special version of the firmware with debugging turned on. After all of that, it was an offhand conversation between Vince and Josh that created the breakthrough.
With this new phone, the activation process is started by clicking to popup a menu and then choosing “Activate”. Now, the only other option available is “Exit”, and since there are only two things in the list, one has a light background and the other a dark one and all you can do is roll the thumbwheel between the two. The truth is, you really have no idea which one is going to be selected when you click because the highlight could work either way. Yup, you guessed it, every time the customer was selecting “Exit”.
My congratulations to the extended team that worked our production issue yesterday through to a decisive conclusion. While the problem is solved and an extended solution in place to prevent it from happening again, I find myself stuck on the breakthrough moment and how we spent our day reaching that conclusion. Having observed many of the adhoc subteams in action throughout the day, all of the elements needed to find the solution where present and all of the information needed to combine them into a solution was available as early as 9 am that morning. Unfortunately, we were all working within our areas of expertise and it seemed to take marshalling everyone together and laying out our options to trigger the chain of events that led to the solution. Others involved in the process might have a different point of view – I would be interested in hearing what you think catalyzed the solution.
Happy hunting!
