Marcato

When you focused on an iterative process that will bring about “non-functional requirements” (like performance!) it is good to be as crisp as you can be in defining what success looks like.

I’ve found that it can be useful to express performance goals (or current operational characteristics) in three parts…

The first part of a performance goal is a transaction rate or volume as a function of time.  You might say something along the lines of: “On any given day, we expect message rates building up from 0 messages/sec at 7:00AM to 180 messages/sec at noon and then back down to 0 message/sec at 10:00PM, roughly in the form of a bell curve.”  Or you might say, “Every morning at 8:00AM, 5000 customer service representatives log on to the system and perform, on average, 3 inquiries per representative per minute until closing time at 5:00PM, 7 days per week.”

The second part of a performance goal should be a maximum acceptable latency, or in some cases, a fixed-length time constraint.  You might say,

• “95% of all requests must be satisfied in 20 seconds or less.”
• “All inquiries must return in less than three seconds.”
• “All messages that arrive between 9:00pm and 4:00am must be completely processed, with enough reserve capacity to process previous night’s batch in same window.”

The final part of the goal should speak to the distribution in the type of work (if it isn’t already homogenous).  You might say,

• “35% of all system interactions represent an open-ended search, 40% represent direct retrieval of a record, and the remaining interactions are new orders.”
• “10% of all interactions represent a new client, while all others represent an existing client.”
• “90% of all interactions are in the happy path, while 10% require alternate path handling.”

Once you have constrained your goals along these three axes (throughput, latency, and content distribution), you’ll find it is easier to formulate testing plans and to keep the team focused on the immediate problem at hand.  There will always be parts of your system that are sub-optimal and represent clear opportunity for tuning.  (I’ve got a system that is thrashing the garbage collector at the moment.)  But if that tuning doesn’t represent the current shortest path to achieving your goal, it needs to wait.  Don’t lose time to premature optimization.