Shortly after logging a bug this morning, my business analyst kindly asked, “Is this the same as Bug10223, that I logged in October of 2011?”.

…It was!

Ordinarily, prior to logging production bugs that have been around for a while, I run a bug report query to search for open bug reports by keywords.  It takes about 10 seconds to determine if the bug I’m about to log has already been logged.  This morning I got lazy (and cocky) and just logged it.

It probably took 20 minutes of my time and my business analyst’s time to create a bug report that already existed, communicate about the confusion, and reject my duplicate bug report.

Look before you Log!

This sucks.  I’ve been testing all day and I haven't found a single problem. 

No, wait…

This is good, right? Clean software is the goal.  Alright, cool, we rock!  Looks like we’re deploying to prod tomorrow morning…just one more test…Dammit! I just found a problem!  I hate finding problems at the final hour.  This sucks.

No, wait…

This is good, right?  Better to have caught it in QA today than in prod tomorrow.  That’s what they pay me for.  Hey, here’s another bug!  And another!  I rock.  I just found the mother load of bugs.  This is awesome!!!

No, wait…

This is bad, right?  We’re either going to have to work late or delay tomorrow’s prod release.  I totally should have caught these problems earlier, it would have been so much cheaper.  I suck. 

What’s that?  The product owners are rejecting my bugs?  Really?  How humiliating.  I hate when my bugs get rejected!

No, wait…

This is good, right? It’s great that my bugs got rejected.  Less churn.  Now I don’t have to retest everything.

No, wait…I want to retest everything.

No, wait…maybe I don’t.

Ahhhhhhh!

Every once in a while, progs amaze me by casually offering some off-the-cuff solution to my major testing headaches.

I was trying to recreate a complex user scenario involving precise timing.  I needed a way to make a service hang (blocking other services), sneak some specific test actions through in the meantime, then unhang the service.  After assuming this would be way too complicated for me, my prog offered, “just use a SQL hint to put an exclusive lock on the table”.

A SQL hint is an addition to the query that instructs the database engine to do something extra, overriding its normal decisions.  The tabblock hint, wrapped in a transaction, allows you to put an exclusive lock on a table, preventing other transactions from reading or modifying said table.  I’m using MS SQL but Oracle supports a similar technique.

Here is how it works in a generic test example:

1. State A exists.
   
2. Lock the table:
   
   begin TRANSACTION
   UPDATE        tblCustomers
   WITH (TABLOCK)
   SET           Name = 'Fred'
   WHERE        (ID = 10)
   
   Note: The transaction remains open.  The update statement is irrelevant because we are going to roll it back.
   
3. Trigger the action you want to hang.  For example: maybe the current UI state is ready for female customers. You trigger a service that returns female customers from tblCustomers to display them on the UI.  Take your time, it won’t complete due to the tablock.
   
4. Perform the action you are trying to sneak in.  For example: maybe you  change the UI to expect male customers.
   
5. Now State B exists instead of State A.
   
6. Unlock the table:
   
   ROLLBACK TRANSACTION
   
   Note: execute the above statement in the same query session as the query in step 1 was executed.  The action that was hanging in step 3 completes, and in this example, female customers attempt to load into a screen expecting male customers.

So the next time you have a test idea that is too complex to execute in real time, try doing it in bullet time (using a tablock hint to slow things down.)

In Part 1 I focused on removing misleading details and unnecessary repro steps from bug reports.  I tried to make the case that a tester’s job is to narrow a bug’s repro steps down to only those actions or data that are actually required to experience the bug.

Now let’s discuss the exceptions and how to handle them.  Here are two reasons to include non-required data in bug reports:

• It is not feasible to rule out certain data as relevant.
• It is helpful for the person following the repro steps to reuse data already identified by the repro steps author, to save time.

IMO, the repro steps (and the bug report itself) should reflect that said data is or may be non-required.  I do this by providing the extra data in a comment for a specific repro step.  For example:

1. Create a new order.
2. Add at least two line items to the new order. (I added Pens and Pencils but it appears that any two line items cause the bug)
3. Submit the new order.

Expected Results: No errors are thrown.
Actual Results: “Object reference not found” error is thrown.

In some cases, there may have been significant searching to find data in a specific state.  One can save time for the bug report reader by providing tips on existing data.  For example, maybe the bug only occurs for orders in an “On Hold” state:

1. Open an “On Hold” order.  (I used Order# 10054)
2. Add at least two line items to the new order.
3. Submit the new order.

Expected Results: No errors are thrown.
Actual Results: “Object reference not found” error is thrown.

Again, the core repro steps more or less capture the relevant data pertaining to the bug.  The notes, speed up the repro process.  Look at how the opposite approach may mislead:

1. Open Order# 10054.
2. Add at least two line items to the new order.
3. Submit the new order.

Expected Results: No errors are thrown.
Actual Results: “Object reference not found” error is thrown.

The above instance makes the bug look like it is just a problem with order# 10054.  A skilled tester should have figured out the “On Hold” order state is key.

In conclusion, start with some repro steps.  Question each of your steps until you narrow them down to only the required data.  Then add any notes to aid the reader if necessary, being careful to offset those notes from the core steps.  That’s what I do.  What’s your approach?

Congratulations, you just found a bug in an office supply commerce application!  It appears that any time you submit an order with more than one line item, an “object reference not found” error is thrown.  Cool!  Here are the repro steps you logged in the bug report:

1. Create a new order.
2. Add Pencils to the new order.
3. Add Pens to the new order.
4. Select UPS for shipping method.
5. Select Credit Card for payment method.
6. Submit the new order.

Expected Results: No errors are thrown.
Actual Results: “Object reference not found” error is thrown.

Those are pretty solid repro steps.  Every time one performs them, the Actual Results occur.  Nevertheless, do you see any problems with the above repro steps?

Hmmmmm….

What if you depend on just the repro steps to understand the bug?

Does this bug only repro if pens and pencils are ordered, if the UPS shipping method is selected, if the Credit Card payment method is selected?

Those details capture what the tester did, but might they mislead?  Anytime you add specific data to your repro steps, you may be implying, to someone, that said data is required to repro the bug (e.g., perhaps the “pens” data in our DB is bad).  A more skilled tester may log the bug like this:

1. Create a new order.
2. Add at least two line items to the new order.
3. Select a shipping method.
4. Select a payment method.
5. Submit the new order.

Okay, that seems considerably better to me.  However, we can still improve it.  Is it reasonable to assume the programmers, testers, and business analysts on our project team know how to submit an order?  Do they already understand that in order to submit an order, one must specify shipping and payment methods?  Yes!

Thus, an even more skilled tester may end up with these repro steps:

1. Create a new order.
2. Add at least two line items to the new order.
3. Submit the new order.

There is nothing extra to cloud our interpretation.  The steps look so minimal and easy, anyone could do them!  Now that’s a set of repro steps we can be proud of.

This clearly makes her the coolest kid at daycare.

If I had Josephine 1000 years ago, she would probably have become a software tester like her dad.  Back then, trades often remained in the family.  But in this era, she can be whatever she wants to be when she grows up.

I doubt she will become a software tester.  However, I will teach her how important software testers are.  Josie will grow up with Generation Z, a generation that will use software for almost everything.  The first appliances she buys will have sophisticated software running them.  She will probably be able to see if she needs more eggs by logging in to a virtual version of her refrigerator from work. 

And why do you think that software will work?  Because of testers!

Josie will be able to process information, herself, at lightning speeds.  So I figure, if I start early enough, she can start making suggestions to improve the way we test. 

But first she has to learn to talk.

Do your kids appreciate your testing job?

For most of us, testing for “coolness” is not at the top of our quality list.  Our users don’t have to buy what we test.  Instead, they get forced to use it by their employer.  Nevertheless, coolness can’t hurt. 

As far as testing for it…good luck.  It does not appear to be as straightforward as some may think.

I attended a mini-UX Conference earlier this week and saw Karen Holtzblatt, CEO and founder of InContext, speak.  Her keynote was the highlight of the conference for me, mostly because she was fun to watch.  She described the findings of 90 interviews and 2000 survey results, where her company asked people to show them “cool” things and explain why they considered them cool.

Her conclusion was that software aesthetics are way less important than the following four aspects:

1. Accomplishments – When using your software, people need to feel a sense of accomplishment without disrupting the momentum of their lives.  They need to feel like they are getting something done that was otherwise difficult.  They need to do this without giving up any part of their life.  Example: Can they accomplish something while waiting in line?
2. Connection – When using your software, they should be motivated to connect with people they actually care about (e.g., not Facebook friends).  These connections should be enriched in some manner.  Example: Were they able to share it with Mom?  Did they talk about it over Thanksgiving dinner?
3. Identity - When using your software, they should feel like they’re not alone.  They should be asking themselves, “Who am I?”, “Do I fit in with these other people?”.  They should be able to share their identity with joy.
4. Sensation – When using your software, they should experience a core sensory pleasure.  Examples: Can they interact with it in a fresh way via some new interface?  Can they see or hear something delightful?

Here are a few other notes I took:

• Modern users have no tolerance for anything but the most amazing experience.
• The app should help them get from thought to action, nothing in between.
• Users expect software to gather all the data they need and think for them.

I guess maybe I’ll think twice the next time I feel like saying, “just publish the user procedures, they’ll get it eventually”.