I can’t imagine testing without multiple computers at my disposal.  You may want to hold on to your old, out of warranty computers if given the choice.  Five quick reasons:

• When Computer#1 hits an impediment such as an unrecoverable error, Computer#2 can start testing immediately as Computer#1 reboots.
• I can use both computers to simulate interesting multi-user tests.  Example: what if two users attempt to acquire locks at roughly the same time.
• I can kick off timely processes, staggered on 3 separate boxes, so as not to sit idle waiting.
• Different OS’s, browser versions, frameworks, and other software running in the background can be informally tested to narrow down variables.
• Computer#1 can support the administrative work of testing (e.g., documenting tests, bugs, emailing), while Computer#2 can stay clean and focus on operating the product under test. 

What is the relationship between these two objects?

How about these two?

This, I’m afraid, is how testers (myself included) often see software modules…like black boxes. Their relationships are hidden from us.  We know the programmer just changed something related to the seeds inside the orange, so we ask ourselves, “How could changing the seeds inside the orange affect the toaster?”.  Hmmmm.  “Well, it sure seems like it couldn’t”.  Then, after a deployment, we’re shocked to discover the toaster starts burning all the toast.

Why didn’t the programmer warn us?  Well, just because the programmer understands the innards of the orange, doesn’t mean they understand the innards of the toaster.  In fact, based on my experiences, if there is enough work to do on the orange, the orange programmer will happily take it over learning to code for the toaster.

So here we are, left with nothing better to do than regression test the toaster, the jointer, and the flash light, every time the orange changes.  No wonder we spend so much time regression testing.

In conclusion, maybe the more we learn about the architecture behind our software system, the fewer regression tests we’ll need to execute.  Instead, we’ll better focus our testing and make the invisible relationships visible for the rest of the team…before development even begins.

I’m trying to fill two technical tester positions.  It’s exhausting.  All the resumes are starting to look the same.  They all tout:

• Extensive knowledge of the SDLC.  Who cares?  I’ve been testing for 15 years and I’ve never encountered a situation where extensive knowledge of the SDLC has come in handy…I’m not even sure what it is.  Who is motivating this?  Are there that many Test Managers out there saying, “what we really need, is a tester who knows the SDLC”?
• Understanding of test automation tools like QTP?  “Understanding of”?
• Ability to map test cases to requirements in Quality Center.  It’s been 6 years since I’ve seen Quality Center but I don’t recall it being that difficult a task.
• Performed different types of tests like Functional, Regression, Smoke, UAT, White Box, Black Box, Grey Box, and End-to-end.  Darn, I was really looking for someone who could write “integration tests”.  Oh well.

One resume said:

• Extensive QA experience via hands-on testing?  Is there a way to gain experience without being hands-on?

Another said:

• Fixed tested bugs and coordinated with developers in release of bug fixes during every Sprint Run.  Hmmm.  Perhaps you should start by testing your resume verbiage.

During an interview I asked:

Me: According to your resume, you worked on a team using Test Driven Development.  Was it effective?
Candidate:  Oh yes.  At the end of Sprints, if the developers had time, they would write some unit tests for Stories we were about to release to production.

During another, I asked the following easy question:

Me: What are some attributes of a good bug report?
Candidate: Documenting the bug is the most important attribute.

Finally, after an interview with me, for a programming position, the candidate remarked, “That’s odd, I’ve never seen a man in a QA role.”.  It reminds me of a little post I made years ago that almost lost me some friends.

BTW - If you live in the Atlanta area, have excellent DB and SQL skills, and are capable of testing something without a UI, please drop me a note.  I may have an awesome job waiting for you.

After watching Elisabeth Hendrickson’s CAST 2012 Keynote (I think), I briefly fell in love with her version of the “checking vs. testing” terminology.  She says “checking vs. exploring” instead. 

I love the simplicity.  I imagine when used in public, most people can follow; “exploring” is a testing activity that can only be performed by humans, “checking” is a testing activity that is best performed by machines.  And the beauty of said terms is…they’re both testing!!!  Yes, automation engineers, all the cool stuff you build can still be called testing.

The thing I’ve always found awkward about the Michael Bolton/James Bach  “checking vs. testing” terminology, is accepting that tests or testing can NOT be automated.  Hendrickson’s version seems void of said awkwardness.  She just says, “exploring” can NOT be automated…well sure, much easier to swallow.

The problem, I thought, was James and Michael’s testing definition was too narrow. Surely it could be expanded to include machine checks as testing.  Thus, I set out to find common “Testing” definitions that would support my theory.  And much to my surprise, I could not.  All the definitions (e.g., Merriam-Webster) I read, described testing as an open-ended investigation…in other words, something that can NOT be automated.

Finally, I have to admit, Hendrickson’s term, “exploring” can be ambiguous.  It might get confused with Exploratory Testing, which is a specific structured approach, as opposed to Ad Hoc testing, which is unstructured.  Hmmm…Elisabeth, if you’re out there, I’m happy to listen to your definitions, perhaps you will change my mind.

So it seems, just when I thought I could finally wiggle away from their painful terminology, I am now squarely back in the James and Michael camp when it comes to “checking vs. testing”.

…Dang!

Per Elisabeth Hendrickson, I’m one of the 80% of test managers looking for testers with programming skills.  And as I sift through tester resumes, attempting to fill two technical positions, I see a problem; testers with programming skills are few and far between!

About 90% of the resumes I’ve seen lately are for testers specialized in manual (sapient) testing of web-based products.  And since most of these resumes are sprinkled with statements like “knowledge of QTP”, I assume most of these testers are doing all their testing via the UI.

And then it hit me…

Maybe the reason so many testers are specialized in manual testing via the UI is because there are so many UI bugs!

This is no scientific analysis by any means.  Just a quick thought about the natural order of things.  But here’s my attempt to answer the question of why there aren’t more testers with programming skills out there.

It may be because they’re too busy finding bugs in the UI layer of their products.

1. Spend time reporting problems that already exist in production, that users have not asked to fix.
2. Demand all your bugs get fixed, despite the priorities of others.
3. Keep your test results to yourself until you’re finished testing.
4. Never consider using test tools.
5. Attempt to conduct all testing yourself, without asking non-testers for help.
6. Spend increasingly more time on regression tests each sprint.
7. Don’t clean up your test environments.
8. Keep testing the same way you’ve always tested.  Don’t improve your skills.
9. If you need more time to test it, ask to have it pulled from the sprint, you can test it during the next sprint.
10. Don’t start testing until your programmer tells you “okay, it’s ready for testing”.

If you made two lists for a given software feature (or user story):

1. all the plausible user scenarios you could think of
2. all the implausible user scenarios you could think of

…which list would be longer?

I’m going to say the latter.  The user launches the product, holds down all the keys on the keyboard for four months, removes all the fonts from their OS, then attempts to save a value at the exact same time as one million other users.  One can determine implausible user scenarios without obtaining domain knowledge.

Plausible scenarios should be easier to predict, by definition.  It may be that only one out of 100 users stray from the “happy path”, in which case our product may have just experienced an implausible scenario.

What does this have to do with testing?  As time becomes dearer, I continue to refine my test approach.  It seems to me, the best tests to start with are still confirmatory (some call these “happy path”) tests.  There are fewer of them, which makes it more natural to know when to start executing the tests for the scenarios less likely to occur.

The chart above is my attempt to illustrate the test approach model I have in my head.  The Y axis is how plausible the test is (e.g., it is 100% likely that users will do this, it is 50% likely that users will do this).  The X axis represents the test order (e.g., 1st test executed, 2nd test executed, etc.).  The number of tests executed is relative.

Basically, I start with the most plausible tests, then shift my focus to the stuff that will rarely happen.  These rare scenarios at the bottom of the chart above can continue forever as you move toward 0% plausibility, so I generally use the “Times Up” stopping heuristic.  One can better tackle testing challenges with this model if one makes an effort to determine how users normally use the product.

I often hear contradictory voices in my head saying, “don’t start with confirmatory tests, the bugs are off the beaten path”.  Okay, but are they really?  If our definition of a bug is “something that bugs someone who matters”, then the problems I find on the bottom of the above chart’s line, may matter less than those found on the top.  Someone who matters, may not venture to the bottom. 

For more on my thoughts (and contrary thoughts) on this position see We Test To Find Out If Software *Can* Work.