I think it’s only people who experience bugs.

Sadly, devs, BAs, other testers, stakeholders, QA managers, directors, etc. seldom appear interested in the fruits of our labor.  The big exception is when any of these people experience a bug, downstream of our test efforts.

“Hey, did you test this?  Did it pass?  It’s not working when I try it.”

Despite the disinterest, us testers spend a lot of effort standing up ways to report test results.  Whether it be elaborate pass/fail charts or low-tech information-radiators on public whiteboards, we do our best.  I’ve put lots of energy into coaching my testers to give better test reports but I often second guess this…wondering how beneficial the skill is.

Why isn’t anyone listening?  These are some reasons I can think of:

• Testers have done such a poor job of communicating test results, in the past, that people don’t find the results valuable.
• Testers have done such a poor job of testing, that people don’t find the results valuable.
• People are mainly interested in completing their own work.  They assume all is well with their product until a bug report shows up.
• Testing is really difficult to summarize.  Testers haven't found an effective way of doing this.
• Testing is really difficult to summarize.  Potentially interested parties don’t want to take the time to understand the results.
• People think testers are quality cops instead of quality investigators; People will wait for the cops to knock on their door to deliver bad news.
• Everyone else did their own testing and already know the results.
• Test results aren’t important.  They have no apparent bearing on success or failure of a product.

What is the relationship between these two objects?

 

How about these two?

This, I’m afraid, is how testers (myself included) often see software modules…like black boxes. Their relationships are hidden from us.  We know the programmer just changed something related to the seeds inside the orange, so we ask ourselves, “How could changing the seeds inside the orange affect the toaster?”.  Hmmmm.  “Well, it sure seems like it couldn’t”.  Then, after a deployment, we’re shocked to discover the toaster starts burning all the toast.

Why didn’t the programmer warn us?  Well, just because the programmer understands the innards of the orange, doesn’t mean they understand the innards of the toaster.  In fact, based on my experiences, if there is enough work to do on the orange, the orange programmer will happily take it over learning to code for the toaster.

So here we are, left with nothing better to do than regression test the toaster, the jointer, and the flash light, every time the orange changes.  No wonder we spend so much time regression testing.

In conclusion, maybe the more we learn about the architecture behind our software system, the fewer regression tests we’ll need to execute.  Instead, we’ll better focus our testing and make the invisible relationships visible for the rest of the team…before development even begins.

My system 1 thinking says “no”.  I’ve often heard separation of duties makes testers valuable.

Let’s explore this.

A programmer and a tester are both working on a feature requiring a complex data pull.  The tester knows SQL and the business data better than the programmer.

If Testers Write Source Code:

The tester writes the query and hands it to the programmer.  Two weeks later, as part of the “testing phase”, the tester tests the query (they wrote themselves) and finds 0 bugs.  Is anything dysfunctional about that? 

If Testers do NOT Write Source Code:

The programmer struggles but manages to cobble some SQL together.  In parallel, the tester writes their own SQL and puts it in an automated check.  During the “testing phase”, the tester compares the results of their SQL with that of the programmer’s and finds 10 bugs.  Is anything dysfunctional about that? 

Yes, Michael Bolton is one of my biggest mentors.  And you’ve read a lot of fanboy posts on this blog.  But before I start spewing stuff from my RST notes, I want to post a disagreement I had with Michael Bolton (and RST).  After a 15 minute discussion, he weakened my position.  But I still disagree with this statement:

We don’t test to find out if something works.  We test to find out if it doesn’t work.

Here is a reason I disagree:  Knowing at least one way software can work, may be more valuable than knowing a thousand ways it can NOT work.

Example: Your product needs to help users cross a river.  Which is more valuable to your users? 

• “hey users, if you step on these exact rocks, you have a good chance of  successfully crossing the river”
• “hey users, here are a bunch of ways you can NOT cross the river: jump across, swim past the alligators, use the old rickety bridge, swing across on a vine, drain the river, dig a tunnel under it, etc.”

Users only need it to work one way.  And if it solves a big enough problem, IMO, those users will walk across the rocks.

Sure, finding the problems is important too.  Really important!  But if someone puts a gun to my head, and says I only get one test.  It’s going to be a happy path test. 

Bolton referred us to the following discussion between James Bach and Michael Kelly (http://michaeldkelly.com/media/  then click on “Is there a problem here?”).  I thought it would change my mind, as most James Bach lessons do.  It hasn’t…yet. 

I might be wrong.

In my Don’t Give Test Cases To N00bs post I tried to make the argument against writing test cases as a means to coaching new testers.  At the risk of sounding like a test case hater, I would like to suggest three contexts that may benefit from detailed test cases.

These contexts do not include the case of a mandate (e.g., the stakeholder requires detailed test cases and you have no choice).

1. Automated Check Design: Whether a sapient tester is designing an automated check for an automation engineer or an automation engineer is designing the automated check herself, detailed test cases may be a good idea.  Writing detailed test cases will force tough decisions to be made prior to coding the check.  Decisions like: How will I know if this check passes?  How will I ensure this check’s dependent data exists?  What state can I expect the product-under-test to be in before the check’s first action?
2. Complex Business Process Flows:  If your product-under-test supports multiple ways of accomplishing each step in its business process flows, you may want to spec out each test to keep track of test coverage.  Example: Your product’s process to buy a new widget requires 3 steps.  Each of the 3 steps has 10 options.  Test1 may be: perform Step1 with Option4, perform Step2 with Option1, then perform Step3 with Option10.
3. Bug Report Repro Steps: Give those programmers the exact foot prints to follow else they’ll reply, “works on my box”.

Those are the three contexts I write detailed test cases for.  What about you?

You don’t need bugs to feel pride about the testing service you provide to your team.  That was my initial message for my post, Avoid Trivial Bugs, Report What Works.  I think I obscured said message by covering too many topics in that post so I’ll take a more focused stab at said topic.

Here is a list of things we (testers) can do to help feel pride in our testing when everything works and we have few to no bugs to report.  Here we go…

1. Congratulate your programmers on a job well done.  Encourage a small celebration.  Encourage more of the same by asking what they did differently.  Feel pride with them and be grateful to be a part of a successful team.
2. If you miss the ego boost that accompanies cool bug discovery, brag about your coolest, most creative, most technical test.  You were sure the product would crash and burn but to your surprise, it held up.  Sharing an impressive test is sometimes enough to show you’ve been busy.
3. Give more test reports (or start giving them).  A test report is a method of summarizing your testing story.  You did a lot.  Share it.
4. Focus on how quickly whatever you tested has moved from development to production.  Your manager may appreciate this even more than the knowledge that you found a bunch of bugs.  Now you can test even more.
5. Start a count on a banner or webpage that indicates how many days your team has gone without bugs.
6. If the reason you didn’t find bugs is because you helped the programmer NOT write bugs from the beginning, then brag about it in your retrospective.
7. Perform a “self check”; ask another team member to see if they can find any bugs in your Feature.  If they can’t find bugs, you can feel pride in your testing.  If they can find bugs, you can feel pride in the guts it took to expose yourself to failure (and learn another test idea).

What additions can you think of?

Ilya Kurnosov asked an interesting question regarding my Avoid Trivial Bugs, Report What Works post. He was referring to my statement:

 

"Instead of figuring out what works, they are stuck investigating what doesn’t work.”

Ilya asked:

Why did you use "stuck" referring to context of the other testers? Isn't "investigating what doesn’t work" more important than "figuring out what works" (other factors being equal)?

I love that question.  It really made me think.  Here is my answer:

• If stuff doesn’t work, then investigating why it doesn’t work may be more important than figuring out what works.
• If we’re not aware of anything that is broken, then figuring out what else works (or what else is not broken) is more important than investigating why something doesn’t work…because there is nothing broken to investigate.

When testers spend their time investigating things that don’t work, rather than figuring out what does work, it is less desirable than the opposite.  Less desirable because it means we’ve got stuff that doesn’t work!  Less desirable to who?  It is less desirable for the development team.  It means there are problems in the way we are developing software. 

An ultimate goal would be bug free software, right?  If skilled testers are not finding any bugs, and they are able to tell the team how the software appears to work, that is a good thing for the development team.  However, it may be a bad thing for the tester. 

• Many testers feel like failures if they don’t have any issues to investigate. 
• Many testers are not sure what to do if they don’t have any issues to investigate. 
• If everything works, many testers get bored.
• If everything works, there are fewer hero opportunities for many testers. 

I don’t believe things need to be that way.  I‘m interested in exploring ways to have hero moments by delivering good news to the team.  It sounds so natural but it isn’t.  As a tester, it is soooooo much more interesting to tell the team that stuff just doesn’t work.  Now that’s dysfunctional.  Or is it?

And that is the initial thought that sparked my Avoid Trivial Bugs, Report What Works post.

Thanks, Ilya, for making me think.

I’m going back through my STARwest notes and I want to blog about a few other sessions I enjoyed.

One of those was Nancy Kelln and Lynn McKee’s, “Test Estimation and the Art of Negotiation”, in which they suggested a new way to answer the popular question…

How long will testing take?

I figured Nancy and Lynn would have some fresh and interesting things to say about test estimation since they hosted the Calgary Perspectives on Software Testing workshop on test estimation this year. I was right.

In this session, they tried to help us get beyond using what one guy in the audience referred to as a SWAG (Silly Wild-Ass Guess).

1. Nancy and Lynn pointed to the challenge of dependencies. Testers sometimes attempt to deal with dependencies by padding their estimates. This won’t work for Black Swans, which are the unknown unknowns; those events cannot be planned for or referenced.
2. The second challenge is optimism. We think we can do more than we can. Nancy and Lynn demonstrated an example of the impact of bugs on testing time. As more bugs are discovered, and their fixes need to be verified, more time is taken from new testing, time that is often under estimated.
3. The third challenge is identifying what testing means to each person. Does it include planning? Reporting? Lynn suggested to try to estimate how much fun one would have at Disneyland. Does the trip start when I leave my house, get to California, enter the park, or get on a ride? When does it end?

Eventually, Lynn and Nancy suggested the best estimate is no estimate at all.

Instead, it is a negotiation with the business (or your manager). When someone asks, “how long will testing take?”, maybe you should explain that testing is not a phase. Testing is exploration, discovery, learning, and reporting. Testing could end when there are no more interesting questions to answer but stopping testing is a business decision.

They further suggested that testers have a responsibility to help the business understand the trade-offs; if quality expectations are high, it may require more testing than if they are lower. Change your test approach to fit the needs. If you only have one day to test, you can still do your best to find the most mission critical information that you can in one day.

Apart from the session content, Lynn and Nancy were awesome presenters. They used volunteers from the audience for role plays, cartoons, brainstorms, and other interactive techniques to keep the session engaging. I heard they proposed a full day session for STPCon Spring. That would be a fun day.

Thanks Nancy and Lynn!

Question: What metric or easily understood information can my test team provide users, to show our contribution to the software we release?

I just got back from vacation and am looking at a beautiful pie chart that shows the following per iteration:

• # of features delivered
• # of bugs found in test vs. prod
• # of bugs fixed
• # of test cases executed
  

After a series of buggy production releases, my team (or at least the BAs) have decided to provide users with colorful charts depicting how hard we’ve been working each iteration. My main gripe is providing my BAs with a # representing executed test cases.

First, I feel uncomfortable measuring tester value based on test case count, for obvious reasons.

Second, the pie chart looks like all we do is test. One slice lists 400 tests. Another lists 13 features...strange juxtaposition.

Third, I’m not even sure how to provide said count. I certainly don’t encourage my test team to exhaustively document their manual test cases, nor do I care how many artifacts they save distinct tests within. Do I include 900+ automated UI test executions? Do I include thousands more unit test executions? Does the final # speak to users about quality? Does it represent how effective testers are? Not to me. Maybe it does to users...

PR is important, especially when your reputation takes a dive. I, too, want to show the users how hard my QA team works. I want to show it in the easiest possible way. I could provide a long list of tests, but they don't want to read that. What am I missing? What metric or easily understood information can my test team provide users, to show our contribution to the software we release?

I have an open headcount for the highest level tester position at my company. We are hoping to get someone with test automation skills. Most of the candidates are making me yawn. Few are enthusiastic enough to have read anything about their own craft, and fewer are interested in deep discussions.

One resume that grabbed my interest was from a career developer that claims to want to try her hand at being a tester. She is mainly interested in writing test automation code...but she never has. Prior to an interview, my devs and I were enthusiastic. Afterwards, we realized this candidate had no testing experience (other than experimenting with unit tests) and most feared our team could end up with a good developer but a weak test automation stack instead of a good tester who could provide instant gratification via sapient UI tests.

A dev wanting to cross over... How rare of a thing is this? Am I missing a great opportunity by not hiring the dev? Or is this a dev that just couldn’t cut it as an application developer…hoping to cut it as a test automation developer?

I can’t help but wonder, is it easier to teach a developer how to be a good test automator or to teach a good manual tester how to be a good test automator?

Does one require more start-up time than the other?

My department is gearing up for a spike in development this summer. We plan to use temporary, contractor developers and testers.

IMO, the contractor devs do not need to know nearly as much about our systems as the contractor testers do. The devs will be able to make their updates with limited business domain or system knowledge. However, the testers will need to go way beyond understanding single modules; they will need extensive business domain and system knowledge in order to determine what integration tests to write/execute.

Some on my team have a notion that testers can come in and be valuable by running simple tests, with limited knowledge. It scares me so much I would almost consider giving up my entire summer, moving into the office, and doing all the testing myself.

I’m also wondering if contractor testers should be paired with veteran devs and vice versa. If we have contractor testers working with contractor developers, as is the plan, it sure seems like a recipe for disaster.

Have any of you experienced something similar? What advice do you have for me?

Successful test automation is the elephant in the room for many testers. We all want to do it because manual testing is hard, our manager and devs would think we were bad-ass, and…oh yeah, some of us believe it would improve our AUT quality. We fantasize about triggering our automated test stack and going home, while the manual testers toil away. We would even let them kiss the tips of our fingers as we walked out the door.

…sounds good.

So we (testers) make an attempt at automation, exaggerate the success, then eventually feel like losers. We spend more time trying to get the darn thing to run unattended and stop flagging false bugs, while the quality of our tests takes a back seat and our available test time shrinks.

We were testing one product. Now we are testing two.

The two obvious problems are, 1.) Most of us are not developers. 2.) Writing a program to test another program is more difficult than writing the original program. ...Ah yes, a match made in heaven!

I watched an automated testing webinar last week. It was more honest than I expected. The claim was, to be successful at test automation the team should not expect existing testers to start automating tests. Instead, a new team of developers should be added to automate tests that testers” write. This new team would have their own requirement reviews, manage their own code base, and have their own testers to test their test automation stack. This does not sound cheap!

While watching this webinar, something occurred to me. Maybe we don’t need test automation. Why do I think this? Simple. Because somehow my team is managing to release successful software to the company without it. There is no test automation team on our payroll. Regression testing is spotty at best, yet somehow our team is considered a model of success within the company. How is this possible when every other test tool spam email or blog post I read makes some reference to test automation?

In my case, I believe a few things have made this possible:

• The devs are talented and organized enough to minimize the amount of stuff they break with new builds. This makes regression testing less important for us testers.
• The BAs are talented enough to understand how new features impact existing features.
• The testers are talented enough to know where to look. And they work closely with devs and BAs to determine how stuff should work.
• The user support team is highly accessible to users, knowledgeable about the AUT and the business, and works closely with the BAs/devs/testers to get the right prod bugs patched quickly. The entire team is committed to serving the users.
• The users are sophisticated enough to communicate bug details and use workarounds when waiting on fixes. The users like us because we make their jobs easier. The users want us to succeed so we can keep making their jobs easier.
• The possibility of prod bugs resulting in death, loss of customers, or other massive financial loss is slim to none.
  

I suspect a great deal of software teams are similar to mine. I'm interested in hearing from other software teams that do not depend on tester-driven test automation.

I do use test automation to help me with one of my simple AUTs which happens to lend itself to automated testing (see JART). However, from my experiences, there are few apps that are easy to automate with simple checks.

Many (if not most) test teams claim to perform test case reviews. The value seems obvious, right? Make sure the tester does not miss anything important. I think this is the conventional wisdom. On my team, the review is performed by a Stakeholder, BA, or Dev.

Valuable? Sure. But how valuable compared to testing itself? Here are the problems I have with Test Case Reviews:

• In order to have a test case review in the first place, one must have test cases. Sometimes I don’t have test cases…
• In order for a non-tester to review my test cases, the test cases must contain extra detail meant to make the test meaningful to non-testers. IMO, detailed test cases are a huge waste of time, and often invaluable or misleading in the end.
• From my experiences, the tests often suggested by non-testers are poorly designed tests or tests already covered by existing tests. This becomes incredibly awkward. If I argue or refuse to add said tests, I look bad. Thus, I often just go through the motions and pretend I executed the poorly conceived tests. This is bad too. Developers are the exception, here. In most cases, they get it.
  
• Forcing me to formally review my test cases with others is demeaning. Aren’t I getting paid to know how to test something? When I execute or plan my tests, I question the oracles on my own. For the most part, I’m smart enough to know when I don’t understand how to test something. In those cases, I ask. Isn’t that what I’m being paid for?
• Stakeholders, BAs, or Devs hate reading test cases. Zzzzzzzzzz. And I hate asking them to take time out of their busy days to read mine.
• Test Case Reviews subtract from my available test time. If you’ve been reading my blog, you know my strong feelings on this. There are countless activities expected of testers that do not involve operating the product. This, I believe, is partly because testing effectiveness is so difficult to quantify. People would rather track something simple like, was the test case review completed? Yes or No.
  

I’m interested in knowing how many of you (testers) actually perform Test Case Reviews on a regular basis, and how you conduct the review itself.

So you’ve got 10 new features to test in about 25% of the time you asked for…just another day in the life of a tester. How do you approach this effort?

Here is how I approach it.

• First, I sift through the 10 features and pick out the one that will have the most critical bugs (call it FeatureA). I test FeatureA and log two or three critical bugs.
• Next, I drop FeatureA and repeat the above for the feature that will have the next most critical bugs (call it FeatureB). I know FeatureA has undiscovered bugs. But I also know FeatureA’s critical bug fixes will trigger FeatureA testing all over again. I also assume some non-discovered FeatureA bugs will be indirectly fixed by the first batch of bug fixes. I am careful not to waste time logging “follow-on-bugs”.
• When bug fixes are released, I ignore them. I repeat the above until I have tested all 10 new Features with the first pass.
• At this point something important has occurred. The devs and BAs know the general state of what they are most interested in.
• Finally, I repeat the above with additional passes, verifying bug fixes with each feature. As the features gradually become verified I communicate this to the team by giving the features a status of “Verified”. I use my remaining time to dig deeper on the weak features.
  

Okay, nothing breakthrough here, but there are two tricks that should stand out in the above.

Trick 1 – Don’t spend too much time on individual features in the first pass. You want to provide the best info to your devs as early as possible for all 10 Features. It’s way too easy to run out of time by picking one Feature clean.

Trick 2 – Ignore those bug fixes until you get through your first pass with all 10 Features. I know it’s hard. You’re so anxious to see the damn thing fixed. However, IMO, the unknowns of untested Features are more valuable to chase down than the unknowns of whether bugs are fixed. In my experiences, when I log bugs well, verifying them is a brainless rubber-stamping-activity.

How do you get the job done?

Most bug readers will agree, a simple “Expected Results” vs. “Actual Results” statement in the bug description will remove all ambiguity. But what about the bug title? Is a bug title supposed to include the expected results, actual results, or both? Every time I log a bug, I pause to consider the various title possibilities. I want a unique but concise title that will summarize the bug, making the description as unnecessary as possible. My head swims with choices…

• If user does X after midnight, user gets Y.
• If user does X after midnight, user should not get Y.
• If user does X after midnight, user should get Z.
• If user does X after midnight, user gets Y instead of ZUser should get Z when they do X after midnight.
• etc…

Here is what I think.

There is no “best” bug title. Any bug title is good enough as long as it:

• is unique within my bug tracking system
• describes the problem to some extent
• includes key words (these words will be used someday to find this bug; searching for bugs where title contains some text)

So unless someone convinces me otherwise, with a comment on this post, I have decided to just use the first distinct bug title popping into my mind and stop worrying about bug title perfection.

I attended Robert Sabourin’s “To Infinity And Beyond” class at STPCon. This guy’s sessions are fun because of his sense of humor and pop culture references. He played us a screen captured video of someone sending an email and we were asked to write down as many boundaries as we could think of within the video. I had nearly 40 in five minutes but many testers got more.

The point was to start a conversation about boundary tests that go beyond the obvious. The first take-away I embraced was a simple mnemonic Robert gave us for thinking about boundaries. I usually hate mnemonics because most of them are only useful as PowerPoint slide filler. However, I’ve actually started using this one:

“Take AIM”

A – Application logic
I – Inputs
M – Memory storage

These are three ways to think about boundaries for test ideas. Most features you attempt to test will have these types of boundaries. The other cool thing about this mnemonic is it prioritizes your tests (i.e., test the application logic first, if you have it).

The typical example is testing a text box. But let's apply it to something different. If we apply Robert’s mnemonic to the spec, “The system must allow the user to drop up to four selected XYZ Objects on a target screen area at the same time”, we may get the following boundary tests.

Application logic (i.e., business rules)

• Drop one XYZ object.
• Drop four XYZ objects.
• Attempt to drop five XYZ objects.

Inputs

• Attempt to drop one ABC object.
• Attempt to drop a selection set of three XYZ objects and one ABC object.
• Can the objects get to the target screen area without being dropped? If so, try it.

Memory storage

• Attempt to drop 1,000,000 XYZ objects (there may be a memory constraint in just evaluating the objects)
• Refresh the screen. Are the four dropped XYZ objects still in the target screen area?
• Reopen the AUT. Are the four dropped XYZ objects still in the target screen area?

What other boundary tests can you think of and which AIM category do they fit into?

Verifying fixed bugs logged by you is easy. You’re a tester. You logged a proper bug, with repro steps, expected results, and actual results. You remember the issue.

But what about those bugs logged by non-testers that you never even had a chance to experience? You certainly need to verify their fix. However, how can you be sure you understand the bug?

Here is what I try to do.

First, I track down to person who logged the bug. I ask them how it was discovered. Knowing this info will solidify the repro steps (if there were any).

Second, I experience the bug in its unfixed build. This sometimes means preventing the fix from being released to the environment I need to verify it on (so I have time to repro it). If I experience the problem I am usually confident I can verify its fix.

Third, I determine what was done to fix the bug? I can usually identify the dev from the bug history. Even if the bug has the dev comments about its fix, devs are always thrilled to discuss it at length.

Finally, while talking to the dev, I squeeze in the important question, “What related functionality is at risk of breaking?.” The answer to this question helps narrow down my regression test goal to something realistic.

Am I missing anything?

My first job out of college was teaching software applications like MS Access, FoxPro, and Act!. Back then, in the late 90's, demand for these types of classes was much higher than it is today. This, I believe, is because today's software users are more sophisticated. Most have already been exposed to some flavor of word processing, spreadsheet, or email applications. Many can even teach themselves software or look online for answers.

After accepting the above, it's not too great a leap to also accept that modern software users are aware that software is not perfect. They have experienced application hangs and strange system errors and many users learn to avoid these bugs or recover via a reboot or similar.

If the above is true, why can't all bug lists be public? The culture of my dev team prefers to keep the bug list hidden because they believe users will have trust issues if we admit to known production bugs. I disagree. In fact, if properly facilitated, I think a public bug list can actually build user trust. Users are smart enough to see the value in having their software earlier, even at the expense of known bugs.

What do you think?