I noticed one of our development teams was creating new Jira Issues for each bug found during the development cycle.  IMO, this is an antipattern. 

These are the problems it can create, that I can think of:

• New Jira Issues (bug reports) are creating unneccessry admin work for the whole team. 
• We see these bug reports cluttering an Agile board.
• They may have to get prioritized.
• We have to track them, they have to get assigned, change statuses, get linked, maybe even estimated.
• They take time to create. 
• They may cause us to communicate via text rather than conversation.
• Bug reports mislead lazy people into tracking progress, quality, or team performance by counting bugs.
• It leads to confusion about how to manage the User Story.  If the User Story is done except for the open bug reports, can we mark the User Story “Done”?  Or do we need to keep the User Story open until the logged bugs get fixed…”Why is this User Story still in progress?  Oh yeah, it’s because of those linked logged bugs”.
• It’s an indication our acceptance criteria is inadequete.  That is to say, if the acceptance criteria in the User Story is not met, we wouldn’t have to log a bug report.  We would merely NOT mark the Story “Done”.
• Bug reports may give us an excuse not to fix all bugs…”let’s fix it next Sprint”, “let’s put it on the Product Backlog and fix it some other day”…which means never.
• It’s probably a sign the team is breaking development into a coding phase and a testing phase.  Instead, we really want the testing and programming to take place in one phase...development. 
• It probably means the programmer is considering their code “done”, throwing it over the wall to a tester, and moving on to a different Story.  This misleads us on progress.  Untested is as good as nothing.

If the bug is an escape, if it occurs in production.  It’s probably a good idea to log it.

On a production support kanban development team, a process dilema came up.  In the case where something needs to be tested by a tester:

1. Should the tester perform the testing first in a development environment, then in a production-like environment after the thing-under-test has been packaged and deployed?  Note: in this case, the package/deploy process is handled semi-manually by two separate teams, so there is a delay.
2. Or, should the tester perform all the testing in a production-like environment after the thing-under-test has been packaged and deployed?

Advantage of scenario 1 above:

• Dev environment testing shortens the feedback loop.  This would be deep testing.  If problems surface they would be quicker and less risky to fix.  The post-package testing would be shallow testing, answering questions like: did the stuff I deep tested get deployed properly?

Advantage of scenario 2 above:

• Knock out the testing in one environment.  The deep testing will indirectly cover the package/deployment testing.

From the surface, scenario 2 looks better because it only requires one testing chunk, NOT two chunks separated by a lengthy gap.  But what happens if a problem surfaces in scenario 2?  Now we must go through two lengthy gaps.  How about a third problem?  Three gaps.  And so on.

My conclusion: Scenario 1 is better unless this type of thing-under-test is easy and has a history of zero problems.

We never have enough of them.  They never mirror production.  They never work.

My opinions at my current company:

1. Who should own test environments?  Testers.
2. Who should build test environments?  NOT testers.  DevOps.
3. Who should request test environments?  Testers.
4. Who should populate, backup, or restore the test data in test environments?  Testers.
5. Who should configure test environments to integrate with other applications in the system?  NOT testers.  DevOps.
6. Who should deploy code to test environments?  NOT testers. Whoever (or whatever) deploys code to production.
7. Who should control (e.g., request) code changes to test environments?  Testers.
8. Who should create and maintain build/deploy automation?  NOT testers.  DevOps.
9. Who should push the “Go” button to programmatically spin up temporary test environments?  Testers or test automation.

Fiddling with test environments is not testing work, IMO.  It only subtracts from test coverage.

About five years ago, my tester friend, Alex Kell, blew my mind by cockily declaring, “Why would you ever log a bug?  Just send the Story back.”

Okay.

My dev team uses a Kanban board that includes “In Testing” and “In Development” columns.  Sometimes bug reports are created against Stories.  But other times Stories are just sent left; For example, a Story “In Testing” may have its status changed to “In Development”, like Alex Kell’s maneuver above.  This normally is done using the Dead Horse When-To-Stop-A-Test Heuristic. We could also send an “In Development” story left if we decide the business rules need to be firmed up before coding can continue.

So how does one know when to log a bug report vs. send it left?

I proposed the following heuristic to my team today:

If the Acceptance Test Criteria (listed on the Story card) is violated, send it left.  It seems to me, logging a bug report for something already stated in the Story (e.g., Feature, Work Item, Spec) is mostly a waste of time.

Thoughts?

Have you ever been to a restaurant with a kitchen window?  Well, sometimes it may be best not to show the customers what the chicken looks like until it is served.

A tester on my team has something similar to a kitchen window for his automated checks; the results are available to the project team.

Here’s the rub: 

His new automated check scenario batches are likely to result in…say, a 10% failure rate (e.g., 17 failed checks).  These failures are typically bugs in the automated checks, not the product under test.  Note: this project only has one environment at this point.

When a good curious product owner looks through the kitchen window and sees 17 Failures, it can be scary!  Are these product bugs?  Are these temporary failures? 

Here’s how we solved this little problem:

• Most of the time, we close the curtains.  The tester writes new automated checks in a sandbox, debugs them, then merges them to a public list.
• When the curtains are open, we are careful to explain, “this chicken is not yet ready to eat”.  We added an “Ignore” attribute to the checks so they can be filtered from sight.

1. Spend time reporting problems that already exist in production, that users have not asked to fix.
2. Demand all your bugs get fixed, despite the priorities of others.
3. Keep your test results to yourself until you’re finished testing.
4. Never consider using test tools.
5. Attempt to conduct all testing yourself, without asking non-testers for help.
6. Spend increasingly more time on regression tests each sprint.
7. Don’t clean up your test environments.
8. Keep testing the same way you’ve always tested.  Don’t improve your skills.
9. If you need more time to test it, ask to have it pulled from the sprint, you can test it during the next sprint.
10. Don’t start testing until your programmer tells you “okay, it’s ready for testing”.

ATDD Sans Automated Tests.  Why Not?

Every time I hear about Acceptance Test Driven Development (ATDD), it’s always implied that the acceptance tests are automated.  What’s up with that?  After all, it’s not called Automated Acceptance Test Driven Development (AATDD).  It seems to me, that ATDD without automated tests, might be a better option for some teams.

This didn’t occur to me until I had the following conversation with the Agile consultant leading the discussion at the ATDD 2013 STAReast discussion lunch table.  After a discussion about ATDD tools and several other dependencies to automating acceptance tests, our conversation went something like this:

Agile Consultant: ATDD has several advantages.  While writing the acceptance tests as a team, we better understand the Story.  Then, we’ll run the tests before the product code exists and we’ll expect the tests to fail.  Then we’ll write the product code to make the tests pass.  And one of the main advantages of ATDD is, once the automated acceptance tests pass, the team knows they are “done” with that Story.

Me: Sounds challenging.  Are you saying there must be an automated check written first, for every piece of product code we need?

Agile Consultant:  Pretty much.

Me: Doesn’t that restrict the complexity and creativity of our product?  I mean, what if we come up with something not feasible to test via a machine.  Besides, aren’t there normally some tests better executed by humans, even for simple products?

Agile Consultant:  Yes, of course.  I guess some manual tests could be required, along with automated tests, as part of your “done” definition.

Me: What if all our tests are better executed by a human because our product doesn’t lend itself to automation?  Can we still claim to do ATDD and enjoy its benefits?

Agile Consultant:  …um…I guess so…  (displaying a somewhat disappointed face, as if something does not compute, but maybe she was just thinking I was an annoying nutcase)

And that got me thinking: And doesn’t this save us a lot of headaches, pain, and time because we wouldn’t have to distill our requirements into rigid test scripts with specific data (AKA “automatic checks”)?  We wouldn’t have to ask our programmers to write extra test code hooks for us.  We wouldn’t have to maintain a bunch of machine tests that don’t adapt as quickly as our human brains do?

Let’s call this Human Acceptance Test Driven Development (HATDD).  I’ve stated some of the advantages above.  The only significant disadvantage that stands out is that you don’t get a bunch of automated regression checks.  But it seems to me, ATDD is more about new Feature testing than it is about regression testing anyway.

So why aren’t there more (or any) Agile consultants running around offering HATDD?

Shortly after logging a bug this morning, my business analyst kindly asked, “Is this the same as Bug10223, that I logged in October of 2011?”.

…It was!

Ordinarily, prior to logging production bugs that have been around for a while, I run a bug report query to search for open bug reports by keywords.  It takes about 10 seconds to determine if the bug I’m about to log has already been logged.  This morning I got lazy (and cocky) and just logged it.

It probably took 20 minutes of my time and my business analyst’s time to create a bug report that already existed, communicate about the confusion, and reject my duplicate bug report.

Look before you Log!

My system 1 thinking says “no”.  I’ve often heard separation of duties makes testers valuable.

Let’s explore this.

A programmer and a tester are both working on a feature requiring a complex data pull.  The tester knows SQL and the business data better than the programmer.

If Testers Write Source Code:

The tester writes the query and hands it to the programmer.  Two weeks later, as part of the “testing phase”, the tester tests the query (they wrote themselves) and finds 0 bugs.  Is anything dysfunctional about that? 

If Testers do NOT Write Source Code:

The programmer struggles but manages to cobble some SQL together.  In parallel, the tester writes their own SQL and puts it in an automated check.  During the “testing phase”, the tester compares the results of their SQL with that of the programmer’s and finds 10 bugs.  Is anything dysfunctional about that? 

When bugs escape to production, does your team adjust?

We started using the following model on one of my projects.  It appears to work fairly well.  Every 60 days we meet and review the list of “escapes” (i.e., bugs found in production).  For each escape, we ask the following questions:

1. Could we do something to catch bugs of this nature?
2. Is it worth the extra effort?
3. If so, who will be responsible for said effort?

The answer to #1 is typically “yes”. Creative people are good at imagining ultimate testing. It’s especially easy when you already know the bug.  There are some exceptions though. Some escapes can only be caught in production (e.g., a portion of our project is developed in production and has no test environment).

The answer to #2 is split between “yes” and “no”.  We may say “yes” if the bug has escaped more than once, significantly impacts users, or when the extra effort is manageable.  We may say “no” when a mechanism is in place to alert our team of the prod error; we can patch some of these escapes before they affect users, with less effort than required to catch them in non-prod environments.

The answer to #3 falls to Testers, Programmers, BAs, and sometimes both or all.

So…when bugs escape to production, does my team adjust?  Sometimes.

In response to my What I Love About Kanban As A Tester #1 post, Anonymous stated:

“The whole purpose of documenting test cases…[is]…to be able to run [them] by testers who don’t have required knowledge of the functionality.”

Yeah, that’s what most of my prior test managers told me, too…

“if a new tester has to take over your testing responsibilities, they’ll need test cases”

I wouldn’t be surprised if a secret QA manager handbook went out to all QA managers, stating the above as the paramount purpose of test cases.  It was only recently that I came to understand how wrong all those managers were.

Before I go on, let me clarify what I mean by “test cases”.  When I say “test cases”, I’m talking about something with steps, like this:

1. Drag ItemA from the catalog screen to the new order screen.
2. Change the item quantity to “3” on the new order screen.
3. Click the “Submit Order” button.

Here’s where I go on: 

• When test cases sit around, they get stale.  Everything changes…except your test cases.  Giving these to n00bs is likely to result in false fails (and maybe even rejected bug reports).
• When test cases are blindly followed, we miss the house burning down right next to the house that just passed our inspection.
• When test cases are followed, we are only doing confirmatory testing.  Even negative (AKA “unhappy”) paths are confirmatory testing.  If that’s all we can do, we are one step closer to shutting down our careers as testers.
• Testing is waaaay more than following steps.  To channel Bolton, a test is something that goes on in your brain.  Testing is more than answering the question, “pass or fail?”.  Testing is sometimes answering the question, “Is there a problem here?”. 
• If our project mandates that testers follow test cases, for Pete’s sake, let the n00b’s write their own test cases.  It may force them to learn the domain.
• Along with test cases comes administrative work.  Perhaps time is better spent testing.
• If the goal is valuable testing from the n00b, wouldn’t that best be achieved by the lead tester coaching the n00b? And if that lead tester didn’t have to write test cases for a hypothetical n00b, wouldn’t that lead tester have more time to coach the hypothetical n00b, should she appear.  Here’s a secret: she never will appear.  You will have a stack of test cases that nobody cares about; not even your manager.

In my next post I’ll tell you when test cases might be a good idea.

...regression testing is optional.  What?  The horror!!!

Back in the dark ages, with Scrum, we were spending about 4 days of each iteration regression testing.  This product has lots of business logic in the UI, lots of drag-and-drop-type functionality, and very dynamic data, so it has never been a good candidate for automation.  Our regression test approach was to throw a bunch of humans at it (see my Group Regression Testing and Chocolate post).  With Scrum, each prod deployment was a full build, including about 14 modules, because lots of stuff got touched.  Thus, we always did a full regression test, touching all the modules.  Even after an exhaustive regression test, we generally had one or two “escapes” (i.e., bugs that escaped into production).

Now, ask me how often regression tests failed?  …not very often.  And, IMO, that is a lot of waste.

With Kanban, each prod release only touches one small part of the product.  So we are no longer doing full builds.  Think of it like doing a production patch.  We’ve gotten away from full regression tests because, with each release, we are only changing one tiny part of the product.  It is far less risky.  Why test the hell out of bits that didn’t change? 

So now we regression test based on one risk: the feature going to prod.  Sometimes it means an hour of regression tests.  Sometimes it means no regression tests.  So far, it’s a net loss of time spent on regression tests.  And this is a good thing.

We switched to Kanban in February.  So far, not a single escape has made it to prod (yes, I’m knocking on wood).

This success may just be a coincidence.  Or…maybe it’s easier for teams to prevent escaped bugs when those teams can focus on one Feature at a time.   Hmmmmmm…

...no testing deadlines…the freedom to test as long as I want.

Back in the dark ages, with Scrum, all the sprint Features had to be tested by the end of the iteration.  Since programming generally continued until that last minute (we couldn’t help ourselves), testers were sometimes forced to cut corners.  Even in cases where the whole team (e.g., programmers, BAs) jumped in to help test, there was still a tendency to skimp on testing that would otherwise be performed.  The team wants to be successful.  Success is more easily measured by delivered Features than Feature quality.  That’s the downside of deadlines.

With Kanban, there are no deadlines…you heard me!  Testers take as long as they need.  If the estimates are way off, it doesn’t leave gaps or crunches in iterations.  There are no iterations!  Warning: I fear unskilled testers may actually have a difficult time with this freedom.  Most testers are used to being told how much time they have (i.e., “The Time’s Up! Heuristic”).  So with Kanban, other Stopping Heuristics may become more important.

I believe testers have the power to either slow down the rate of production deployments or speed them up, without adversely affecting their testing value.

1. My test career began as a “Quality Cop”.  I believed a large responsibility of my job was preventing things from going to production. 
2. After talking Michael Bolton’s Rapid Software Testing class, I stopped trying to assure quality, stopped being the team bottleneck, and became a tester.  At this point I was indifferent to what and when things went to production.  I left it in the hands of the stakeholders and did my best to give them enough information to make their decision obvious.
3. Lately, I’ve become an “Anti-bottleneck Tester”.  I think it’s possible to be an excellent tester, while at the same time, working to keep changes flowing to production.  It probably has something to do with my new perspective after becoming a test manager.  But I still test a considerable amount, so I would like to think I’m not completely warped in the head yet.

Tell me if you agree.  The following are actions testers can do to help things flow to production quicker.

• When you’re testing new FeatureA and you find bugs that are not caused by the new code (e.g., the bug exists in production), make this clear.  The bug should probably not slow down FeatureA’s prod deployment.  Whether it gets fixed or not should probably be decoupled from FeatureA’s path.  The tester should point this out.
• Be a champion of flushing out issues before it hits the programmer’s desk.  Don’t get greedy and keep them to yourself.  Don’t think, “I just came up with an awesome test, I know it’s going to fail!”.  No no no tester!  Bad tester!  Don’t do this.  Go warn somebody before they finish coding.
• Be proactive with your test results.  Don’t wait 4 days to tell your stakeholders what you discovered.  Tell them what you know today!  You may be surprised.  They may say, “thanks, that’s all we really needed to know, let’s get this deployed”.
• Help your programmers focus.  Work with them.  I’m NOT talking about pair programming.  When they are ready for you to start testing, start testing!  Give them immediate feedback, keep your testing focused on the same feature.  Go back and forth until you’re both done.  Then wrap it up and work on the next one… together.  When possible, don’t multi-task between user stories.
• Deployments are something to celebrate, not fear.  This relates more to Kanban than Scrum.  If you have faith in your testing then don’t fear deployments.  We have almost daily deployments on my Kanban project now.  This has been a huge change for testers who are used to 4 week deployments.  Enthusiastic testers who take pride in rapid deployments can feel a much needed sense of accomplishment and spread the feeling to the rest of the team.
• Don’t waste too much time on subjective quality attributes.  Delegate this testing to users or other non-testers who may be thrilled to help. 
• Don’t test things that don’t need testing.  See my Eight Things You May Not Need To Test post.

Every other development team is running around whining “we’re overworked”, “our deadlines are not feasible”.  Testers have the power to influence their team’s success.  Why not use it for the better?

Last week we celebrated two exciting things on one of my project teams:

1. Completing our 100th iteration (having used ScrumBut for most of it).
2. Kicking off the switch to Kanban.

Two colleagues and I have been discussing the pros and cons of switching to Kanban for months.  After convincing ourselves it was worth the experiment, we slowly got buy-in from the rest of the project team and…here we go!

Why did we switch?

• Our product’s priorities change daily and in many cases users cannot wait until the iteration completes.
• Scrum came with a bunch of processes that never really helped our team.  We didn’t need daily standups, we didn’t like iteration planning, we spent a lot of time breaking up stories and arguing about how to calculate iteration velocity.  We ran out of problems to discuss in retrospectives and in some cases (IMO) forced ourselves to imagine new ones just to have something to discuss.
• We’re tired of fighting the work gaps at the start and end of iterations (i.e., testers are bored at the iteration start and slammed at the end, programmers are bored at the iteration end and slammed at the start).
• Deploying entire builds, filled with lots of new Features forced us to run massive regression tests, and deploy on weekends, during a maintenance window (causing us to work weekends, and forcing our users to wait for Features until weekends).
• Change is intellectually stimulating.  This team has been together for 6 years and change may help us to use our brains again to make sure we are doing things for the right reasons.  One can never know if another approach works better unless one tries it.

As I write this, I can hear all the Scrum Masters crying out in disgust, “You weren’t doing Scrum correctly if it didn’t work!”  That’s probably true.  But I’ll give part of the blame to the Scrum community, coaches, and consultants.  I think you should strive to do a better job of explaining Scrum to the software development community.  I  hear conflicting advice from smart people frequently (e.g., “your velocity should go up with each iteration”, “your velocity should stay the same with each iteration”, “your velocity should bounce around with each iteration”).

When I was a young kid, my family got the game “Video Clue”.  We invited my grandpa over to play and we all read through the instructions together.  After being confused for a good 30 minutes, my grandpa swiped the pieces off the table and said, “anything with this many rules can’t possibly work”.

Anybody else out there using Kanban?

You don’t need bugs to feel pride about the testing service you provide to your team.  That was my initial message for my post, Avoid Trivial Bugs, Report What Works.  I think I obscured said message by covering too many topics in that post so I’ll take a more focused stab at said topic.

Here is a list of things we (testers) can do to help feel pride in our testing when everything works and we have few to no bugs to report.  Here we go…

1. Congratulate your programmers on a job well done.  Encourage a small celebration.  Encourage more of the same by asking what they did differently.  Feel pride with them and be grateful to be a part of a successful team.
2. If you miss the ego boost that accompanies cool bug discovery, brag about your coolest, most creative, most technical test.  You were sure the product would crash and burn but to your surprise, it held up.  Sharing an impressive test is sometimes enough to show you’ve been busy.
3. Give more test reports (or start giving them).  A test report is a method of summarizing your testing story.  You did a lot.  Share it.
4. Focus on how quickly whatever you tested has moved from development to production.  Your manager may appreciate this even more than the knowledge that you found a bunch of bugs.  Now you can test even more.
5. Start a count on a banner or webpage that indicates how many days your team has gone without bugs.
6. If the reason you didn’t find bugs is because you helped the programmer NOT write bugs from the beginning, then brag about it in your retrospective.
7. Perform a “self check”; ask another team member to see if they can find any bugs in your Feature.  If they can’t find bugs, you can feel pride in your testing.  If they can find bugs, you can feel pride in the guts it took to expose yourself to failure (and learn another test idea).

What additions can you think of?

I’ve been testing this darn thing all morning and I haven't found a single bug, or even an issue.  My manager probably thinks I’m not testing well enough.  My other tester colleagues keep finding bugs in their projects.  Maybe I’m not a very good tester.  My next scrum report is going to be lame.  This sucks, man.

Wrong!  It probably doesn’t suck.  Not finding bugs may be a good thing.  Your team may be building stuff that works.  And you get to be the lucky dude who delivers the good news. 

If there is lots of stuff that works and no bugs, you have even more to report than testers who keep finding bugs.  Testers who keep finding bugs are probably executing fewer tests than you so they know less about their products than you.  Instead of figuring out what works, they are stuck investigating what doesn’t work.  They’ll still need to figure out what works eventually, it’s just going to take them a while to get there.  And that sucks.

My manager is probably looking at my low bug count metric, thinking I’m not doing anything.  Logging bugs makes me feel like a bad ass.  There must be something I can log…hmmm…I know, I’ll log a bug for this user message; it’s not really worded as well as it could be…it has been like that for the last four years.

No!  No!  No!  That’s gaming the system.  It’s not going to work.  You’re going to get a reputation as a tester who logs trivial bugs.  Your manager is only counting bugs because you’re not giving her anything else.  She just wants to know what you’re doing.  Help your manager.  Show her where to find your test reports, session sheets, or test execution results.  Invite her to your scrum meetings. Tell her how busy you’ve been knocking out tests and how bad ass your entire project team is.

Think about it. 

Reporting what works may be better than reporting trivial bugs.

This article will be published in a future addition of the Software Test Professionals Insider – community news.  I didn’t get a chance to write my blog post this week so I thought I would cheat and publish it on my own blog first.

I will also be interviewed about it on Rich Hand’s live Blog Talk Radio Show on Tuesday, January 31st at 1PM eastern time. 

My article is below.  If it makes sense to you or bothers you, make sure you tune in to the radio show to ask questions…and leave a comment here, of course.

Don’t Test It

As testers, we ask ourselves lots of questions:

• What is the best test I can execute right now?
• What is my test approach going to be?
• Is that a bug?
• Am I done yet?

But how many of us ask questions like the following?

• Does this Feature need to ever be tested?
• Does it need to be tested by me?
• Who cares if it doesn’t work?

In my opinion, not enough of us ask questions like the three above.  Maybe it’s because we’ve been taught to test everything.  Some of us even have a process that requires every Feature to be stamped “Tested” by someone on the QA team.  We treat testing like a routine factory procedure and sometimes we even take pride in saying...

“I am the tester.  Therefore, everything must be tested...by me...even if a non-tester already tested it...even if I already know it will pass...even if a programmer needs to tell me how to test it...I must test it, no exceptions!”

This type of thinking may be giving testers a bad reputation.  It emphasizes testing importance because of a thoughtless process rather than a service to provide the most valuable information to someone. 

James Bach came up with the following test execution heuristic:

Basic Heuristic:  “If it exists, I want to test it”

I disagree with that heuristic, as it is shown above and often published.  However, I completely agree with the full version James published when he introduced it in his 7/8/2006 blog post:

“If it exists, I want to test it. (The only exception is if I have something more important to do.)”

The second sentence is huge!  Why?  Because often we do have something more important to do, and it’s usually another test!  Unfortunately, importance is not always obvious.  So rather than measuring importance, I like to ask the three questions above and look for things that may not be worth my time to test.  Here are eight examples of what I’m talking about:

1. Features that don’t go to production -  My team has these every iteration.  These are things like enhancements to error logging tables or audit reports to track production activity.  On Agile teams these fall under the umbrella of Developer User Stories.  The bits literally do not go to production and by their nature cannot directly affect users. 
2. Patches for critical production problems that can’t get worse - One afternoon our customers called tech support indicating they were on the verge of missing a critical deadline because our product had a blocking bug.  We had one hour to deliver the fix to production.  The programmer had the fix ready quickly and the risk of further breaking production was insignificant because production was currently useless.  Want to be a hero?  Don’t slow things down.  Pass it through to production.  Test it later if you need to.
3. Cosmetic bug fixes with timely test setup - We fixed a spelling mistake that had shown up on a screen shot of a user error message.  The user was unaware of the spelling mistake but we fixed it anyway; quick and easy.  Triggering said error message required about 30 minutes of setup.  Is it worth it?
4. Straight forward configuration changes - Last year our product began encountering abnormally large production jobs it could not process.  A programmer attempted to fix the problem with an obvious configuration change.  There was no easy way to create a job large enough to cross the threshold in the QA environment.  We made the configuration change in production and the users happily did the testing for us.
5. Too technical for a non-programmer to test - Testing some functionality requires performing actions while using breakpoints in the code to reproduce race conditions.  Sometimes a tester is no match for the tools and skills of a programmer with intimate knowledge of the product code.  Discuss the tests but step aside.
6. Non-tester on loan - If a non-tester on the team is willing to help test, or better yet, wants to help test a certain Feature, take advantage of it.  Share test ideas and ask for test reports.  If you’re satisfied, don’t test it.
7. No repro steps - Occasionally a programmer will take a stab at something.  There are often errors reported for which nobody can determine the reproduction steps.  We may want to regression test the updated area, but we won’t prevent the apparent fix from deploying just because we don’t know if it works or not.
8. Inadequate test data or hardware - Let’s face it.  Most of us don’t have as many load balanced servers in our QA environment as we do in production.  When a valid test requires production resources not available outside of production, we may not be able to test it.

Many of you are probably trying to imagine cases where the items above could result in problems if untested.  I can do that too.   Remember, these are items that may not be worth our time to test.  Weigh them against what else you can do and ask your stakeholders when it’s not obvious.

If you do choose not to test something, it’s important not to mislead.  Here is the approach we use on my team. During our Feature Reviews, we (testers) say, “we are not going to test this”.  If someone disagrees, we change our mind and test it.  If no one disagrees, we “rubber stamp” it. Which means we indicate nothing was tested (on the work item or story) and pass it through so it can proceed to production.  The expression “rubber stamping” came from the familiar image of an administrative worker rubber stamping stacks of papers without really spending any time on each.  The rubber stamp is valuable, however.  It tells us something did not slip through the cracks.  Instead, we used our brains and determined our energy was best used elsewhere.

So the next time you find yourself embarking on testing that feels much less important than other testing you could be doing, you may want to consider...not testing it.  In time, your team will grow to respect your decision and benefit from fewer bottlenecks and increased test coverage where you can actually add value.

Who tests the automated tests? Well, we know the test automation engineer does.  But per our usual rationale, shouldn’t someone other than the test automation engineer test the automated tests?  After all, they are coded…in some cases by people with less programming experience than the product programmers.

We’re experimenting with manual testers testing automated tests on one of my project teams.  The test automation engineer hands each completed automated test to a manual tester.  The manual tester then executes the automated test.  At this point the manual tester is testing two things at the same time.  They are:

1. Testing the automated test and
2. testing whatever the automated test tests

Or, to be more precise, we can use Michael Bolton speak and say the tester is:

1. Testing the automated check and
2. checking whatever the automated check checks

Whatever you call it, during this exercise, it’s important to distinguish the above two activities.  If the automated test's execution results in a “Fail”, it doesn’t mean the test of the automated test fails…but it may.  Are you with me still?  An automated test’s execution result of “Fail” may, in fact, mean the automated test is a damn good test.  It may have found a bug in the product under test.  But that is completely up to the manual tester who is testing the automated test.  One cannot trust the expected result of an automated test until one has finished testing the automated test.

Thus, the tester of the automated test will need to evaluate the test somehow and declare it to be a good test.  They may be able to do this several ways:

• Manipulate the product to see if the test both passes and fails under the right conditions.
• Execute the automated test by itself then as part of the test suite to determine if the setup/teardown routines adapt sufficiently.
• Read the automated test’s code (e.g., does the Assert check the intended observation correctly?).
• Manually test the same thing the automated test checks.
• Manipulate the product such that the test cannot evaluate its check. Does the test resolve as “Inconclusive”?

It would be nice to have the luxury of time/resources to test the automated checks thoroughly but in the end, I suspect we will have to draw the line somewhere and trust the test automation engineer’s test of their check.  In the meantime, we’ll see where this gets us.

My tech department had a little internal one day idea conference this week.  I wanted to keep it from being a programmer fest so I pitched Crowdsource Testing as a topic.  Although I knew nothing about it, I had just seen James Whittaker’s STARwest Keynote, and the notion of having non-testers test my product had been bouncing around my brain.

With a buzzword like “crowdsource” I guess I shouldn’t be surprised they picked it.  That and maybe I was one of their only submissions.

Due to several personal conflicts I had little time to research, but I did spend about 12 hours alone with my brain in a long car ride.  During that time I came up with a few ideas, or at least built on other people’s.

• Would you hang this on your wall?
  
  
  
  …only a handful of people raised their hands. But this is a Jackson Pollock!  Worth millions! It’s quality art!  Maybe software has something in common with art.
• Two testing challenges that can probably not be solved by skilled in-house testers are:
• Quality is subjective. (I tip my hat to Michael Bolton and Jerry Weinberg)
• There are an infinite amount of tests to execute. (I tip my hat to James Bach and Robert Sabourin)
• There is a difference between acting like a user and using software. (I tip my hat to James Whittaker).  The only way to truly measure software quality is to use the software, as a user.  How else can you hit just the right tests?
• Walking through the various test levels: Skilled testers act.  Team testers act (this is when non-testers on your product team help you test).  User acceptance testing is acting (they are still pretending to do some work).  Dogfooders…ah yes!  Dogfooders are finally not acting.  This is the first level to cross the boundary between pretending and acting.  Why not stop here?  It’s still weak in the infinite tests area.
• The next level is crowdsource testing.  Or is it crowdsourced testing?  I see both used frequently.  I can think of three ways to implement crowdsource testing:
• Use a crowdsource testing company.
• Do a limited beta release.
• Do a public beta release.
• Is crowdsource testing acting or using? If you outsource your crowdsource testing to a company (e.g., uTest, Mob4hire, TopCoder), now you’re hiring actors again.  However, if you do a limited or public beta release, you’re asking people to use your product.  See the difference?
• Beta…is a magic word.  It’s like stamping “draft” on the report you’re about to give your boss.  It also says, this product is alive!  New!  Exciting!  Maybe cutting edge.  It’s fixable still.  We won’t hang you out to dry.  It can only get better!
• Who should facilitate crowdsource testing?  The skilled in-house tester. Maybe crowdsource testing is just another tool in their toolbox.  If they spent this much time executing their own tests…
  
  
  
  …maybe instead, they could spend this much time (blue) executing their own tests, and this much time (yellow) collecting test information from their crowd…
  
  
  
• How do we get feedback from the crowd?  (from least cool to coolest) The crowd calls our tech support team, virtual feedback meetings, we go find the feedback on twitter etc., the crowd provides feedback via tightly integrated feedback options on the product’s UI, we programmatically collect it in a stealthy way.
• How do we enlist the crowd?  Via creative spin and marketing (e.g., follow Hipster’s example).
• Which products do we crowdsource test?  NOT the mission critical ones.  We crowdsource the trivial products like entertainment or social products.
• How far can we take this?  Far.  We can ask the crowd to build the product too.
• What can we do now?
• BA’s: determine beta candidate features.
• Testers: spend less time testing trivial features.  Outsource those to the crowd.
• Programmers: Decouple trivial features from critical features.
• UX Designers: Help users find beta features and provide feedback.
• Managers: Encourage more features to get released earlier.
• Executives: Hire a crowdsource test czar.

There it is in a nutshell.  Of course, the live presentation includes a bunch of fun extra stuff like the 1978 Alpo commercial that may have inspired the term “dogfooding”, as well as other theories and silliness.