Have you ever been to a restaurant with a kitchen window?  Well, sometimes it may be best not to show the customers what the chicken looks like until it is served.

A tester on my team has something similar to a kitchen window for his automated checks; the results are available to the project team.

Here’s the rub: 

His new automated check scenario batches are likely to result in…say, a 10% failure rate (e.g., 17 failed checks).  These failures are typically bugs in the automated checks, not the product under test.  Note: this project only has one environment at this point.

When a good curious product owner looks through the kitchen window and sees 17 Failures, it can be scary!  Are these product bugs?  Are these temporary failures? 

Here’s how we solved this little problem:

• Most of the time, we close the curtains.  The tester writes new automated checks in a sandbox, debugs them, then merges them to a public list.
• When the curtains are open, we are careful to explain, “this chicken is not yet ready to eat”.  We added an “Ignore” attribute to the checks so they can be filtered from sight.

BDD/ATDD is all the rage these days.  The cynic in me took a cheap shot at it here.  But the optimist in me really REALLY thinks it sounds cool.  So I set off to try it….and failed twice.

First Fail:

I’m not involved in many greenfield projects so I attempted to convince my fellow colleagues to try BDD with their greenfield project.  I started with the usual emails, chock full of persuasive BDD links to videos and white papers.  Weeks went by with no response.  Next, we scheduled a meeting so I could pitch the idea to said project team.  To prepare, I read Markus Gartner’s “ATDD By Example” book, took my tester buddy, Alex Kell, out to lunch for an ATDD Q & A, and read a bunch of blog posts.

I opened my big meeting by saying, “You guys have an opportunity to do something extraordinary, something that has not been done in this company.  You can be leaders.”  (It played out nicely in my head before hand) I asked the project team to try BDD, I proposed it as a 4 to 6 month pilot, attempted to explain the value it would bring to the team, and suggested roles and responsibilities to start with. 

Throughout the meeting I encountered reserved reluctance.  At its low point, the discussion morphed into whether or not the team wanted to bother writing any unit tests (regardless of BDD).  At its high point, the team agreed to do their own research and try BDD on their prototype product.  The team’s tester walked away with my “ATDD By Example” book and I walked away with my fingers crossed.

Weeks later, I was matter-of-factly told by someone loosely connected to said project team, “Oh, they decided not to try BDD because the team is too new and the project is too important”.  It’s that second part that always makes me shake my head.

Second Fail:

By golly I’m going to try it myself!

One of my project teams just started a small web-based spin-off product, a feedback form.  I don’t normally have the luxury of testing web products and it seemed simple enough so I set out to try BDD on my own.  I choose SpecFlow and spent several hours setting up all the extensions and NuGet packages I needed for BDD.  I got the sample Gherkin test written and executing and then my test manager job took over, flinging me all kinds of higher priority work.  Three weeks later, the feedback form product is approaching code complete and I realize it just passed me by.

…Sigh.

…are not always the full truth.  Is that hurting our craft? 

Last week, I attended the first Software Testing Club Atlanta Meetup.  It was organized by Claire Moss and graciously hosted by VersionOne.  The format was Lean Coffee, which was perfect for this meeting.

Photo by Claire Moss

I’m not going to blog about the discussion topics themselves.  Instead, I would like to blog about a familiar Testing Story pattern I noticed:

During the first 2 hours, it seemed to me, we were telling each other the testing stories we wanted to believe, the stories we wanted each other to believe.  We had to make first impressions and establish our personal expertise, I guess.  But during the 3rd hour, we started to tell more candid stories, about our testing struggles and dysfunctions.  I started hearing things like, “we know what we should be doing, we just can’t pull it off”.  People who, at first impression, seemed to have it all together, seemed a little less intimidating now.

When we attend conference talks, read blog posts, and socialize professionally, I think we are in a bubble of exaggerated success.  The same thing happens on Facebook, right?  And people fall into a trap: The more one uses Facebook, the more miserable one feels.  I’m probably guilty of spreading exaggerated success on this blog.  I’m sure it’s easier, certainly safer, to leave out the embarrassing bits.

That being said, I am going to post some of my recent testing failure stories on this blog in the near future.  See you soon.

My data warehouse project team is configuring one of our QA environments to be a dynamic read-only copy of production.  I’m salivating as I try to wrap my head around the testing possibilities.

We are taking about 10 transactional databases from one of our QA environments, and replacing them with 10 databases replicated from their production counterparts.  This means, when any of our users perform a transaction in production, said data change will be reflected in our QA environment instantly.

Expected Advantages:

• Excellent Soak Testing – We’ll be able to deploy a pre-production build of our product to our Prod-replicated-QA-environment and see how it handles actual production data updates.  This is huge because we have been unable to find some bugs until our product builds experience real live usage.
• Use real live user scenarios to drive tests – We have a suite of automated checks that invoke fake updates in our transactional data bases, then expect data warehouse updates within certain time spans.  The checks use fake updates.  Until now.  With the Prod-replicated-QA-environment, we are attempting to programmatically detect real live data updates via logging, and measure those against expected results.
• Comparing reports – A new flavor of automated checks is now possible.  With the Prod-replicated-QA-environment, we are attempting to use production report results as a golden master to compare to QA report results sitting on the pre-production QA build data warehouse.  Since the data warehouse data to support the reports should be the same, we can expect the report results to match.

Expected Challenges:

• The Prod-replicated-QA-environment will be read-only.  This means instead of creating fake user actions whenever we want, we will need to wait until they occur.  What if some don’t occur…within the soak test window?
• No more data comparing? - Comparing transactional data to data warehouse data has always been a bread and butter automated check we’ve performed.  These checks check data integrity and data loading.  Comparing a real live quickly changing source to a slowly updating target will be difficult at best.

I can’t imagine testing without multiple computers at my disposal.  You may want to hold on to your old, out of warranty computers if given the choice.  Five quick reasons:

• When Computer#1 hits an impediment such as an unrecoverable error, Computer#2 can start testing immediately as Computer#1 reboots.
• I can use both computers to simulate interesting multi-user tests.  Example: what if two users attempt to acquire locks at roughly the same time.
• I can kick off timely processes, staggered on 3 separate boxes, so as not to sit idle waiting.
• Different OS’s, browser versions, frameworks, and other software running in the background can be informally tested to narrow down variables.
• Computer#1 can support the administrative work of testing (e.g., documenting tests, bugs, emailing), while Computer#2 can stay clean and focus on operating the product under test. 

What is the relationship between these two objects?

How about these two?

This, I’m afraid, is how testers (myself included) often see software modules…like black boxes. Their relationships are hidden from us.  We know the programmer just changed something related to the seeds inside the orange, so we ask ourselves, “How could changing the seeds inside the orange affect the toaster?”.  Hmmmm.  “Well, it sure seems like it couldn’t”.  Then, after a deployment, we’re shocked to discover the toaster starts burning all the toast.

Why didn’t the programmer warn us?  Well, just because the programmer understands the innards of the orange, doesn’t mean they understand the innards of the toaster.  In fact, based on my experiences, if there is enough work to do on the orange, the orange programmer will happily take it over learning to code for the toaster.

So here we are, left with nothing better to do than regression test the toaster, the jointer, and the flash light, every time the orange changes.  No wonder we spend so much time regression testing.

In conclusion, maybe the more we learn about the architecture behind our software system, the fewer regression tests we’ll need to execute.  Instead, we’ll better focus our testing and make the invisible relationships visible for the rest of the team…before development even begins.

I’m trying to fill two technical tester positions.  It’s exhausting.  All the resumes are starting to look the same.  They all tout:

• Extensive knowledge of the SDLC.  Who cares?  I’ve been testing for 15 years and I’ve never encountered a situation where extensive knowledge of the SDLC has come in handy…I’m not even sure what it is.  Who is motivating this?  Are there that many Test Managers out there saying, “what we really need, is a tester who knows the SDLC”?
• Understanding of test automation tools like QTP?  “Understanding of”?
• Ability to map test cases to requirements in Quality Center.  It’s been 6 years since I’ve seen Quality Center but I don’t recall it being that difficult a task.
• Performed different types of tests like Functional, Regression, Smoke, UAT, White Box, Black Box, Grey Box, and End-to-end.  Darn, I was really looking for someone who could write “integration tests”.  Oh well.

One resume said:

• Extensive QA experience via hands-on testing?  Is there a way to gain experience without being hands-on?

Another said:

• Fixed tested bugs and coordinated with developers in release of bug fixes during every Sprint Run.  Hmmm.  Perhaps you should start by testing your resume verbiage.

During an interview I asked:

Me: According to your resume, you worked on a team using Test Driven Development.  Was it effective?
Candidate:  Oh yes.  At the end of Sprints, if the developers had time, they would write some unit tests for Stories we were about to release to production.

During another, I asked the following easy question:

Me: What are some attributes of a good bug report?
Candidate: Documenting the bug is the most important attribute.

Finally, after an interview with me, for a programming position, the candidate remarked, “That’s odd, I’ve never seen a man in a QA role.”.  It reminds me of a little post I made years ago that almost lost me some friends.

BTW - If you live in the Atlanta area, have excellent DB and SQL skills, and are capable of testing something without a UI, please drop me a note.  I may have an awesome job waiting for you.