If your AUT has logic that checks for null values, make sure removed values get set back to null. If they get set to blank, you may have a nice crunchy bug.

Here are the pseudo steps for the test case using a generic order form example. But you can use anything in your AUT that programmatically makes a decision based on whether or not some value exists.

Feature: Orders with missing line item quantities can no longer be submitted.

1. Create a new line item on an order. Do not specify a quantity. Crack open the DB and find the record with the unpopulated quantity. Let’s assume the quantity has a null value.
2. Populate the quantity via the AUT .
3. Clear the quantity via the AUT. Look at the DB again. Does the quantity have a blank or null value?
   
   Expected Results: Quantity has a null value.

If your dev only validates against null quantity values, your user just ordered nothing...yikes!

We’ve been interviewing to fill a couple QA positions on our team. My favorite part of each interview is my “test this light switch” exercise. It reveals interesting skills about each test candidate.

I point to the light switch in the room and say “test this light switch”. Here is a sampling of how candidates have responded:

• some asked if there are any requirements (this is a great way to start!)
• some just start testing with lots of assumptions (no so great)
• one candidate smiled and thought I was kidding. After asking lots of questions to prime him, he stared uncomfortably at the light switch and offered me close to nothing (embarrassing for both of us)
• one candidate walked up to the light switch and began testing it as she walked me through her thought process. After some solid high level tests, she wanted to see electrical schematics for the building and asked me all kinds of questions about emergency backup power, how many amps the room’s lights draw, and what else was on that circuit. She wanted to remove the trim plate to check the wiring for electrical code standards. She asked if the room’s lights could be controlled by a master switch somewhere else or an energy saver timer for off-hours. (these types of questions/tests make her a good fit for my team because my AUT’s weakest test area is its integration with other systems)
• one candidate was good at coming up with cosmetic and usability tests (e.g., Is the switch labeled well? Can I reach it from the doorway when the room is dark? Does the trim plate match the room’s trim in color and style?)…not so important for my AUT but good tests for others perhaps.
• one candidate went right for stress tests. He flipped the lights on/off as quickly as he could. He tried to force the switch to stay in the halfway-off-halfway-on position to see if it sparked or flickered the lights.
  

More was revealed about the confidence of each candidate, their creativity, how technical their brain was, how quickly their mind worked, their persistence, and finally how interested they were in determining their mission and what I thought was important to know.

Most bug reports include Severity and Priority. On my team, everyone is interested in Priority (because it affects their work load). Severity is all but ignored. I propose that testers stop assigning Priority and only assign Severity.

Priority is not up to the tester. It is usually a business decision. It wastes the tester’s time to consider Priority, takes this important decision away from someone more suited to make it, and finally, it may misguide workflow.

Bugs without Priority have to be read and understood by the customer team (so they can assign priority themselves). This is a good thing.

What about blocking bugs, you ask?

Some bugs are important to fix because they block testing. These bugs are best identified as blocking bugs by testers. They can be flagged as “Blocking Bugs” using an attribute independent of the Priority field. Think about it…if BlockingBugA is blocking testing that is less important than other, non-blocked testing, perhaps BlockingBugA only deserves a low priority.

Tell me where I’m wrong.

I recently read about 15 resumes for tester positions on my team. None of them told us anything about how well the candidate can test.

Here is what I saw:

• All candidates list a ton of ”technologies” they are familiar with (e.g., .Net, Unix, SQL, XML, MS Office)
• They also list a bunch of off-the-shelf testing tools (e.g., TestDirector, LoadRunner, QuickTest Pro, SilkTest, BugZilla)

…So far I don’t know anything about how well they can test.

• All candidates string together a bunch of test buzz words…something like, “I know white box testing, gray box testing, black box testing, stress testing, load testing, functional testing, integration testing, sanity testing, smoke testing, regression testing, manual testing, automated testing, user acceptance testing, etc.”
  

…as if I would be thinking, “yes, but do you know Glass Box Testing? That’s really what we’re looking for.”

• Some candidates will say something like “I wrote a 50-page test plan”, or “I’m responsible for testing an enterprise application used by 1000 users”

…okay, so how well can you test? To be fair, this is a difficult skill to convey in a resume and perhaps I am just not good at reading between the lines and determining which candidates would thrive as testers on my team. However, the candidate would have probably gotten an instant interview if they had included any of these:

• My approach to testing is as follows…
• See my software testing blog for my opinions on how to test.
• My favorite testing books and blogs are…
• I enjoy testing because…
  

Sigh. I guess the modern resume has not advanced far enough to reflect candidate traits to said extent. That's why the interview questions will be so important. I get to participate in my first interview tomorrow and I have come up with a list of fun questions/activities to help me see how well the candidate tests. One of them will be to "Test that light switch over there on the wall completely." (if the candidates are cool enough to read my blog, they'll have a head start).

What are your favorite tester interview questions?

'Twas the night before prod release, when all through the build,
Not a unit test was failing, the developers were thrilled;

The release notes were emailed to users with care,
In hopes that new features, soon would be there;

The BA was nestled all snug in her chair,
With visions of magnitude ready to share;

And I on my QA box, trying not to be stressed,
Had just settled down for a last minute’s test;

When during my test there arose such a clatter,
I opened the error log to see what was the matter;

And what to my wondering eyes should appear,
But an unhandled fault, with its wording unclear;

When I showed it to dev, and he gave me a shrug,
I knew in a moment it must be a bug;

More rapid than eagles, dev’s cursing it came,
And he shouted at testers, and called us by name;

“Now, Jacobson! Now, Zacek! Now, Whiteside and Surapaneni!
On, Cagle! On, Addepalli, on Chang and Damidi!

Stop finding bugs in my web service call!
Now dash away! dash away! dash away all!"

And then, in a twinkling, I heard from the hall,
The tester who showed me, scripts can’t test it all;

As I rejected the build, and was turning around,
Into my cube, James Bach came with a bound;

He was dressed really plain, in a baseball-like cap,
And he patted my back for exploring my app;

He had a big white board and a little round belly,
That shook when he diagrammed like a bowlful of jelly;

He was chubby and plump, a right jolly old elf,
And he laughed when he saw RST on my shelf;

Then he spoke about testing, going straight to his work,
And attempted traspection, though he seemed like a jerk;

His eyes -- how they twinkled! his dice games, how merry!
He questioned and quizzed me and that part was scary!

He told me of lessons he taught at STARWEST,
Made an SBT charter and then told me to test;

Then I heard him exclaim, ere he walked out of sight
“Happy testing to all! ...just remember I’m right!”

Yesterday we had to roll back some production code to its previous build. It was essentially a performance problem. The new bits kept timing out for the types of requests users were making from the system.

Our poor, battered support team had to send out a user notice that said something like “Sorry users, for all the timeouts. We have just rolled back the code until we can fix the problems. Oh and by the way, that means you’ll no longer have these new features we just gave you.”

Shortly thereafter, the head of my department, indirectly asked me why these issues were not caught in testing. …This is the stuff nightmares are made of. I felt like throwing up, resigning, or crying. Once my heart started beating again, I realized I had logged a bug for these problems. Exhale now.

The bug was fairly accurate, stating the users would get timeout errors if they attempted anything beyond a specific threshold. During a bug meeting, it was down-graded in priority and the team decided to move forward with the release. In hindsight, there was a bit of group-think going on. We were so anxious to get the release out, we figured the users would put up with the performance problems. Boy were we wrong.

Being a tester is scary. At any given moment, all hell may break loose in production. And when it does, asking why the tester didn’t find it is, of course, the fairest of questions.

I brainstormed with my report dev and we came up with two valuable JART expansions.

The first is coded and has already found two bugs (and saved me from days of bordom). Comprehensive data checking in the report output is complicated but what about checking for data that is missing? My reports display the string “[Not Found]” when data fields get mapped incorrectly or other datasource problems occur. It only took about an hour to write a function that scans all report results looking for the string. Since JART already exported each report result to a text file, I simply automated Notepad’s Find functionalty to look for specific strings passed into my SearchReportResultsForString function. This was way easy. Now JART checks about 2240 distinct data columns across 113 reports for this issue.

I use said function to also look for the “No Data Found” string. If I find this string it means the report results returned but other than the cover page, no data matching the search criteria exists. This check gets reported as a “Warning” or inconclusive result. I use it to help me determine when I need to adjust my filter criteria parameters on each report.

The second expansion is to build a routine that saves off the report results of each iteration as “Iteration N Baselines”. Then, using the same report filter criteria and same data store, save off the next iteration’s report results as “Iteration N+1 Baselines”. Once JART has at least two iterations of baselines, JART will compare the files from each baseline to ensure nothing has changed. If I can pull this off, it will be HUGE. I’m expecting it to only support about 75% of my reports. The other 25% is time sensitive data that may not have a way to remain frozen in time.

Yesterday, a dev pointed out that the reports display the report execution date (i.e., the current date) in the header and footer. Thus, they should never match. Ooops! I think I can still work around it, but it does complicate things a bit.