The users found a bug for something that was marked “PASSED” during the test cycle. I made this remark to my programmers and one of them said “Ha ha, testers just mark tests ‘passed’ without doing anything”.
He was joking but he raised a frightening thought. What if he were right? What if you gave testers a list of 100 tests and they just marked them PASSED without operating the product? How would you know? The sad truth is, 95 of those tests probably would have passed, had the product been operated, anyway. The other 5 could arguably have been tested under different conditions than those deemed to raise the bug. And this is what makes testing both so unsatisfying to some testers and so difficult to manage.
If you can get the same result doing nothing, why bother?
We have an on-going debate about working from home. Let a programmer work from home and at the end of the day you can see what code they checked in. You have something substantially more tangible than that of a tester (even if that tester takes detailed test notes). It’s not an issue of trust. It’s an issue of motivation.
I believe testers who are uninterested in their job can hide on test teams easier than other teams, especially if their managers don’t take the time to discuss the details of what they actually tested. They may be hiding because they’ve never gotten much appreciation for their work.
Having a conversation with a tester about what they tested can be as easy as asking them about a movie they just saw:
  • How did George Clooney do?
  • Was it slow paced?
  • Was the story easy to follow?
  • Any twists or unexpected events?
  • Was the cinematography to your liking?
If they didn’t pay much attention to the movie, you’ll know.

I’ve got a test automation engineer and a bunch of manual testers. The plan, of course, is to teach the manual testers to write automated tests were appropriate; a plan I have seen fail several times. I figured I would take a crack at it anyway. In the meantime, I’m trying to keep the automation engineer busy writing valuable tests.
Everybody knows, choosing the right tests to automate is tricky. But what I’ve really been struggling with is finding the right skillset to actually design (determine the detailed steps) of each test automation candidate.
It turns out, the test automation engineers suck at it, and so do the manual testers. Who knew?
You see, manual testers live in a dream world, where test setup can be customized on the fly, validation can be performed with whatever info the tester decides is important at that moment, and teardown can be skipped altogether. Try asking those manual testers to stub out some repeatable tests to automate. I suspect you will run into a problem.
The test community loves to poke fun at traditional, script-happy, manual testers. Some testers (myself included) talk about exploratory testing or test case fragments as being the cool way to test. They scoff at testers who use rigorous step-by-step test cases. Have you ever encountered said traditional testers? I certainly haven’t. I’m sure they exist somewhere. But I think most of them only exist in a myth we’ve created in order to feel like test innovators.
Why am I such a skeptic about these folks? The truth is, writing repeatable, detailed, step-by-step test cases is really really really hard. If you’ve attempted to automate end-to-end business facing test cases for a complex system, you’ll know exactly what I mean.
On the other side, the test automation engineers are bored with manual testing and obsessed with trying to achieve the near-impossible ROI goals imposed by management. They want the manual testers to write them detailed tests because they don’t have time to learn the AUT.
A tester who makes time for manual testing and test automation has the magic skillset necessary to be an effective test automation author. Each of us should strive to achieve this skillset.

Have you seen this?
Type “psr” in your Windows 7 run prompt.
Click “Start Record” and every input you perform will get described, along with screen shots, then packaged into an MHTML zip file. The file can be viewed in IE or just as raw XML (for more details like mouse click coordinates). The little “Add Comment” feature is a good way to embed Expected vs. Actual results.

I used Problem Steps Recorder (PSR) today, to add details to one of my bug reports. PSR may also work as a personal test tracking tool. IMO, we can do a better job of capturing the relevant info ourselves in most cases. I rarely capture my entire screen and normally don't bother capturing steps unrelated to issues. Nevertheless, it's a good tool for your toolbox.

We encountered the following fixed bug today. Call it Bug100.
“The word ‘Unknown’ is spelled incorrectly in the dialog the user sees when sending an error report.”
Bug100 was spun off the larger issue (i.e., why the error was thrown). Bug100 was logged by a BA after noticing the spelling error, and easily fixed by a developer.
Unfortunately, the developer could not determine an easy way to trigger the error message described by Bug100. After a few failed attempts by the tester, the tester and I had a brief discussion. We decided to rubber stamp Bug100 and spend our time elsewhere. “Rubber stamp” is the expression we use to describe situations where the tester does not really do any testing, but they still move the bug report to the “Tested” status so its fix can proceed to production. We make a note on the bug report that says nothing was tested.
Would you have bothered to test this bug fix? Use the voting buttons below.

Would you bother testing this fix?

I promised I would post about my Lightning Talk, Programmer Profiling, per my Our First Tester Lightning Round post.
I came up with the notion of programmer profiling after listening to the Intelligence Squared podcast called “U.S. AIRPORTS SHOULD USE RACIAL AND RELIGIOUS PROFILING”
The TSA is responsible for finding bombs among some 3 million people participating in 20 thousand US airline flights per day. The TSA takes heavy criticism from people who believe racial profiling is wrong. But the TSA also takes heavy criticism for the opposite, searching little old ladies and children. Some firmly believe racial and religious profiling is one approach that should be on the table. Based on prior terrorist attempts, searching old ladies may not be the best use of the TSA’s time.
I noticed some vague similarities between the TSA and software testers.
  • The TSA protects passengers by finding bombs among 3 million people.
  • I protect users by finding bugs among 3 million lines of code.
Then I realized I already practice my own form of profiling to determine which areas may need more of my test attention. I profile the programmers. Not just based on their prior code quality history, but also based on their current behaviors.
For example:
  • If I ask ProgrammerA how she tested something and she shows me a set of sound unit tests, and a little custom application she wrote to help her test better, I gain a certain level of confidence in her code.
  • On the other hand, if I ask ProgrammerB how she tested something and she shrugs and says “that’s your job”, I gain a different level of confidence in her code.
When the clock is ticking, and all other things are equal, where do you think I’ll focus my time?
As is the case with potential TSA racial profiling, it should only be used as one of many approaches to finding the problems. It should be balanced with other considerations. But perhaps it should be on the table.

I call this “Programmer Profiling” and I think testers should not be afraid to use it.

Data warehouse (DW) testing is a far cry from functional testing. As testers, we need to let the team know if the DW dimension, fact, and bridge tables are getting the right data from all the source databases, storing it in such a way as to allow users to build reports, and keeping it current.
We are using automated tests as one of our test approaches. Contrary to popular belief, we found DW testing to be an excellent place to build and reuse automated tests. We came up with a simple automated test template that is allowing testers who have never done test automation to build up a pretty sweet library of automated tests.
Here's how it works:
Our automation engineer built us a test with two main parameters we pass in; a SQL statement for source data and a SQL statement for target data. The test compares the two record sets, including all data values and asserts they are equal. As the programmers build the various DW dimension, fact, and bridge tables, the testers copy and paste said automated test, and swap out the SQL with their SQL for the current test subject.
One of our most important tests is to rebuild the programmer's SQL ourselves, which is a manual process. If the tester's datasets don't match the DW, there may be a problem. This is where most of the defects are found. As we manually build our SQL, we plug it into an automated test that actually compares every record and column data value on millions of rows, and tells us where the differences are. The byproduct is an automated regression test.
Sure, rebuilding much of what the programmers did is challenging. We've had to learn case statements, cross DB joins, data type casting, and keep our heads straight while working through queries that include 30 or more joins. But normally it's fun and easy to get help from Google or our programmers.
We've been getting one or more automated tests completed for each dimension, fact, or bridge table; sometimes as early as in the development environment. When the DW is deployed to various environments, we normally just execute our automated tests and interpret the results. It's quite beautiful.
I’ll blog about some of the other tests we execute. There appears to be a shortage of pragmatic DW test ideas out there.

We did our first tester Lightning Round yesterday, which consisted of ten testers, who each had five minutes to make their point to a room full of other testers. To keep it fun, and I think, in the tradition of typical Lighting Rounds, we displayed a digital clock on the wall to track each speaker’s time. When a speaker used more than their five minutes, we threw squishy balls at them to make them stop.

The point of a Lightning Round is to give the audience one specific action item, without them having to endure a one hour lecture. This also gives the audience a variety of topics in a short time span. The advantage for the speakers is the short preparation time required to speak for five minutes.

Yesterday’s Lightning Round was an experimental version of something we’ve been doing for the last couple of years at my company. Testers from various projects and departments meet to exchange ideas. But normally we meet to attend an hour-long presentation about a single testing topic. Yesterday’s feedback indicated Lightning Rounds were way better to attend than the usual hour-long-single-topic session. Although, several testers said five minutes was too short. Some suggested five speakers with ten minutes each.

Personally, I think it was the five minute limit that kept the energy high and kept the audience from getting bored. But I’m willing to compromise. The larger problem, however, is getting testers to volunteer for these. I had to talk five of the speakers into doing it just to get my ten. Apparently, there are few testers willing to share their ideas with any conviction.

Anyway, the topics in yesterday’s section were the following:

  • Stop Writing Test Cases
  • Regression Testing Importance – Fact or Fiction?
  • How SOX Changed Our QA Process
  • What Programmer Profiling Taught Me About Choosing the Best Tests
  • Jing, A Favorite Test Tool
  • How to Increase Your Focus During Testing
  • Using Automation to Generate Large Amounts of Data
  • Why It’s Better to Have More Short Test Scripts Than Fewer Long Test Scripts
  • How Usability Testing is Making a Comeback with the Surge of Human-Computer Interaction Engineering
  • Smoke Testing vs. Sanity Testing
Mine was the Programmer Profiling topic. A respected colleague told me he didn’t care for it. Maybe I’ll blog about it later.



Copyright 2006| Blogger Templates by GeckoandFly modified and converted to Blogger Beta by Blogcrowds.
No part of the content or the blog may be reproduced without prior written permission.