As a tester, while striving for the impossible goal of perfect software, I sometimes feel stupid. How valuable am I to the team? Do I really have any hard skills different than the next guy? Am I a testing failure?

I feel stupid when…

• production bugs have to be patched (the kind I should have caught).
• devs talk about code or architecture I don’t understand.
• non-testers log bugs.
• I have to execute brainless tests that the guy on the street could execute.
• I can’t remember if I tested a certain scenario and my executed test documentation is incomplete.
• the team celebrates individual dev accomplishments for feature sets and QA is not recognized.
• my bug is rejected by dev for a legitimate reason.
• I read a software testing blog post about some tester with 95% of her tests automated.

As a fellow tester, maybe you have felt stupid at times too. Feeling stupid is not fun and eventually will lead to disliking your job. I guess there are two solutions; 1.) find a new job or 2.) try not to feel stupid.

I talk my way out of feeling stupid as a tester the same way I do outside of work during conversations with doctors, physicists, CEOs or other potentially intimidating experts of some field. I remember that everyone is an expert at something…just something different. In the examination room, the doctor may be the expert at prescribing the treatment, but put the doctor and me at the bottom of a 300-foot-deep pit in a wet cave, and suddenly the doctor is asking me for help (I’m a caver).

When it comes to testing, we don’t know the same things the developers or BAs know but we shouldn’t feel stupid about it. It doesn’t mean we should stop learning, we just need to put things in perspective instead of feeling inadequate. Faking your knowledge is way worse than saying “I don’t know”.

Don’t second guess your skills as a tester.

In a future post, I'll tell you when I feel awseome as a tester.

“Added a validation in the service codes to make sure either "DSP" or "Already in GE" is true for Accession.”


Do your devs write bug resolution comments? If not, ask for them. All bug tracking systems have a spot for devs to write notes about what they did to fix the bug. It’s tough to get devs to write their resolution but it’s well worth the pain.

When you verify fixed bugs you should be asking yourself what regression testing is necessary; what did this bug fix break? If you don’t know how the dev fixed it, you don’t really know what could have broken as a result. Sometimes, devs are cool enough to provide tips of where other bugs may be lurking…


“This problem is bigger than the shoot value hiding. In fact, the entire preCat is not reloading after a save which is hiding other bugs (UI refresh and service layer issues.)”


Some of my devs are terrible at writing resolution comments. Sometimes I have no idea what they did to fix it. My emotions tell me to Reopen the bug. But after listening to my brain, and asking the dev what they did, I usually discover an unexpected fix that still solves the problem. Dev comments would have been nice.

You may also discover your devs share some of your angst when dealing with scatterbrain stakeholders, as evident in comments I’ve recently seen such as the following:


“UI now allows multiple preCats and DSP check box, even though earlier requirements conflict with both of these.”

“Yet another requirements change. The need dsp check box has always been there. It was expressed that it should only be visible when there IS and accession, but I guess we are changing that.”


Poor dev guy. I feel your pain. I miss the good old pre-agile days too.

One of my new AUTs has stakeholders who are obsessed with cosmetics. Despite having an AUT full of business processes gaps and showstopper bugs, during stakeholder meetings their first priority is to rattle off a list of all the cosmetic things they want changed. For example:

• titles should left align
• read-only fields should be borderless
• certain fields should be bigger/smaller
• less white space
• no scroll bars
• don’t like text color or font
• buttons should be same width regardless of their names

Theoretically, Agile is supposed to address this kind of perpetual scope creep. But I hate it because even after listening to the stakeholders, it still becomes awkward for the dev to code and the tester to verify.

Something truly lacking in custom in-house (not shrink-wrap) apps is the ability for users to customize UIs until they’ve bored themselves to death. I’ve never been one to bother to “change skins” on my apps or even to change my desktop background. But cosmetics is a major concern for some users. Forcing me to test someone’s notion of what they think looks good on a UI is not interesting to me, as a tester. Let’s write software that lets users make their own cosmetic changes on their own time. I’ll test its engine. That sounds interesting.

JART found a bug this morning. But it wasn't in my AUT.

JART had been happily smoke testing our pre-production environment this morning for an hour. I was eagerly awaiting the results for a group of anxious managers. After seeing QTP’s auto-generated test results consistently for 144 previous test runs, QTP suddenly decided to give me this instead of the results:



I didn’t change any versions of any software on this box, of course. After waiting another hour while JART repeated all the tests, the next results file was fine. …Annoying.

We’ve been spinning our wheels investigating a prod bug that corrupted some data yesterday. Once we cracked it, we realized the bug had been found and fixed more than a year ago. …Depressing. My first thought? Why didn’t I catch this when it broke?

Perfecting regression testing is a seemingly impossible task. Some of you are thinking, “just use test automation...that's what they said in that Agile webinar I just attended”. If my team had the bandwidth to automate every test case and bug we conceived of, the automation stack would require an even larger team to maintain. And it would need its own dedicated test team to ensure it properly executed all tests.

It’s even more frustrating if you remove the option of automated regression testing. Each test cycle would need to increase by the same amount of time it took to test the new features in the last build, right? So if iteration 4 is a two week iteration, and I spend a week testing new features. That means, iteration 5 needs to be a three week iteration; I’ll need that extra week so I can run all the iteration 4 tests again. They’ll give me eight weeks to test iteration 10, right?

Wrong? You mean I have the same amount of test time each iteration, even though the amount of tests I have to execute are significantly increasing? This is a reality that somehow we all deal with.

Obviously, none of us have "perfect" regression testing. The goal is probably "good enough" but the notion of improving it is probably driving you crazy, as it is me. This topic is glossed over so much, I wonder how many testers have an effective strategy.

What is your regression test strategy?

During a recent phone call with Adam White, he said something I can’t stop thinking about. Adam recently took his test team through an exercise to track how much of their day was actually spent testing. The results were scary. Then Adam said it, “If you’re not operating the product, you’re not testing”…I can’t get that out of my head.

Each day I find myself falling behind on tests I wanted to execute. Then I typically fulfill one of the following obligations:

• Requirement walkthrough meetings
• System design meetings
• Write test cases
• Test case review meetings
• Creating test data and preparing for a test
• Troubleshooting build issues
• Writing detailed bug reports
• Bug review meetings
• Meetings with devs b/c tester doesn’t understand implementation
• Meetings with devs b/c developer doesn’t understand bug
• Meetings with business b/c requirement gaps are discovered
• Collecting and report quality metrics
• Managing official tickets to push bits between various environments and satisfy SOX compliancy
• Update status and other values of tested requirements, test case, and bug entities
• Attempt to capture executed exploratory tests
• Responding to important emails (arriving multiple per minute)

Nope, I don’t see "testing" anywhere in that list. Testing is what I attempt to squeeze in everyday between this other stuff. I want to change this. Any suggestions? Can anyone relate?

If your test automation doesn’t verify anything useful, it is essentially worthless. First, there are some basic tests I decided to programmatically verify with JART. These are tests that often fail during manual testing.

• Can I access the report app for a given environment?
• Does a working link to each report exist?
• Does each report’s filter page display?
• Does each report’s filter page display the expected filter controls I care about?
  
  

The above can be verified without even executing any reports. Piece of cake!

Next, I need to verify each report executes with some flavor of expected results. Now I’m bumping it up a notch. There are an unlimited amount of results I can expect for each report and these all require knowledge or control of complex reportable business data. This also means I have to examine the report results, right? My AUT uses MS ActiveReports and displays results in an object not recognized by QuickTest Pro. According to the good folks at SQA Forums, the standard way to extract info from the results is to use the AcrobatReaderSDK, which I don’t have. The workaround, which I use, is to install a free app that converts pdf files to text files. I wrote a little procedure to save my report results as pdf files, then convert them to text files, which I can examine programmatically via QuickTest Pro. So far, it works great. The only disadvantage is the extra 5 seconds per report conversion.

So what am I examining in the report results for my verifications? So far, I am just looking at each report’s cover page, which displays the specified filter criteria returned, along with its filter name (e.g., “Start Date = 3/20/2006”). If it returns as expected, I have verified the AUT’s UI is passing the correct filter parameters to the report services. This has been a significant failure point in the past, which is no surprise because the UI devs and service devs are poor communicators with each other.

Currently, JART verifys 59 Reports and up to 9 filters on each. It takes about 1 hour to complete. JART is ready to perform my next sanity test when we go live. So far I have put in about 24 hours of JART development.

I’ll discuss the simple error handling JART uses in a future post.

Note: The failures from the test run result summary above were the results of QuickTest Pro not finding the text file containing the converted report results. I couldn’t repro this JART error but now I may have to invest time researching the fluke and determining how to handle it. This is time not spent testing my AUT.