Another of Robert Sabourin’s STPCon sessions I attended was “Deciding What Not to Test”.

Robert did not give me the magic, simple answer, I was hoping for. His main point was, it’s better to not test certain things because you decided not to, rather than because you ran out of time.

I agree but I’m not convinced this is practical. In order to decide what not to test, one must spend time determining lots of extra tests that will be thrown out. I don’t like that. If there are an infinite number of tests to execute, when do I stop coming up with them? Instead, I prefer to come up with them as I test within my given testing time. My stopping point then becomes the clock. I would like to think, as a good tester, I will come up with the best tests first. Get it?

Maybe “which tests should I NOT execute?” is the wrong question. Maybe the better question is “which tests should I execute now?”.

At any rate, it is feasible that a tester often finds themselves in a situation where they have too many tests to execute in the available time, er…time they are willing to work. When this situation arises, Robert suggests a few questions to help prioritize:

1.) What is the risk of failure?

2.) What is the consequence of failure?

3.) What is the value of success?

Here are my interpretations of Robert’s three questions:

1.) A tester can answer this. Does this dev usually create bugs with similar features? How complex is this feature? How detailed were the specs or how likely is it that the correct info was communicated to the dev?

2.) It should be answered from the stakeholder’s perspective. Although, a good tester can answer it as well. It all comes down to this question: will the company lose money?

3.) This one should also be answered from the stakeholder’s perspective. If this test passes, who cares? Will someone be relieved that the test passed?

So if you answer “high” to any of Robert’s three questions, I would say you had better execute the test.

Do you have any better advice on knowing what not to test? If so, please share!

Bob Galen’s STPCon session, entitled “Agile Testing within SCRUM”, had an interesting twist I did not expect. After a Scrum primer, Bob suggested that test teams can use a Scrum wrapper around their test activities, regardless of what the dev methodology may be.

In other words, even if you’re testing for one or more non-Scrum dev teams, you may still use Scrum to be a better test team. This is kind of a fun idea because I’ve been chomping at the bit to be part of a Scrum team. The idea is that your QA team hold the daily stand-up meetings, create a sprint backlog list, track sprint progress with a burndown chart, and end each sprint with a review meeting to reflect on sprint success/failure. You can add as many Scrum practices as you find valuable (e.g., invite project stateholders like devs/customers to prioritize sprint backlog items or attend daily meetings).

Wrapping QA practices with Scrum is actually not that difficult. For example, sprint backlog items can be bugs to retest, features to test, or test cases to write. Daily stand-up reports can be “Yesterday I tested 5 features and logged 16 bugs, today I will test these other features, and Bug13346 is blocking me from executing several tests.”



My QA team actually started holding Scrum meetings (see picture) about three months ago and it seems to help us stay more focused each day. What’s lacking is a formal sprint goal and means to track progress towards it. Bob Galen’s little session has convinced me it’s worth a try. At least to tide me over till all my devs implement Scrum!

Many of my notes from Hans Buwalda’s STPCon session are test design tips that can also apply to manual testing. One of my favorite tips was to remember to go beyond requirement-based testing. A good QA Manager should say “I know you have tested these ten requirements, now write me some tests that will break them”.

As testers, we should figure out what everyone else forgot about. These are the good tests. These are where we can shine and provide extra value to the team. One way to do this is to take a simple test and make it more aggressive.

Example Requirement: A user can edit ItemA.

Requirement-based Test: UserA opens ItemA in edit mode.

How can I make this test more aggressive? Let’s see what happens if:

• UserA and UserB both open ItemA in edit mode at the same time.
• UserA opens ItemA in edit mode when UserA already has ItemA in edit mode.
• UserA opens ItemA in edit mode, makes changes, goes home for the weekend, then attempts to save changes to ItemA on Monday.
• UserA opens ItemA in edit mode, loses network connectivity, then attempts to save ItemA.

What else can you think of?

Here are 10 things I heard Hans Buwalda say about test automation. I have thought about each of these to some extent and I would love to discuss any that you disagree with or embrace.

1. Stay away from “illegal checks”. Do not check something just because an automated test is there. Stay within the scope of the test, which should have been defined by the test designer. There should be a different test for each thing to check.
   
2. If bugs found by tests will not be fixed soon, do not keep executing those tests.
   
3. All Actions (AKA Keywords) with parameters should have defaults so the parameters do not have to be specified. This makes it easier for the test author to focus on the target.
   
4. Group automated tests into modules (i.e., chunks of tests that target a specific area). These tests should not be dependent on other modules.
   
5. Do not use copy and paste inside your automation code. Instead, be modular. Instead of copying low-level steps to another test, use a procedure that encapsulates those low-level tests. This prevents a maintenance nightmare when the AUT changes.
   
6. Remove all hard-coded wait times. Instead use active timing. Never tell a test to wait 2 seconds before moving on. If it takes 3 seconds your test breaks. Instead, test for the ready state using a loop.
   
7. Ask your devs to populate a specific object property (e.g., “accessibility name”) for you. If not, you will waste time determining how to map to each object.
   
8. Attempt to isolate UI tests such that one failed test will not fail all the other tests.
   
9. Something I didn’t expect Hans to say was not to worry about error handling in your automation framework. He says not to waste time on error handling because the tests should be written to “work”. At first I disagreed with this. But later I realized, in my own experiences with error handling, that it made me lazy. Often, instead of automating a solid test, I relied on error handling to keep my tests passing.
   
10. When Hans recommends an “Action-Based” test automation framework, IMO what he means is that it should support both low-level and high-level descriptions for the test steps. Hans considers “Keyword-Driven” automation to be low-level; the keywords being things like “click button”, “type text”, “select item”. Hans also considers Business-Template-Driven automation to be high-level; things like “submit order”, “edit order”. Action-Based test automation uses all of the above. One reason is to build a test library that can check low-level stuff first. If the low level stuff passes, then the high-level tests should execute.

What do you think about these?

I attended Robert Sabourin’s “To Infinity And Beyond” class at STPCon. This guy’s sessions are fun because of his sense of humor and pop culture references. He played us a screen captured video of someone sending an email and we were asked to write down as many boundaries as we could think of within the video. I had nearly 40 in five minutes but many testers got more.

The point was to start a conversation about boundary tests that go beyond the obvious. The first take-away I embraced was a simple mnemonic Robert gave us for thinking about boundaries. I usually hate mnemonics because most of them are only useful as PowerPoint slide filler. However, I’ve actually started using this one:

“Take AIM”

A – Application logic
I – Inputs
M – Memory storage

These are three ways to think about boundaries for test ideas. Most features you attempt to test will have these types of boundaries. The other cool thing about this mnemonic is it prioritizes your tests (i.e., test the application logic first, if you have it).

The typical example is testing a text box. But let's apply it to something different. If we apply Robert’s mnemonic to the spec, “The system must allow the user to drop up to four selected XYZ Objects on a target screen area at the same time”, we may get the following boundary tests.

Application logic (i.e., business rules)

• Drop one XYZ object.
• Drop four XYZ objects.
• Attempt to drop five XYZ objects.

Inputs

• Attempt to drop one ABC object.
• Attempt to drop a selection set of three XYZ objects and one ABC object.
• Can the objects get to the target screen area without being dropped? If so, try it.

Memory storage

• Attempt to drop 1,000,000 XYZ objects (there may be a memory constraint in just evaluating the objects)
• Refresh the screen. Are the four dropped XYZ objects still in the target screen area?
• Reopen the AUT. Are the four dropped XYZ objects still in the target screen area?

What other boundary tests can you think of and which AIM category do they fit into?

Sorry about the blog lapse. I just got back from STPCon and a 10-day vacation in the Pacific Northwest.

I’ll share my STPCon ideas and takeaways in future posts. But first I have to mention my STPCon highlight.

During the conference, keynote speaker James Bach invited me to participate in an experiment to see how quickly testers can learn testing techniques. He approached me before lunch and I joined him and another tester, Michael Richardson, in the hotel lobby. James asked us each to play a different game with him (simultaneously). Michael played the “Dice Game”, in which Michael had to roll random dice of Michael’s choosing, from several different dice styles, to determine a pattern that James made up.

I played a game much like “20 Questions”, in which I had to explain what happened based on vague information I was given. I could ask as many Yes/No questions as I wanted. During the games, James coached us on different techniques. After about 45 minutes, Michael jumped in and bailed me out. Afterwards, James asked me to play again with a different premise. I was able to crack the second premise in less than 5 minutes. I would like to think I actually learned something from my first, painful, attempt.

These games share similarities to software testing because they require skillful elimination of infinite variables. Some of the techniques James suggested were:

• Focus then defocus. Sometimes you should target a specific detail. Other times, you should step back and think less narrow. For me, defocus was the most important approach.
• Forward then backward. Search for evidence to backup a theory. Then search for a theory based on evidence you have determined.
• Dumb questions can lead to interesting answers. Don’t be afraid to ask as many questions as you can, even if they are seemingly stupid questions.
• Flat lining. If you are getting nowhere with a technique, it’s time to stop. Try a different type of testing.

Later, James asked Michael to think up a challenging dice game pattern. James was able to solve it in about 30 seconds using only one die. Obviously, having played before, he knew it would eliminate most of the variables. I just used this exact idea today to understand a software behavior problem I initially thought complex.

After the stress of performing these challenges with James, we sat back and enjoyed a conversation I will not forget. James was more personable and less judgemental than I expected. Despite being a high school dropout, he is currently writing a book about learning (not about testing). I also thought it cool that during STPCon he headed across the street to the Boston Public Library and began reading from various history books. He was trying to determine why people with power would ever choose to give it up. I guess he’s got more on his mind than just testing.

For me, spending time with James was a huge privilege and I was grateful for the opportunity to discuss testing with him, as well as getting to know the person behind the celebrity.

Note: Michael Richardson was also an interesting guy. He once did QA for a fish company and was literally smacked in the face with a large fish due to poor testing.