After three days at CAST2011, I finally caught up on the #CAST2011 Twitter feed.  It was filled with great thoughts and moments from the conference, which reflects most of what I experienced.  There was only one thing missing; critical reaction.

In Michael Bolton’s thought provoking keynote, I was reminded of Jerry Weinberg’s famous tester definition, “A tester is someone who knows that things can be different".  Well, before posting on what I learned at CAST2011, I’ll take a moment to document four things that could have been different.

Here are some things I got tired of hearing at CAST2011.

  • Commercial test automation tools are the root of all evil.  Quick Test Pro (QTP) was the one that took the most heat (it always is).  Speakers liked to rattle off all the commercial test automation tools they could think of and throw them into a big book-burning-fire.  The reason I’m tired of this is I’ve had great success using QTP as a test automation tool and I didn’t use any of its record/playback features.  I’ve been using my QTP tests to run some 600 checks for the last 24 iterations and it has worked great.  I think any tool can suck when used in the wrong context.  These tools can also be effective in the right context.
  • Physical things are shiny and cool and new all over again. One presentation was about different colored stickies on a white board instead of organizing work items on a computer (you’ve heard that before).  One was about writing tests on different colored index cards.  Someone suggested using giant Lego blocks to track progress.  In each of these cases, one can see the complexity grow (e.g., let’s stick red things on blue things to indicate the blue things are blocked, one guy entered his index card tests into a spreadsheet so he could sort them).  Apparently Einstein used to leave piles of index cards all over his house to write his ideas down on.  I’m thinking maybe that was because Einstein didn’t have an iphone.  IMO, this obsession with using office supplies to organize complex work is silly.  This is why we invented computers after all.  Use software!
  • Down with PowerPoint!  It’s popular these days to be anti-PowerPoint and CAST2011 speakers jumped on that too.  Half the speakers I saw did not bother to use PowerPoint.  I think this is silly.  There is a reason PowerPoint grew to such popularity.  It works!  I would much rather see an organized presentation that someone took the time to prepare, rather than watching speakers fumble around through their file structure looking for pictures or videos to show, which is what I saw 3 or 4 times.  One speaker actually opened PowerPoint, mumbled something about hating it, then didn’t bother to use slideshow mode.  So we looked at his slides in design view.  PowerPoint presentations can suck, don’t get me wrong, but they can also be brilliant with a little creativity.  Just watch some TED talks.
  • Traditional scripting testers are wrong.  You know the ones, those testers who write exhaustive test details so a guy off the street can execute their tests.  Oh wait…maybe you don’t know the ones.  Much time was spent criticizing that approach.  I’m tired of it because I don’t really think those people are much of a threat these days.  I’ve never worked with one and they certainly don’t attend CAST.  Why spend time bashing them?

I’m not bitter.  I learned from and loved all the speakers.  Jon Bach and his brother, James, put on an excellent tester conference that I was extremely grateful to attend.  I was just surprised we couldn’t get beyond the above.

Positive posts to come. I promise.

BTW - Speaking of candor… to Jon Bach’s credit, he opened Day2 with a clever self-deprecating bug report addressing conference concerns he had collected on Day1.  Things like the name tag print being to small and the breakfast lacking protein.  Most of these issues were addressed and he even used PowerPoint to address them.  Go Jon!  I was very impressed.

In my experiences, 95% of the test cases we write are read and executed only by ourselves.  If we generally target ourselves as the audience, we should strive…

…to write the least possible to remember the test.

I like the term “test case fragment” for this.  I heard it in my Rapid Software Testing class.  On the 5% chance someone asks us about a particular test, we should be able to confidently translate our chicken scratches into a detailed test.  That’s my target.

If we agree with the above, couldn’t we improve our efficiency even more by coming up with some type of test case short hand? 

For example: 

  • Instead of “Expected Results”, I write… “E:”. 
  • Instead of writing statements like “the user who cancelled their order, blocks the user who logged in first”, I prefer to assign variables, “UserA blocks UserB”.
  • Instead of multiple steps, I prefer one step test cases.  To get there I make some shortcuts.
    • Rather than specifying how to get to the start condition (e.g., find existing data vs. create data), I prefer the flexibility of word “with” as in “With a  pending order”.  How the order becomes pending is not important for this test.
    • To quickly remember the spirit of the test, I prefer state-action-expected as in “With a pending order, delete the ordered product. E: user message indicates product no longer exists.”
  • Deliberately vague is good enough.  When I don’t have enough information to plug in state-action-expected, I capture the vague notion of the test, as in “Attempt to corrupt an order.”.  In that case I drop the “E:” because it is understood, right? Expected Result: corruption handled gracefully.

It may be a stretch to call these examples “shorthand”, but I think you get the idea.

What test case short hand do you use?

When we find a bug with repro steps we typically leave the deep investigation to our Progs and move on to the next test. But sometimes it’s fun to squeeze in a little extra investigation if time permits.

My service calls were not returning the expected results so my Prog showed me a cool little trick; how to capture the SQL being passed into my DB for specific service calls. This technique is otherwise known as performing a SQL trace. We ran a SQL trace and with a bit of filtering and some timing, we easily captured the SQL statements being triggered via the called service. We extracted the SQL from the trace and used it to perform direct DB calls. Said isolation helped us find the root problem; a bad join.

Ask your Progs or DBAs which tools they use to perform SQL Traces. We used SQL Profiler and SQL Server 2008 R2. Make sure they show you how to filter down the trace (e.g., only show TSQL, to a certain DB, sent from a specific trigger like a user or service) so you don’t have to sift through all the other transactions.

As testers, we typically open screens in our products and focus on the data that is presented to us.  Does it meet our expectations?  We are sometimes forced to be patient and wait for said data to present itself.  Don’t forget to step back and determine what happens if the data doesn’t get a chance to present itself.

Test Steps:

1. Open a screen or window on the UI.  This bug is easier to repro in places where lots of data needs to load on the screen or window.  Look for asynchronous data loading (e.g., the user is given control after some data presents itself, while the rest of the data is still being fetched)

2. Close the screen or window before the data has finished loading.  Be quick.  Be creative; if the screen you opened doesn’t close on command, try its parent.

Expected Results:  No errors are thrown.  The product handles it gracefully by killing the data load process and releasing the memory.

After reading my Testing for Bug Bucks & Breaking The Rules post, In2v asked, “I was wondering if your team went on with that game and how it turned out. Would you update us?”

Sure.  The Bug Bucks game/experiment failed miserably.  It sounded good on paper but in practice it was too awkward.  When testers found bugs, they didn’t have the heart to ask Progs for Bug Bucks.  Go figure…

We like to joke about testers and Progs being caught up in some kind of rivalry. But when we’re actually performing our jobs on serious work, we want each other to succeed.  Both tester and Prog had an understanding that nobody was at fault; they were both doing their jobs.  When I asked TesterA why he didn’t take any Bug Bucks from ProgA, I was told ProgA would have been in Bug Buck debt (because of all the bugs) but it didn’t seem fair because many of the bugs were team oversights.

We could have adjusted the rules but I could see it was too awkward.   And just before management was getting ready to order some Bug Buck goods for purchase, I suggested we pull the plug.

So now I have a stack of worthless Bug Bucks on my desk, and a recurring hope to come up with a more fun way to perform our jobs.  I’m open to suggestions…

From this post forward, I will attempt to use the term “Prog” to refer to programmers.

I read a lot of Michael Bolton and I agree, testers are developers too.  So are the BAs.  Testers, Programmers, and BAs all work together to develop the product.  We all work on a software development team. 

Before I understood the above, I used the word “Dev” as short hand for “developer” (meaning programmer).  Now everybody on my development team says “Dev” (to reference programmers).  It has been a struggle to change my team culture to get everyone to call them “programmers”.  I’m completely failing and almost ready to switch back to “Dev”, myself.  I don’t much like the word, “programmer”…too many syllables and letters.

Thus, I give it one last attempt.  “Prog” is perfect!  Please help me popularize this term. It’s clearly short for “programmer”, easy to spell, and fun to say.  It also reminds me of “frog”, which is fitting because some progs are like frogs.  They sit all day waiting for us to give them bugs.

We have a recurring conversation on both my project teams.  Some Testers, Programmers, BAs believe certain work items are “testable” while others are not.  For example, some testers believe a service is not “testable” until its UI component is complete.  I’m sure most readers of this blog would disagree.

A more extreme example of a work item, believed by some to not be “testable”, is a work item for Programmer_A to review Programmer_B’s code.  However, there are several ways to test that, right?

  • Ask Programmer_A if they reviewed Programmer_B’s code.  Did they find problems?  Did they make suggestions?  Did Programmer_B follow coding standards?
  • Attend the review session.
  • Install tracking software on Programmer_A’s PC that programmatically determines if said code was opened and navigated appropriately for a human to review.
  • Ask Programmer_B what feedback they received from Programmer_A.

IMO, everything is testable to some extent.  But that doesn’t mean everything should be tested.  These are two completely different things.  I don’t test everything I can test.  I test everything I should test.

I firmly believe a skilled tester should have the freedom to decide which things they will spend time testing and when.  In some cases it may make more sense to wait and test the service indirectly via the UI.  In some cases it may make sense to verify that a programmer code review has occurred.  But said decision should be made by the tester based on their available time and queue of other things to test.

We don’t need no stinkin’ “testable” flag.  Everything is testable.  Trust the tester.



Copyright 2006| Blogger Templates by GeckoandFly modified and converted to Blogger Beta by Blogcrowds.
No part of the content or the blog may be reproduced without prior written permission.