Warning: This is mostly a narcissistic post that will add little value to the testing community.

I’ve been pretty depressed about my proposal not getting picked for Let’s Test 2014.  Each of my proposals have been picked for STPCon and STAR over the past three years; I guess I was getting cocky.  I put all my eggs in one basket and only proposed to Let’s Test.  My wife and I were planning to make a vacation out of it…our first trip to Scandinavia together.

Despite my rejection, my VP graciously offered to send me as an attendant but I wallowed in my own self pity and turned her down.  In fact, I decided not to attend any test conferences in 2014.  Pretty bitter, huh?

I know I could have pulled off a kick-ass talk with the fairly original and edgy topic I submitted.  I dropped names.  I got referrals from the right people.  My topic fit the conference theme perfectly, IMO.  So why didn’t I make the cut?

The Let’s Test program chairs have not responded to my request for “what I could have done differently to get picked”.  Lee Copeland, the STAR program chair was always helpful in that respect.  But I don’t blame the Let’s Test program chairs.  Apparently program chairs have an exhausting job and they get requests for feedback from hundreds of rejected speakers.

Fortunately, my mentor and friend, Michael Bolton read my proposal and gave me some good honest feedback on why I didn’t get picked.  He summarized his feedback into three points which I’ll paraphrase:

1. A successful pitch to Let’s Test involves positioning your talk right in the strike zone of an experience report.  You seemed to leave out the teensy, weensy little detail that you’re an N-year test manager at Turner, and that you’re telling a story about that here.
2. Apropos of that, tell us about the story that you’re going to tell.  You’ve got a bunch of points listed out, but they seem disjointed and the through line isn’t clear to me.  For example, what does the second point have to do with the first?  The fourth with the third?
3. Drop the dopey idea of “learning objectives”, which is far less important at Let’s Test than it may be at other conferences.

Bolton also directed me to his tips on writing a killer conference proposal, which make my How To Speak At a Testing Conference look amateur at best.

So there it is.  One of my big testing-related failure stories.  Wish me luck next year when it give it another go, for Let’s Test 2015…man that seems a long ways off.

…are not always the full truth.  Is that hurting our craft? 

Last week, I attended the first Software Testing Club Atlanta Meetup.  It was organized by Claire Moss and graciously hosted by VersionOne.  The format was Lean Coffee, which was perfect for this meeting.

Photo by Claire Moss

I’m not going to blog about the discussion topics themselves.  Instead, I would like to blog about a familiar Testing Story pattern I noticed:

During the first 2 hours, it seemed to me, we were telling each other the testing stories we wanted to believe, the stories we wanted each other to believe.  We had to make first impressions and establish our personal expertise, I guess.  But during the 3rd hour, we started to tell more candid stories, about our testing struggles and dysfunctions.  I started hearing things like, “we know what we should be doing, we just can’t pull it off”.  People who, at first impression, seemed to have it all together, seemed a little less intimidating now.

When we attend conference talks, read blog posts, and socialize professionally, I think we are in a bubble of exaggerated success.  The same thing happens on Facebook, right?  And people fall into a trap: The more one uses Facebook, the more miserable one feels.  I’m probably guilty of spreading exaggerated success on this blog.  I’m sure it’s easier, certainly safer, to leave out the embarrassing bits.

That being said, I am going to post some of my recent testing failure stories on this blog in the near future.  See you soon.

ATDD Sans Automated Tests.  Why Not?

Every time I hear about Acceptance Test Driven Development (ATDD), it’s always implied that the acceptance tests are automated.  What’s up with that?  After all, it’s not called Automated Acceptance Test Driven Development (AATDD).  It seems to me, that ATDD without automated tests, might be a better option for some teams.

This didn’t occur to me until I had the following conversation with the Agile consultant leading the discussion at the ATDD 2013 STAReast discussion lunch table.  After a discussion about ATDD tools and several other dependencies to automating acceptance tests, our conversation went something like this:

Agile Consultant: ATDD has several advantages.  While writing the acceptance tests as a team, we better understand the Story.  Then, we’ll run the tests before the product code exists and we’ll expect the tests to fail.  Then we’ll write the product code to make the tests pass.  And one of the main advantages of ATDD is, once the automated acceptance tests pass, the team knows they are “done” with that Story.

Me: Sounds challenging.  Are you saying there must be an automated check written first, for every piece of product code we need?

Agile Consultant:  Pretty much.

Me: Doesn’t that restrict the complexity and creativity of our product?  I mean, what if we come up with something not feasible to test via a machine.  Besides, aren’t there normally some tests better executed by humans, even for simple products?

Agile Consultant:  Yes, of course.  I guess some manual tests could be required, along with automated tests, as part of your “done” definition.

Me: What if all our tests are better executed by a human because our product doesn’t lend itself to automation?  Can we still claim to do ATDD and enjoy its benefits?

Agile Consultant:  …um…I guess so…  (displaying a somewhat disappointed face, as if something does not compute, but maybe she was just thinking I was an annoying nutcase)

And that got me thinking: And doesn’t this save us a lot of headaches, pain, and time because we wouldn’t have to distill our requirements into rigid test scripts with specific data (AKA “automatic checks”)?  We wouldn’t have to ask our programmers to write extra test code hooks for us.  We wouldn’t have to maintain a bunch of machine tests that don’t adapt as quickly as our human brains do?

Let’s call this Human Acceptance Test Driven Development (HATDD).  I’ve stated some of the advantages above.  The only significant disadvantage that stands out is that you don’t get a bunch of automated regression checks.  But it seems to me, ATDD is more about new Feature testing than it is about regression testing anyway.

So why aren’t there more (or any) Agile consultants running around offering HATDD?

It’s a cliché, I know.  But it really gave me pause when I heard Jeff “Cheezy” Morgan say it during his excellent STAReast track session, “Android Mobile Testing: Right Before Your Eyes”.  He said something like, “instead of looking for bugs, why not focus on preventing them?”.                 

Cheezy demonstrated Acceptance Test Driven Development (ATDD) by giving a live demo, writing Ruby tests via Cucumber, for product code that didn’t exist.  The tests failed until David Shah, Cheezy’s programmer, wrote the product code to make them pass. 

(Actually, the tests never passed, which they later blamed on incompatible Ruby versions…ouch.  But I’ll give these two guys the benefit of the doubt. )

Now back to my blog post title.  I find this mindshift appealing for several reasons, some of which Cheezy pointed out and some of which he did not:

• Per Cheezy’s rough estimate 8/10 bugs involve the UI.  There is tremendous benefit to the programmer knowing about these UI bugs while the programmer is writing the UI initially.  Thus, why not have our testers begin performing exploratory testing before the Story is code complete?
• Programmers are often incentivized to get something code completed so the testers can have it (and so the programmers can work on the next thing).  What if we could convince programmers it’s not code complete until it’s tested?
• Maybe the best time to review a Story is when the team is actually about to start working on it; not at the beginning of a Sprint.  And what do we mean when we say the team is actually about to start working on it?
• First we (Tester, Programmer, Business Analyst) write a bunch of acceptance tests.
• Then, we start writing code as we start executing those tests.
• Yes, this is ATDD, but I don’t think automation is as important as the consultants say.  More on that in a future post.
• Logging bugs is soooooo time consuming and can lead to dysfunction.  The bug reports have to be managed and routed appropriately.  People can’t help but count them and use them as measurements for something…success or failure.  If we are doing bug prevention, we never need to create bug reports.

Okay, I’m starting to bore myself, so I’ll stop.  Next time I want to explore Manual ATDD.

• Measuring your Automation might be easy.  Using those measurements is not.  Examples:
• # of times a test ran
• how long tests take to run
• how much human effort was involved to execute and analyze results
• how much human effort was involved to automate the test
• number of automated tests
• EMTE (Equivalent Manual Test Effort) – What effort it would have taken humans to manually execute the same test being executed by a machine.  Example: If it would take a human 2 hours, the EMTE is 2 hours.
• How can this measure be useful? It is an easy way to show management the benefits of automation (in a way managers can easily understand).
• How can this measure be abused?  If we inflate EMTE by re-running automated tests just for the sake of increasing EMTE, when are misleading.  Sure, we can run our automated tests everyday, but unless the build is changing every day, we are not adding much value.
• How else can this measure be abused?  If you hide the fact that humans are capable of noticing and capturing much more than machines.
• How else can this measure be abused?  If your automated tests can not be executed by humans and if your human tests can not be executed by a machine.
• ROI (Return On Investment) – Dorothy asked the students what ROI they had achieved with the automation they created.  All 6 students who answered, got it wrong; they explained various benefits of their automation, but none were expressed as ROI.  ROI should be a number, hopefully a positive number. 
• ROI=(benefit-cost)/cost
• The trick is to convert tester time effort to money.
• ROI does not measure things like “faster execution”, “quicker time to market”, “test coverage”
• How can this measure be useful?  Managers may think there is no benefit to automation until you tell them there is.  ROI may be the only measure they want to hear.
• How is this measure not useful?  ROI may not be important.  It may not measure your success.  “Automation is an enabler for success, not a cost reduction tool” – Yoram Mizrachi.  You company probably hires lawyers without calculating their ROI.
• She did the usual tour of poor-to-better automation approaches (e.g., capture playback to advanced key-word driven framework).  I’m bored by this so I have a gap in my notes.
• Testware architecture – consider separating your automation code from your tool, so you are not tied to the tool.
• Use pre and post processing to automate test setup, not just the tests.  Everything should be automated except selecting which tests to run and analyzing the results.
• If you expect a test to fail, use the execution status “Expected Fail”, not “Fail”.
• Comparisons (i.e., asserts, verifications) can be “specific” or “sensitive”.
• Specific Comparison – an automated test only checks one thing.
• Sensitive Comparison – an automated test checks several things.
• I wrote “awesome” in my notes next to this: If your sensitive comparisons overlap, 4 tests might fail instead of 3 passing and 1 failing.  IMO, this is one of the most interesting decisions an automator must make.  I think it really separates the amateurs from the experts.  Nicely explained, Dorothy!

If you want to have test automation
And don't care about trials and tribulation
Just believe all the hype
Get a tool of each type
But be warned, you'll have serious frustration!

(a limerick by Dorothy Graham)

I attended Dorothy Graham’s STARCanada tutorial, “Managing Successful Test Automation”.  Here are some highlights from my notes:

• “Test execution automation” was the tutorial concern. I like this clarification; sets it apart from “exploratory test automation” or “computer assisted exploratory testing”).
• Only 19% of people using automation tools (In Australia) are getting “good benefits”…yikes.
• Testing and Automating should be two different tasks, performed by different people.
• A common problem with testers who try to be automators:  Should I automate or just manually test?  Deadline pressures make people push automation into the future.
• Automators – People with programming skills responsible for automating tests.  The automated tests should be able to be executed by non-technical people.
• Testers – People responsible for writing tests, deciding which tests to automate, and executing automated tests.  “Some testers would rather break things than make things”.
• Dorothy mentioned “checking” but did not use the term herself during the tutorial.
• Automation should be like a butler for the testers.  It should take care of the tedious and monotonous, so the testers can do what they do best.
• A “pilot” is a great way to get started with automation.
• Calling something a “pilot” forces reflection.
• Set easily achievable automation goals and reflect after 3 months.  If goals were not met, try again with easier goals.
• Bad Test Automation Objects– And Why:
• Reduce the number of bugs found by users – Exploratory testing is much more effective at finding bugs.
• Run tests faster – Automation will probably run tests slower if you include the time it takes to write, maintain, and interpret the results.  The only testing activity automation might speed up is “test execution”.
• Improve our testing – The testing needs to be improved before automation even begins.  If not, you will have poor automation.  If you want to improve your testing, try just looking at your testing.
• Reduce the cost and time for test design – Automation will increase it.
• Run regression tests overnight and on weekends – If your automated tests suck, this goal will do you no good.  You will learn very little about your product overnight and on weekends.
• Automate all tests – Why not just automated the ones you want to automate?
• Find bugs quicker – It’s not the automation that finds the bugs, it’s the tests.  Tests do not have to be automated, they can also be run manually.
• The thing I really like about Dorothy’s examples above, is that she helps us separate the testing activity from the automation activity.  It helps us avoid common mistakes, such as forgetting to focus on the tests first.
• Good Test Automation Objectives:
• Free testers from repetitive test execution to spend more time on test design and exploratory testing – Yes!  Say no more!
• Provide better repeatability of regression tests – Machines are good checkers.  These checks may tell you if something unexpected has changed.
• Provide test coverage for tests not feasible for humans to execute – Without automation, we couldn’t get this information.
• Build an automation framework that is easy to maintain and easy to add new tests to.
• Run the most useful tests, using under-used computer resources, when possible – This is a better objective than running tests on weekends.
• Automate the most useful and valuable tests, as identified by the testers – much better than “automated all tests”.