Most of the testers at my new company do not have programming skills (or at least are not putting them to use).  This is not necessarily a bad thing.  But in our case, many of the products-under-test are perfect candidates for automation (e.g., they are API rich).

We are going through an Agile transformation.  Discussions about tying programmatic checks to “Done” criteria are occurring and most testers are now interested in getting involved with automation.  But how?

I think this is a common challenge.

Here are some ways I have had success getting manual testers involved in automation.  I’ll start with the easiest and work my way down to those requiring more ambition.  A tester wanting to get involved in automation can:

1. Do unit test reviews with their programmers.  Ask the programmers to walk you through the unit tests.  If you get lost ask questions like, “what would cause this unit test to fail?” or “can you explain the purpose of this test at a domain level?”.
2. Work with automators to inform the checks they automate.  If you have people focused on writing automated checks, help them determine what automation might help you.  Which checks do you often repeat?  Which are boring?
3. Design/request a test utility that mocks some crucial interface or makes the invisible visible.  Bounce ideas off your programmers and see if you can design test tools to speed things up.  This is not traditional automation.  But it is automation by some definitions.
4. Use data-driven automation to author/maintain important checks via a spreadsheet.  This is a brilliant approach because it lets the test automater focus on what they love, designing clever automation.  It lets the tester focus on what they love, designing clever inputs.  Show the tester where the spreadsheet is and how to kick off the automation.
5. Copy and paste an automated check pattern from an IDE, rename the check and change the inputs and expected results to create new checks.  This takes 0-to-little coding skills.  This is a potential end goal.  If a manual tester gets to this point, buy them a beer and don’t push them further.  This leads to a great deal of value, and going further can get awkward.
6. Follow an automated check pattern but extend the framework.  Spend some time outside of work learning to code. 
7. Stand up an automation framework, design automated checks.  Support an Agile team by programming all necessary automated checks.  Spend extensive personal time learning to code.  Read books, write personal programs, take online courses, find a mentor. 

Egads!  It’s been several months since my last post.  Where have I been? 

I’ve transitioned to a new company and an exciting new role as Principal Test Architect.  After spending months trying to understand how my new company operates, I am beginning to get a handle on how we might improve testing.

In addition to my work transition, myself and each member of my family have just synchronously suffered through this year’s nasty flu, and then another round of stomach flu shortly thereafter.  The joys of daycare…

And finally, now that my son, Haakon, has arrived, I’ve been adjusting to my new life with two young children.  1 + 1 <> 2.  

It has been a rough winter.

But alas, my brain is once again telling me, “Oh, that would make a nice blog post”.  So let’s get this thing going again!

If you read part 1, you may be wondering how my automated check performed…

The programmer deployed the seeded bug and I’m happy to report, my automated check found it in 28 seconds! 

Afterwards, he seeded two additional bugs.  The automated check found those as well.  I had to temporarily modify the automated check code to ignore the first bug in order to find the second.  This is because the check stops checking as soon as it finds one problem.  I could tweak the code to collect problems and keep checking but I prefer the current design.

Here is the high level generic design of said check:

Build the golden masters:

• Make scalable checks - Before test execution, build multiple golden masters per coverage ambition.  This is a one-time-only task (until the golden masters need to be updated per expected changes).
• Bypass GUI when possible – Each of my golden masters consist of the response XML from a web service call, saved to a file.  Each XML response has over a half a million nodes, which are mapped to a complex GUI.  In my case, my automated check will bypass the GUI.  GUI automation could never have found the above seeded bug in 28 seconds.  My product-under-test takes about 1.5 minutes just to log in and navigate to the module being tested. Waiting for the GUI to refresh after the countless service calls being made in the automated check would have taken hours.
• Golden masters must be golden! Use a known good source for the service call.  I used Production because my downstream environments are populated with data restored from production.  You could use a test environment as long as it was in a known good state.
• Use static data - Build the golden masters using service request parameters that return a static response.  In other words, when I call said service in the future, I want the same data returned.  I used service request parameters to pull historical data because I expect it to be the same data next week, month, year, etc.
• Automate golden master building - I wrote a utility method to build my golden masters.  This is basically re-used code from the test method, which builds the new objects to compare to the golden masters.

Do some testing:

• Compare - This is the test method.  It calls the code-under-test using the same service request parameters used to build the golden masters.  The XML service response from the code-under-test is then compared to that of the archived golden masters, line-by-line.
• Ignore expected changes - In my case there are some XML nodes the check ignores.  These are nodes with values I expect to differ.  For example, the CreatedDate node of the service response object will always be different from that of the golden master.
• Report - If any non-ignored XML line is different, it’s probably a bug, fail the automated check, report the differences with line number and file (see below) references and investigate.
• Write Files - For my goals, I have 11 different golden masters (to compare with 11 distinct service response objects).  The automated check loops through all 11 golden master scenarios, writing each service response XML to a file.  The automated check doesn’t use the files, they are there for me.  This gives me the option to manually compare suspect new files to golden masters with a diff tool, an effective way of investigating bugs and determining patterns.

I’m feeling quite cocky at the moment.  So cocky that I just asked my lead programmers to secretly insert a bug into the most complex area of the system under test.

Having just finished another epic automated check based on the Golden Master approach I discussed earlier, and seeing that most of the team is on pre-Thanksgiving vacation, this is the perfect time to “seed a bug”.  Theoretically, this new automated check should help put our largest source of regression bugs to rest and I am going to test it.

The programmer says he will hide the needle in the haystack by tomorrow.

I’m waiting…

You’ve got this new thing to test. 

You just read about a tester who used Selenium and he looked pretty cool in his bio picture.  Come on, you could do that.  You could write an automated check for this.  As you start coding, you realize your initial vision was too ambitious so you revise it.  Even with the revised design you’re running into problems with the test stack.  You may not be able to automate the initial checks you wanted, but you can automate this other thing.  That’s something.  Besides, this is fun.  The end is in sight.  It will be so satisfying to solve this.  You need some tasks with closure in your job, right?  This automated check has a clear output.  You’ve almost cracked this thing…cut another corner and it just might work.  Success!  The test passes!  You see green!  You rule!  You’re the Henry Ford of testing!  You should wear a cape to work!

Now that your automated thingamajig is working and bug free, you can finally get back to what you were going to test.  Now what was it?

I’m not hating on test automation.  I’m just reminding myself of its intoxicating trap.  Keep your eyes open.

You must really be a sucky tester.

I’m kidding, of course.  There may be several explanations as to why an excellent tester like yourself is not finding bugs.  Here are four:

• There aren’t any bugs!  Duh.  If your programmers are good coders and testers, and perhaps writing a very simple Feature in a closely controlled environment, it’s possible there are no bugs.
• There are bugs but the testing mission is not to find them.  If the mission is to do performance testing, survey the product, determine under which conditions the product might work, smoke test, etc., it is likely we will not find bugs.
• A rose by any other name… Maybe you are finding bugs in parallel with the coding and they are fixed before ever becoming “bug reports”.  In that case, you did find bugs but are not giving yourself credit.
• You are not as excellent as you think.  Sorry.  Finding bugs might require skills you don’t have.  Are you attempting to test data integrity without understanding the domain?  Are you testing network transmissions without reading pcaps?

As testers, we often feel expendable when we don’t find bugs.  We like to rally around battle cries like:

“If it ain’t broke, you’re not trying hard enough!”

“I don’t make software, I break it!”

“There’s always one more bug.”

But consider this, a skilled tester can do much more than merely find a bug.  A skilled tester can also tells us what appears to work, what hasn’t broken in the latest version, what unanticipated changes have occurred in our product, how it might work better, how it might solve additional problems, etc. 

And that may be just as important as finding bugs.

Hey testers, don’t say:

“yesterday I tested a story.  Today I’m going to test another story.  No impediments”

Per Scrum inventor, Jeff Sutherland, daily standups should not be “I did this…”, “I’ll do that…”.  Instead, share things that affect others with an emphasis on impediments.  The team should leave the meeting with a sense of energy and urgency to rally around the solutions of the day.  When the meeting ends, the team should be saying, “Let’s go do this!”.

Here are some helpful things a tester might say in a daily standup:

• Let’s figure out the repro steps for production Bug40011 today, who can help me?
• I found three bugs yesterday, please fix the product screen bug first because it is blocking further testing.
• Sean, I know you’re waiting on my feedback on your new service, I’ll get that too you first thing today.
• Yesterday I executed all the tests we discussed for Story102, unless someone can think of more, I am done with that testing.  Carl, please drop by to review the results.
• I’m getting out of memory errors on some test automation, can someone stop by to help?
• If I had a script to identify data corruption, it would save hours.
• Paul, I understand data models, I’ll test that for you and let you know something by noon.
• The QA data seems stale.  Don’t investigate any errors yet.  I’m going to refresh data and retest it today.  I’ll let you know when I’m done.
• Jolie, if you can answer my question on expected behavior, I can finish testing that Story this afternoon.

Your role as a tester affects so many people. Think about what they might be interested in and where your service might be most valuable today.