Sometimes, the most feasible way to test something, is to let it soak in an active test environment for several weeks.  Examples:

• No repro steps but general product usage causes data corruption.  We think we fixed it.  Release the fix to an active test environment, let it soak, and periodically check for data corruption.
• A scheduled job runs every hour to perform some updates on our product.  We tested the hourly job, now let’s let it run for two weeks in an active test environment.   We expect each hourly run to be successful.

Per Google, soak testing involves observing behavior whilst under load for an extended period of time.  In my case, load is normally a handful of human testers, as opposed to a large programmatic load of thousands.  Nevertheless, the term is finally catching on within my product teams.

Who cares about the term?  I like it because it honestly describes the tester effort, which is very little.  It does not mislead the team into thinking testers are spending much time investigating something.  It’s almost like not testing.  But yet, we still plan to observe from time to time and eventually make an assessment of success or failure.

Be sure to over-annunciate the “k” in “soak”.  People on my team thought I was saying “soap” test.  I’m not sure what a soap test is…but I’m sure it exists too!

The Proceedings of the National Academy of Sciences just published a study, “Extraneous factors in judicial decisions”, that finds decision making is mentally taxing and when people are forced to continually make difficult decisions, they get tired and begin opting for the easiest decision.

Eight parole board judges were observed for 10 months, as they ruled whether or not to grant prisoners parole.  The study noticed a trend.  Near the end of work periods, prisoners being granted parole dropped significantly.  The decision of granting parole takes much longer to explain and involves more work than the decision to deny parole. 

(For those of you reading my blog from prison, try to get your parole hearing scheduled first thing in the morning or right after lunch.)

What does this remind you of?

Testing!  I don’t have the ambition to perform said study on test teams, but I have certainly experienced the same pattern.  I’m guessing fewer bugs get logged latter in the day.  The decision that you found a bug is a much more difficult decision than denial.  Deciding you found a bug means investigating, logging a report, convincing people sometimes, testing the fix, regression testing what broke, etc.

I wrote about Tester Fatigue and suggested solutions in How To Combat Tester Fatigue.  But according to the above study, taking breaks from testing is paramount.  Therefore, I will now head downstairs for some frozen yogurt.

When programmers log bugs, us testers are grateful of course.  But when programmer-logged bugs travel down their normal work flow and fall into our laps to verify, we’re sometimes befuddled…

”Hey! Where are the repro steps?  How can I simulate that the toolbar container being supplied is not found in the collection of merged toolbars?”   

I used to insist that every bug fix be tested by a tester.  No exceptions!  Some of these programmer-logged bugs were so technical, I had to hold the programmer’s hand through my entire test and test it the same way the programmer already tested it.  This is bad because my test would not find out anything new.   Later I realized I’m not only wasting the programmer’s time, I’m also wasting my time; from other new tests I could be executing.

Sometimes, it’s still good to waste time for the sake of understanding but don’t make it a hard and fast rule for everything.  Instead, you may want to do as follows:

• Ask the programmer how they fixed it and tested their fix.  Does it sound reasonable?
• Ensure the critical regression tests will be run in the patched module, before production deployment.

Then rubber stamp the bug and spend your time where you can be more helpful.

I did a Force Field Analysis brainstorm session with my team.  We wrote down what we like and don’t like about our jobs as testers.  Then, in the “didn’t like” column, we circled the items we felt we had control over changing.  Here are what my testers and I don’t like about our jobs.

Items We May Not Have Control Over:

• When asked what needs to be tested as a result of a  five minute code change,  programmers often say “test everything”.
• Stressful deadlines.
• Working extra hours.
• Test time is not adequately considered by Programmers/BAs when determining iteration work.  Velocity does not appear to matter to team.

Items We May Have Control Over:

• Testers don’t have a way to show the team what they are working on.  Our project task boards have a column/status for “Open”, “In Development”, “Developed”, and “Tested”.  It’s pretty easy to look under the “In Development” column to see what programmers are working on.  But after that, stuff tends to bunch up under the “Developed” column.  Even though their may be 10 Features/Stories in “Developed”, the testers may only be working on two of them.  A side affect is testers having to constantly answer the question “What are you working on?”...my testers hate that question.
• Testers don’t know each other’s test skills or subject matter expertise.  We have some 20 project teams in my department.  Most of the products interact.  Some are more technical to test than others.  Let’s say you’re testing ProductA and you need help understanding its interface with ProductB.  Which tester is your oracle?  Let’s say you are testing web services for the first time and you’re not sure how to do this.  Which tester is really good at testing web services and can help you get started?
• Testers lose momentum when asked to change priorities or switch testing tasks. A programming manager once told me, “each time you interrupt a programmer it takes them 20 minutes to catch back up”.  Testers experience the same interruption productivity loss, but arguably, to a larger degree.  It is annoying to execute tests in half-baked environments, while following new bugs that may or may not be related, along the way.

We will have a follow-up brainstorm on ways to deal with the above.  I’ll post the results.

During a Support Team/Test Team collaboration, I was coaching a technical support team member through logging a bug.  I suggested my usual template.

Repro Steps:

1. Do this.
2. Do that.

Expected Results: Something happens.
Actual Results: Something does not happen.

Afterwards, I said “That’s it.”  She looked over the Spartan bug report and was reluctant to let it go.  she said, “It looks too bare bones.”  She wondered if we should add screen captures and more commentary.  IMO, when a bug has repro steps, that’s all you need.  And the fewer the better!  Sure, a picture is worth a thousand words.  But sometimes it only takes about 14.

If you’ve ever participated in bug triage meetings, you’ll probably agree.  People love to fill bug reports with non-essential comments that obscure the spirit of the bug and discourage anyone from reading it.

“After those steps I tried it again and got the same error.”

“Maybe it is failing because I am doing this instead of that.”

“I remember it used to not throw an error, but now it does.”

“We could fix it by doing this instead of that.”

It’s a bug report, stupid.  Report the bug and get back to testing.  Opinions and essays are for blogs.

…Praise for the simple, bare bone bugs.

Sounds too good to be true, huh?

One of my programmers is cool enough to request a code review with me when something critical is at stake.  I can’t really read code and he knows it.  So why does he keep suggesting we do code reviews?

He realizes said activity will force him to explain his code at a translated-to-layman-terms-level.  Occasionally, I’ll ask questions; “Why do you need that statement?”, “What if the code flow takes an unexpected path?”.  But often I start to daydream…not about my weekend or other non-work things but about a previous statement the programmer made.  My brain gets hung up trying to grok the code.  And while I’m lost in thought, the programmer keeps zooming through their code, until…

…all of a sudden, the magic happens.

The programmer says, “Hold on…I see a mistake”.  It happens every time!  I kid you not!  A different programmer on my team found two errors while explaining his unit tests to me Monday.

Now I’m not suggesting testers should daydream their way through code reviews.  And of course, a tester capable of actually reading/understanding code is more valuable here.  But I am suggesting this:

If you’re a tester who feels too uncomfortable or inadequate to perform a code review with your programmers, remember, all you have to do is listen and occasionally ask dumb questions.  And even if you’re bored to death looking at someone’s code, and you fall into a daydream, who knows?  You may have indirectly helped your programmer save the world.

Our Data Warehouse uses Change Data Capture (CDC) to keep its tables current.  After collaborating with one of the programmers we came up with a pretty cool automated test template that has allowed our non-programmer testers to successfully write their own automated tests for CDC. 

We stuck to the same pattern as the tests I describe in my Automating Data Warehouse Tests post.  Testers can copy/paste tests and only need to update the SQL statements, parameters, and variables.  An example of our test is below.  If you can’t read C#, just read the comments (they begin with //).

BTW – if you want a test like this but are not sure how to write, just ask your programmers to write it for you.  Programmers love to help with this kind of thing.  In fact, they will probably improve upon my test.

Happy Data Warehouse Testing!

[TestMethod]
        public void VerifyCDCUpdate_FactOrder()
        {//get some data from the source
            var fields = DataSource.ExecuteReader(@"
SELECT     TOP (1) OrderID, OrderName
FROM         Database.dbo.tblOrder");
            var OrderID = (int)fields[0];
            var originalValue = (string)fields[1];

            //make sure the above data is currently in the Data Warehouse
            var DWMatch = new DataSource("SELECT OrderID, OrderName FROM FactOrder WHERE OrderID = @OrderID and OrderName = @OrderName",
                                              new SqlParameter("@OrderID", OrderID),
                                              new SqlParameter("@OrderName", originalValue));

            //fail test is data does not match.  This is still part of the test setup.
            DataSourceAssert.IsNotEmpty(DWMatch, "The value in the datawarehouse should match the original query");

            try
            {
                // Set a field in the source database to something else
                var newValue = "CDCTest";

                DataSource.ExecuteNonQuery(
                    @"UPDATE Database.dbo.tblOrder SET OrderName = @NewValue WHERE OrderID = @OrderID",
                    new SqlParameter("@NewValue", newValue),
                    new SqlParameter("@OrderID", OrderID));

                var startTime = DateTime.Now;
                var valueInDW = originalValue;
                while (DateTime.Now.Subtract(startTime).Minutes < 10)
                {
                    // Verify the value in the source database is still what we set it to, otherwise the test is invalid
                    var updatedValueInSource = DataSource.ExecuteScalar<string>(@"SELECT OrderName FROM Database.dbo.tblOrder WHERE OrderID = @OrderID",
                            new SqlParameter("@OrderID", OrderID));

                    if (updatedValueInSource != newValue)
                        Assert.Inconclusive("The value {0} was expected in the source, but {1} was found.  Cannot complete test", newValue, updatedValueInSource);

                    //start checking the target to see if it has updated.  Wait up to 10 minutes (CDC runs every five minutes).  This is the main check for this test.  This is really what we care about.
                    valueInDW = DataSource.ExecuteScalar<string>(@"SELECT OrderName FROM FactOrder WHERE OrderID = @OrderID",
                                              new SqlParameter("@OrderID", OrderID));

                    if (valueInDW == newValue)
                        break;
                    Thread.Sleep(TimeSpan.FromSeconds(30));
                }
                if (valueInDW != newValue)
                    Assert.Fail("The value {0} was expected in DW, but {1} was found after waiting for 10 minutes", newValue, valueInDW);
            }
            finally
            {
                // Set the value in the source database back to the original
                // This will happen even if the test failes
                DataSource.ExecuteNonQuery(
                    @"UPDATE Database.dbo.tblOrder SET OrderName = @OriginalValue WHERE OrderID = @OrderID",
                    new SqlParameter("@OriginalValue", originalValue),
                    new SqlParameter("@OrderID", OrderID));
            }
        }