Your mission is to test a new data warehouse table before its ETL process is even set up.  Here are some ideas:

Start with the table structure.  You can do this before the table even has data.

• If you’ve got design specs, start with the basics;
• Do the expected columns exist?
• Are the column names correct?
• Are the data types correct?
• Is the null handling type correct?
• Do the columns logically fit the business needs?  This was already discussed during design.  Even if you attended the design, you may know more now.  Look at each column again and visualize data values, asking yourself if they seem appropriate.  You’ll need business domain knowledge for this.
• Build your own query that creates the exact same table and populates it with correct values.  Don’t look at the data warehouse source queries!  If you do, you may trick yourself into thinking the programmer must be correct.

Once you have data in your table you can really get going.

• Compare the record set your query produced with that of the data warehouse table.  This is where 70% of your bugs are discovered.
• Are the row counts the same?
• Are the data values the same?  This is your bread and butter test.  This comparison should be done programmatically via an automated test so you can check millions of columns & rows (see my Automating Data Warehouse Tests post).  Another option would be to use a diff tool like DiffMerge.  A third option, just spot check each table manually.
• Are there any interesting columns?  If so, examine them closely.  This is where 20% of your bugs are hiding.  This testing can not be automated.  Look at each column and think about the variety of record scenarios that may occur in the source applications; ask yourself if the fields in the target make sense and support those scenarios.
• Columns that just display text strings like names are not all that interesting because they are difficult to screw up.  Columns that are calculated are more interesting.  Are the calculations correct?
• Did the design specify a data type change between the source and the target?  Maybe an integer needed to be changed to a bit to simplify data…was it converted properly?  Do the new values make sense to the business?
• How is corrupt source data handled?  Does the source DB have orphaned records or referential integrity problems?  Is this handled gracefully?  Maybe the data warehouse needs to say “Not Found” for some values.
• Build a user report based on the new data warehouse table.  Do you have a trusted production report from the transactional DB?  Rebuild it using the data warehouse and run a diff between the two.

What am I missing?

Sometimes, the most feasible way to test something, is to let it soak in an active test environment for several weeks.  Examples:

• No repro steps but general product usage causes data corruption.  We think we fixed it.  Release the fix to an active test environment, let it soak, and periodically check for data corruption.
• A scheduled job runs every hour to perform some updates on our product.  We tested the hourly job, now let’s let it run for two weeks in an active test environment.   We expect each hourly run to be successful.

Per Google, soak testing involves observing behavior whilst under load for an extended period of time.  In my case, load is normally a handful of human testers, as opposed to a large programmatic load of thousands.  Nevertheless, the term is finally catching on within my product teams.

Who cares about the term?  I like it because it honestly describes the tester effort, which is very little.  It does not mislead the team into thinking testers are spending much time investigating something.  It’s almost like not testing.  But yet, we still plan to observe from time to time and eventually make an assessment of success or failure.

Be sure to over-annunciate the “k” in “soak”.  People on my team thought I was saying “soap” test.  I’m not sure what a soap test is…but I’m sure it exists too!

The Proceedings of the National Academy of Sciences just published a study, “Extraneous factors in judicial decisions”, that finds decision making is mentally taxing and when people are forced to continually make difficult decisions, they get tired and begin opting for the easiest decision.

Eight parole board judges were observed for 10 months, as they ruled whether or not to grant prisoners parole.  The study noticed a trend.  Near the end of work periods, prisoners being granted parole dropped significantly.  The decision of granting parole takes much longer to explain and involves more work than the decision to deny parole. 

(For those of you reading my blog from prison, try to get your parole hearing scheduled first thing in the morning or right after lunch.)

What does this remind you of?

Testing!  I don’t have the ambition to perform said study on test teams, but I have certainly experienced the same pattern.  I’m guessing fewer bugs get logged latter in the day.  The decision that you found a bug is a much more difficult decision than denial.  Deciding you found a bug means investigating, logging a report, convincing people sometimes, testing the fix, regression testing what broke, etc.

I wrote about Tester Fatigue and suggested solutions in How To Combat Tester Fatigue.  But according to the above study, taking breaks from testing is paramount.  Therefore, I will now head downstairs for some frozen yogurt.

When programmers log bugs, us testers are grateful of course.  But when programmer-logged bugs travel down their normal work flow and fall into our laps to verify, we’re sometimes befuddled…

”Hey! Where are the repro steps?  How can I simulate that the toolbar container being supplied is not found in the collection of merged toolbars?”   

I used to insist that every bug fix be tested by a tester.  No exceptions!  Some of these programmer-logged bugs were so technical, I had to hold the programmer’s hand through my entire test and test it the same way the programmer already tested it.  This is bad because my test would not find out anything new.   Later I realized I’m not only wasting the programmer’s time, I’m also wasting my time; from other new tests I could be executing.

Sometimes, it’s still good to waste time for the sake of understanding but don’t make it a hard and fast rule for everything.  Instead, you may want to do as follows:

• Ask the programmer how they fixed it and tested their fix.  Does it sound reasonable?
• Ensure the critical regression tests will be run in the patched module, before production deployment.

Then rubber stamp the bug and spend your time where you can be more helpful.

I did a Force Field Analysis brainstorm session with my team.  We wrote down what we like and don’t like about our jobs as testers.  Then, in the “didn’t like” column, we circled the items we felt we had control over changing.  Here are what my testers and I don’t like about our jobs.

Items We May Not Have Control Over:

• When asked what needs to be tested as a result of a  five minute code change,  programmers often say “test everything”.
• Stressful deadlines.
• Working extra hours.
• Test time is not adequately considered by Programmers/BAs when determining iteration work.  Velocity does not appear to matter to team.

Items We May Have Control Over:

• Testers don’t have a way to show the team what they are working on.  Our project task boards have a column/status for “Open”, “In Development”, “Developed”, and “Tested”.  It’s pretty easy to look under the “In Development” column to see what programmers are working on.  But after that, stuff tends to bunch up under the “Developed” column.  Even though their may be 10 Features/Stories in “Developed”, the testers may only be working on two of them.  A side affect is testers having to constantly answer the question “What are you working on?”...my testers hate that question.
• Testers don’t know each other’s test skills or subject matter expertise.  We have some 20 project teams in my department.  Most of the products interact.  Some are more technical to test than others.  Let’s say you’re testing ProductA and you need help understanding its interface with ProductB.  Which tester is your oracle?  Let’s say you are testing web services for the first time and you’re not sure how to do this.  Which tester is really good at testing web services and can help you get started?
• Testers lose momentum when asked to change priorities or switch testing tasks. A programming manager once told me, “each time you interrupt a programmer it takes them 20 minutes to catch back up”.  Testers experience the same interruption productivity loss, but arguably, to a larger degree.  It is annoying to execute tests in half-baked environments, while following new bugs that may or may not be related, along the way.

We will have a follow-up brainstorm on ways to deal with the above.  I’ll post the results.

During a Support Team/Test Team collaboration, I was coaching a technical support team member through logging a bug.  I suggested my usual template.

Repro Steps:

1. Do this.
2. Do that.

Expected Results: Something happens.
Actual Results: Something does not happen.

Afterwards, I said “That’s it.”  She looked over the Spartan bug report and was reluctant to let it go.  she said, “It looks too bare bones.”  She wondered if we should add screen captures and more commentary.  IMO, when a bug has repro steps, that’s all you need.  And the fewer the better!  Sure, a picture is worth a thousand words.  But sometimes it only takes about 14.

If you’ve ever participated in bug triage meetings, you’ll probably agree.  People love to fill bug reports with non-essential comments that obscure the spirit of the bug and discourage anyone from reading it.

“After those steps I tried it again and got the same error.”

“Maybe it is failing because I am doing this instead of that.”

“I remember it used to not throw an error, but now it does.”

“We could fix it by doing this instead of that.”

It’s a bug report, stupid.  Report the bug and get back to testing.  Opinions and essays are for blogs.

…Praise for the simple, bare bone bugs.

Sounds too good to be true, huh?

One of my programmers is cool enough to request a code review with me when something critical is at stake.  I can’t really read code and he knows it.  So why does he keep suggesting we do code reviews?

He realizes said activity will force him to explain his code at a translated-to-layman-terms-level.  Occasionally, I’ll ask questions; “Why do you need that statement?”, “What if the code flow takes an unexpected path?”.  But often I start to daydream…not about my weekend or other non-work things but about a previous statement the programmer made.  My brain gets hung up trying to grok the code.  And while I’m lost in thought, the programmer keeps zooming through their code, until…

…all of a sudden, the magic happens.

The programmer says, “Hold on…I see a mistake”.  It happens every time!  I kid you not!  A different programmer on my team found two errors while explaining his unit tests to me Monday.

Now I’m not suggesting testers should daydream their way through code reviews.  And of course, a tester capable of actually reading/understanding code is more valuable here.  But I am suggesting this:

If you’re a tester who feels too uncomfortable or inadequate to perform a code review with your programmers, remember, all you have to do is listen and occasionally ask dumb questions.  And even if you’re bored to death looking at someone’s code, and you fall into a daydream, who knows?  You may have indirectly helped your programmer save the world.