Data warehouse (DW) testing is a far cry from functional testing. As testers, we need to let the team know if the DW dimension, fact, and bridge tables are getting the right data from all the source databases, storing it in such a way as to allow users to build reports, and keeping it current.

We are using automated tests as one of our test approaches. Contrary to popular belief, we found DW testing to be an excellent place to build and reuse automated tests. We came up with a simple automated test template that is allowing testers who have never done test automation to build up a pretty sweet library of automated tests.

Here's how it works:

Our automation engineer built us a test with two main parameters we pass in; a SQL statement for source data and a SQL statement for target data. The test compares the two record sets, including all data values and asserts they are equal. As the programmers build the various DW dimension, fact, and bridge tables, the testers copy and paste said automated test, and swap out the SQL with their SQL for the current test subject.

One of our most important tests is to rebuild the programmer's SQL ourselves, which is a manual process. If the tester's datasets don't match the DW, there may be a problem. This is where most of the defects are found. As we manually build our SQL, we plug it into an automated test that actually compares every record and column data value on millions of rows, and tells us where the differences are. The byproduct is an automated regression test.

Sure, rebuilding much of what the programmers did is challenging. We've had to learn case statements, cross DB joins, data type casting, and keep our heads straight while working through queries that include 30 or more joins. But normally it's fun and easy to get help from Google or our programmers.

We've been getting one or more automated tests completed for each dimension, fact, or bridge table; sometimes as early as in the development environment. When the DW is deployed to various environments, we normally just execute our automated tests and interpret the results. It's quite beautiful.

I’ll blog about some of the other tests we execute. There appears to be a shortage of pragmatic DW test ideas out there.

We did our first tester Lightning Round yesterday, which consisted of ten testers, who each had five minutes to make their point to a room full of other testers. To keep it fun, and I think, in the tradition of typical Lighting Rounds, we displayed a digital clock on the wall to track each speaker’s time. When a speaker used more than their five minutes, we threw squishy balls at them to make them stop.

The point of a Lightning Round is to give the audience one specific action item, without them having to endure a one hour lecture. This also gives the audience a variety of topics in a short time span. The advantage for the speakers is the short preparation time required to speak for five minutes.

Yesterday’s Lightning Round was an experimental version of something we’ve been doing for the last couple of years at my company. Testers from various projects and departments meet to exchange ideas. But normally we meet to attend an hour-long presentation about a single testing topic. Yesterday’s feedback indicated Lightning Rounds were way better to attend than the usual hour-long-single-topic session. Although, several testers said five minutes was too short. Some suggested five speakers with ten minutes each.

Personally, I think it was the five minute limit that kept the energy high and kept the audience from getting bored. But I’m willing to compromise. The larger problem, however, is getting testers to volunteer for these. I had to talk five of the speakers into doing it just to get my ten. Apparently, there are few testers willing to share their ideas with any conviction.

Anyway, the topics in yesterday’s section were the following:

Mine was the Programmer Profiling topic. A respected colleague told me he didn’t care for it. Maybe I’ll blog about it later.

As I struggle to find a tool that is current enough to automate tests for a Silverlight 4.0 app, my former QA manager and thoughtful testing buddy, keeps spouting the same message to me. Alex Kell keeps saying my first mistake was to let the programmers pick SL4 in the first place. We have tools that can automate SL2 and even SL3. Alex says, why not convince the programmers to use the older versions of SL.

His point being, if testing is really so important, then why shouldn’t we select our product’s coding technologies based on the ease of testing them? It’s that thing we always hear about called, “testability”.

When the initial SL4 decision was made, I sheepishly took Alex’s advice. I asked the lead architect if he could use SL3 instead. I explained how easy it would be to convert my existing automated test framework to drive SL3. He paused for two seconds, then said “no”, the SL4 bindings were superior and more favorable for our project. Can you blame him? I can’t. Of course, any team starting a greenfield project would want to begin with the latest supported code version.

So now I'm getting busy rewriting my framework using Microsoft’s CodedUI platform, which has SL4 support and is used elsewhere on my team.

In the meantime I’m haunted by Alex’s advice and wonder how hard I should have pushed. On the other hand, there is merit in adaptability. To be able to support your team with the best testing possible, based on their design choices…

…well that is something too.

What do you think?

Can you test something too much? Sure.

If you spend all your time testing FeatureA and four other Features go untested, you’ve probably tested FeatureA too much.

Part of what makes some testers better than others is their ability to know how much testing effort to put into each Feature and when. This is a tricky decision and may require the tester to stop testing FeatureA, even though FeatureA is still yielding bugs. What?!! Think about it. If you know FeatureA is yielding bugs but you don’t know anything about FeatureB, where should you spend your time next? I say, FeatureB. It just feels wrong, doesn’t it?

A lot of testers have a hard time with this. They get wrapped up in completing exhaustive testing on one Feature while the unknowns sit and collect dust. They follow their nose for the reward of bug discovery in comfortable areas. They underestimate how much time it will take to test the unknowns and before they know it, opps, not enough time to test everything!

Have you ever seen one of those plate spinning circus acts? You know the ones, the guy is frantically running around trying to keep all the plates spinning on sticks so they don’t fall. As a tester, you should be like a plate spinner. The Features are the plates. I don’t know what the sticks are, don’t worry about it. But if your programmers are working on Features (e.g., fixing bugs, finding bugs, refactoring, testing), your plates are spinning.

Keep your plates spinning!

At Monday’s retrospective, my answer to the question, “What should we do differently”, was to "have more fun". We have been cranking out releases for 85 iterations and the new year seemed like a good time to try something fresh. One of my testers came up with Bug Bucks.

Although it violates several agile practices, we’re going to give it a try. Here’s how it works:

So what can one do with a Bug Buck? We’re still working on that but pending budget approval, we will let them be exchanged for gifts, or more likely, one minute of personal time off, which should work out to a free day off per quarter for ambitious players!

There are several problems we are aware of; such as, will this discourage collaboration? Um…yes.

But will it really? In the big picture? We've already been collaborating on the rules. Anyway, our current approach is not to over-engineer this game, but to try to have some fun. If our velocity changes for the worst, we’ll pull the plug... just have to wait and see.

I'll report back.

Song to the famous Christmas Song, "Let it Snow, Let it Snow, Let it Snow"

The performance test stats are frightful,

Useability is not delightful,

I expect the bug count to grow,

App is slow! App is slow! App is slow!

It doesn't show signs of stopping,

And this build is really flopping,

My confidence is way down low,

App is slow! App is slow! App is slow!

When the hour glass goes away,

And the screen finally starts to repaint,

There's a timeout error trying to say,

Ready for prod, this thing ain't!

Available memory is slowly dying,

But, my devs, are still denying,

They say it's my box, but I know,

App is slow! App is slow! App is slow!

Last year's Twas the Night Before Prod Release was much better.

Last week a group of us testers, programmers, and business analysts volunteered to sort, inspect, and pack donated food, beverages and health products at the Atlanta Community Food Bank. Most cities have similar gigs and this is a fun way of doing volunteer work that is related to the old school usage of the term QA. If you’re a tester, volunteer to be an inspector.

As an inspector, you’ll go through all the donated items and decide which get packaged up to become meals, which get thrown in the trash, or which need special treatment. This is QA at its most primitive and I found it incredibly fun because unlike software testing, I actually got to make my own decisions as far as what would go to production and what would not.

Things that helped us succeed:

...if only we could weed through software bits this quickly.