Posted by Eric Jacobson at Wednesday, October 27, 2010
After attempting to use Microsoft Test Manager 2010 for an iteration, we quickly decided not to use it. Here is why.
About 3 years ago we finally managed to stop using HP Quality Center (AKA Test Director). We started managing our test cases and bugs as work items in Microsoft Team Foundation Server (TFS). The brilliance behind this transition was that it gave us the ability to attach our tests and bugs to the iteration’s Features/Stories and Tasks. This meant, as Features move in and out of iterations, the related tests and bugs follow; a beautiful thing! Additional benefits are, 1.) Programmers and BAs can easily review our test cases and write reports based on their execution status and, 2.) no more attempting to synch the TFS Features to Quality Center Requirements…the horror! I totally hated that!
It turns out, Microsoft Test Manager 2010, which sits on Microsoft TFS, took away many of the above TFS benefits and added as much overhead as Quality Center.
Stuff We Didn’t Like About Microsoft Test Manager 2010:
- Before one can write tests, one must set up a Test Suite. Test Suites do not update when the Feature set changes. Thus, one must manually keep one’s Test Suite synched with one’s iteration. To further frustrate, without customization, Test Manager does not let one write test cases for Task work items.
- Test cases in Test Manager follow a different workflow than those of TFS. The result is, nobody on your team can see which of your tests pass or fail unless they open Test Manager, which they probably don’t have a license for (unless they are testers). The reasoning behind this is probably Test Manager’s test cases can have test case run execution history (e.g., TestA could be passed in one build and failed in another build at the same time ). Test case run execution history is actually cool. In fact, it was one of the biggest motivators for us to try Test Manager. However, we were hoping Test Manager would trickle down the results to TFS so the whole team could benefit.
- One of the most annoying parts of our Test Manager trial may have been its usability and performance. The screens frequently hung, causing some testers to force quit and re-launch. The navigation was also awkward. It took too many clicks to get anything done.
- To update the execution status of Test Manager tests, one must go through the ridiculous test case executor. This is the thing that shows you each test case step and asks you for a pass/fail result. I can’t imagine anyone actually testing like this. Quality Center had something similar but provided an alternate method of updating execution status (i.e., one could bulk update via a grid).
- Our other gripe about updating the execution status of Test Manager tests is that the test case “summary” does not show. Most testers like to write their test cases as fragments, using the test case work item’s free-form summary tab. The summary tab is preferred over the grid, which forces tests into individual steps with expected results. The big joke is, if you write your tests in the summary tab, it is not possible to see them while running the silly test case executor. So you are presented with a blank test step and asked if it passes or fails.
Stuff We Think We Like About Microsoft Test Manager 2010:
There are two things some project teams are planning on using Test Manager for in the near future:
- Calling our CodedUI test methods and passing in parameters. According the idealistic demo I saw at the Stareast Microsoft booth, this allows manual testers to write/execute automated tests without coding.
- Using test case run execution for regression testing.
I guess TFS's simple work item model fits our needs for flexible lightweight test documentation with little administrative overhead. Maybe someone can convince me otherwise.