Here’s another failure story, per the post where I complained about people not telling enough test failure stories.
Years ago, after learning about Keyword-Driven Automation, I wrote an automation framework called OKRA (Object Keyword-Driven Repository for Automation). @Wiggly came up with the name. Each automated check was written as a separate Excel worksheet, using dynamic dropdowns to select from available Action and Object keywords in Excel. The driver was written in VBScript via QTP. It worked, for a little while, however:
- One Automator (me) could not keep up with 16 programmers. The checks quickly became too old to matter. FAIL!
- An Automator with little formal programming training, writing half-ass code with VBScript, could not get help from a team of C# focused programmers. FAIL!
- The product under test was a .Net Winforms app full of important drag-n-drop functionality, sitting on top of constantly changing, time-sensitive, data. Testability was never considered. FAIL!
- OKRA was completely UI-based automation. FAIL!
Later, a product programmer took an interest in developing his own automation framework. It would allow manual testers to write automated checks by building visual workflows. This was a Microsoft technology called MS Workflow or something like that. The programmer worked in his spare time over the course of about a year. It eventually faded into oblivion and was never introduced to testers. FAIL!
Finally, I hired a real automator, with solid programming skills, and attempted to give it another try. This time we picked Microsoft’s recently launched CodedUI framework and wrote the tests in C# so the product programmers could collaborate. I stood in front of my SVP and project team and declared,
“This automation will shave 2 days off our regression test effort each iteration!”
- The automator was often responsible for writing automated checks for a product they barely understood. FAIL!
- Despite the fact that CodedUI was marketed by Microsoft as being the best automation framework for .Net Winform apps, it failed to quickly identify most UI objects, especially for 3rd party controls.
- Although, at first, I pushed for significant amounts of automation below the presentation layer, the automator focused more energy on UI automation. I eventually gave in too. The tests were slow at best and human testers could not afford to wait. FAIL! Note: this was not the automators failure, it was my poor direction.
At this point, I’ve given up all efforts to automate this beast of an application.
Can you relate?