Last week we had an awesome Tester Lightning Talk session here at my company. Topics included:
- Mind Maps
- Cross-Browser Test Emulation
- How to Bribe Your Developers
- Performance Testing Defined
- Managing Multiple Agile Projects
- Integration Testing Sans Personal Pronouns
- Turning VSTS Test Results Files Into Test Reports
- Getting Back to Work After Leave
- Black Swans And Why Testers Should Care
The “Performance Testing Defined” talk inspired me to put my own twist on it and blog. Here goes…
The terms in the above graphic are often misused and interchanged. I will paraphrase from my lightning talk notes:Baseline Testing – Less users than we expect in prod. This is like when manual testers perform a user scenario and use a stopwatch to time it. It could also be an automated load test where we are using less than the expected number of users to generate load.
Load Testing – # of users we expect in prod. Real-world scenario. Realistic.
Stress Testing – More users than we expect in prod. Obscene amount of users. Used to determine the breaking point. After said test, the tester will be able to say “With more than 2000 users, the system starts to drag. With 5000 users, the system crashes.”
Stability Testing – Run the test continuously over a period of time (e.g., 24 hours, 1 week) to see if something happens. For example, you may find a memory leak.
Spike Testing – Think TicketMaster. What happens to your system when it suddenly jumps from 100 simultaneous users to 5000 simultaneous users for a short period of time?
There. Now you can talk like a performance tester and help your team discuss their needs.
As far as building these tests, at the most basic level, you really only need one check (AKA automated test). Said check should simulate something user-like, if possible. In the non-web-based world (which I live in) this check may be one or more service calls. In the non-web-based world, you probably do not want to use an automated check at the UI level; you would need an army of clients to load test. After all, your UI will only have a load of 1 user, right? What you’re concerned with is how the servers handle the load. So your check need only be concerned with the performance before the payload gets handed back to the client.
The check is probably the most challenging part of Performance testing. Once you have your check, the economies of scale begin. You can use that same check as the guts for most of your performance testing. The main variables in each are user load and duration.
Warning: I’m certainly an amateur when it comes to performance testing. Please chime in with your corrections and suggestions.