Top 10 Questions to Ask During Your Evaluation of Web Test Tool Vendors
1. Can I test my Web applications and services without having to script, please?
If I have to write code to make my scripts work the first time, does that mean I also have to constantly rework and debug these scripts as my application changes? Can other team members easily take part in the testing process or do they also have to learn a complicated proprietary language that requires weeks of training to get up to speed?
2. How quickly can I get results and do I need to be a performance guru to make sense of them?
Conduct your own evaluation to determine how labor intensive the tool is to work with. Will using this tool extend your project schedule by days, weeks or months? Will you need to hire dedicated personnel just to support the tool or even worse pay through the nose for a fleet of expensive consultants to work through complex data analysis when the results prove too difficult to understand?
3. So, about how much will this stuff cost anyway?
Get a ballpark quote early on in the evaluation process. That way you can find out early on in the process if someone is trying to gouge you. Make sure everyone understands your requirements so that you can be sure of an apples-to-apples comparison. Often times, functionality that one vendor charges for separately is included with another vendor’s core package.
4. Have your tools been built specifically to test Web applications?
Most vendors offer legacy test tools that were originally designed to test client server applications. Will they be able to keep up with the highly dynamic, quickly evolving Web applications you need to test?
5. Can I easily leverage your tools across my entire organization?
Let’s face it, testing is a team sport. It crosses departmental lines and sometimes geographic boundaries. Does the vendor offer browser-based access along with a flexible licensing model that allows you to easily centralize your test data and share a common testing resource across multiple teams and office locations? Or is their solution just to sell you more departmental licenses leaving you with multiple installation points to manage?
6. How do I know your tools are reliable and the results are accurate?
Is the vendor’s technology based on open industry standards or their own proprietary framework? The former is tried, proven and accepted by the IT community at large ensuring both accuracy and interoperability with your existing systems, while the latter is closed and often unverified leaving you at risk to technology obsolescence and unreliable or misleading results. Don’t make the mistake of deploying your application with a false sense of confidence. After all, your job may depend on it.
7. How many machines will I need to run your tools?
Legacy test tool vendors did not originally design their tools to accommodate the unlimited, or open, demand of the Web and as result may only allow you to generate a small number of users per machine. That means you'll need to purchase and maintain a bank of servers to run your load tests, which will drive up your total cost of ownership for the tool.
8. Do I get training included or am I on my own?
Does the vendor bundle all of the tools, support and services that you'll need to be successful with your software purchase? Or will you need to pay exorbitant training and consulting fees on top of already inflated product prices before you start getting value out of the tool?
9. When I call support, can I deal with an experienced test engineer right away?
Or will you have to talk to a ‘handler’ who will triage the severity of your question and route you to someone who may barely be able to spell ‘Web app’ let alone test one. Does the vendor have a guaranteed support response time?
10. Am I going to like dealing with you? Trust your instincts. How are you going to feel when you have to interact with this company day in and day out? Will they treat you as a valued customer or just another number? If you want straight answers – answers that won’t make you wince or scratch your head – call OpenDemand now at 1-973-735-0547. Or start your own evaluation of our products by visiting our