Web Services Load Testing

Now-a-days large number of web applications are either consuming web services or providing their business functions as web services.  Like the performance of accessing the application services from user interface (web pages or desktop forms) is important for satisfaction of the end users, same should be with web services that application is providing.  I found performance requirements are not explicitly mentioned in requirements documents, testers should ensure the performance requirements are mentioned in requirement document similar to the way the performance requirements are mentioned for web pages and servers.

We can test performance of web services like the way we test the performance of normal web applications. We can track metrics like Response Time, Throughput, Memory Consumption, Application Server performance, I/O performance for web services performance testing. I have worked on testing of some applications where  complete  application is built on web services. It's a smart client application which interacts with the server via web services. In such scenarios it is difficult to performance test the application from user interface for huge loads. In such cases we can actually test the core of the application which are web services, rather than testing the client. Performance testing of web services is supported many open source and commercial tools.  There are some open source and free tools available which can server basic performance testing requirements. I found JMeter is good choice among open source tools, JMeter offers good extensibility and scalability and has good options to performance test web services.  In one of the projects, I had to test the web services with 3000 concurrent users with huge data transfers, which is difficult to test with some SOAP testing tools as the data exchange and processing involved with each request is huge. JMeter is the good choice and has good extensibility of the performance testing.

The challenges in load testing of web services is, verification of response of each web service, parameterization of input data, and finding error rate of the requests. We can add assertions on web service response to check whether response is correct or not. This may cause some overhead on JMeter to verify each response and may affect the correct metrics. If your web services have robust logging mechanism, the verification for correctness of response or output can be verified from log files or log message tables.

Some considerations while designing performance tests are, to include think time between calling of different web service methods to when trying to achieve a business flow,  providing appropriate ramp up time, adding more remote machines to JMeter is necessary step if we want to test with large number of virtual concurrent users, configuring the proper Heap memory of JMeter to accommodate more load.

Read More

Automation Coverage

Automation Coverage is widely asked metric by management to see how much of test case repository is automated. Managers use this metric for estimation of manual efforts and automation scripts execution time etc.

Automation coverage percentage cannot give good picture unless what kind of test cases are automated, how they are automated, how well are they maintainable and context in which automation is going to be used. I believe automation is more useful in sanity tests, happy path test cases, post production hot fix testing, test data creation, pre-requisite setup creation, repeatable data driven testing, and to support regression testing. Hence, instead of using one coverage metric for automation, I think it is better to use different automation coverage metrics for each type of testing, like automation coverage of sanity tests, automation coverage of happy path test cases, automation coverage of regression test cases etc.

The power of automation lies in the way the test cases are automated. Automation should add more value to the manual test cases. Automation to take care of the areas where manual testing finds it difficult. The framework on which automation tests are builts needs to be robust and easy to use. The efficiency of automation scripts should be increased to complete the tests faster so that automation can cover more tests. Automation scripts need to reveiwed time to time in order to improve their efficiency in finding bugs. Definitely automation can drastically reduce the efforts where test cases to be executed with volumes of data, they can ensure that positive paths are working fine as a final check, and in test data creation. Automation can be of great help in certain areas of testing if they cover the areas including what manual tester cannot easily do. Automation is a good aid in testing when it is meticulously planned, efficiently used as demanded by context, and treated it as good as software development.

In summary, test management tools to provide more realistic metrics on automation coverage than simply calculating the percentage of automated test cases of total test repository.

Read More

How mind maps are useful in testing?

I have been using mind maps since many months, but recently I started using them extensively using in test design, and test ideas. As visual presentation is more powerful than linear representation, I prefer using mind maps whereever I use multi level multiple bullet lists.

Earlier I used to write my test ideas as a linear list of bullet points. I found this approach is not much productive when system under test has integrations with multiple third party systems or system itself has lot of interfaces.

Mind maps are useful in Lean Test Design, where it can save a lot of time in documenting the test cases in test management tools. This technique is handy when I explain test ideas with peers and dev team. Dev team can use mind map created by tester as a checklist for sanity testing before delivering the code to test team. Mind map gives a quick snapshot of how entire system looks like, how quickly I can test the system.

In one of my projects, the system has integration with 4 external systems and all these systems are integrated with one another. I found designing tests and writing test cases in traditional model do not add value and also tough to visualize how all these systems are integrated. When I used mind map to represent the test design, more test ideas started flowing in and everyone in the team has more clarity on the system. More I use mind maps, I get more ideas and better the understanding of the system.

I will write more on mind mapping with examples in next blog post.

Tools: Open source desktop tools like FreeMind, XMind or web based tools like Mind Meister. I use XMind and Mind Meister in my work.

Further Help: Darren McMillan written excellent articles on using Mind Maps in Lean Test Design and other areas of application.

Read More