The Context
A new player in the home automation space had come up with a smart home device to capitalize on the smart home revolution. The pre-launch marketing campaign had yielded hundreds of pre-booking for the product. Tismo was brought on board to set up a scalable IoT architecture and carry out performance testing of the IoT cloud application.
The Problem
With a robust architecture set up on AWS IoT Platform, performance testing of the IoT cloud application was the last checkpoint before market launch. The challenge for the Tismo team was to simultaneously simulate thousands of end devices, each exhibiting different behavior and functionality, and measure the performance of the web server. This involved investigating the server behavior when thousands of users start using the system. Aspects such as the server’s request handling capability, crashing conditions and the performance of REST API were to be examined.
Products with poor performance metrics almost never meet expected sales goals. Thus, this was the ‘break-or-make’ moment for the client, an up and coming player in the market.
What is Performance Testing
Performance Testing ensures that software applications perform optimally under their expected workload. In addition to testing the features and functionality supported by the system, performance indicators like response time, stability, scalability, reliability etc are evaluated to avoid performance bottlenecks. Performance testing unearths what needs to be improved before the product hits the market.
The Solution
Tismo understood that the requirement called for a specialized performance testing tool to analyze and measure the performance of the IoT cloud application. Apache JMeter was chosen due to its tailor-made features for performance testing of IoT cloud applications and servers and the comprehensive test coverage that it offered.
The basic element in a JMeter test is referred to as a thread. Each thread executes the test plan, completely independent of other test threads. Tismo created multiple threads to run performance testing of the IoT cloud application while simultaneously sending requests corresponding to every one of the numerous device functionalities. Loop controllers to send requests to individual devices were set up and the requests to test device functionalities were sourced from predefined CSV files. Tismo ensured that the server responses were accurate by running HTTPS code assertions.
Loop controllers for performance testing sourced from predefined CSV file
A pronounced advantage of using JMeter for performance testing is its highly extensible core, that allows virtually unlimited testing capabilities. Tismo used the following plugins to carry out performance testing of the server using JMeter:
- JSON Path Extractor plugin to execute JSON Path expressions against JSON responses and store the result in a JMeter Variable
- Inter-Thread Communication plugin to enable the managing of data between different thread groups. This was useful for scenarios where access tokens needed to be shared between thread groups, implement asynchronous downloads etc
- Transactions per Second plugin to display the graph of the number of successful and failed transactions per second during a test run
Tismo was able to identify performance bottlenecks and optimize the architecture to support thousands of devices. Performance variation was kept minimal by tweaking certain server-side configurations.
About JMeter
The Apache JMeter is an open source software, designed to analyze and measure the performance of web applications and servers. It simulates user behavior and returns statistical information about the target server through graphical outputs.
Features of JMeter
- Since it is an open source software, it is freely available.
- It has a simple and intuitive GUI.
- JMeter can be used for performance testing of many different server types ?Web – HTTPS, HTTP, SOAP, Database via LDAP, JDBC, JMS, Mail – POP3, etc.
- It is a platform-independent tool
- It has Swing and lightweight component support
- JMeter stores its test plans in XML format. This makes it possible to generate a test plan using any text editor.
- The multi-threading framework allows simultaneous sampling of different functions by separate thread groups and concurrent sampling by many threads
- It is highly extensible using plugins and enjoys support from a huge community of developers
Summary
Tismo carried out performance testing of an IoT cloud application using Apache JMeter. Multiple threads and loop controllers were set up and simultaneous requests, corresponding to each device functionality, were sent to the server. HTTPS code assertions were carried out and various plugins were used to ensure maximum coverage for performance testing.