Context

In our world where IoT products are multiplying by day, performing stress tests on the server is essential in ensuring a seamless user experience. One of Tismo’s IoT clients sought Tismo’s assistance in creating a simulator that can support 100K devices per instance. Their current implementation of the simulator was not stable.

Challenge

The simulator needs to send messages in bursts at specific intervals, also with a slight jitter to model real world applications. It needs to generate traffic to the server that mimics installation with up to 100K devices. The generated traffic dynamics needs to closely resemble real life installation scenarios. Different devices generate different types of messages. The test setup, the number of devices simulated, the message types should all be configurable at test time.

Technology Used

Tismo considered a few popular choices for these kinds of simulators. Python, Go, Rust and C#/.NET were the candidates. In order to exploit the multi-core test system, ease of development and for concurrency support, Tismo selected C# / .NET.

Features that made the .NET Framework the ideal choice were:

1. Cross-platform compatibility.
2. Extensive support for concurrency.
3. Ease of development.
4. Well maintained libraries.

Solution

The obvious choice in design of using a separate thread per device simulated was ruled out considering the large number of devices that need to be simulated. The packet traffic can be bursty and the simulator needs to adhere to tight real time deadlines to mimic real world scenarios. Tismo chose the discrete time simulation technique.

Message packets would be created beforehand and scheduled according to a priority determined by their dispatch time. CPU cores that are free would be utilized to form the message packets along with their time of dispatch. The dispatch engine would push the real time traffic out to the server.

Priority based concurrent queue

A priority based concurrent queue allowed all the device nodes to queue up the message packets to the scheduler.

So one thread to queue the messages and one to send it bringing it down to just 2 threads from the initial 500. This method of crafting messages and scheduling them beforehand allowed for a much more precise and efficient dispatch of messages, achieving high throughput from the application.

Here is an example of the messages sent by the application recorded every 10 minutes.

Graph showcasing messages sent every 10 minutes

Here we can see that the messages are sent at a consistent rate by the application. The randomly added configurable jitter can be clearly seen in the traffic pattern.

Tismo also added quality of life features such as a command line to start or stop the nodes, collection of message statistics and node info.

Summary

An application to simulate a large number of nodes was developed by Tismo, adhering to all the client requirements and exceeding the performance requirements on the simulator.

Note: The featured image for this project is AI generated.