Home

IoT load testing versus IoT performance testing – 3 differences

  • by Steve Hilton

    • MachNation is often asked about the differences between IoT load testing and IoT performance testing. Because Tempest, our testing software, was built to do both types of simulations, we thought it would be helpful to distinguish between the two. IoT load testing is a subset of IoT performance testing, so making the comparison is important. 

    Differences between IoT load testing and IoT p

    Difference 1: test purpose

    The ultimate goal of IoT load testing is to discover and mitigate an IoT platform’s scalability shortcomings. The goal of IoT performance testing is to validate and optimize an IoT solution or use case. Load testing is generally focused on finding infrastructure shortcomings and points-of-failure in an IoT platform. Enterprises load test their infrastructure to proactively show customers the overall durability of an IoT platform or as part of an internal testing process to plan for worst-case scenarios. In contrast, IoT performance testing is designed to validate an IoT solution implementation and ultimately to find ways to optimize the underlying technology to improve performance. Enterprises conduct IoT performance testing to ensure that their implementations meet expectations or to validate that an IoT proof-of-concept is ready to scale to production.

    Difference 2: test scale

    Load testing is conducted at the extremes of measurement, while performance testing is conducted at a normal operating range to simulate a real-world IoT use case. The goal of load testing is to determine the ultimate scalability of an IoT solution. During load testing, the IoT solution is pushed to and beyond its capacity in order to understand its ability to operate at maximum scale. The goal of performance testing is to simulate a real-world IoT solution at real-world operating levels. This type of testing ensures that an IoT solution will deliver a customer’s key performance indicators (KPIs) while meeting service-level agreements (SLA).

    Difference 3: test metrics collected

    IoT load testing metrics are generally related to success and failure, while IoT performance testing metrics measure how well an IoT solution meets a set of KPIs. Enterprises collect various metrics to help calculate the success and failure of a test for a given volume. It also collects more granular metrics like the rate of failure during ramp-up and ramp-down. IoT performance testing collects all the metrics of load testing plus various IoT solution-specific latency metrics, such as event processing latency, rules engine latency, device-update latency, and more.

    Test examples

    Examples of IoT load and performance testing

    Here are a few examples of IoT load and performance testing based on projects that MachNation has done or is currently completing.

    Testing the scalability of an MQTT broker is an example of a typical IoT load test. MachNation has completed this type of testing on an MQTT broker recently. We captured data on both failure rates and message latency, although this type of test was an IoT load test. Our goal was to push an MQTT broker to its point of failure—around 2 million simultaneous connections.

    An experiment to understand the impact of bursty loads on an IoT platform is another type of IoT load test. Bursty loads—loads that scale-up rapidly and represent a very large percentage increase over average platform load—can tax IoT platform resources and lead to platform and message failure. Bursting large volumes of data at a platform and measuring success and failure of the platform to handle the load is a common test that MachNation performs.

    Validating an IoT solution is a typical IoT performance test. An example might be validating that a connected wind turbine solution is performant at production scale. Typical IoT performance tests would include tests on streaming data and various complex event process (CEP) functions to ensure the solution can meet the required SLAs for performance and control.

    Validating that a connected HVAC system (or any type of building management solution) is capable of at-scale operations is another example of IoT performance testing. Typical IoT performance tests of this type would simulate message flows at scale, a mass disconnect and reconnect of connected assets, and bulk SOTA or FOTA updates. 

    Want to learn more?

    Scroll to top