MachNation reached another milestone with its IoT performance testing software. MachNation testers successfully ran high-scale IoT performance tests of 500,000 simulated IoT devices against one MQTT broker to determine suitability for various-sized IoT projects. [UPDATE: As of 14 April 2020, MachNation successfully completed testing of 1 million devices pushing 4000 messages per second! Stay tuned for that data later.]
Read on to learn more about our exciting findings.
Or for more information about Tempest, MachNation’s IoT performance testing software,
- Visit our product landing page and get a copy of our Tempest Solution Overview sheet
- Download our latest performance testing infographic
- Listen to our Tempest podcasts
- Contact us to learn about MIT-E Pf
Findings
Using Tempest, the industry’s first IoT-centric performance testing software, MachNation simulated an IoT solution of 500,000 connected devices streaming through one instance of the VerneMQ MQTT broker. We chose VerneMQ because it is well known, open source, and claims high performance levels. We found that the MQTT broker performed well relative to other brokers and platforms we have tested. We measured ramp-up and ramp-down rates for message generation, message failure rates, and latency rates to determine suitability for an IoT solution.
1. Ramp-up and ramp-down of devices
MachNation testers chose a fairly typical ramp up and ramp down of connected virtual IoT devices. Using Tempest, MachNation testers are able to adjust the ramp-up and ramp-down rate which can have an impact on overall IoT solution performance.
Over the course of the test, Tempest sustained an average of approximately 1000 messages per second flowing from Tempest to the MQTT broker or vice versa. It took approximately 6.9 seconds to ramp-up from 0 to 1000 messages per second and similar time to ramp down from 1000 to 0 messages per second. Overall, it took approximately 8.5 minutes to connect all 500,000 virtual IoT devices and another 8.5 minutes to disconnect all devices. The entire test run lasted approximately 26 minutes, 38 seconds.
2. Calculation of message failure rate
We calculated message failure rates. Overall, no observations failed to make the pub-sub loop. To fully gauge VerneMQ’s ability to meet extremely high reliability SLAs for an applicable IoT solution, MachNation recommends more thorough testing including multiple times of day, days of the week, and various load schedules.
3. Inspection of outliers
Third, upon inspection of the performance data, we chose to remove 2 datapoints as outliers. Both outliers had latency values greater than 100 milliseconds (ms): one was 209 ms and one was 211 ms. This number of outliers implies that if a customer expected an SLA of 99.99% that latency would not exceed 100 ms, VerneMQ would successfully fulfill the SLA for the period of the limited test run by MachNation. To fully gauge VerneMQ’s ability to meet a 4-nines latency-related SLA, MachNation recommends more thorough testing including multiple times of day, days of the week, and various load schedules.
4. Statistical summary of data
We calculated typical summary statistics. Latency was fairly normally distributed about a mean of 5.54 ms with standard deviation of 3.31 ms. This means that in continuous testing or a production environment mimicking MachNation’s test environment, we would expect 95% of the latency values to be less than 12.16 ms. These statistical summary data show that a single instance of the deployed MQTT broker exhibited low average latencies and the spread about the average was fairly small. Enterprise IoT solutions that depend on consistently rapid transfer of IoT data through an MQTT broker would find this particular open-source broker well suited to their requirements.
We also calculated the maximum latency during every 8-second period of the performance test. When graphically depicted, maximum latency for each time period appears rather variable, although this sort of visualization is quite common when reviewing maximum latency data.
5. Analysis of quartiles
We created a graphical representation of the upper, middle, and lower quartiles of the VerneMQ performance data. The middle quartile (i.e., the median) was 4 ms; upper and lower quartiles were 7 ms and 3 ms, respectively; and the maximum and minimum (measured as 1.5 times the interquartile range above and below the upper and lower quartiles, respectively) were 13 ms and 0 ms. This analysis shows that 50% of the actual data for this performance test were between 3 ms and 7 ms, a fairly tight distribution. This reiterates the tight spread of latency data for this performance test. Although, as depicted on the graph below, the upper and middle quartile increased over the course of the performance test, possibly pointing to some performance degradation of the MQTT broker at peak load.
Overall, VerneMQ performed relatively well compared to other MQTT brokers and platforms tested by MachNation at this load schedule.
Feel free to contact us for more information about IoT performance testing.