Tl;dr — More Load Impact customers than ever are continuously running load tests. In response, we’ve just shipped our “Performance Thresholds” feature. Now, engineers set Pass/Fail metrics for their scheduled tests in order to hone in on the data that’s most important to them. Here’s how we think that helps you:
- The binary status of Pass/Fail metrics is great for automated Continuous Integration flows
- Set thresholds for VU load time, page load time or any of your custom metrics
- Know right away if a test passed or failed your expectations
- Configure tests to abort when a metric fails and avoid completely crashing your application
In the spirit of Continuous Delivery and automation, there’s a wave of new Load Impact users scheduling load tests at all times of day and night. With an increased number of performance tests in the software development lifecycle, reliable notifications and triggers when things go wrong become extremely important.
That’s why we’re excited to announce our new Performance Thresholds feature.
The idea here is for testers to use their performance goals to set Pass/Fail criteria for their load tests. For example:
- When VU load time is more than 1,000 ms
- When the response time of specific resource or page is higher than desired
- When a server response takes longer than expected
Thresholds can be created for VU Load Time, Failure Rate, page metrics and any of your defined custom metrics.
Aborting a Failed Test vs. Letting it Run
Users have the option to either immediately abort a test when a metric has failed or allow the test to complete. There are really two simple reasons for this.
- Aborting a test when a metric fails: You have a figure in mind — let’s say a VU load time of 500 ms — and you don’t particularly care about other metrics in that moment. Why continue the test if the metric you care most about has failed?
- Letting a test run after a metric fails: Sometimes you want to see what happens when a metric has surpassed acceptable levels, or maybe you want to see how your app recovers
Be in the Know!
Use our Notifications feature (which was released in April to rave reviews) to know exactly when your load tests have completed and whether or not they’ve failed.
For Load Impact users who are already using notifications, you’ll see when a test has “failed by threshold” in the status code, which is already included in the data we send in every notification.
As the CD, CI and DevOps methodologies have picked up substantial steam the last few years, more engineers using Load Impact started regularly scheduling load tests. Here’s an example of what a bunch of our great users are doing:
- Scheduling smaller tests throughout the day based on how much traffic they typically handle
- Users figure out how much traffic they average by the hour from Google Analytics data, then they run a series load tests mirroring the typical user flow throughout the day (some as often as every 15 minutes)
- After a few test runs, engineers establish solid benchmarks for their web performance. Then it’s time to test the resilience of their website or application with a large test
- It’s a good idea to occasionally run a max capacity test to see just how much traffic your product can handle. Then you at least have an idea of what needs to be done if you get an unexpected wave of traffic