It’s one of our most commonly asked questions: what about CI integrations for performance testing? The short answer is simple: we have them, you should have them, and you should make sure you’ve implemented them.
The longer answer is similar but more complex. Your CI integration for performance testing or any testing should support your tool of choice. Your preferred tool could be one of these:
- Team City
- AWS CodeBuild
- Travis CI
- Atlassian Bamboo
- AWS CodePipeline
- Jenkins / Cloudbees
- Microsoft Visual Studio Team ServicesMicrosoft Visual Studio
- Gitlab CI
- BitBucket Pipelines
- Solano Labs
We support all of those and more: check out the latest on our CI integrations here.
Of course, you can always hook up your own integration: we offer tools for bash (*nix and MacOS), Windows Powershell, a Java SDK, a Python SDK and even a REST API.
When you’re looking for your CI integrations, make sure the performance testing solution you choose at minimum provides integration tools like an API or SDK (or scripting) - if not a custom, prebuilt integration you can use “off the rack.”
Considerations for Integrations
It’s easy to fall into the trap of wanting “all the things” when you’re considering a CI integration for load testing. It’s true, your load testing and performance testing tool should be robust and flexible. However, before you start thinking you need many integration points, consider how you’re going to use your performance testing tool with your CI tool.
It’s likely, if you follow our best practices, that you’re not running your performance tests after every commit - and not even after every build if you’re doing them more frequently than daily. Instead, you’re likely running your performance tests on a schedule.
We recommend that you run baseline performance tests daily as part of a daily build sequence. How you do that may vary from CI tool to CI tool. No matter how you trigger it, in essence, your CI integration will tell your CI tool to trigger the external performance test. In turn, your performance test script will perform the test or tests you want and return its results. In most CI integrations, that result will be a simple pass/fail.
It’s up to you in your CI integration to decide if a fail passback is enough to fail the whole build. Regardless, you’ll also want to be notified of a test failure via your collaboration tool of choice - like Slack or email.
In that scenario, your CI integration needs to trigger the performance test, receive the test pass/fail notification, behave appropriately based on that passback notification, log the result, and, in some cases, notify you via a communications channel.
That does represent more than one integration point, but with a little perspective, you can see those are the most crucial CI integration points, and you can evaluate your performance testing tool accordingly.
These tips should help with your CI integration and performance testing. We’re here to help if you have any other questions: just ask!