TL;DR: Run your load tests with your nightly builds.
Running load tests with nightly builds is the sweet spot for how often you should load test. We’ve tried it more frequently; we’ve tried it less frequently. In our experience, nightly is the best frequency. Even as load test tool developers, we believe more often is too much — but most people don’t load test enough.
It’s easy to get used to the quick response of unit and functional tests. They are the kinds of tests you want to run with every single code change. A load test takes more time, though, at least to get usable results, so running them with every build is too often.
We regularly test a small part of our LoadImpact API with 700ish unit tests that take only a couple of minutes. That’s not blazingly fast, but it’s fast enough that we haven’t bothered to optimize it. A full API build for us, including unit tests, takes only a few minutes, so we’ve still got our heads in the code when the tests complete.
To start, we recommend you start with small, "unit load tests.” These are designed to run on specific units of code, and may take only 10-30 seconds each. Run them regularly while developing to gauge how code changes affect performance. When you test all the units, you’re looking at a longer test time - more of a comprehensive test.
Compare: a functional unit test may take a fraction of a second to run, but a unit load test will take maybe 10 seconds. That’s at least 100 x slower than the functional unit test.
You may be running a test suite with many small unit load tests or running one larger, more comprehensive load test. Either way, the total time spent on load testing will be too long for it to be practical to run ANY load tests on each commit.
A good, comprehensive load test, with enough results to make statistically valid optimizations, takes some time. Ideally, you’d run all the tests in your test suite for at least 10 minutes, if not longer, and that’s too long to run for every commit. If you’re like us, by the time you’ve run a load test for 30 or 60 minutes, you’ve mentally moved on and have switched contexts. Your head’s no longer in the code.
Still, some recommend you run comprehensive load tests with every commit. As much as we love load tests, this won’t work. Besides the context switching issue, if your team is bigger than a couple of developers, you’d be running too many tests too often. Frequent, concurrent load tests could provide skewed results that wouldn’t help anyway, depending on how well you have separated your pre-production environment.
We tried running load tests on every commit, because that was the easiest solution. That didn’t work because it made the builds so slow for everyone. Instead, we found a way to schedule load tests with nightly builds in CircleCI.
The best practices we recommend: Run your unit and functional tests with every commit. Run your load tests with nightly builds.