The tools we tested
|Hey (previously "Boom")||snapshot 2017-09-19 (Github)||https://github.com/rakyll/hey|
|k6||snapshot 2017-09-14 (Github)||https://k6.io/|
- Has there been any change in the relative performance of the tools?
- How does our own tool - k6 - perform in comparison with the other tools?
Real scripting, or not?
- Hey (previously known as "Boom")
The above tools do not have a built-in scripting language. Well, not a real one at least. Some of them (Artillery, Jmeter, Gatling and Tsung) have DSLs that allow the user to create logic in their test scenarios, but these DSLs are not as powerful and flexible as a "real" language, and are much clumsier to use.
However, the following tools were executing script code during the test:
- Grinder (executing Python/Jython)
- Locust (executing Python)
- Wrk (executing Lua)
- If you want to look at detailed results from the testing, here is a spreadsheet: https://docs.google.com/spreadsheets/d/13ZCxDxy06LlhDD0vuys9R04ppGv787c7DkZT1XO5-Gc/edit?usp=sharing
- The tools perform pretty similarly to last time we benchmarked them. No major surprises or changes.
- k6 seems to do pretty well in terms of performance. It is actually the best performer of the tools that execute a real scripting language.
Tips for choosing a tool
- If the only requirement is to achieve the absolute max in terms of RPS numbers, I'd choose Wrk or Apachebench
- If I have very simple needs - only want to hit a single, static URL and don't need detailed statistics - I'd choose one of the simpler tools: Apachebench, Hey, Wrk
- If I want a more versatile tool, with both decent performance and more functionality/flexibility, I'd choose one of: Gatling, Grinder, Jmeter, k6 or Tsung. They all perform very well and provide a lot of configuration- and results output options.
- If I want to write my test cases using real code, I'd choose Grinder or k6. If I don't care about tool performance I may also be able to use Locust.
Repeat the tests yourself
It is actually very simple! We just released an updated load testing tool benchmark test suite that makes benchmarking the tools (or just trying them out) ridiculously simple:
$ git clone https://github.com/loadimpact/loadgentest
$ cd loadgentest
That's it. The shell script runtests.sh will show you a menu that looks like this:
You enter a target URL using option "1", then you can stress that URL using different tools and settings, simply by pressing a button.
You have to have docker installed, because the various load testing tools are run as docker images (based on Alpine Linux), fetched from Docker hub. You also have to have some kind of target system you can make life miserable for, of course - don't load test any public site out there that isn't yours!
(Note that runtests.sh is able to use Linux netem to add simulated network delay - option "R" - but this only works if you're root, so start the script as root if you want to use that feature)
Would be great if people helped adding more tool support to the https://github.com/loadimpact/loadgentest repo as there are several load testing tools out there that we haven't been testing.