As some may already know, Load Impact recently let loose upon the world yet another load testing tool - k6
Given that there are tons of load testing tools out there, both proprietary and open source (k6 is of the latter kind), a reasonable question to ask is why we decided to create one more?
The reason, as it turns out, is not singular; Instead, there are a bunch of small reasons to create a new tool, and those reasons all add up to a very compelling case, in our opinion. First, we need some background:
The rise of developer testing
In many organizations up until now, QA professionals have performed the bulk of software testing. Load testing has been done by even more specialized "performance testers”, who have had the time and inclination to learn all the complex tools and techniques. The end result has been that load testing has largely been reserved for a very small minority of enterprises that have the scale to be able to afford such a specialized workforce. Overall, very little load testing has been done historically because of this, despite the fact that most IT professionals see the benefits of load testing and state a desire to do it.
Now we’re seeing more and more people who identify themselves as "regular developers” getting into testing. This is all part of the DevOps trend where development and operations meld together and one set of people takes full end-to-end responsibility for the applications they build.
So what about the tools?
Most of the tools that exist today were clearly created for the QA professional, and not for the developer. We believe it is likely that the lack of developer-friendly tools has cemented the QA professional as the only person to go to for load testing. Developers today, without prior load testing experience, often find it hard to get started with load testing.
What is the issue with the existing load testing tools then? As we see it, they often belong to one of two categories:
- Basic command-line tools with too little sophistication to provide real value. Lack of scripting means they’re unable to simulate anything remotely resembling real traffic onto a site, app or API and they often produce fixed and limited statistics. These tools are often released as open source.
- Complex, GUI-based tools, often for Windows that offer old and clunky UIs, obsolete scripting languages, if any, and are not at all easy to integrate into a CI environment. This category contains both free and open-source tools, such as Jmeter, and proprietary tools such as HP Loadrunner. Apart from being old and often quite painful to use, these tools also tend take a lot of time to learn properly. Time that developers rarely have.
Category 1) above is clearly not so interesting because while those tools are easy to get started with, you immediately run into their limitations that prevent you from getting real value out of your testing.
And category 2) is simply not suitable for developers, who tend to want to write test cases in real code, who want to automate testing, and who do not want huge point-and-click applications (and even if they did, they would not feel it was worth the time investment to learn those applications).
We do have some tools released in the past few years that show a little bit of promise, like e.g. Gatling and Locust, but they have their problems too: Gatling doesn’t have real scripting but its own, slightly clunky DSL implemented as Scala classes. Scala is obscure, and the DSL obfuscates any kind of logic you want to put in your test code. Gatling is also dependent on a JVM, which is an annoyance to many who are not Java developers. Locust has a very nice Python scripting API, but fails in the performance department: it is slower than perhaps any other tool out there and its inefficiency means measurements are notoriously unreliable. Also, it doesn’t produce results output in a format that is usable to developers who want to automate things.
So what is left for a developer then? Not much. There is an old tool called The Grinder which is not so bad, but it is not very actively developed and also dependent on a JVM. There is Artillery, which has good automation support but is based on Node.js, making it almost as slow and producing almost as unreliable measurements as Locust.
Then there is...nothing.
Enter k6
Putting developers’ load testing problems aside, we needed a next-generation load generator engine for loadimpact.com and looking at what was available out there, we didn’t really see anything we liked (maybe because we’re developers ourselves). We then realized we had a chance to hit two birds with one stone: create a really good, new load generator for our online service, and give the developer community a tool built from the ground up for their needs.
This new load generator would differ from the existing ones in that it would be:
- Scriptable in a popular, high-level language (most likely JavaScript)Providing developers with power and flexibility, plus their preferred way of writing tests (real code, no home-built DSL).
- Written in a popular language So that we might get community contributions in the open-source project.
- Suitable for automation Automation-friendly scripting API, pass/fail results, standardized output format that can be parsed by a computer, etc.
- High-performing (which we worried might rule out JS as scripting language) So that it could be run on limited hardware (e.g. a laptop) and still be useful.
We sat down and thought a bit about these requirements and how to best fulfill them (expect a follow-up article about that soon), then we created k6 which is:
- Scriptable in JavaScript (ES6)
- Written in Golang Being new to Go before this project, we’re very impressed with the language, and it has huge momentum among developers.
- Automation-friendlyScripting API with automation support, pass/fail results, usable output options, simple and intuitive command line UI, no external dependencies.
- Performing on par with the better tools out thereOnce we found the right JS engine, performance turned out to be quite good.
The feedback so far has been fantastic, with k6 amassing almost 3,000 Github stars in the past month or so and users already starting to interact with us on Slack and Github.
Please try k6 out and tell us what you think! Or help spread the word to people who are interested in performance/load testing, or a combination of that and functional testing (k6 is designed to handle both performance- and functional testing).