k6 as alternative for Azure and Visual Studio load tests

By Pepe Cano. On 2019-07-02

At the end of 2018, Microsoft stated in a Preview Release note that 2019 is the last version of Visual Studio with load testing features; then only a few months later they formally announced the deprecation of the Azure load testing service would occur on March 31st, 2020.

This announcement came as a bomb to the community. They have been using the tools and service for years, some members even asking for MS to open source their load testing tool.

For those that follow this market, this news was not a complete surprise; we think open source tools and other vendors are providing better performance testing alternatives; this is in line with Microsoft’s opinion about the reason for shutting down its service.

When I look around at other offerings in this space (open source as well as commercial offerings that sometimes include consulting services) I honestly feel they are now better suited to meet the needs of our customers.

Jamie Cool, Director of Program Management, Azure DevOps

Following the announcement, a large number of users from the Azure and Visual Studio load testing community have been asking questions about our Load Testing platform in particular. This post intends to present how to use k6 and LoadImpact as an alternative to replace the Azure load testing service.

k6 and LoadImpact

Before digging on how our Load Testing platform compares to the Azure service, let’s start with an introduction of the main components of our platform:; k6 and the LoadImpact cloud service.

k6: an open source load testing tool; a modern alternative to JMeter. Key benefits include:

  • Test scripts are written in Javascript.
  • An easy-to-use CLI and robust set of built in APIs.
  • Automation friendly, to easily facilitate the inclusion of your performance tests into CI/CD pipelines.

You can install k6 and start running performance tests in your machine for free. By default, k6 prints the result of your test to the console. Additionally, it can store its data into multiple alternatives for the better visualization and analysis of your test results.

LoadImpact: is our cloud service built with k6 at it’s core. We’ve designed our commercial offering to:

  • Run and scale your tests in the cloud with no infrastructure setup.
  • Provide a User-Friendly Interface to organize, configure, and run your tests.
  • Manage your team members and permissions.
  • Provide a convenient and powerful interface for the visualization of your test result.
  • Leverage intelligent algorithms to identify performance issues.
  • Analyze and compare the performance of your application over time.

We have briefly covered the components of our Load Testing platform: k6, the open source tool, and LoadImpact, the cloud service. Next, we are going to describe how to run the same type of Azure load tests with our cloud service or k6.

Azure load testing service

Azure load testing service has the following options to create and run your load tests:

Azure Load Tests

URL load test

A URL test is one of the simplest ways to create load tests; you can add multiple URLs, select its HTTP method, add headers, and query string values for each URL. The URL Load Test accesses each of these URLs using the specified parameters multiple times depending on your load test settings.

Azure URL test

 

 

The LoadImpact cloud service provides the same mechanism for configuring your load tests with the Request Builder Interface. These tests are configured through the LoadImpact web app, utilizing an interface that should feel familiar to Azure’s offering.

Request Builder

 

You can read more about the different options on the Request Builder documentation.

The Request Builder is an excellent tool to run a load test without writing any Javascript code - we generate that all for you!. It can also be very helpful to onboard new users into the k6 scripting API. By clicking the script button at the top right corner, you can find the k6 script related to your test configuration.

Script in Request Builder

 

HTTP Archive load test

The HTTP Archive aka HAR is a format for logging a user interaction of a website.

Recording and downloading a HAR file is a straightforward process. Here, is a guide to generate a HAR file from a browser session.

 

Downloading HAR file

 

Note that in Chrome, the Preserve log option set up the network request log to not be cleared upon a page change. After we download the HAR file, we are ready to start creating our load test.

In Azure, you have to select `New HTTP Archive based test` and the web interface will convert the HAR file content into individual HTTP requests on the HTTP URL interface.

 

HAR file GIF

 

LoadImpact also provides the ability to create a load test from a HAR file. The following image shows the view from you could upload the HAR file, configure other test settings, and run your load test.

 

LI HAR file GIF

 

For users preferring CLI tools or those without a LoadImpact account, we also have open source a har-to-k6 tool to convert HAR files to k6 scripts.

# install the converter
npm install -g har-to-k6

# convert HAR file
har-to-k6 my-user-session.har -o loadtest.js

Now, you could use k6 to run a local or cloud test of the converted k6 script.

# local execution
k6 run loadtest.js

# cloud execution
k6 cloud loadtest.js

JMeter test

Azure allows you to run JMeter tests on its Load Testing Cloud Service. This is done by selecting the New Apache JMeter test, uploading your jmx file, and running your test from the web interface.

Azure JMeter tests

 

JMeter is the most popular open source load testing tool in the market. The first release was launched in 1998 and it has been the standard tool for performance testing.

We thank the JMeter community for providing a great load testing tool and paving the way for performance testing in the software industry, but we built k6 because we believe an easy-to-use alternative with a best in class developer experience is necessary in today’s world.

For users having implemented their performance tests in JMeter, we have built a tool to convert JMeter tests into k6 scripts.

# install the converter
npm install -g jmeter-to-k6

# convert JMeter load test
jmeter-to-k6 loadtest.jmx -o loadtest

Now, you could use k6 to run a local or cloud test of the converted k6 script.

# local execution
k6 run ~/loadtest/test.js


# cloud execution
k6 cloud ~/loadtest/test.js

 

Visual Studio web performance test

Visual Studio web performance tests (.webtest files) is a Microsoft technology to perform web tests simulating many users accessing a server at the same time. While it was not designed for API testing, users have also used them for running API tests.

A web test project provides many options to configure your load test. Comparing both technologies deserves its dedicated blog post having in-depth explanations. The following section quickly describes how some webtest options can be configured in k6.

 

Add artificial human interaction pauses in your scenario

You can use the sleep function to simulate think-time by suspending the test execution for a period of time.

import { sleep } from "k6";
import http from "k6/http";

export default function() {
  http.get("https://loadimpact.com");
  sleep(Math.random() * 30);
  http.get("https://loadimpact.com/features");
};

 

Specify the number of virtual users for your scenario

You can specify the number of Virtual Users and duration of your load test in your k6 script, via the CLI or using the web interface.

k6 run --vus 10 --duration 30s script.js
export let options = {
    vus: 10,
    duration: "3m"
};
UI test options

 

Configure test iteration settings for your scenario

You can configure the number of iterations a VU can make on a similar way than duration and VUs.

export let options = {
    vus: 10,
    iterations: 100
};

You can read more about all the k6 options.

 

Configure the probability of a virtual user running a test in the scenario

There is Work In Progress for providing the ability to configure probability for different scenarios. The current workaround is to use conditional statements in your k6 script.

 

Configure the desired network mix for your scenario

k6 does not provide yet the ability to simulate network conditions. Some users have been using chaos engineering tools while running their Load Tests to replicate the same type of network conditions.

 

Select the appropriate Web browser mix for your scenario

Set the User Agent header on the HTTP requests of your k6 script to simulate a particular browser generating the requests.

let headers = { "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0" };

http.get("https//test.loadimpact.com", { headers: headers });

Additionaly, the batch option configures the maximum number of simultaneous/parallel connections in total that an http.batch() call in a VU can make.

export let options = {
    batch: 15
};

 

Specify a threshold rule using counters for your load test

Checks provide an API to define assertions in your load tests. Thresholds allow configuring performance expectations in your load test to make your test pass or fail.

You could define thresholds for ensuring performance goals, for example:

  • 99th percentile response time must be below 300 ms.
  • Minimum response time must be below 100 ms
  • The response content must have been OK more than 95% of the time.
  • ...

 

Use context parameters

You can use Environment variables to pass parameters to your load tests.

k6 run -e MY_HOSTNAME=test.loadimpact.com script.js

 

We don't aim to cover an in-depth comparison of both technologies in this post; instead, we decided to write a quick overview of how k6 supports some of the webtest options included on the webtest documentation. For further questions on this or any other topic, we encourage to reach the LoadImpact support team or the k6 community.

Test results

The web interface of the Azure DevOps service and Visual Studio allows analyzing the result of load tests. You can visualize the test summary, configuration, errors and more, analyze individual load metrics, graphs and trends, create reports, compare tests, and much more.

Azure load test result summary Azure load test result charts

 

k6 provides the --out option to output your load test results to different sources. This option allows you to have multiple choices for visualizing your test results.

# json
k6 run --out json=my_test_result.json script.js

# InfluxDB/Grafana
k6 run --out influxdb=http://localhost:8086/k6 script.js

# Apache Kafka
k6 run --out kafka=brokers=broker_host:8000,topic=k6

# statsd
k6 run --out statsd script.js

# datadog
k6 run --out datadog script.js

 

Our cloud service also provides an interface for the visualization of your load test results. The test result interface has been designed to:

  • providing a convenient and powerful interface for the visualization of your test result.
  • leverage intelligent algorithms to identify performance issues.
  • analyze and compare the performance of your application over time.

You can read more about all the sections of the test result interface in this article.

Load test result in LoadImpact

 

Azure Pipelines

Both locally executed k6 tests and cloud tests can be run from your Azure Pipelines process. Running load tests as part of your CI pipeline ensures that you'll catch any performance regressions that occur when your application or infrastructure changes.

If you ever thought about including your performance tests in your CI/CD Pipelines, read our step-by-step guide to integrating k6 and LoadImpact with Azure Pipelines.

Conclusion

We have presented k6 and the LoadImpact cloud service as an alternative to the departing Azure load testing service. This post intends to provide a high-level overview of both load testing platforms to show you how you can run the same type of load tests on our platform.

If you are evaluating Load Testing platforms, the best way is to try them running your own performance tests. LoadImpact offers a free trial of our cloud service, which we encourage you to explore.

If you have any feedback about the Azure DevOps service or any load testing experience, don’t hesitate to include your comment, contact our support team, or the k6 community.

Loading...