Load testing with Azure Pipelines

By Rafael Muñoz Cárdenas. In Tutorials. On 2018-10-11

Load testing with Azure Pipelines

In this tutorial, you will discover how easy and fast is to integrate k6 tests into Azure Pipelines.

k6 is a free and open-source testing tool for load and performance testing of APIs, microservices and websites.

Azure Pipelines is a continuous integration (CI) and continuous delivery (CD) to continuously test and build your software and deploy it to any target.

Performance issues can quickly turn expensive. Whether that’s because of unexpected load, a piece of code that’s suddenly introduced a regression, or a misconfiguration of network components. Integrating k6 into your CI pipeline will allow you to know about performance degradations before your user experience them in production.

The roadmap of this tutorial is going to follow the outline below:

Write your performance test

We will create a k6 test for our demo that will simulate a progressive ramp up from 0 to 15 virtual users (VUs) over a duration of 10 seconds. Then the 15 virtual users will remain at 15 VUs 20 seconds and finally it will ramp down over a duration of 10 seconds to 0 virtual users.

Each user will request the front page (without any static files linked to it), wait 5 seconds, and restart the default function in a loop, until the test is over.

export let options = {
    stages: [
        { duration: "10s", target: 15 },
        { duration: "20s", target: 15 },
        { duration: "10s", target: 0 }
    ],
};

export default function() {
    let res = http.get("http://test.loadimpact.com/");
    sleep(5);
};

A core prerequisite to performance testing is to have defined service level objectives (SLOs). SLOs are a vital aspect of ensuring the reliability of your systems and applications. In some cases, having a proper testing may ensure that you are capable of fulfilling a certain service level agreement (SLA) with your customers and avoid expensive costs and risky situations for your business.

Thresholds allow you to define a clear criteria of what is considered a test success or failure. Let’s add a new threshold to our options object:

export let options = {
    // ...
    thresholds: {
        "http_req_duration": ["p(95)<250"]
    }
};

In this case, 95th percentile response time must be below 250 ms. If the response time is higher, then the test will fail. This is the signal to our CI tool that this step has failed and thus the build requires further attention.

Thresholds are extremely versatile, you can set them to evaluate just about any numeric aggregation of the response time returned. The most common are maximum or 95th/99th percentile metrics. The documentation can be found in the Thresholds section.

Now let’s put it together in a single script:

import { sleep } from "k6";
import http from "k6/http";

// Test configuration
export let options = {
    stages: [
        { duration: "10s", target: 15 },
        { duration: "20s", target: 15 },
        { duration: "10s", target: 0 }
    ],
    thresholds: {
        "http_req_duration": ["p(95)<250"]
    }
};

export default function() {
    let res = http.get("http://test.loadimpact.com/");
    sleep(5);
};

Add loadtest.js and commit the changes to your repository.

If you have installed k6 in your local machine, you could run your test locally in your terminal using the command: k6 run loadtest.js.

Running local tests on Azure Pipelines

Azure Pipelines allows you to automate processes writing scripts in a really simple manner.

For learning purposes, we are going to show a minimal example that we will execute in Azure’s virtual machine the test we wrote before:

# azure-pipelines.yml

pool:
  vmImage: 'ubuntu-18.04'

steps:
- script: |
    docker run -v `pwd`:/src -i loadimpact/k6 run /src/loadtest.js
  displayName: Run k6

We run an Ubuntu image with Docker preinstalled, so we can use it to easily run our test. This will be the equivalent of executing: k6 run loadtest.js.

This example standalone may not correspond to a real pipeline, but it will give you ideas to easily extend your current CI scripts.

Do not forget to add azure-pipelines.yml and commit the changes to your repository. After pushing your code, head over to the Azure dashboard and visit the job that triggered the git push. Below is the succeeded job:

Azure Pipeline Job running a local load test

Running cloud tests on Azure Pipelines

Running your tests in our cloud is totally transparent for the user, because no changes are needed in your scripts. Just run k6 cloud loadtest.js and you are ready to go!

Cloud tests can be useful in these common cases:

  • You want to run a test from one or multiple geographic locations (load zones).
  • You want to run a test with high-load that will need more compute resources than provisioned by the CI server.
  • Get automatic analysis of the results.

By default, the cloud service will run the test from N. Virginia (Ashburn). But, we will add some extra code to our previous script to select other load zone:

export let options = {
    // ...
    ext: {
        loadimpact: {
            name: "test.loadimpact.com",
            distribution: {
                loadZoneLabel1: { loadZone: "amazon:ie:dublin", percent: 100 }
            }
        }
    }
};

This will create 100% of the virtual users in Ireland. If you want to know more about the different Load Zones options, read more here.

Now, we recommend you to test how to trigger a cloud test from your machine. Execute the following command in the terminal:

k6 login cloud
k6 cloud loadtest.js

And, let’s create a pipeline script:

# azure-pipelines.yml

pool:
  vmImage: 'ubuntu-18.04'

steps:
- script: |
    docker pull loadimpact/k6
  displayName: Pull k6 image
- script: |
    docker run -i -e K6_CLOUD_TOKEN=$(k6cloud.token) -v `pwd`:/src loadimpact/k6 cloud /src/loadtest.js
  displayName: Run k6 cloud

To make this script work, you need to get your account token from the Integrations section and add it as a variable. The Azure Pipeline command will pass your account token to the Docker instance as the K6_CLOUD_TOKEN environment variable, and k6 will read it to authenticate you to the LoadImpact Cloud Service automatically.

Azure Pipeline Variable for k6 API Token

 

With that done, now we can go ahead and git add, git commit and git push the changes we have made in the code and initiate the CI job.

It is essential to know that the Azure Pipeline task prints the URL of the test result in the LoadImpact Cloud Service. You could navigate to this URL to see the result of your cloud test.

Azure Pipeline Job running a cloud load test

 

k6 cloud test result

 

We recommend that you define your performance thresholds in the k6 tests in a previous step. If you have configured your thresholds properly and your test passes, there should be nothing to worry about. If the test fails, you will want to visit the test result on the cloud service to find the cause of the failure.

Nightly build

It’s common to run some load tests during the night when users do not access the system under test. For example, to isolate larger tests from other types of testing or to generate periodically a performance report.

Triggering a subset of performance tests at a specific time is considered a best practice for automating your performance testing.

Below is the first example configuring a scheduled trigger that runs at midnight (UTC) of everyday, but only if the code has changed on master since the last run.

# azure-pipelines.yml

pool:
  vmImage: 'ubuntu-18.04'

steps:
- script: |
    docker run -i -v `pwd`:/src loadimpact/k6 run /src/loadtest.js
  displayName: Run k6

schedules:
- cron: "0 0 * * *"
  displayName: Daily midnight build
  branches:
    include:
    - master

The k6-azure-pipelines-example repository contains additional examples to setup k6 with Azure Pipelines.

Summary

Writing Azure Pipelines executing k6 just takes a few minutes to set up.

Integrate performance testing into your CI/CD process and it will help your software to comply with your SLO or SLA requirements and detect performance regressions.

If you are a Microsoft user looking for an alternative to their deprecated load testing service, we created another article for presenting k6 as an alternative to Visual Studio Load Tests.

Loading...