Integrating k6 with Apache Kafka

📖What you will learn

• How to setup and start Apache Kafka and ZooKeeper on your local machine
• How to configure Kafka to handle k6 output
• How to configure a k6 test to send the test output to Kafka

⚠️ This tutorial is outdated

This tutorial was written for k6 0.34 and below.

You can now use the xk6-output-kafka extension and follow the instructions in this tutorial.

Introduction

Apache Kafka is an open-source event streaming platform. It can be used to store and process millions of messages per second, making it a good fit for distributed real-time data processing solutions. Although it provides features like stream-processing and horizontal scalability, it remains easy to use: you send your messages to it (produce) and later pick them up (consume) for further processing.

If you want read more about Kafka, I recommend you to have a look at the official documentation.

k6 is an open-source load testing tool. It can produce hundreds of messages per second, as a result of testing a system or platform. These messages can be sent to a number of different destinations, like a JSON file or, as in this case, Apache Kafka.

Setup

To be able to send messages from k6 to Apache Kafka, we’d go with a very simple setup, with only one instance. In our scenario, k6 would act as a simple producer of JSON messages and we'll try to consume message on terminal, using the built-in simple kafka consumer. Here I will use the official binary provided by Apache and would leave the docker setup up to you to try. To give you a clue, using Lenses.io's fast-data-dev, you can have a complete Docker environment with Kafka, ZooKeeper, Schema Registry, Kafka-Connect, Landoop Tools and more than 20 connectors in an easy to use package.

Since we're using the binary package, no special installation is needed, and you can directly invoke the binary from the command line. I have used Debian 10.2 to run these commands. Simply follow the instructions below:

Download and extract Apache Kafka

The current version as of this writing is 2.3.0, but you can choose your own desired version.

$ wget http://apache.mirrors.spacedump.net/kafka/2.3.0/kafka_2.12-2.3.0.tgz
$ tar xvf kafka_2.12-2.3.0.tgz
$ cd kafka_2.12-2.3.0

Starting ZooKeeper and Kafka

As you can see, when you start the ZooKeeper, it will start listening on all interfaces (0.0.0.0) on port 2181.

$ bin/zookeeper-server-start.sh config/zookeeper.properties

Now start Kafka server in a new terminal. It will connect to your local ZooKeeper instance on port 2181 and will start listening for new connections on port 9092.

$ bin/kafka-server-start.sh config/server.properties
[2019-11-15 12:43:54,672] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
...
[2019-11-15 12:43:55,366] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
...
[2019-11-15 12:43:55,666] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Creating a k6 topic

On another terminal, create a topic, for example k6-output, to receive k6 messages on. This topic would be a single partition topic with no replication.

$ bin/kafka-topics.sh --create \
    --bootstrap-server localhost:9092 \
    --replication-factor 1 \
    --partitions 1 \
    --topic k6-output

Executing our test

Running k6 with Kafka output

Using the following script in a file named scenario.js, we’ll start k6 to send its output to Kafka:

import { check, sleep } from 'k6';
import http from 'k6/http';

export const options = {
  duration: '1m',
  vus: 100,
};

export default function () {
  let res;
  res = http.get('https://httpbin.org/get');
  check(res, { 'status is 200': (r) => r.status === 200 });

  res = http.get('https://httpbin.org/bearer', {
    headers: { Authorization: 'Bearer da39a3ee5e6b4b0d3255bfef95601890afd80709' },
  });
  check(res, {
    'status is 200': (r) => r.status === 200,
    'is authenticated': (r) => r.json()['authenticated'] === true,
  });

  res = http.get('https://httpbin.org/base64/azYgaXMgYXdlc29tZSE=');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'k6 is awesome!': (r) => r.body === 'k6 is awesome!',
  });

  sleep(1);
}

$ k6 run \
    --logformat=raw \
    --out kafka=brokers=localhost:9092,topic=k6-output,format=json \
    scenario.js

Consuming messages

To view messages of k6 output on your terminal, use the following built-in consumer:

$ bin/kafka-console-consumer.sh \
    --bootstrap-server localhost:9092 \
    --topic k6-output

As you can see, we’ve sent our messages with JSON format to Kafka, so you'll now be able to build a JSON consumer in your own desired programming language (e.g. Python) and start consuming and using the data provided by k6 to Kafka.

There are other possibilities, like using platforms that can import or consume data from Apache Kafka, like Apache Spark or Kafka Connect from Confluent Platform, to name a few. You can also use visualization platforms that has support for Kafka as input. I recommend reading the following articles about it:

• Just Enough Kafka for the Elastic Stack, Part 1 and Part 2
• How the k6 Load Testing Tool is Leveraging Grafana

Final words

I hope you've found this article helpful and were able to send your k6 output to Apache Kafka, an industry-proven technology.

If you have any suggestions or comments, I'd really like to hear from you.