Introducing Distributed Load Testing Using locust on Azure Kubernetes Service ( AKS )

Today I’m open-sourcing distributed-load-testing-using-locust-on-aks, making an easy to scalable distributed load testing using Azure Kubernetes Service (AKS) and locust.

If you’ve used a performance tools, then you are probably familiar with the concept of Distributed Load Testing. Distributed Load Testing lets you check your application for High Availability and Scalability, and Resiliency, and Reliability.

However, Azure currently does not offer a managed service for load test, so you need to prepare a performance tool on Azure. If you want to check performance, you can quickly get a distributed load testing by using distributed-load-testing-using-locust-on-aks.

Table of Contents

  1. What does distributed-load-testing-using-locust-on-aks solve?
  2. What does distributed-load-testing-using-locust-on-aks do?
  3. Getting started with distributed-load-testing-using-locust-on-aks
  4. How does distributed-load-testing-using-locust-on-aks work?
  5. Conclusion

1. What does distributed-load-testing-using-locust-on-aks solve?

Two prominent issues led me to develop distributed-load-testing-using-locust-on-aks:

1 Azure currently does not have a managed service for load testing. So you need to prepare a tool for load testing yourself.

I want to select a simple tool and also can run on my laptop pc.

2 Even if you prepared a performance tool, your performance tool might have a bottleneck. So your application might not be a correct result of performance. Currently, Web Applications often have auto-scale.

You need to solve the bottleneck of this tool. Then, performance tests can confirm where broken in your application. And, your applications can demonstrate degenerate driving methods appropriate. And the application can establish scaling at the right time.

2. What does distributed-load-testing-using-locust-on-aks do?

It precisely solves the two problems listed above:

  1. distributed-load-testing-using-locust-on-aks solves a complex operation using locust on Azure Kubernetes Service (AKS):
  2. distributed-load-testing-using-locust-on-aks builds a scalable environment for your workload using locust on Azure Kubernetes Service (AKS) :

3. Getting started with distributed-load-testing-using-locust-on-aks

distributed-load-testing-using-locust-on-aks has the bootstrap script. So you can run bootstrap script.

$ script/bootstrap

If you have Azure Kubernetes Service (AKS) environment, you might apply kubernetes/ manifest files to your Azure Kubernetes Service (AKS).

you need run kubectl port-forwardcommand because you can access WebUI.

$ kubectl port-forward svc/locust-master 8089:8089 -n locust(Command+t)$ open http://localhost:8089# Set your target URL

Please check out distributed-load-testing-using-locust-on-aks and consider trying out the project and giving feedback.

I did a load test on a sample application in a repository called myapp, assuming a scenario that I experienced in the past. This scenario is based on a spike that occurred when the application was introduced on TV.

Statistics
Request Transition Graph

I checked myapp can scalable application because of application can scale until request peak.

$ kubectl get hpa -A -w
NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
locust locust-hpa Deployment/locust-worker 100%/40% 2 100 5 5m15s
myapp myapp-hpa Deployment/myapp 29%/40% 1 100 1 2m31smyapp myapp-hpa Deployment/myapp 59%/40% 1 100 1 2m47slocust locust-hpa Deployment/locust-worker 99%/40% 2 100 5 5m32smyapp myapp-hpa Deployment/myapp 59%/40% 1 100 2 3m2slocust locust-hpa Deployment/locust-worker 99%/40% 2 100 10 5m47smyapp myapp-hpa Deployment/myapp 59%/40% 1 100 2 3m18smyapp myapp-hpa Deployment/myapp 73%/40% 1 100 2 3m48slocust locust-hpa Deployment/locust-worker 95%/40% 2 100 10 6m32slocust locust-hpa Deployment/locust-worker 95%/40% 2 100 20 6m47slocust locust-hpa Deployment/locust-worker 95%/40% 2 100 24 7m3slocust locust-hpa Deployment/locust-worker 76%/40% 2 100 24 7m33smyapp myapp-hpa Deployment/myapp 89%/40% 1 100 2 4m49slocust locust-hpa Deployment/locust-worker 76%/40% 2 100 41 7m47smyapp myapp-hpa Deployment/myapp 89%/40% 1 100 4 5m3smyapp myapp-hpa Deployment/myapp 89%/40% 1 100 5 5m19slocust locust-hpa Deployment/locust-worker 59%/40% 2 100 41 8m33smyapp myapp-hpa Deployment/myapp 59%/40% 1 100 5 5m49slocust locust-hpa Deployment/locust-worker 59%/40% 2 100 47 8m48slocust locust-hpa Deployment/locust-worker 59%/40% 2 100 47 9m34smyapp myapp-hpa Deployment/myapp 59%/40% 1 100 5 6m50slocust locust-hpa Deployment/locust-worker 59%/40% 2 100 47 10mmyapp myapp-hpa Deployment/myapp 59%/40% 1 100 5 7m51slocust locust-hpa Deployment/locust-worker 59%/40% 2 100 47 11mmyapp myapp-hpa Deployment/myapp 59%/40% 1 100 5 8m52smyapp myapp-hpa Deployment/myapp 59%/40% 1 100 5 9m53slocust locust-hpa Deployment/locust-worker 60%/40% 2 100 47 13mmyapp myapp-hpa Deployment/myapp 59%/40% 1 100 5 10m

As you can see, it scaled from 2 locust workers to a maximum of 47. Also, one of myapp’s responses went down due to the sudden load, but myapp could scale up to 5 workers before the peak.

When the performance test is done, let’s clean up.

$ script/clean

4. How does distributed-load-testing-using-locust-on-aks work?

distributed-load-testing-using-locust-on-aks builds locust on Azure Kubernetes Service (AKS). locust on Azure Kubernetes Service (AKS) has master and some workers. In addition, locust on Azure Kubernetes Service (AKS) has included hpa(horizontal pod autoscaler) setting for the worker. Therefore, distributed-load-testing-using-locust-on-aks realizes distributed load testing depending on the load.

Why use Kubernetes?

To increase the scalability of the load tool, I used hpa of kubernetes because manual scale is very difficult. Nowadays, web applications support scalability as a matter of course. I solved a bottleneck in my load tool this way.

Why use Azure Kubernetes Service (AKS)?

  • If you have an account in Azure, you can easily create and delete the Kubernetes infrastructure using az commands.
  • If you have a Web Application on Azure, you can performance test in Azure.

5. Conclusion

Using distributed-load-testing-using-locust-on-aks, you can easily use distributed load testing on Azure and get the experience that you’re used to in terms of end-to-end performance.

I hope this article explains why and how I built distributed-load-testing-using-locust-on-aks. Please make sure to check out the GitHub repository, star/fork it, and give feedback through the issue tracker or on Twitter(@koudaiii).

Cloud Solution Architect at Microsoft Japan Co., Ltd. Blogs are my own.