On-premise test tools, APMs, CEMs and server/network based monitoring solutions may not be giving you a holistic picture of your system's performance; cloud-based continuous testing can.
The answer might seem obvious to all you industry experts out there, but it's a question we get, and it's a reasonable question for a developer to ask. There are different tools used for different things, so an explanation is necessary.
Load Testing Tools
With a load testing tool, you create a large amount of traffic to your website and measure what happens to it. The most obvious measurement is to see how the response time differs when the web site is under the load created by the traffic. Generally, you want to find out either how many concurrent users your website can handle or you want to look at the response times for a given amount of concurrent users.
Think of it as success simulation: What happens if I have thousands of customers in my web shop at the same time. Will it break for everyone or will I actually sell more? Knowing a bit about how your website reacts under load, you may want to dig deeper and examine why it reacts the way it does.
When doing this, you want to keep track of various indicators on the web site itself while it receives a lot of traffic. How much memory is consumed? How much time spent waiting for disk reads and write? What's the database response time? (and there are plenty more questions to be asked after that)
Load Impact offers server metrics as a way to help you do this. By watching how your webserver (or servers) consume resources, you gradually build better and better understanding about how your web application can be improved to handle more load or just to improve response times under load.
Next up, you may want to start using the load testing tool as a development tool. You make changes that you believe will change the characteristics of your web application and then you make another measurement. As you understand more and more about the potential performance problems in your specific web application, you iterate toward better performance.
A site monitoring tool like our friends at New Relic might be somewhat of a distant relative to us, but it's definitely a different creature.
A site monitoring tool will send requests to your web site on a regular interval. If your web site doesn't respond at all or, slightly more advanced, answers with some type of error message, you will be notified. An advanced site monitoring tool can check your web site very often, once every minute for instance.
It will also test from various locations around the world to be able to catch network problems between you and your customers. A site monitoring tool should be able to notify you by email and SMS as soon as something happens to your site. You are typically able to set rules for when your are notified and for what events, such as "completely down," "slow response time" or "error message on your front page."
In recent years, functionality has become more advanced, and aside from just checking if your web site is up, you can test if entire work flows are working. For instance, if your customers can place an item in the shopping cart and check out. Most site monitoring tools also include reporting so you can find out what your service level have been like historically.
It's not unusual to find out that the web site you thought had 100 percent uptime actually has a couple of minutes of down time every month. By proper reporting, you should be able to follow if downtime per month is trending. Sounds like a good tool right?
We think it deserves to be mentioned that whenever you detect downtime or slow response time with a site monitoring tool, you typically don't know why it's down or slow. But you know you have problems and that's a very good start.
One or the Other?
Having a bit more knowledge about the difference between these types of tools, we also want to shed some light on how these can be used together. First of all, you don't choose one or the other type of tool, they are simply used for different things.
Like measuring tape and a saw, when you're building a house, you definitely want both. We absolutely recommend that if you depend on your website being accessible, you should use a site monitoring tool.
When fine tuning your web site monitoring tool, you probably want to set a threshold for how long you allow for a web page to load. If you have conducted a proper load test, you probably know what kind of response times are acceptable and when the page load times actually indicate that the web server has too much load.
Then, when your site monitoring tool suddenly alerts you about problems and you want to dig down and understand why, that's when the load testing tool becomes really useful. As long as the reason for your down time can be traced back to a performance problem with the actual web server, a load testing tool can help you a long way.
Recently, I had a client that started getting customer service complaints about the web site not working. The first step was to set up a website monitoring tool to get more data in place. Almost directly, the website monitoring tool in use was giving alerts. The site wasn't always down, but quite often it was rather slow. The web shop was hosted using a standard web hosting package at one of the local companies.
I quickly found out that the problem was that the web shop software was just using a lot of server resources, and this was very easy to confirm using a load testing tool. Now the client is in the process of moving the site to a Virtual Private Server where resources can be added as we go along. Both types of tools played an important role in solving this problem quickly.
Using Both Tools is Easier Than Ever!
Our favorite part about the API Economy is that the best tools in every facet of web development can work together if they choose, and that's exactly what we've done with New Relic.
Load Impact users can get a full view of their application and server performance under load by adding New Relic metrics to their load tests!
Check out the Support documentation here, and feel free to contact us if you have any additional questions.
***This post was updated 10/30/2015***