When to Leverage Commercial Load Testing Services, and When to Go it Alone

How and where you execute load and performance testing is a decision that depends on a number of factors in your organization and even within the application development team.

It is not a clear cut decision that can be made based on the type of application or the number of users, but must be made in light of organizational preferences, cadence of development, timeline and of course the nature of the application itself and technical expertise currently on staff.

In this post we will provide some context around some of the key decision points that companies of all size should consider when putting together load NS performance testing plans.

This discussion is really an amalgamation of On-Premise versus SaaS/Open-Source versus Commercial Services.

In the load testing space there are commercial offerings that offer both SaaS and on-premise solutions as well as many SaaS only solutions for generating user load.

From an open source perspective, JMeter is the obvious choice (there are other less popular options such as FunkLoad, Gatling, Grinder, SOAPUI, etc). Having said that, let’s look at the advantages and challenges of the open source solution,  JMeter, and contrast that with a cloud-based commercial offering.

Key JMeter Advantages:

  1. 100% Java application so it can be run on any platform (windows, osx, linux) that can run Java.
  2. Ability to test a variety of types of servers – not just front end HTTP servers.  LDAP, JMS, JDBC, SOAP, FTP are some of the more popular services that JMeter can load test out of the box.
  3. Extensible, plug-in architecture. The open source community is very active in development around JMeter plugins and many additional capabilities exist to extend reporting, graphing, server resource monitoring and other feature sets.  Users can write their own plugins if desired as well.  Depending on how much time and effort is spent there is little that JMeter can’t be made to do.
  4. Other than the time to learn the platform there is no software cost of course since it is open source.  This may be of particular value to development teams with limited budget or who have management teams who prefer to spend on in-house expertise versus commercial tools.
  5. It can be easy to point the testing platform at a development server and not have to engage the network or server team to provide external access for test traffic.  It’s worth noting that while this is easier it is also less realistic in terms of real world results.

Key JMeter Disadvantages:

  1. Being that it is open source you do not have an industry vendor to rely upon for support, development or expertise.  This doesn’t mean that JMeter isn’t developed well or that the community isn’t robust – quite the opposite. Depending on the scope of the project and visibility of the application it can be very helpful to have industry expertise available and obligated to assist.  Putting myself in a project manager’s shoes, would I be comfortable telling upper management, “we thoroughly tested the application with an open source tool with assistance from forums and mailing lists?” if there were to be a major scale issue discovered in production?
  2. It’s very easy to end up with test results that aren’t valid.  The results may be highly reliable – but reliably measuring bottlenecks that have nothing to do with the application infrastructure isn’t terribly useful.  Since JMeter can be run right from a desktop workstation, you can quickly run into network and CPU bottlenecks from the testing platform itself – ultimately giving you unrealistic results.
  3. Large scale tests – not in the wheelhouse of JMeter.  Right in the documentation (section 16.2 of best practices) is a warning about limiting numbers of threads.  If a truly large scale test is required you can build a farm of test servers orchestrated by a central controller, but this is getting pretty complicated, requires dedicated hardware and network resources, and still isn’t a realistic real-world scenario anyway.
  4. The biggest disadvantage is inherent in all on-premise tools in this category in that it is not cloud based.  Unless you are developing an in-house application and all users are on the LAN, it does not makes a ton of sense to rely (entirely) on test results from inside your network.  I’m not suggesting they aren’t useful but if users are geographically distributed then testing in that mode should be considered.
  5. Your time: doing everything yourself is a trap many smart folks fall into, and often times at the expense of project deadlines, focus. Your time is valuable and in most cases it could be better spend somewhere else.

This discussion really boils down to if you like to do things yourself or if the project scope and criticality dictate using commercial tools and expertise.

For the purposes of general testing, getting familiar with how load testing works and rough order of magnitude sizing, you can certainly use open source tools on your own – with the caveats mentioned.  If the application is likely to scale significantly or have users geographically distributed, then I do think using a cloud based service is a much more realistic way to test.

In addition to the decision of open source versus commercial tools is if professional consulting services should be engaged.  Testing should be an integral part of the development process and many teams do not have expertise (or time) to develop a comprehensive test plan, script and configure the test, analyse the data and finally sort out remediation strategies on their own.

This is where engaging experts who are 100% focused on testing can provide real tangible value and ensure that your application scales and performs exactly as planned.

A strategy I have personally seen work quite well with a variety of complex technologies is to engage professional services and training at the onset of a project to develop internal capabilities and expertise, allowing the organization to extract maximum value from the commercial product of choice.

I always recommended to my customers to budget for training and service up front with any product purchase instead of trying to shoe-horn it in later, ensuring new capabilities promised by the commercial product are realized and management is satisfied with the product value and vendor relationship.

——

Peter CannellThis post was written by Peter Cannell. Peter has been a sales and engineering professional in the IT industry for over 15 years. His experience spans multiple disciplines including Networking, Security, Virtualization and Applications. He enjoys writing about technology and offering a practical perspective to new technologies and how they can be deployed. Follow Peter on his blog or connect with him on Linkedin.

Don’t miss Peter’s next post, subscribe to the Load Impact blog by clicking the “follow” button below. 

#UnsexyTech and trying to make it a little sexier

We think what we do is pretty cool. I mean come on! Performance and load testing, who doesn’t get excited at the idea?!

Well, apparently not everyone. Some have even said performance testing is a bit like selling health insurance: most people know it’s important to have, but you don’t reap the benefits of having it until something unexpected happens.

In any event, we wanted to try and find a way to explain what we do in a more relatable and humorous way. Framing our somewhat “unsexy tech”  in a way that connects back to everyone’s everyday lives.

Well, here is is. With the help of our video producers Adme (props to a job well done), we made this nifty short video to explain what we do and, if possible, make you chuckle a little. Enjoy!

 

 

Are you working with unsexy tech? Let us know why you think your tech is super sexy in the comments below.

Performance & Scalability Test: Pantheon -VS- Other Drupal-Based Platforms

In this post I will demonstrate how to  setup a Drupal 7 site from scratch using Pantheon. Then, using Load Impact, I will attempt to compare the performance of Pantheon versus other Drupal-based platforms (Aberdeen Cloud platform and VPS).

Pantheon

For those who don’t know Pantheon, it’s a cloud platform for web development using different frameworks such as Drupal or WordPress. It provides development, testing and live environments for easy deployment using git. And since it’s built using containers instead of virtual machines, it’s performance and scalability is considerably better than traditional hosting providers.

Pantheon-howItWorks copy

Pantheon also provides a platform with preconfigured tools such as Varnish, Redis, Apache Solr, automated backups and application updates, etc.

For this performance test, we are going to need to create a Pantheon user account.

As a registered Pantheon user, we now see the Pantheon control panel:

1-your-sites

Now we need to create a site in order to install Drupal and run a performance test.

Steps to creating a Drupal 7 site using Pantheon

1. After registering an account with Pantheon, click “Create a site now” link.

2-account-ready

2. Next, provide some information about the site, such as the name and the access URL. Later, you can easily redirect your existing domain if you need it.

Press the “Create Site” button to continue with the process.

3-create-site

 

3. Create or import a site

You can create a site from scratch using one of the available web application cores for Drupal or WordPress, or you can import an existing one – packing your application files and database dump.

For this test, we are going to start a Drupal 7 site from scratch, but feel free to import your existing application if you want to compare its performance in Pantheon.

Select “Start from scratch” option and one of the available cores – in this case Drupal 7.

Press “Install Drupal 7” button to continue with the site installation.

Pantheon will setup the site with the best suitable configuration for Drupal 7.

6-create-drupal-site

 

4. Use the dashboard and configure your application

Once the site is completely installed you will have access to the site Dashboard where you can configure different aspects of your application, create backups and use the Pantheon tools.

10-dashboard

Now it’s time to install Drupal. As you can see, the site installer only copied Drupal 7 files but it didn’t execute Drupal installation.

You have to do it manually, so just press “Visit Development Site” button.

11-visit-development-site

5. Install Drupal

In this case, you are going to install Drupal 7 using a Pantheon profile – a custom Drupal distribution preconfigured with Pantheon modules and custom configuration that enables Drupal caching and provides better performance, and an Apache Solr integration.

Press “Save and continue” button to go to next installation step where you can select default language for Drupal. You can install Drupal 7 in other languages – but that can be done later.

Press “Save and continue” button again and Drupal will install with pre-selected Pantheon configuration.

After the installation you have to configure site details, such as site name or administrator username, password and email.

Once you’ve completed the form details, Drupal should be installed and ready to use.

12-drupal-choose-profile

 

6. Start using Drupal

To start using Drupal just press “Visit your new site” link and you will access your brand new Drupal 7 website.

7.  Setup your git repository in your local machine

Now it’s time to set up your git repository in your local machine so you can add or modify your Drupal website.

First of all, you need to add your SSH public key into your Pantheon user account. Just go to “Sites & Account” in your user menu.

18-sites-account

If you don’t already have a SSH key you have to create it. You can find more information about SSH key generation here: http://helpdesk.getpantheon.com/customer/portal/articles/366938-generate-ssh-keys

Once you have created your SSH key, you should add your Public Key just by pressing the “Add key” button. For more information about SSH keys, visit this page: http://helpdesk.getpantheon.com/customer/portal/articles/361246-loading-ssh-keys

Now you can connect to Pantheon git repository using SSH.

8. Configure the repository in your local machine

To configure the repository in your local machine you should have git previously configured. If you need to install git just follow this guide for all platforms https://help.github.com/articles/set-up-git

First you should copy the path to connect to your server using SSH, because you are going to need it later. You can find it in your project dashboard.

20-dashboard

We are going to use Git as connection mode so be sure that it’s selected and copy the URL you find in the textbox. As you can see in the following screenshot:

21-connection-mode

Go to your terminal and execute the following command from the location where you want to create your project repository folder (Don’t forget to use the URL you previously copied):

$ git clone ssh://codeserver.dev.xxx@codeserver.dev.xxx.drush.in:2222/~/repository.git drupal-loadimpact

If everything goes well, the command will create a folder named drupal-loadimpact with all the files corresponding to Drupal 7 core.

9. Install devel module

Now we are going to install devel module. This Drupal module will help us to creat thousands of nodes in our Drupal website.

A node is a Drupal piece of content with multiple fields and properties, and we are going to use them for our performance test.

You can download the module using wget or drush, but remember to place the module files into sites/all/modules directory in your repository folder.

Then add the new files for commit using the following command:

$ git add sites/all/modules/devel

Commit the changes into your local repository using a descriptive message:

$ git commit -m"Added devel module."

And finally push the changes to the remote repository:

$ git push origin master

You can check that everything went well by checking the commit log in your project dashboard.

22-devel-module-commit

You can see the new commit with your descriptive message.

10.  Install devel module by enabling it in Drupal

Select “Modules” menu option from the top menu in your Drupal site.

Enable only Devel and Devel generate modules.

23-drupal-modules

 

11. Generate content 

Now select “Configuration” menu option from the top menu. And then select “Generate content” option.

25-devel-menu

 

In this example, we are going to create 5000 nodes of type Article for our performance test. It should be enough to have a high amount of content in the Drupal database and create different performance tests.

27-devel-generate-content

Press “Generate” button and relax because it’s going to take a while. You can take a cup of coffee ;)

.... (coffee break)

So, here you have your 5000 recently created articles. To see them just select “Content” from the top menu.

29-drupal-content

12. Create page listing all content (for performance testing purposes)

Now we are going to create a page that will list all the content so it will create a high amount of database queries and render a lot information in the page.

For that we will need Views (https://www.drupal.org/project/views) and Chaos Tools Suite (https://www.drupal.org/project/ctools) modules.

You can download them using the same method you used to download Devel module. Don’t forget to place them in sites/all/modules folder to let Drupal to find them and be able to install them.

Once you have downloaded the modules into your local repository, you can add and commit the modules into your local repository as we did before:

$ git add sites/all/modules/views
$ git commit -m”Added views module.”
$ git add sites/all/modules/ctools
$ git commit -m"Added ctools module."

Now push the changes to your remote repository:

$ git push origin master

Once again, you can see your commits in the Commit Log in your project dashboard and verify that everything went well.

To enable the modules, select “Modules” option in your top menu:

Enable Chaos tools module because it’s a dependency of Views module:

32-chaos-tools

Also enable Views and Views UI modules:

33-views-module-enable

Now select “Structure” option form the top menu to start creating your view page.

Then select “Views” option.

On this page you can find a list of preconfigured views. We are going to create our own view page with a list of a specific number of content articles.

35-views-list

 

 

13. Create and configure a new View

Select “Add new view” link to start.

Create a new view using the following configuration and press “Continue & edit” button.

37-views-create

This page is the configuration panel for your view. You can add fields, sorting order and filters as well as arguments and other custom features.

38-views-add-fields

We are going to add a few fields to display more content in the page.

Now you can see that there is only Title field available. Press the “Add” button in “Fields” section.

In the next window select Body and Image fields and press “Apply (all displays)” button with default settings for all fields.

40-views-select-fields

Now your “Fields” section should contain the following:

  • Content: Title
  • Content: Body(body)
  • Content: Image(image)

Press “Save” button to save your new view.

14. Visit your new page

Now you should be able to visit your page. The page we created for testing is here: http://dev-drupal-loadimpact.gotpantheon.com/performance-test

You should see a page with 100 article nodes -  which is a normal amount of data for Drupal to load.

It will create a considerable number of database queries and processes to render all the content into the page.

Consider to increase or decrease the number of items in the “Pager” section of your view if you want to test with different amount of data loaded.

You can select different values to adapt the performance test to your real needs.

43-views-pager-config

 

We have to take into consideration the default configuration that Pantheon profile applies to Drupal.

If you select “Configuration” in the top menu you can find “Performance” menu item:

44-performance-link

In this page you can find configuration settings related to Drupal performance. You can see that there are a few settings already enabled.

Page and block caches are enabled for 15 minutes. Also, CSS and JavaScript are aggregation is enable - what packs all CSS and JavaScript files and build them together in order to decrease the number of requests to the server to download all the files.

45-performance

It’s important to clear caches every time you run a new performance test to be sure that you don’t have cached pages. Just press “Clear all caches” button.

Testing the performance of Drupal-based platforms

Now it’s time for performance testing using Load Impact. Create a user account if you don’t already have one.

After that visit “My tests” to start creating your performance test and press “Start a new test” button. This will execute a rather small test of 50 concurrent users for 5 minutes.

47-my-tests

Type your test page URL into the text box and press “Start test” button.

You can also configure a larger test from the "Test configurations" page (e.g. with ramp-up/down; additional IPs, multi-geo load generation; mobile network emulation, server metrics, etc.)

48-start-test

The test will start and you will see how Load Impact is creating requests from different hosts.

50-test-running

 

Pantheon results

The following are the performance test results for the same Drupal site (using the same configuration and number of content nodes) hosted in Pantheon and Aberdeen Cloud platforms and a Virtual Private Server:

51-pantheon-test

Pantheon results (user load time)

Aberdeen Cloud 

52-aberdeencloud-test

Aberdeen Cloud results (user load time)

VPS (CPU: 1.5 Ghz. - RAM: 1 Gb.)

53-VPS-test

VPS results (user load time)

 

You can observe that Pantheon keeps user load time between 1.5 and 3 seconds, meanwhile with Aberdeen Cloud platform and the VPS, the user load time stays between 3 and 4 seconds.

Based solely on these few simple load tests, it seems Pantheon manages to serve Drupal pages at least one second faster than the other tested platforms. 

Now it’s time for you to try different configurations for the Drupal site, like number of content nodes, disable caching or file aggregation and see how it affects performance and scalability.

Read more about load and performance testing with Pantheon: http://helpdesk.getpantheon.com/customer/portal/articles/690128-load-and-performance-testing

--------

This blog post was written by Ruben Teijero. Follow the Load Impact blog (see below) for more great articles by Ruben and many other fantastic developers, testers and operations pros.

5 Lessons Learned After Self-Hosting Goes Haywire

When things start to go wrong it can sometimes be impossible to contain the unravelling – as if the problematic situation quickly gains momentum and begins to ‘snowball’ into an even worse situation.

This happened to me recently. And since much of what went wrong could have been prevented with good process controls, I believe I have some valuable lessons to share with you.

At the very least this post will be entertaining, as I assume many of you reading this will think to yourself: “yep, been there done that”.

I’ll start by mentioning how I met the folks at Load Impact and started working with their product and writing for them.

I was doing some shopping for a hosting provider for my personal and business website, and ran across someone else’s blog post that tested the performance of all the major players in the ‘affordable web hosting’ segment.  We are talking the $8/month type deals here – the bare bones.

This author used Load Impact to quantify the performance of all these providers and provided great insight into how they faired from a performance and scalability perspective.

My first thought was: awesome! – I’ll use that same tool and test a few out myself, and then compare it to the performance of a self-hosted site.  I already had a bunch of VMs running on an ESXI server so adding a turnkey wordpress site would be super easy.

It turns out that my self hosted site was much faster and scaled all I needed (thanks to Load Impact), so in the end I decided to just self host.

I’m not making any money from the sites – no ecommerce or ads – so it doesn’t really matter from a business perspective.It’s also easier to manage backups and control security when you manage the whole environment.

But it’s also much more likely that the whole thing will get screwed up in a major time-consuming way.

I imagine there are many SMBs out there that self host as well, for a variety of reasons.  It could be that you like having control of your company assets, it was faster and cheaper, or you just like doing everything yourself.

It’s often very difficult for smart people to avoid doing things they can do but probably shouldn’t do as it might not be the best use of their time.

In this blog post I’ll demonstrate how how quickly this situation can go wrong and then go from bad to worse:

Problem #1: my ISP screwed me!

If you are in business long enough, your ISP will screw you too.  I made a change to my service plan (added phone line) which I did the week before we went out of town.

For some reason nothing happened so I decided called my provider while 300 miles away from my house. Of course, this is exactly when things started to unravel.

Instead of provisioning my modem correctly, they removed my internet service and added phone.  No internet.  To make matter worse,  I’m not at home so I can’t troubleshoot.

Lesson #1 – don’t make changes with your ISP unless you can be onsite quickly to troubleshoot.

It was nearly impossible for me to troubleshoot this issue as I couldn’t VPN into my network, there wasn’t a connection at all.

I even had a neighbor come in and manually reboot both my firewall and modem.  That didn’t work, so my only recourse was a dreaded call to customer support.

The first time I called it was a total waste of time, the Customer Support agent had no idea what was going on so that call ended.

Call number two the next day was slightly more productive in that it ended 45 minutes later and a level 2 support ticket was opened.

Finally, upon getting a level 2 engineer on the line (I was home at this point), they immediately recognized that my modem was mis-provisioned and was setup for phone only!  It only took minutes to properly provision the modem and get it back online.

Lesson #2 – if you are technically savvy, then immediately demand a level 2 support engineer. Time spent with first line support is usually a totally frustrating time suck.

giphy

Problem #2: Some things start working again and others mysteriously don’t 

After the final problem-resolving phone call was complete I was tired, hot (AC was off while out of town) and irritated. So when the internet connection finally came back up, I wasn’t exactly in a “I’m making great decisions” mindset.

So I go to check my site and one works fine, but this one it not up at all.  I reboot the VM but still no response from the server.

I’m not sure what is going on.

Lesson #3 – Don’t start making significant changes to things when tired, hot and irritated.  It won’t go well.

This is exactly the point at which I should have made a copy of the VM in it’s current state to make sure I don’t make things worse.  But instead I immediately go to my backup server (Veeam) and try to restore the VM in question.

Well guess what?  That didn’t work either, some sort of problem with the storage repository for Veeam.  Unfortunately, the problem is that some of the backup data is corrupt.

I ended up with a partially restored but completely unusable webserver VM.

giphy-2

Lesson #4 – Test your backups regularly and make sure you have more than one copy of mission critical backups.  

At some point in this whole fiasco, I remembered what this package I had on my desk was.  It was a replacement hard drive for my ZFS array because one of my 4 drives in the RAIDZ1 array was “failing”.

I figured that now would be the perfect time to swap that drive out and allow the array to heal itself.

Under normal circumstances this is a trivial operation, no big deal.  Not this time!

This time, instead of replacing the failing hard drive, I accidentally replace a perfectly good drive!

So now I have a really tenuous situation with a degraded array that includes a failing hard drive and no redundancy whatsoever.

Fortunately there wasn’t any real data loss and eventually I was able to restore the VM from a good backup source.

Finally back online!

Lesson #5 – Be extra diligent when working on your storage systems and refer to Lesson #3.

The overall message here is most, if not all, of these issues could have been easily avoided. But that is the case 99% of the time in IT – people make mistakes, there is a lack of good well documented processes to handle outages, and of course hardware will fail.

It’s also worth noting that in large enterprises mechanisms for change control are usually in place – preventing staff from really messing things up or making changes during business hours.

Unfortunately, many of smaller businesses don’t have those constraints.

So what does this have to do with Load Impact?  Nothing directly…but I think it’s important for people to be aware of the impact that load and performance testing can have on the infrastructure that runs your business and plan accordingly when executing test plans.

Just like you wouldn’t do something stupid like changing network configs, ISP settings or Storage without thoroughly thinking it through, you should also not unleash a world-wide load test with 10,000 concurrent users without thinking about when you should execute the test (hint – schedule it) and what the impact will be on the production systems.

Hopefully there is a test/dev or pre-production environment where testing can take place continuously, but don’t forget many times there are shared resources like firewalls and routers that may still be affected even if the web/app tiers may not be.

And always remember Murphy’s lawAnything that can go wrong will go wrong. 

———-

Peter CannellThis post was written by Peter Cannell. Peter has been a sales and engineering professional in the IT industry for over 15 years. His experience spans multiple disciplines including Networking, Security, Virtualization and Applications. He enjoys writing about technology and offering a practical perspective to new technologies and how they can be deployed. Follow Peter on his blog or connect with him on Linkedin.

Don’t miss Peter’s next post, subscribe to the Load Impact blog by clicking the “follow” button below. 

Load Testing Validates Performance Benefits of CDN – 400% Improvement (CASE STUDY)

Ushahidi used Load Impact to greatly improve the performance of its software. Through comparing “before” and “after” test results it was possible to see the performance impact of optimization efforts – like the use of a CDN. 

Ushahidi is a non-profit tech company that specializes in developing free and open source software for information collection, visualization and interactive mapping. Such software is deployed during disasters so that real time information can be shared across the web. Like WordPress, the software can be self hosted or hosted on the company’s server.

Case:

Ushahidi software is generally used for crisis and disaster situations so optimization is absolutely crucial. An earthquake reporting site based on Ushahidi software (http://www.sinsai.info/) received a spike in traffic after the earthquake and tsunami in Japan and it went down several times, causing service outage at the time the service was needed the most.

Ushahidi were interested in using a load testing tool to test the performance of their software before and after optimization efforts, to determine what effect the optimizatons had had.

Test setup:

There were four load tests run on two different versions of the Ushahidi software. The software was hosted on Ushahidi’s servers. The first two test runs used ramp-up configurations up to 500 concurrent users on the test sites to test performance differences between Ushahidi 2.0.1 and Ushahidi 2.1. The results were revealing, showing performance graphs that were practically identical. There hadn’t been any change in performance from 2.0.1 to 2.1.

From these tests, it was also found out that the theoretical total number of concurrent users for Ushahidi on a typical webserver is about 330 clients but may be lower, depending on configuration. Load times at the 330-client level were very high, however, and defining the largest acceptable page load time to be 10 seconds meant that a more realistic figure would be 100 concurrent users on the typical webserver.

Finally, Ushahidi wanted to measure the potential performance gain when using a CDN (content delivery network). The Ushahidi 2.1 software was modified so that static resources were loaded from Rackspace’s CDN service instead of the Ushahidi server, then the previous load test was executed again.

The result was a major increase in the number of concurrent users the system could handle. Where previous tests had shown a significant slowdown after 60-100 concurrent users, and an absolute max limit of about 330 concurrent users, the CDN-enabled site could handle more than 300 concurrent users before even starting to slow down. To find out the extreme limit of the site with CDN enabled, a final test was run with even higher load levels, and it was found that the server now managed to serve content at load levels up to 1,500 concurrent users, although with the same high load times as in the 330-client case with no CDN.

Service environment:

  • Apache
  • PHP
  • MySQL
  • Linux (CentOS 5.0)

Challenges:

  • Find load limits for 2 different software versions
  • Find load limits with/without CDN enabled for static files
  • Detect potential problems in the infrastructure or web app before they affect customers

Solution:

  • Run ramp-up tests with identical configurations on the 2.01 and the 2.1 software. See which one performs better or worse
  • Run ramp-up tests with identical configurations on the 2.1 software with CDN enabled, and without CDNenabled. See which performs better or worse.
  • Run final, large-volume ramp-up test for the CDN-enabled software, to find out its theoretical maximum concurrent user limit.

Results:

  • Ushahidi found out that there was a significant performance gain when using CDN to serve their static files.
  • Load test measured that performance increased by 300% – 400% when using the CDN
  • Load times started to increase only after 334 concurrent users when using the CDN, and the server timed out at around 1500 concurrent users.
  • Faster time to verify CDN deployment. Test also quantified % increase in performance which leads to justification for additional cost of CDN service.
  • Test showed no changes in load time between version 2.01 and 2.10.

Is Your Application as Mobile and Global as You Claim it is? – Prove it!

Your application has been localized, your website is responsive, you’ve even built a mobile app – how about your performance?! 

It takes more than a mobile app, responsive design and localization to stay ahead of the game, make sure your performance can also meet the demands of an increasingly mobile and global user-base.

Regardless of whether your applications are in a highly virtual, cloud based environment or a self-hosted single datacenter, realistic performance testing must take into account all the complexities that exist between applications and end users. In today’s highly mobile world, users can literally be anywhere in the world coming across connections that vary widely in quality and speed.

A successful application deployment must take into account factors that influence this Quality of Experience (QX) and integrate continuous testing that best simulates a wide variety of situations.

Not long ago, load testing was a simple and typically one-time test done to size hardware before a roll-out. Testing was nearly always done in-house and did not take into effect what the end user experience was like and how those variables could significantly affect not only user experience but server resources as well.

Gone are the days of users only using your application from a desktop, connected to their DSL at home, and located within the same national borders as your business. Depending on who you ask, by 2020 75% of commercial transactions and 50% of consumer spend will be mobile.

Already today, mobile accounts for 25% of all web usage globally – and 80% for China only. With internet penetration soaring in countries like China, Indonesia and Brazil, its no surprise that nearly all big US-based internet properties are seeing a larger portion of their traffic and users coming from abroad.

The 2014 Mary Meeker Internet Trends report revealed that 6 of the top 10 US-based internet properties that have global operations have more than 86% of their users coming from outside the US.

MaryMeeker copy

This shouldn’t come as a major shock to most applications teams, those who now know they must design either a mobile responsive page or a mobile app in addition to the traditional desktop browser to stay competitive, let alone make sure that a users’ experience is consistent regardless of geographic location.

So if applications teams are so focused on designing around an increasing mobile and global user base, wouldn’t it make sense to performance test in that mode as well – using geographically distributed load, simulating mobile networks, browsers and connections?

Here are a few key considerations and benefits of what a global/mobile approach will bring:

 1.  Browser Simulation

Users interact with applications from a wide variety of desktop and mobile browsers (or apps) today and there are very real differences in how each use case impacts scale.  It’s not good enough to simply assume every browser will follow caching and compression directives the same and that TCP connections issues will be consistent across the whole user base.

Additionally you have to take into account iPhone and Android OS types and multiple browsers on each platform.  Bottom line here is to use multiple user scenarios that include different browsers and platforms mixed in!

A realistic testing platform should simulate both desktop & mobile browsers

A realistic testing platform should simulate both desktop & mobile browsers

2.  Network Connections

One thing that’s for sure these days is an inconsistency when it comes to how users connect to an application.  Some users will have super low latency, google fiber connections (one can dream) that probably eclipse your datacenter circuit performance and others will be on a roaming 3G connection with tons of packet loss.

Even more challenging is what happens when a mobile user hands off from cellular data to WiFi and what that means to server resources (think  lots of FIN & WAIT TCP states) and experience.  A realistic test should include simulations for multiple connection types – DSL, 3G, LTE, unlimited, etc.  Even better would be a system that can introduce jitter and packet loss to mobile connections for the ultimate in realism and impact to server resources.

Being able to simulate different connection types and associated connection quality is also important

Being able to simulate different connection types and associated connection quality is also important

3.  Geo-Distributed Users

Users are going to be geographically distributed for just about any application these days, even intra-net only corporate applications. And they should expect a great user experience regardless of where they are.  At a bare minimum, testing within the continent where 80% of users will be located is recommended – going global is even better.  Being able to test from multiple geographies simultaneously during a single test is very valuable since you can then see exactly the differences in performance and user experience with the only variable being the user location.

If users are primarily US based then test multiple locations within the US - at least

If users are primarily US based then test multiple locations within the US – at least

However if users (or company execs) frequently travel abroad then test it!

However if users (or company execs) frequently travel abroad then test it!

A great user experience (sub 1-sec load times for example) is great but if that performance drops off a cliff as you move away from the datacenter then looking into a CDN (or a better CDN) may become a high priority.  If you are using distributed server resources and a complex CDN strategy, this is a great way to validate that all is working properly and you are getting the best value from the provider of choice.

The bane of most Ops teams’ existence is the “the app is slow” ticket, and the last thing a user will want to hear from a support reply is “not from here it’s not!”  A great way to identify early potential performance issues on a geographic basis is to continually test (ok maybe hourly or daily) and automate that process.

If a baseline is created then when performance numbers well outside of that reference range occur you can be proactive and not reactive.  If performance is slow from users in the UK but no where else and you have a quantitative analysis in hand, discussions with hosting and CDN providers takes on a much more productive tone.  Think of all the unnecessarily steps and level-1 troubleshooting that can be eliminated, all potentially before the first support ticket is opened for the UK slowness that you already were working on.

Consistently slower page load times from Australia might mean it's time for a new hosting resources or a CDN upgrade

Consistently slower page load times from Australia might mean it’s time for a new hosting resources or a CDN upgrade

With the tools available today, application teams have the ability to continuously test load and performance with highly realistic and sophisticated test scenarios. Performing this testing using a cloud based test platform removes on-premise test tool cost and deployment hassles and allows teams to test at every phase of a deployment including after the app goes live.

This type of approach can also help evaluate different hosting and CDN offerings well before the application goes live, and determine which providers offer the best value in the regions of the country or world you care most about. Taking a pro-active approach to monitoring the performance of applications, especially mobile applications where you are certain to face TCP connection issues, roaming from 4G to WiFI and a host of other mobile-centric challenges will go a long way to ensuring deployment success in a Dev-Ops fashion.

 

 

———–

Peter CannellThis post was written by Peter Cannell. Peter has been a sales and engineering professional in the IT industry for over 15 years. His experience spans multiple disciplines including Networking, Security, Virtualization and Applications. He enjoys writing about technology and offering a practical perspective to new technologies and how they can be deployed. Follow Peter on his blog or connect with him on Linkedin.

Don’t miss Peter’s next post, subscribe to the Load Impact blog by clicking the “follow” button below. 

We’re Hiring! Front End Lead Based in Stockholm

We are looking for a front end expert to take on the role of lead front end developer for the loadimpact.com site and application, being a part of our upcoming redesign of the whole site, service and user flow.

What is loadimpact.com then?

Load Impact is the world’s most widely used online load testing service, with over 100,000 users from 190+ countries and over a million executed load tests – Google “web load test”!

The company is small but very experienced in developing applications for performance testing and monitoring. We started out as a software development consulting company way back in 2000, developing performance test and measurement software for customers such as Nasdaq and the European space agency. We wrote both hardcore, low-level networking software and high-level web user interfaces and became quite good at both of those things. We pride ourselves on understanding the lower levels of a networked application while at the same time realizing how incredibly important UX is. In 2009 we took the step and launched loadimpact.com, becoming a product company rather than a consulting company, and we have never looked back.

We are located in Stockholm, Sweden and San Francisco, USA. This job opening is in Stockholm.

This is a chance to get the best of both worlds – you get to build something practically from scratch and decide how to do things while at the same time you’re supported by a successful, global business with a very strong user base and sound financial backing. You will be part of a small and very competent dev team, using the latest and greatest technologies and working in a modern and agile environment.

Being part of a small team means you get a lot of say in what happens in general, and can influence your own work situation. As a company, we are very “flat” (no hierarchies) and that means you’ll have a say in most decisions we make as a company, if you want to.

You need to be really, really good at:

HTML, CSS & Javascript

You need to be be knowledgeable about:

Usability / UI design
Common JS libraries/frameworks
Browser support/portability/quirks
Common data formats: JSON, XML
Responsive design

You should have some knowledge or experience of:

Client-side optimization (minification, sprites, lazy-loading etc)
Git / Github
SEO basics (the common-sense stuff, no black hat tricks thank you)
Common web servers: Apache, Nginx
Web analytics & growth hacking
SQL
Cloud operations (e.g. using Amazon EC2 and/or other AWS services)
Web application protocols: HTTP, HTTPS

The office is located in Stockholm, Sweden. A very scenic city, with excellent infrastructure, clean streets, low crime, many bars, fantastic coffee. A bit cool in winter, but who cares, we just buy more coffee. There is also a very vibrant tech startup scene in Stockholm, much thanks to companies like Spotify, Skype, King.com (Candy crush), Mojang (Minecraft) and others that started and are still operating here. The city is attracting more and more world-class IT entrepreneurs and developers – it is a very dynamic environment to work in, with a lot of industry colleagues to hang out with and learn from. Our office is very centrally located, by the way, with a fantastic view of the city harbour :)

 

photo

Actual view from our desks!

How to apply:

Email your resume and info to ragnar@loadimpact.com.

We look froward to hearing from you!

Deadline July 1, 2014.

Exception Handling in an AngularJS Web Application (Tutorial)

During this tutorial I will implement best practices for exception handling in an AngularJS web application.

I’ll start creating the structure of a demo application, throw some test exceptions and intercept them.

The second part of this tutorial will cover some frameworks that will help you organize your exception logs using Raven with Sentry.

Creating web app structure

Note: You can skip this section if you already know how to setup your web application with Yeoman and AngularJS.

In order to use Yeoman to create a basic structure of your web app with AngularJS, you need to have installed Yeoman on your computer. A basic tutorial to install it can be found here: http://yeoman.io/gettingstarted.html

Once installed, run the following command and follow the instructions. When asked to install SASS, reply “N”. Rest of settings/questions can be set as default.

$ mkdir exceptionHandling
$ cd exceptionHandling
$ yo angular

The command will create a structure where you can find the structure of your web app and other files to configure it with Grunt and Bower (for more information, check http://gruntjs.com/ and http://bower.io/).

Now you are ready to execute your AngularJS application. Open a terminal and navigate to the root of your application. Run the following command to keep the web application running in the background:

$ grunt serve

Now, open your browser with the following url:

http://localhost:9001

Exceptions and Promises

Exceptions

In order to keep a good structure of your exceptions, create an object for every throw exception. The structure of this thrown object should look like this (this is just an example):

function MyException(message) {
  this.name = 'MyException';
  this.message= message;
}
MyException.prototype = new Error();
MyException.prototype.constructor = MyException;

Then, you can throw the exceptions as error objects:

throw new MyException('Something was wrong!');

Promises

Promise is an interface class that helps you to control if the call to a function - that may be executed asynchronously - has finished successfully or with an error. It is very similar to have a try-catch-finally block for async calls.

This is just an example:

function myAsyncFunction() {
  var deferred = $q.defer();
  // Doing something async...
  setTimeout(function() {

    // Notify AngularJS model about changes
    scope.$apply(function() {
      deferred.notify('Async method has ended.');
      
      /* This is just a random boolean value for the demo */
      var randomResult = (Math.random(2).toFixed(1) * 10) % 2;

      if (randomResult) {
        deferred.resolve('OK');
      } else {
        deferred.reject('FAIL');
      }
    });
  }, 1000);
  return deferred;
}

Then, you can call to your async function obtaining the results with the promise interface:

myAsyncFunction()
  .then(successCallback)
  .catch (errorCallback)
  .finally(alwaysCallback);

Throwing some exceptions

Now add to your controller a function that raises an exception. Open the file under the path app/scripts/controllers/main.js and write this content:

'use strict';

angular.module('exceptionHandlingApp')
  .controller('MainCtrl', ['$scope', '$q',
    function($scope, $q) {
      function MainCtrlInitException(message) {
        this.name = 'MainCtrlInitException';
        this.message = message;
      }
      MainCtrlInitException.prototype = new Error();
      MainCtrlInitException.prototype.constructor = MainCtrlInitException;

      function init() {
        /* We just reject the promise of this function. Only for the demo */
        return $q.reject("Cannot init");
      }

      init().
      catch (
        function(cause) {
          throw new MainCtrlInitException(cause);
        });
    }
  ]);

As you can see in the example, in the function “init”, I reject the promise that will cause the catch interface to be called. Then, I throw a custom exception.

Opening the console of your browser, you should see the results of this exception.

Intercepting exceptions with Decorators

AngularJS has its own exception handler. You can override it with a Decorator that helps you to extend the functionality of an object in AngularJS.

You can create a new Decorator using Yeoman Angular Generator in the terminal applying the following command:

$ yo angular:decorator customExceptionHandler

This command will create a new file under the path: app/scripts/decorators/customExceptionHandlerDecorator.js

Replace its content with the following code:

'use strict';

angular.module('exceptionHandlingApp')
  .config(function($provide) {
    $provide.decorator('$exceptionHandler', ['$log', '$delegate',
      function($log, $delegate) {
        return function(exception, cause) {
          $log.debug('Default exception handler.');
          $delegate(exception, cause);
        };
      }
    ]);
  });

In your browser, you should see the new result or your exception handler:

Image-One-2

Intercepting exceptions with Sentry

Sentry is a system that helps you organize your exceptions using a web application. You can setup an account on getsentry.com or install it on your own server. You will need to create a new project and obtain the API key to interact with your application.

Image-2

Be sure that you configure your hosts file to point your localhost as another domain.

In windows, you can find this file under “%SYSTEMROOT%\System32\drivers\etc\hosts” or in Linux/Mac under “/etc/hosts”

For example, you can use:

127.0.0.1   example.com

And configure Sentry to accept calls from example.com

Image-3

In order to interact with Sentry, we will need to use RavenJS with Sentry. On the bottom of your app/index.html file, before the last body tag, add the following line:

<scriptsrc="//cdn.ravenjs.com/1.1.14/jquery,native/raven.min.js"></script>

Then, modify app/scripts/app.js using the API key obtained from Sentry:

/* global Raven:true */
'use strict';

angular
  .module('exceptionHandlingApp', [
    'ngCookies',
    'ngResource',
    'ngSanitize',
    'ngRoute'
  ])
  .config(function ($routeProvider) {
      Raven.config('https://7be...............491@app.getsentry.com/2...2', {
        logger: 'Error Handling Demo',
      }).install();

    $routeProvider
      .when('/', {
        templateUrl: 'views/main.html',
        controller: 'MainCtrl'
      })
      .otherwise({
        redirectTo: '/'
      });
  });
 .otherwise({
        redirectTo: '/'
      });
  });

You will create a Decorator that interacts with Sentry, sending the exceptions to that server. You can use Yeoman Angular Generator:

$ yo angular:decorator sentryExceptionHandler

And replace the content of the file under app/scripts/decorators/sentryExceptionHandlerDecorator.js

/* global Raven:true */
'use strict';

angular.module('exceptionHandlingApp')
  .config(function($provide) {
    $provide.decorator('$exceptionHandler', ['$log', '$delegate',
      function($log, $delegate) {
        return function(exception, cause) {
          $log.debug('Sentry exception handler.');
          Raven.captureException(exception);
          $delegate(exception, cause);
        };
      }
    ]);
  });

Congratulations! Now you will see your exceptions appear on Sentry:

Image-4

 

Image-5

 

References

-------------

avatar

Miguel is a professional software developer that has consulted for dozens of companies through his firm - Eridem.se. His focus is on mobile application (android) development, web front-end development (javascript, css, html5) and back-end (mvc, .net, java). He is also continuously learning new software approaches, mostly related with UX, mobile and DevOps, and creating new ventures such as Apprendy.com.

Feel free to leave your comments and questions for Miguel in the comment section below.

Subscribe to the Load Impact blog below. We post something new every week from a different developer or tester.

If you like this blog post and are an experienced front-end developer, we want to hear from you. Check-out our latest job opening: http://www.arcticstartup.com/jobs/1876

[Case Study] How One Digital Agency Guaranteed Performance BEFORE a Big Release

JWT, one of the largest advertising agencies in the United States and the fourth-largest in the world, used Load Impact to perform load tests to verify that their new campaign site could handle up to 120,000 visitors/hour.

Background:

According to an independent global research study undertaken by Vanson Bourne, even minor delays to website response times can have a sizable impact on customer satisfaction, page views, conversion rates and site abandonment. Despite this, an astonishing 68% of website owners experienced performance or stability problems and 32% of organizations do not know if their website is monitored on a 24×7 basis*. To make matters worse, 47% of PC visitors, 69% of tablet visitors and 34% of smartphone visitors expect response times equal to or below 2 seconds**. 

In an effort to ensure quality performance of a new campaign website built for a client in the pharmaceutical industry, Load Impact was commissioned to establish that the backend for the website could handle the expected traffic – 120,000 visitors per hour – while exhibiting load times that were within acceptable limits.

The campaign-site was built with a user signup/login procedure and offers an interactive online game. The backend for the service is hosted in Texas, and all static content is distributed through a CDN which makes calls to the backend servers. There is also an external partner which handles the user database including registration, signup etc.

Test setup:

For the purpose of testing the backend only, a set of specific user actions were defined, such as “user registration”, “user sign-in”, and other actions where the backend systems had to be involved. These actions were activated through requesting certain URLs, one for each type of action, that were created specifically for the load test. In practice it meant that a simple API was created only for running the load test.

The simulated users in the test were configured to perform a series of these predefined user actions, resulting in corresponding calls to the backend systems. The static content, normally served through CDN operators, was ignored in the test.

The test was run as a series of 5 minute ramp-up tests (simulating approximately 8.33 clients per second), individually tuned depending on results of previous tests and designed to find out the breaking point of the target system.  

Service environment:

  • Apache
  • PHP
  • MySQL
  • Linux

The tested environment consisted of an HTTP load balancer plus web-, application- and database servers.

Challenges:

There were numerous challenges that the test was designed to detect. First of all, there was a need to validate that the system could handle the expected amount of traffic and establish a performance baseline. The test was also set-up to detect potential problems in the infrastructure and web app. 

  • Validate that the system could handle the expected amount of traffic
  • Detect potential problems in the infrastructure or web app
  • Establish a performance baseline

Solution:

The solution agreed upon was to generate and measure load from multiple geographic locations as well as to measure response times, throughput and customer experience.  

  • Load generation and measurements from multiple geographic locations
  • Application response time, throughput and customer experience analysis provided for JWT

Results:

The results of the load test revealed that the campaign website could withstand the expected traffic and there were no specific performance problems with the site. Therefore, a baseline was established at about the required level of 120k visitors/hour.



The external service provider of user registration and sign-in functionality had no problems and their response times remained constant during the tests while the other backend services exhibited response times that were stable until just over the required level of 120,000 visitors/hour, after which response times started to increase rapidly and exponentially.

Specifically, the response times for the start page were under 1 second for up to 2,000 concurrent visitors. Response times for the configured tests, which included the set of specific user actions, were under 2 seconds for up to 2,000 concurrent visitors. Considering that the average response time for similar campaigns of this size is above 4 seconds*, these results were impressive. 

The campaign site was launched successfully on YouTube.

*Source: State of Web Readiness Report, Load Impact, 2013

**Source: How To Deliver Fast, Engaging Responsive Web Design Sites, Akamai, 2012

Saving Your Super Bowl Bacon: How One Advertiser Performance Tested Early and Avoided Disaster

SuperBowl2014__140201231903During Super Bowl  XLVII, Coca Cola, Axe, Sodastream, Calvin Klein had their hands full. Not so much serving online visitors as running around looking for quick fixes for their crashed websites. As reported by Yottaa.com, no fewer than 13 of the companies that ran ads during that Super Bowl saw their websites crash just as they needed them the most.

If anything in this world is ever going to be predictable, a large spike in traffic when you show your ad to a Super Bowl audience must be one of those things.

Thats why one leading digital agency – ROKKAN –  worked with us to guarantee the stability of their client’s campaign website just days ahead of their Super Bowl XLVIII TV commercial.

Getting this level of performance assurance not only eased their client’s concerns, it also showed how committed ROKKAN was to the quality of execution.

As with any advertising campaign, its not only about the glitz and glamour of the ad, its about the ability to motivate action on the part of the audience. And if that motivation is to visit a website, you better be damn well sure that website can take the heat and deliver an exceptional experience. Anything less translates to lost customers, degraded reputation and fewer sign ups – as was this advertiser’s campaign objective.

Background:

The auto industry was the biggest sponsor of the Super Bowl for the third straight year this year. No fewer than seven brands advertised during Fox TV’s telecast of the National Football League’s championship game on Sunday, February 2, 2014.

On average, the carmakers spend as much as $6 million per 30-second spot, for the chance to get in front of an estimated 100+ million viewers. 

“When these types of investments are made, if your website doesn’t work, it becomes very ugly, very quickly for all parties involved in the campaign,” said Jim Blackwelder, chief technical officer for ROKKAN, a Publicis Groupe company that’s done award winning work for American Express, jetBlue, Chipotle, Ceasars Palace, and TAG Heuer. “Everybody just expects technology to work these days, and if it doesn’t, everyone notices.”

Test setup:

The Super Bowl Sunday challenge for ROKKAN – a 2014 Advertising Age A-List Agency for the third consecutive year – could not have been clearer: the target user load for its automaker client’s microsite promoting the features of an upcoming luxury sedan, was over 250,000 visitors per hour coming in from across the country and around the world.  But initial performance tests of the website using Load Impact failed far short of that goal.

“That first test was a total eye-opener,” said Blackwelder.  “Load Impact showed us we had a lot of work to do to get performance where it needed to be.”

At the advice of Load Impact, ROKKAN split performance testing to separately hit the website run by the hosting company Rackspace. The tests were done by going through and around the Rackspace content delivery network (CDN) to uncover crucial performance and configuration issues with both scenarios.

Daily meetings and testing ensued. 

Challenges:

“The team at Load Impact realized that without a full court press – a really integrated effort with all of the stakeholders including ROKKAN, their service providers, as well as Load Impact – the success of the project was far from guaranteed,” said Charles Stewart, Load Impact’s general manager for U.S. operations. 

Solution:

With less than one week from kickoff, ROKKAN made changes and optimizations to problems identified by Load Impact at the caching layer, application layer and hardware layer, that brought page load times to under 4 seconds with just over 10,000 simultaneous users.  

It was 20 times better than when they started, but still a long way from their goal. As an added complication, eleventh hour changes requested by the automaker adversely impacted performance dramatically.

Load Impact and the ROKKAN team examined the data and identified network bandwidth as the likely cause of the performance degradation, and produced reports to support the findings.

With data in hand, Blackwelder was able to go to Rackspace’s network team, which quickly identified and resolved the remaining bottlenecks, increased throughput to the site, and gave ROKKAN’s platform the capacity it needed to scale.  

Results:

The automaker’s website was ready for game day.

“Had we not done performance testing we never would have uncovered a range of issues with both our work and our partners,” said Blackwelder. “Without Load Impact our hitting the project goals and schedule would have been at risk. But it wasn’t at all.  It was a huge success.”

The automaker’s site performed perfectly under the Super Bowl traffic, far exceeding expected connections per second without breaking a sweat, and delivered the customer sign-up conversions that were the automaker’s key success metric.

“For many of ROKKAN’s projects we thought of performance testing as a nice-to-have. But for this project and any of its type, it’s an absolute need-to-have. We needed large scale capacity testing, real-time viewing, and robust reporting, with a platform that could respond to change quickly, and Load Impact delivered on all counts.” - Jim Blackwelder. 

———–

This is a perfect example of how NOT to waste $4 million (not including production costs). If you’ve got a big marketing campaign coming up, make sure your audience can actually fulfill the action you want them to take – performance test your campaign website/app early. 

About Load Impact

Load Impact on-demand services detect, predict, and analyze performance problems – providing the information businesses need to proactively optimize their websites, apps and APIs for customers.
 
With its roots in work for NASDAQ and the European Space Agency, Load Impact has been redefining load testing since 2011 by making it cost-effective, instantly available, automated, and very easy to use.
 
Test your website, app or API at loadimpact.com

Follow us on Twitter

Enter your email address to follow this blog and receive notifications of new posts by email.

Follow

Get every new post delivered to your Inbox.

Join 51 other followers