Internet giants sometimes approach testing differently than the rest of the world should. For example, in a recent Bloomberg BusinessWeek article, Alibaba Cloud senior product manager He Yunfei talks about Singles Day as the ultimate load test, basically. “We use it as a drill. It sharpens us,” he says, according to the article.
You likely had the same response we did: “What?!?” Stress-testing your site or your service by pushing live when you know your customers will be depending on you for significant, revenue-related service is perhaps imprudent. Now, we don’t believe Alibaba Cloud meant it as the article implied: instead, we suspect that the day is meant as a validation of all of the testing they’ve already performed.
The story has a happy ending, as you might have imagined. The Bloomberg BusinessWeek article cited above was published before Singles Day; on Singles Day it performed admirably. As many sources report, Alibaba set sales records, selling $1 Billion USD in sales in the first two minutes (article here).
But the broader point is still valid. How many times have you seen someone deploy an update to production with the hope that the changes made won’t break the site or service? If you’re like us, too many times.
Hope is not a strategy.
This turn of phrase, often credited to a speech by former NYC mayor Rudy Giuliani in 2008, perfectly encapsulates the reason consistent, constant testing eliminates the need for hope.
Of course, the unexpected can always occur. The unexpected is minimized, and the degree of hope is as well, when consistent and constant testing is part of your development cycle. In continuous integration and continuous delivery (CI and CD), testing is part of the build cycle.
Functional and unit testing is essential. However, the idea of using a major deployment before you anticipate a big traffic spike should not be a “drill.” A “drill” should be practice under simulated traffic, under simulated load (that’s why it’s called load testing). A “drill” shouldn’t be undertaken using real customers with real businesses depending on your performance.
The bottom line: don’t mistake these kinds of stories as practices worth emulating. Hope is not a strategy. Only regular, consistent load testing helps you sharpen your site/service performance before the real world demands it.