In parts 1 and 2 of this series, we focused on monitoring software, evaluating which systems architecture is right for you and optimising your code. Software doesn’t stay the same though: It’s something we constantly need to maintain and improve. This is why the next step in building a high-performance system is to understand the importance of performance and load testing – and why you need to do it often to stay ahead.
If there is one thing that all strong racing teams have in common, it’s the amount of time they spend testing. Yes, these days Formula 1 teams are restricted by the amount of track testing they can do. That doesn’t stop them from collecting reams of data in these sessions, though, and applying them in wind tunnels and their own tracks to find ways of improving their performance.
With software, we need to be prepared to test performance, often and always. Many teams make the mistake of only performing efficient load or stress testing on occasion, or towards the end of a release cycle.
The problem with this is that not only is it too late to be making performance improvements at that stage of the development cycle, but you also miss out on understanding how your software performs over time and at varying loads.
You might think software is binary and therefore consistent, but every day there's a chance that underlying servers are facing different strain or interactions that could be exposing different problems. Having all of this data from regular testing allows you to properly benchmark every aspect of your system so that when something does start to degrade – even slightly – you can pinpoint where it is and why.
Now, you might remember I started off this series of articles by looking at monitoring and how important it is. What makes performance testing different is that it’s more focused on measuring load and send time across different times of the day and different loads. It also includes a variety of load testing, which can help you understand how your system performs at different intervals, as well as stress testing, which can identify the performance breaking points of your software.
These tests (except for stress testing) can be done in a test environment and production on a regular basis. It might sound time-consuming and expensive, but if there is one lesson that you can learn from Formula 1 teams and supercars, it's that performance isn’t cheap!
If you aren’t constantly investing in testing it, you’re unlikely to know how to effectively improve your systems.
Here are some tips to keep in mind when testing for performance.
1. Test in production
Variables such as firewalls, specific load balancing configurations and a server's memory configuration, amongst other things, may severely influence performance. This is why it is best to test in production or simulated production.
Testing will need to be done in a non-destructive manner, but the principle is consistent: Test everything – physical components, hardware, network settings etc – to get a full picture of what will happen when the product is launched and used.
Tip: Having a simulated production environment might sound like overkill, but if your business relies on performance, it’s a cost that is worthwhile if you want to ensure stability at all times.
2. The importance of Hybrid Load
If a functional defect can only occur under a specific load, it will be extremely difficult to find and reproduce that defect. Functional testing during protocol level load testing, or hybrid load, can help measure and assess the user experience during different load profiles.
The load scenarios are functional in their essence but they are not checked on the GUI level and don’t vary. As such, the complementary functional testing that is suggested here is not limited to the scenarios that are simulated in the load.
3. Evolution of performance testing to system performance monitoring
Leverage your performance scripts to run constantly. With proper correlations, you will be able to drive Business Intelligence – for example, you'll be able to see how poor vs. good performance affects revenue, or how different usage scenarios affect your system. Generating reports with a load testing tool will help you gain an understanding of the important questions, such as ‘what’, ‘when’, and ‘why’.
With this knowledge, you can plan and prioritise potential problem scenarios, and better understand performance impact when your system needs to scale.
Tip: Not all performance needs are equal, though, and at crunch time, it's probably best to keep your focus on the most important problem at hand.
4. Embed performance testing into your Continuous Integration
Once you are comfortable with your performance, stress and load testing, and that your system can meet both the demands of intense load and sustained performance, you'll need to identify a subset of tests that are readily executed every single day.
This can consist of high-priority use cases and quick-to-execute performance tests that will prevent performance leaks from entering your system.
While it’s a little difficult to include a sizeable production simulation into your CI process, it is certainly possible to cover a wide variety of scenarios.
5. Back-end database testing/monitoring
If you ignore the performance of the database that drives your application, your end user's experience will suffer. Setting and monitoring performance baselines for your database, both as a stand-alone unit under test, and as a part of the system, through roll-out is absolutely essential.
This may include running production system stress tests to ensure your database can handle new data loads, setting thresholds to avoid inefficient or poorly performing queries, and tracking real user response times to ensure a consistent user experience throughout the roll-out process.
Tip: It’s obvious to take an end-to-end approach but spelling out the consistent elements that need to be tested will ensure you have covered all your bases.
6. Perform log analysis – app and client side
Log files contain valuable information on the performance of your application. Because not all problems are visible to users, and not all users have the ability to understand the root cause of problems, especially the non-reproducible, log files will be your most accurate resource for fixing things.
Log analysis tools let you monitor servers and services, generate usage and error reports on the client side, build compliance analytics, and more. Collecting log data from servers, networks, devices and security systems will be able to give you a comprehensive idea of what is going on when an application is launched and used.
Log analysis should be deployed during development, functional testing, load testing and production. It’s much better to get a comprehensive understanding of error logs before users get them, and stop using an application that you have spent so long developing.
7. Stress test more often than you think
Most teams avoid stress testing, or placing the system under extreme load to determine its breaking points, but it is a useful way to learn more about how your system behaves.
Tip: Even if the system is not designed to ever hit the loads you stress test it with, seeing how it fails, at what point it fails and why, can alert you to other flaws in the software.
If there is one key thing to take away from this series of articles, it's that writing high-performing software is not easy. It takes effort and dedicated focus. However, if you are prepared to invest in your monitoring, put the right architecture in place, optimise your code where necessary, and test continuously, you will produce software that performs in the best way possible – and who doesn’t want software that is slight, fast and responsive?
When not changing the world of software development, Craig can also be found writing for Critical Hit as well as his own LinkedIn blog, designing board games for his company Risci Games and running long distances for no apparent reason whatsoever. He likes to think that his many passions only add to his magnetic charm and remarkable talent – at least, that’s what he tells himself when he goes to sleep at night!
