Best Practices for Performance Testing in Continuous Integration
by Scott Reece, on Jul 31, 2019 2:00:00 PM
Enterprise organizations are adding new and complex features into development faster than ever before. But with business timelines taking precedence over all else, these rapid changes often result in features getting pushed to production without proper quality control. That risks customers and users experiencing negative performance issues that could have been avoided with proper testing.
What is Continuous Integration?
The principles of Continuous Integration (CI) were created to help solve this problem. CI involves developers frequently checking in their code changes and building those changes to make sure that new code integrates with the main codebase and often involves automated testing.
The goal of CI to identify as early as possible any code that doesn’t function or that doesn’t integrate with the application and needs more work.
What Is Performance Testing?
Performance testing is one part of the CI process. This type of testing involves stress testing your application to make sure it can handle traffic, users, and data exchanges in an acceptable manner. Measuring both application performance AND machine stress is critical.
Performance testing works by creating virtual users to interact with your application in order to test response times and data transmission. Testing starts with a few ‘users’ and adds more and more ‘users’ over time to measure how an application performs at different levels of activity. Testing also measures machine stress and how much each level adds to the over-all stress of a server to determine when a server is maxed out and can’t perform in an acceptable manner.
- If you’re not running performance tests, the first time you’ll hear about slow applications and time-out errors will be from users who are unhappy with your software.
- If you’re not measuring machine stress, your application isn’t scalable when more people need your solution.
This is why creating a testing process to integrate performance testing into a DevOps and CI process, and using proper tools like BuildMaster to automate the process as early as possible, is essential.
Performance Testing Shouldn’t Be a Second-Class Testing Citizen
Despite the importance, performance testing often takes a back seat to functional testing in software and application development. There are a few reasons for this, but the main one is simply due to perceived utility. Functional testing makes sense: an application or feature can’t be said to be complete if it doesn’t do what it was designed to do. Performance, on the other-hand, is more of a gray area.
Of course, this view doesn’t hold up to scrutiny. If a feature ‘works’ but makes the application so slow and painful to use it drives people away from using it, does it really work?
Bottom line, performance testing in a CI set-up shouldn’t be a secondary thought or something that is done 'if there's time.' It needs to be part of a Continuous Integration model to ensure reliability and performance of applications.
5 Best Practices for Performance Testing and Continuous Integration
To help you get started optimizing your performance testing, we’ve laid out some of the best practices you should follow.
#1 Test Early
Performance testing shouldn’t begin at the end of a delivery pipeline. If you wait until the end stage of a pipeline to test performance and tests fail, you’ve created extra work for your team because the whole process will need to start over.
To be effective, performance testing should be part of your CI process early in your pipeline. It can even be run in parallel with automated unit tests at the very earliest stages since new features or other code changes can have unexpected effects on front- and back-end performance.
#2 Automate Your Testing
Automating performance tests allows you to put testing right at the front of your CI environment to ensure minimum acceptable standards before a build goes any further. Just as failed unit testing would reject a build, performance tests that don’t meet minimum requirements should also fail a build, preventing it from progressing through to release.
#3 Set Benchmarks
Automating tests at early stages necessitates having metrics to measure against. Projects should have a known baseline performance (likely how the application performs in production) as well as a minimum acceptable performance standards.
While an application may be released with minimum performance standards, that knowledge will help you avoid disaster when future changes are released that may degrade performance even further.
#4 Only Test Pieces of Your Application
At the earliest of stages it is also not necessary to test the entire application, which can be expensive and time-consuming. Rather, focus on known areas of slowness or areas that are the most vulnerable to be stressed under load. Focusing only on the most "at-risk" parts allows some certainty of performance with minimal cost and time spent.
#5 Make Reporting Part of the Process
No matter the tool used for performance testing, a requirement should be that it generates consistent and useful reports. These reports should make it clear if the current build of the application meets minimally acceptable performance standards and how much stress the server is under at each phase of testing.
Not only do these reports provide change documentation to help identify fragile areas of your application, but they can also help create buy-in from other dev teams about why this type of testing is important and help make the case for more resources or time to be devoted to performance testing.
Performance Testing and DevOps
The goal of DevOps is to deliver code changes to production faster and to increase quality. That is impossible unless performance testing is included in any CI set-up.
Making performance testing part of your Continuous Integration practices will help ensure that the changes being made by the development staff are of known quality. Knowing this up front before human testing, QA, and approvals saves time and development cycles by identifying changes that will cause unacceptable slowdowns and bottlenecks before they ever leave the CI stage of your pipeline.
To learn more about how Inedo products can help with performance testing—and every stage in the CI process—contact firstname.lastname@example.org.