Mistakes to Avoid for Performance Testing in Continuous Integration
by Marisa Vesel, on Feb 4, 2020 12:00:00 PM
Continuous Integration (CI) involves developers frequently checking their code changes to ensure the new code integrates successfully with the main codebase and has become an integral, almost universal part of modern development shops.
Performance testing is an important part of the CI process that tests applications to make sure they can handle traffic, users, and data exchanges in an appropriate manner. Integrating performance testing within CI allows organizations to catch errors faster, making sure they are delivering high-quality software in a timely manner that meets end-users' expectations.
Note: If you’re not running performance tests, feedback from unhappy users will be the first time you hear about slow applications and time-out errors. Don't let this happen! Make sure performance testing is part of the CI process.
Performance testing is obviously important, but there are common mistakes that are easy to make when onboarding this important testing into your CI process. Being aware of these mistakes will make them easier to avoid in the future.
Mistake 1. Treating Performance Testing as a Second-Class Citizen
Functional testing, which looks to see if an application or feature is able to successfully do what it was built to do, has a more obvious perceived utility than performance testing. This often leads to organizations deprioritizing performance testing.
While performance testing's utility may be hard to see, it is no less important than other types of testing. Underperforming applications result in slow applications, time-out errors, and frustrated users. This especially affects organizations who use SaaS or whose platform is the Internet, since users have an expectation of speed and fast loading times. Failing to deliver can cost you customers.
For example, when Facebook first rolled out their "Like" button, they failed to perceive exactly how much usage and traffic this new feature would receive. As a result, Facebook users were unable to use the feature, and several news and retail sites were slowed down. Facebook did not run the necessary performance testing to ensure their new feature could handle heavy traffic.
Organizations need to create a testing process that ensures performance testing, as well as functional testing, is part of an organization's DevOps and CI process. Organizations need to consider what bare minimum performance is necessary in the application to ensure a satisfied end-user and to test all those components to make sure requirements are being met.
2. Testing Too Late in the Pipeline
Performance testing should be an integral part of an organization's CI process, at the front of the software development lifecycle. Unfortunately, some organizations complete their performance testing at the end of their software delivery pipeline.
When performance testing fails, organizations create extra work for the whole team. When this failure occurs at the end of the pipeline, there’s often no choice but to start entire projects over. Not only is this frustrating and time-consuming, but it also delays application and feature releases and costs the organization unnecessary extra money.
Performance testing should be integrated earlier into the pipeline as part of the CI process. In fact, performance testing can even be run in parallel with automated unit tests at the very earliest stages since new features or other code changes can have unexpected effects on front- and back-end performance.
If your organization is able to push testing left, testing earlier in the process allows issues to be fixed up-front, allowing organizations to stay on track with release dates and save time and money by having less work to redo.
3. Testing the Entire Application at Once
Some organizations who are practicing performance testing make the mistake of only testing the entire application at once, and all at once, instead of breaking the application into critical areas for testing.
Testing full applications is an overwhelming, expensive, and time-consuming process. If multiple parts fail at once, this can lead to whole projects having to be redone in order to create an acceptable application-especially if the root cause is not properly identified.
Rather than testing the whole application, focus on known areas of slowness or areas that are the most vulnerable to be stressed under load within the application. Using this method of focusing only on the most "at-risk" components of an application allows organizations certainty of performance with more minimal cost and time spent.
For performance testing, risks will usually stem from user and functional requirements. It's important that there is a process in place, that is followed, for documenting changes, identifying risk, and making sure these aspects are properly tested.
4. Not Reporting During the Process
It doesn't matter how good your organization is at completing the testing process if that process is not properly reported. While some organizations are good at ensuring their applications are tested, they run into problems when reporting is not accurate.
With no accurate record being kept, organizations are forced to communicate results on their own, often between teams and sometimes spanning geographical locations! As information is passed between people, organizations run the risk of inaccurate information being spread or even going missing entirely.
To make sure accurate information is available for team use and organizational use, use a DevOps tool that can automatically generate consistent and useful reports.
Not only do these reports provide change documentation to help identify fragile areas of your application, but they can also help create buy-in from other development teams about why this type of testing is important and help make the case for more resources or time to be devoted to performance testing.
These reports should make it clear if the current build of the application meets minimally acceptable performance standards and how much stress the server is under at each phase of testing.
Performance Test Early and Often
Although performance testing may seem less important than other tests during the development process, performance testing done right can protect your end-users from frustration and delays, ultimately protecting your business’s market share.
For further information on prioritizing performance testing within continuous integration, check out "Best Practices for Performance Testing in Continuous Integration."