5 QA Best Practices for DevOps
by Scott Reece, on Nov 5, 2019, 12:30:00 PM
DevOps is about more than speed! If an organization focuses solely on release velocity and lead times they risk poor code getting to production.
Customers want features fast—but only if they actually work. In addition, leaving security vulnerabilities in the development lifecycle risks internal process failures that can have devastating consequences.
Quality Assurance can’t be a second-class citizen in any DevOps organization, and teams need to integrate a few, key Quality Assurance (QA) best practices into their DevOps approach for success.
We put together 5 QA best practices for you to keep in mind when building your CI/CD pipelines.
1. Match Testing Priorities to Code Structure
QA has finite resources, and to be most effective, those resources should be used in the most meaningful, purposeful way (rather than, for example, on a first-come, first-served basis). While there is no fool-proof method, being strategic about where a team deploys its resources gives it the best chance to ensure issues are address BEFORE they are released to users.
Releases are planned code deployments, so changes are defined well before they are made live. Focusing on those updated, or new features gives QA the very best opportunity to ensure good quality before release. If a release consists of a few feature updates then QA should be focused on those updates, and not the remainder of the application.
Of course, software is complicated and a change in one place may have unexpected results in another. Focusing on the place most likely to be affected by a change is the best place to start. Even if known issues are released, the rest of the team (development, and support for example) can be prepared for them.
2. Use Feature Toggling and Have a Rollback Strategy
Having feature toggling in production environments gives the entire team real-world user data about new or updated features. Allowing a sub-set of all users (opt-in beta testers in general) to experience, use, and test updates live provides incredibly meaningful feedback to the entire development team. No matter how well-scoped, or how well-written a user-story is real-world users will always find edge cases and different ways of using your software than you planned.
At best, feature toggling confirms that a new, or updated, feature is production-ready. At worst, it demonstrates why it’s not ready to be rolled out to the entire user-base. Users that experience new features and updates before other users won’t catch everything. The more people that use an application, the more edge-cases are found, some of which can be significant enough to warrant removal or a release delay.
When a feature is released to the entire user-base and it needs to be withdrawn due to a significant issue, you need a rollback plan. A built-in rollback feature like the one in BuildMaster simplifies the rollback process to a single button click. If your release tool doesn't include a rollback feature, you need to establish a plan for manually rolling back, because, without a plan, rolling back can introduce even more issues for users.
3. Security Testing
When providing SaaS, users rely on your organization to protect them and their data from bad actors out in the world. While every company touts their security and safety, security testing often doesn't get the prominence and resources it deserves. Management can often see the time and effort spent making sure “what-if’s” don’t happen as a waste of resources. But without this security testing, PR disasters occur—which are far more costly to an organization.
A dedicated security team is a standard practice for SaaS organizations. This team should have specific training and be up-to-date on a host of compliance regulations (HIPPA, SOX, etc.) to protect users from malicious attacks and to ensure full compliance and prepare your organization for governmental audits.
The team should also have training in penetration testing. Working with your cloud providers is the best first step. Not only will they have information specific to their services (including best practices), but also notifying them about your testing will make sure you don’t get flagged as a bad actor when you conduct penetration testing on your sites and services.
4. Move Security Left
By making sure that security is a priority from the very beginning (or the "left" end of a release pipeline) of any release process, you ensure security is part of the release culture. You also spend less time implementing security: Running automated testing like static analysis as early as possible and other, earlier security tests mean that problems are caught earlier, when they will have less of an impact on the business and are less expensive to resolve.
Ideally, your automated tools would review all code upon check-in. This would identify any issues fast, allowing them to be addressed sooner. Finding issues as soon as possible means better visibility on how those issues are introduced.
You should apply the same practice of targeted testing based on priority to security and testing. Even automated testing takes time and resources. Using automation to test narrowly at first allows the best balance. Wider testing can be performed when there is less load on the system (at nights generally).
Implementing promotion gates as part of a release pipeline also introduces security checks into the process. Making sure that code doesn’t move beyond a specific stage without approval helps ensure code quality and feature completeness. Promotion gates also allows for better logging and visibility into the release process, so that regulation compliance is a built-in part of the process.
Promotion gates are a key feature when regulation is involved, and while developers may feel they mute release velocity, they are essential to maintain compliance. WebMD enforced these gates so that only QA engineers could deploy to higher environments to maintain their HITRUST certification.
5. Ensure Testing and Production Environments Match
Testing for feature completeness and for security is a necessity, but that testing is only meaningful if the environments in which the testing is taking place match the real-world production environment. If you’ve created a testing environment that is bullet-proof to penetration testing, for example but that doesn’t reflect the actual environment that the application is deployed to, you’re not seeing real results and, worse yet, getting a false sense of security.
Ensuring that testing environments and production environments match as closely as possible provides the most useful data when issues are found, and fixes are more easily replicated in staging when users find bugs in the production environment. Environment matching also provides performance insights before changes are released. A feature may test as complete and issue-free, but if that feature slows down the entire application, it’s not ready for release.
Environment matching also helps identify and reduce server drift across a pipeline. Access is generally more restricted the further along a release pipeline an application gets, but it is not uncommon for testing servers to be configured differently (sometimes significantly differently) than production servers, which can introduce problems. Though sometimes differences are defined for legitimate reasons (for example, a slightly slower but cheaper testing configuration might make sense for a very limited use-case), sometimes changes are implemented by a well-meaning developer, administrator, or Operations personnel to fix an issue quickly through some "creative coding." If these changes aren’t documented, the expected configuration and the actual configuration become further and further apart, leading to false testing and differences between your environments.
Managing Configuration drift and using defined configurations is a much simpler task with tools like Otter. These tools allow you to define and re-use server configurations, so that your testing environments can be configured exactly like your production environments. Tools like Otter also monitor for drift, so that you can be alerted quickly when an environment has changed.
QA: A DevOps Must-Have
Quality has to be considered a key part in any meaningful DevOps culture. Users won’t tolerate broken releases, security vulnerabilities, or constantly lagging platforms. If QA isn’t fully accepted as part of a DevOps culture, the entire organization will suffer and eventually fail.
Inedo DevOps tools maximize developer time, minimize release risk, and empower stakeholders to bring their vision to life faster. All with the people and technology you have right now. To get help streamlining your CI/CD processes, contact firstname.lastname@example.org.