Using Package Promotion for Quality Assurance
by Jim Borden, on Dec 10, 2019 10:00:00 AM
In this day and age as a .NET developer, you are almost certainly either creating or consuming NuGet packages. It has become the standard method of handling dependencies in the modern .NET development world. If you are distributing packages to the public then it is imperative to test them to the best of your ability without hampering development (at least not too awfully much!). There are so many different processes that different companies follow, and it's likely that the smaller the company is, the less distinction made between development and QA. But this post assumes that they are two distinct teams.
In this article, I will describe the flow that I use as part of the Couchbase Lite team. ProGet aids us greatly, helping us reduce missteps in a very powerful way.
Let's first take a bird's eye view of the process.
We on the development team do what we do, making commits to the source control repository of choice (you are using source control, right?). We develop features and write some rudimentary unit/integration testing that can assert to a reasonable degree that what we did is sane, correct, and didn't cause any obvious regressions.
After that is done, the product in that state is sent to the QA team for further analysis and higher-level functional and system testing. This involves more than just a single process test suite, and the QA tool is able to orchestrate together multiple processes to test things like syncing between machines. If QA is okay with it, the build is ready to be unleashed onto the world.
Things That Can Go Wrong
There are a number of places in this pipeline where things can take unexpected turns. How are the developer written tests run? Does that process guarantee that the exact thing that was tested moves on to QA, if appropriate? How is the communication between dev and QA? Did someone hand them a build that was incorrect or didn't pass the initial tests and end up wasting their time? Did packaging cause some unexpected breakage? The questions just keep on coming.
Using ProGet efficiently will help address these questions and protect the integrity of the bits being tested so that you can be sure that what you are putting out is exactly what you expect!
A Modern Continuous Pipeline
"If you don't test the bits you ship, then you didn't test the product." This is likely the motto of many people who have been burned before by a botched release. I agree with it wholeheartedly, and so in accordance with that the pipeline does not touch the final deliverables (a bit more on what that means throughout the rest of this post) at all from start to finish. Furthermore, until the package is created the build is *not* finished.
The following is a step-by-step description of our pipeline and the benefits at each point of doing it in such a way. I'm going to assume a few things here that are true for the Couchbase Lite team, like you have your code commit system set up in a way that does pull request validation so that you don't end up with unstable main branches and that you have a branching strategy set up that suits your release needs.
Are you ready? Let's start!
- Event 1: Code Commit to a Main Branch
What I mean by "main branch" is a branch that produces builds, such as master (or maybe dev) or release branches. For example, we use `master` to track the perpetual "vNext" release until it comes time to cut a release. At that time, we create a `release/<codename>` branch with the codename of the next release track which will have the next major release and the following minor / patch ones. In any case, a commit that needs a build has just come in, so what should happen from here? Another assumption I have here is that you have a CI server of some sort set up to track when a build is needed and start one. We use Jenkins (for better or worse).
- Step 1: Make a Build
At this point, a build should be kicked off. There is no need to run tests just yet, since we have a chicken-and-egg problem: If we test now and package after, we lose our "ship only what you test" philosophy. Accordingly, at this point a build is made and pushed to a company-internal NuGet feed that I named "CI." This NuGet feed is useful for exactly one thing, and nobody outside of the dev team should be using it for anything except that one thing.
At this point the package should have a prerelease identifier because it is not yet of release quality. Let's say this build is 1.0.0-build.1 (because we are cool and use Semantic Versioning 2.0.) For legacy reasons, we actually use formats like 1.0.0-b0001 (which was needed because of the lexographic ordering of pre SemVer 2.0 NuGet feeds).
- Event 2: Package Upload to CI Feed
Now that the build is done, we have a post-build event that, on success, triggers the next step in the pipeline. Can you guess what it will be? It's time for testing!
- Step 2: Developer Testing
At this point another Jenkins job comes into play, which builds and runs a test runner. The test runner will be set up to pull packages from the CI feed (and the public NuGet feed for any dependencies). The tests will run and if they fail, the pipeline stops and that is the end of the road for this package. It won't progress to the next step. That means QA will never be aware of it and their time won't be wasted. However, if the tests do pass then it's time for ProGet to help us out.
- Event 3: Successful Developer Testing
- Step 3: Promote the Package
ProGet has a set of promotion endpoints that we can take advantage of. They are precisely what we need to do: move a package to another feed. In this case, we have another feed that I named "Internal." This feed is also company-internal only. The package will be moved as-is, and so our rule about not touching the deliverable bits is safe. The package has not yet been tested by QA and so it remains a prerelease package with version 1.0.0-build.1.
- Event 4: New Build(s) in the Internal Feed
In our particular case, the post-build of a successful developer tested build also triggers the next phase of the pipeline, which is to build the QA testing application against the build that was just promoted. As the previous steps guarantee, only builds that pass the developer tests are ever going to make it into this feed, so no bad builds ever enter into QA's realm. As a judgement call, since currently the QA tests take multiple hours to run to completion, we decided against making them run automatically. Instead, QA runs them according to their testing schedule.
- Step 4: QA Tests the Build
This is similar to the previous step, just with a different set of tests. The result of this is either QA finds an issue and circles back to dev, or QA finds no blocking issues and gives the OK for release. If there is a blocking issue, the package stops here and that is the end of the road. However, in the positive case:
- Event 5: QA Accepts the Build
At this point in the pipeline what you have is a prerelease build. Let's assume there was some back-and-forth and now you have 1.0.0-build.64. This cannot be put out as a release because of its prerelease build number, but it also cannot be rebuilt because that would invalidate all of the testing done. Luckily, the version information inside of a NuGet package is stored separately from the actual build.
- Step 5: Repackage & Promote to Public Feed
ProGet also has a set of repackaging endpoints that we can to both promote and change the version of a package without touching its actual executable bits in one API call.
At a high level, changing the version involves the following:
1. Unzip package
2. Edit metadata to replace version
3. Add auditing metadata about the change
4. Re-zip package
As you can see, we still haven't touched the "deliverable bits," since the metadata is only used for NuGet feed purposes and nothing else (the actual thing that gets executed is left untouched). So, because we are tracking version 1.0.0 the version will change from 1.0.0-build.64 to just 1.0.0. The executable contents are identical to when it got built for the scratch feed so you can be assured that what you tested is what you shipped.
Bonus: Release Train
This process can be extended even further to involve some sort of alpha and/or beta prerelease train. Branching off of Event 4 (new builds in the QA feed) we can take the following step:
- Step 4A: Promote to Public Prerelease Feed
Because this is a prerelease package, you can opt to skip the heavy QA testing to improve train cycles. However, it's probably best to keep an independent way of tracking that is different from just build numbers. Otherwise your feed would look like this: 1.0.0-build.25, 1.0.0-build.32, 1.0.0-build.45, etc. This is hard to remember when a certain fix is included, so let's reuse the logic from promoting the final release package. The difference here is that:
1. The package will still have a prerelease version
2. The package will be put into a prerelease feed
So, reusing the logic from changing 1.0.0-build.64 to 1.0.0, instead it can be promoted to, for example, 1.0.0-alpha.1 or 1.0.0-beta.1. The next build after that can be 1.0.0-alpha.2, or 1.0.0-beta.2 (you could even switch from alpha to beta at some point and it would still sort correctly). Once release time arrives, you could put out some release candidates: 1.0.0-rc.1, 1.0.0-rc.2, etc. (just be sure that these ones go through QA first!). If your candidates are accepted by whatever outside users choose to try them, then all you have to do for a release is promote 1.0.0-rc.2 -> 1.0.0, and you have your GA release.
Software is only getting more and more complex, so it would be nice to take some of the complexity out of other areas to make up for it. With a pipeline like this for your library, nobody is confused about the state of any given package. They can simply look at the version, as well as the feed it came from for most cases and know what to expect:
- 1.0.0-build.# from scratch feed: Internal Untested Build
- 1.0.0-build.# from qa feed: Internal Build that passed developer tests (and possibly QA tests)
- 1.0.0-alpha|beta.#: External Prerelease Build that passed developer tests only
- 1.0.0-rc.#: External Prerelease Build that passed all testing
- 1.0.0: External Release Build\
Senior Software Engineer at Couchbase
Jim has been programming computers since the age of 10 years old and has made it a career since 2007. Joining Couchbase in 2015, he has spent the majority of his time as the lead developer of Couchbase Lite .NET and has extensive knowledge of C# / .NET, among other languages.
Couchbase Webpage: https://www.couchbase.com/