Building a Continuous Delivery Pipeline Using Jenkins
by Scott Reece, on Jul 22, 2019 12:30:00 PM
Continuous Delivery refers to the set of practices that development and operations teams use to produce and deliver software for business teams. These practices are not uniformly defined by a standards body such as ISO but instead are explored, implemented, integrated, and continuously refined as part of an organization’s core business processes.
Although specific Continuous Delivery evolves differently within each organization, all organizations tend to experience the same benefits:
- Better quality of software by having less defects make it to production
- Faster delivery of business ideas to market by enabling faster release cycles that allow software to be changed in days or weeks rather than months or quarters
- Cheaper implementation across the entire lifecycle, including less time spent coding, deploying, and testing software changes
These seem almost too good to be true, yet organizations of all sizes, from the leanest startup to the stodgiest enterprise, and organizations in all domains, from banking to games, are reaping these benefits.
What is a DevOps Continuous Delivery Pipeline?
Definition of Continuous Delivery
One of the key components of Continuous Delivery is the deployment pipeline. Pipelines model the software release process by defining the servers and environments that builds will be deployed to, as well as the manual and automatic approvals required at each stage of the process.
Different applications may use different pipelines, or the same application may use different pipelines for different releases.
Stages of a Continuous Delivery pipeline
Contrary to popular myth, there is no such thing as a "universal" continuous delivery pipeline. Not only does every organization have different requirements, but each application within each organization will be different.
For example, a basic web application might use a pipeline with only two stages (testing and production), and simply deploy to a different folder on the same server.
Another application may require a dozen stages, each with multiple targets that go to different environments, and all sorts of automatic and human approvals to meet compliance requirements.
Stages in your continuous delivery pipeline should therefore model the release process that the business is already using, not a technical process that engineers want to use.
No matter what the business structure or business value sought is, continuous delivery pipelines require an input (i.e., the build to deploy), and that input comes from a continuous integration process.
Specifically, this input is a build artifact, essentially a zip file containing the code that will deployed to each server across each environment. When you combine build artifacts created from a continuous integration process and delivery pipelines together, you have the essence of continuous delivery: automated end-to-end software delivery with human judgement.
Bottom line, what’s important is to fit this process to organizational and business needs — not the other way around.
Why is Continuous Delivery Important?
The true value of CI/CD is that organizations, regardless of industry, vertical, geography, market, etc., can adapt to market changes and innovate just as fast (if not faster) than their competitors.
This is accomplished by changing the software that powers the business and then releasing those applications as fast as developers can write new code and testers can verify it.
Prior to CI/CD, software releases are often held up by the unpredictable nature of developing complex changes, bugs/defects during testing, poor code quality, and by processes originally designed for ancient, mainframe-based software systems.
Today, Continuous Delivery pipelines improve this process from start to finish. Despite being released at a higher velocity, applications can also be built and deployed to meet business needs with greater ease than in the past.
How does Jenkins help with Continuous Delivery?
Jenkins is one of the more popular tools currently being used in this process. It was originally designed with one purpose in mind: be a great build automation server. It’s open source, built for developers, and thus has lots and lots of plugins that you can configure to build anything.
At its core, Jenkins operates on the concept of jobs, which were originally designed to run a build script and which are comprised of the following:
- Build Script – a user-supplied ant, maven, shell, batch, or MsBuild script
- [optional] SCM Information - such as CVS or Subversion where your source code resides
- [optional] Trigger – to determine when the job is run
- [optional] Collection Steps – for archiving the artifacts and test results
- [optional] Notification Steps –to notify other people/systems with the build result
Of course, since a build script could just be an arbitrary script, a job could be used to do anything. And thus, Jenkins could be used for anything… even to run a continuous delivery pipeline.
But there are serious limitations for businesses as well. Just as an all-purpose job scheduler would be a poor choice for build automation jobs, Jenkins isn’t equipped to be an end-to-end CI/CD tool because it was never designed as one to begin with.
Therefore, the reality is that while Jenkins is a Continuous Delivery tool, it actually doesn’t create the kind of "universal continuous delivery pipeline" you’ll often hear (erroneously) discussed.
What is a Jenkins Pipeline?
Instead of this mythical "universal" concept, a Jenkins pipeline is actually a special type of job that’s implemented through a plugin originally called workflows.
Essentially, a pipeline job is a sequence of steps that are grouped into stages and that may run on a particular node. Like a job, a step could "do anything," but it’s designed for a single, discrete task and generally is used to run other jobs; there are a handful of built-in steps for things such as running scripts and sending emails.
How do you Build a Continuous Delivery Pipeline Using Jenkins?
The short answer: you can’t. As discussed above, Jenkins on its own is a poor choice for CD. There are a number of reasons why that is the case.
Plugin Volume & Dependencies
The number of Jenkins plugins seems like a blessing, but it can also be a curse. The sheer volume of them is enough to give you choice overload. But, sure, you can make the argument that it isn’t a big problem in and of itself.
However, it can’t be denied that the average number of dependencies you’ll find for plugins is. The more dependencies you have, the greater the risk of something (out of your control) breaking.
Jenkins provides good visibility for small microservices and environments. However, it is problematic as the number of pipelines grows. The evaluation of Shell Script executions does not provide any context about dependencies and global use. That means if one of them fails, you’re sent on a wild goose chase of sorts to figure out what is impacted.
Jenkins cannot natively integrate with your deployment success monitoring setup. To gain visibility into how a deployment to production impacts end-users, you must write ajob or Shell Script. This is an impediment to understanding the full effect of changes to your applications. And in a world where high-velocity release schedules are a must, this risks increasingly brittle pipelines that could cause big problems down the road.
Supplement Jenkins with True Continuous Delivery Tools
Jenkins is a popular tool for a reason. But for all of its strengths, it’s not a good choice for building a Continuous Delivery pipeline. That requires a tool specifically designed to meet the need for high velocity application releases that doesn’t compromise on visibility into the full impact of changes.
Inedo’s BuildMaster was designed with this business need in mind. It allows you to make decisions based on requirements, not release cycles, while giving you full control over permissions and visibility — even with legacy applications. To learn more about how BuildMaster can help your Continuous Delivery pipeline, contact us at firstname.lastname@example.org