DevOps Adoption Approach: Build and Deploy

Build & Deployment processes in a real-time cloud-native project with a Kubernetes CI/CD approach for the enterprise with Serverless Jenkins (Jenkins X).

11 min readDec 16, 2019

**Figure 1:** **Jenkins X Architecture**

The new era of DevOps is about the change that brings developers, IT operations, quality assurance, and business together with a common goal to get projects out faster and of better quality while focusing on collaboration as an extension of the agile principles. Speed is the new norm in software development, driven by an intensifying competitive market, where businesses have to deliver value quickly via software. Having the right culture, tools, processes, and DevOps best practices in place to accelerate the pace of application creation and enhancement is paramount. Now, we need re-imagined CI/CD for cloud-native applications on Kubernetes powered by the cloud, containers, service meshes, microservices, immutable infrastructure, and declarative APIs. Well, Jenkins X is that; it offers a new way by running your CI/CD pipelines on top of Kubernetes. Jenkins X is used as a platform to hide the complexities of Kubernetes from the developer. Developers can now focus on creating high-quality code and excellent user experiences rather than getting bogged down by the intricacies of Kubernetes.

This is the second instalment of a series of two-part blogs on the DevOps adoption approach. In the previous blog, the discussion was on culture, plan, and design. I stated that “DevOps is an outcome-driven culture of practices that supports the agile movement.” That is still the case.

In this blog, I’ll focus on the execution aspects for DevOps by leveraging on the latest breakthrough of automated CI/CD for cloud-native applications on Kubernetes known as Jenkins X. I will explain the basic concepts and draw a parallel between theory and practice by building a pipeline and walk you through the steps required for your first exposure to Jenkins X. It’s essential for you to understand the building blocks and why it’s a perfect fit for cloud-native applications. The instructions will give you a step-by-step guide to Jenkins X and will provide you with prescriptive guidance on how to get started.

Jenkins X is revolutionary and has the potential to change everything from modern applications on customer engagement. It may impact the next generation of development and deployment, similar to what Docker did for the immutable container images and Kubernetes for the orchestration of container services.

Serverless CI/CD pipelines

Jenkins X is an opinionated CI/CD pipelines on top of Kubernetes, a new way to deploy ephemeral pipelines know as serverless pipelines. It is part of the Tekton Pipelines project that provides Kubernetes-style resources for declaring CI/CD-style pipelines.

Modern application development with Jenkins X

Jenkins X “a native cloud version of Jenkins, also knows as Jenkins’ cooler cousin,” is an open-source project that offers automated CI/CD for cloud-native applications on Kubernetes. Jenkins X provides pipeline automation, built-in GitOps, and preview environments to help teams collaborate and accelerate their software delivery at any scale. It makes everything much more straightforward and lets you operate with the power of Jenkins 2.0 by using open-source tools like Helm, Skaffold, Monocular, ChartMuseum, Nexus, KSync, and Docker Registry to build cloud-native applications quickly.

Key Features of Jenkins X

Maintaining dedicated servers for various components involved in CI/CD
Automated Continuous Integration(CI) and Continuous Delivery(CD) tool
Automated management of the environments
Automated preview environments
Naturally integrates with microservice development and release flows
Feedback on issues and pull requests
Manage promotions between environments (Test — Staging — Promotion)

**Figure 3:** **A diagram of the Jenkins X architecture**

Jenkins X —building blocks

Jenkins: CI/CD automation server
Helm: Package manager in form of Charts
Skaffold: Perform the build and push image actions in a pipeline
Nexus: Dependency cache for Node Js & Java Applications and artifact storage
Docker Registry: Container Image Registry (Docker image storage)
ChartMuseum: Registry for Publish Helm Chart (Open-source Helm Chart repository)
Monocular: UI for discovering and running Helm Charts
Prow: Event handlers for GitHub events received through webhooks
Tekton: Management of pipelines natively within Kubernetes
GitOps: Manage the promotions between environments

Kubernetes

Kubernetes is trending, and it is de facto the standard way of installing, upgrading, operating, and managing containerized applications. It’s a container orchestration platform for scheduling and automating the deployment, management, and scaling of containerized applications on any public or hybrid cloud. It has the backing of the top Cloud Service Providers (CSP), such as IBM, Google, Microsoft, and AWS. Each CSP has its offering; to name a few:

IBM Kubernetes Service (IKS)
Google Kubernetes Engine (GKE)
Azure Kubernetes Service (AKS)
Amazon Elastic Container Service for Kubernetes (Amazon EKS)

Kubernetes has its fair share of the market with the infrastructure platform for native cloud computing supported by the Cloud Native Computing Foundation (CNCF); it’s also an integral part of the CI/CD pipeline. Capable of running cloud-native or traditional applications on a single platform. Here are some of the out-of-the-box (OOTB) features:

Networking
Persistence
Service discovery
Rolling upgrades
Cluster elasticity
Resource request / limits
Container scheduling
Container orchestration
Scalability
Container probes
Scale up or down with command, console or automatically
Detached credential configuration management
Self-recovery
Manage the workload and batch execution
Progressive application deployment

What is Helm?

Helm is a Package Manager for Kubernetes like yum and apt-get in the Linux distributions (operating system). In a nutshell, Helm helps manage the deployment of your applications on Kubernetes. It’s a templating tool for Kubernetes components, which means that under it all are just Kubernetes YAMLs with some tools to help install, organize, find, share, and manage applications. The format Helm uses to package applications is called a Chart. A Helm Chart is a collection of files that describe a related set of Kubernetes resources. There are two parts to Helm, a command-line tool (CLI) “client” and a backend server named “Tiller”, which receives the incoming request and installs the appropriate package on the Kubernetes cluster from a single command.

What is Tekton?

Tekton is a set of shared, open-source components for building CI/CD systems, which gives you the ability to build, test, and deploy apps on Kubernetes. Its goal is to provide industry specifications for CI/CD pipelines, workflows, and other building blocks through a vendor-neutral, open-source foundation, by modernizing the Continuous Delivery control plane and moves the brains of software deployment to Kubernetes.

Tekton Pipelines originated from the Knative Build component and was outsourced into a separate repository since August 2018 (it’s relatively new). It focuses only on Kubernetes and is defined by the Kubernetes Custom Resource Definitions (CRDs), which are extended Kubernetes APIs.

Installing Tekton Pipelines to your cluster introduces four new Custom Resources, which will be added to your Kubernestes cluster: Tasks, Pipelines, TaskRuns, PipelineRuns, and PipelineResources.

Task: Defines a set of build steps, such as compiling code, running tests, and building and deploying images.
Pipeline: Defines the set of tasks that compose a pipeline.
PipelineResource: Defines an object that is an input (such as a Git repository) or an output (such as a Docker image) of the pipeline.
PipelineRun: Defines the execution of a pipeline. This resource references the Pipeline to run and which PipelineResource(s) to use as input and output.

What is Prow?

Prow is a collection of utilities used to interact with Git events and schedule build engines. Components include a scalable webhook event handler, automated pull request merger, and many plugins to aid a developer’s experience. Prow provides GitHub automation in the form of policy enforcement, chat-ops via /hook style commands, and automatic PR (Pull Request) merging.

What is GitOps?

GitOps is the conjunction of source control management (SCM) and IT operations born from Git, “a distributed version control system for tracking changes in source code during software development.” And Ops, for IT operations, which is related to infrastructure-as-code (IaC) and deals with computer infrastructure, storage, development pipeline, security, and more. Riding on the success of Git, Git is de facto the automating tool for many company operations. GitOps builds on DevOps with Git as a single source of truth for the whole system, which is also an operating model where changes to infrastructure, including application deployments, are managed via version control system commits.

GitOps means using Git everywhere from development to deployment; it is also opinionated using Kubernetes operating model for prescriptive best practices for cloud-native implementation, management, and monitoring.

IaC is about writing code that describes the infrastructure, similar to writing a program in a high-level language; it brings many benefits, and the most prevailing one is the dynamic infrastructure where resources can easily be created, destroyed, replaced, resized, and moved. GitOps is a smart combination between automation and infrastructure-as-code.

**Figure 10: Example of GitOps pipeline (source: Weaveworks)**

GitOps offer these benefits:

Increased productivity
Enhanced developer experience
Improved stability
Higher reliability
Consistency and standardization
Stronger security guarantees

Putting it all together

Jenkins X brings together all the pieces needed for creating an effective CI/CD pipeline by removing the complexity involved in CI/CD for Kubernetes applications. Through a plethora of automation, tooling, DevOps best practices, open-source projects, such as GitOps, Jenkins, Helm, Tekton, Skaffold, Monocular, ChartMuseum, Nexus, KSync and Pow, all come together in one easy to use native Kubernetes CD platform.

For this demo you will need these tools in place:

A Kubernetes 1.8+ cluster with role-based access control (RBAC) enabled.
The kubectl command-line tool installed on your local machine, configured to connect to your cluster. You can read more about installing kubectl in the official documentation.
helm command-line utility on our local machine, the package manager for K8s. Installing Helm version 2.xx as Jenkins X does not yet support version 3.xx
gitInstalling Git
jx Jenkins X CLI

Putting all the pieces together:

The first step in your journey is to download and install the jx command line tool based on your OS.
Create a Github Access Token by navigating to (https://github.com/settings/tokens/new?scopes=repo,read:user,read:org,user:email,write:repo_hook,delete_repo), give the token a name and save it somewhere safe. You will refer to it during the installation.
Jenkins X requires a Kubernetes cluster to exist, creating a cluster is relatively straightforward, and already well documented on their website.

After installing Jenkins X on an existing Kubernetes Cluster, you will have something like this:

**Figure 11: Jenkins X Cluster deployment**

Create an application on Jenkins X

1- Create a demo application

a) Create a local clone of your fork from https://github.com/ernesen/apiconnectthreeofthree or any other repo

b) Clone the repos git clone https://github.com/$USERNAME/apiconnectthreeofthree or any other repo to your liking.

c) Change directory cd apiconnectthreeofthree

d) Issue command jx import apiconnectthreeofthreeto import existing project into Jenkins:

2- Jenkins X will create a 0.0.1 version of this application and promote it to the staging environment; each pull request will have its ephemeral preview environment.

3- Make a change to the application (editing the index page) and create a PR to the master branch; now our application is deployed in preview environment.

4- Jenkins X will create a preview environment that will trigger a scale event in your cloud provider to add another K8s node for the application to be deployed on; use this environment to review your application before promoting it to production.

5- Approve the PR with an “approved” label — Prow will merge this PR automatically and promote the change to the staging environment. It will trigger an automatic promotion to the staging environment, resulting in a new version of the application:

**Figure 13: Updating the staging environment**

6- Finally, we’re ready for production!

a) Manually using thejx promote prod command.

This was somewhat a quick walk-through with little details, assuming you already master the pull request steps for promoting from staging to a production environment.

Conclusion

It is an exciting time with the number of CI/CD tools available in the market that have been overflowing. In recent years, making this a struggle for teams to make the right decision. DevOps and microservice architecture have allowed teams to be more efficient, delivering projects as an alternative to a traditional monolith architecture. Making the increased adoption of microservices more compelling with higher velocity and ability to deliver applications faster.

Hopefully, I was able to grab your attention and drive my points across. I aimed to entice you to at least look into why you might consider incorporating Serverless Jenkins and Kubernetes’ continuous integration and delivery to solve the following problems into the build and release process of your modern cloud application based on microservices architecture and underlying cloud infrastructure. Jenkins X can help to automatically spin up preview environments for your pull requests so that you can get faster feedback before changes are merged to master and propagate changes from one environment into another.

Along the lines, I hope to have increased your literacy on Serverless Jenkins and development speed, including improving your CI/CD pipeline with a higher level of confidence in change management by removing the barriers to continuous delivery, “a step closer in your continuous evolution”!

Attribution

Special thank you to Amy Xin from Marketing and Communications IBM Garage for proofreading the early draft before publishing. Also, thank you to the IBM Garage Singapore Developer team: Enrique (Ike) Relucio, Ernest Jason C Calalang, Jaren Lim, and Richard Chengbin Yan, who helped test my hypothesis.

References

For more details on the use of Serverless Jenkins (Jenkins X), GitOps, Tekton, DevOps, and CI/CD. Please refer to the links below where you can find many reads on how to master the core concepts that can be adopted quickly and without spending months in trying to figure out “the right way to do stuff”:

Bring your plan to the IBM Garage.
Are you ready to learn more about working with the IBM Garage? We’re here to help. Contact us today to schedule time to speak with a Garage expert about your next big idea. Learn about our IBM Garage Method, the design, development and startup communities we work in, and the deep expertise and capabilities we bring to the table.

Schedule a no-charge visit with the IBM Garage.