CI/CD pipelines enhance the software delivery process by automating key stages such as testing, security scanning, and deployment. Adopting a pipeline-based workflow helps you ship more quickly by passing all code through a consistent set of steps. This guarantees that required standards are adhered to.
You’ll see the greatest CI/CD success when you combine the right set of tools for your team’s situation. With so many platforms available, it can be challenging to select an optimal solution.
In this article, you’ll learn some best practices and techniques for managing CI/CD pipelines with Kubernetes. You can use these strategies to automate deployments straight to your production clusters. Let’s dive in!
Best Practices for CI/CD and Kubernetes
Kubernetes is the leading container orchestrator for deploying, scaling and administering container workloads in production. But how do those workloads reach your cluster? Manually creating them by running Kubectl terminal commands is inefficient and prone to error.
Using CI/CD pipelines with Kubernetes lets you automatically deploy applications whenever your source code changes. You can improve the Kubernetes management experience by automating key tasks. CI/CD also provides an opportunity to verify new deployments meet the standards you expect, such as passing your test suite and being free from security issues.
Nonetheless, integrating Kubernetes into CI/CD workflows can create new problems. Cluster administrators often struggle to see what’s actually running in their environments, while addressing any deployment failures can be time-consuming. Integrations are also potential security risks, as compromising your CI/CD provider could let attackers access your clusters.
Here’s how you can address these drawbacks to combine CI/CD and Kubernetes with confidence.
1. Use GitOps
The term GitOps refers to the practice of keeping your infrastructure configuration as files in your source control repository. Managing everything with Git ensures every essential resource is versioned. It also means you can reference any resource within your pipelines, so you can check the validity of your config and quickly identify any errors.
Manually triggering pipelines from external systems is unreliable and error-prone. Using Git to run a pipeline each time you commit ensures changes can’t slip through to production unnoticed. If you need to revert a deployment, you can check out an older commit and replay the pipeline.
2. Scan Your Container Images
Deploying container images straight to Kubernetes is a security risk. Your images could include zero-day vulnerabilities, accidentally hardcoded secrets, or a malicious package that’s made it into your supply chain.
To stay safe, you can use your pipeline to perform image vulnerability scans. Abort the rest of the pipeline if a fault is found. This will prevent new vulnerabilities from being silently pushed into your cluster.
You can use tools such as Snyk to generate your reports. Snyk also powers the docker scan command that’s integrated into Docker’s CLI.
3. Use Helm to Manage Deployments
Applying Kubernetes manifests individually is problematic because files can get overlooked. Packaging your applications as Helm charts lets you version your manifests and easily repeat deployments into different environments. Helm tracks the state of each deployment as a “release” in your cluster.
Helm also simplifies configuration management. You can easily supply variables when you install your charts, either using YAML files or as command line arguments. This lets you conveniently override specific variables within your pipeline scripts.
Check out Kustomize vs. Helm comparison.
4. Ensure There’s a Rollback Mechanism (and Check It Works!)
Continuous delivery (CD) pipelines are great until they break. When this happens in Kubernetes, it can be unclear how to proceed. Your deployment could be stuck in a failed state, unable to transition into the newly desired one.
Make sure you’ve got a reliable rollback strategy that’s ready to deal with these events. If you’re using GitOps, you should be able to rollback by reverting to a previous commit, then running your pipeline again. The old revision will now be new once more, enabling it to replace the bad state in your cluster.
Stateful workloads, such as databases, might need additional steps in case the failed deployment partially migrated some data while leaving other components stuck on the old version. Your rollback strategy should be capable of preserving and restoring persistent data when required. Finally, make sure your rollback system actually works — too many teams only discover bugs in their rollback scripts during real incidents.
5. Use Immutable Image Tags
Image tag immutability makes your deployments reproducible and helps enable resilient rollbacks. Always deploying my-app:latest is dangerous because the exact image that’s selected could change each time.
It’s much safer to uniquely tag each image you build. Using a truncated commit SHA is a popular strategy, resulting in tags like my-app:0ab43f. This allows you to easily cross-reference commits against the image artifacts and deployments they create. If you need to rollback, you can modify your Kubernetes deployment to refer to the image built from the previous commit.
6. Follow Kubernetes Security Best Practices
It’s important to keep following standard Kubernetes security best practices when you’re integrating your clusters with CI/CD platforms. Harden your environment by enabling etcd encryption, setting up precise RBAC permissions for your CI/CD user accounts, and ensuring any sensitive config values are added to Secrets instead of plain ConfigMaps.
You should check your CI/CD platform only exposes your cluster connections to projects and users you’ve specifically authorized. Remember that anyone who can commit to your project could send malicious code through the pipeline into your Kubernetes cluster.
7. Try Pull-Based CI/CD Workflows
There are two main ways to connect CI/CD systems to Kubernetes:
- Push-based workflows rely on the CI/CD platform being provided with certificates and credentials that let it reach out to your Kubernetes cluster. You then use familiar tools such as Kubectl and Helm to push changes into Kubernetes from the outside.
- Pull-based workflows run an agent utility inside the cluster. The agent is given credentials for your source control system. Its access should be scoped to just the projects it needs to deploy. The agent periodically looks for changes and pulls them into the cluster. This inverts the push-based model.
Pull-based CI/CD is increasingly popular because it provides stronger protection for your cluster. In order to be effective, push-based workflows require privileged Kubernetes credentials to be stored on your source control server. If the server’s compromised, attackers could steal the credentials, access your cluster, and perform arbitrary actions. With a pull-based workflow, attacks against the CI/CD platform won’t grant access to your entire cluster.
💡 You might also like:
5 Tools for Effective CI/CD With Kubernetes
None of these best practices need to be difficult. You should aim to create a toolchain that supports you in efficiently running CI/CD pipelines with Kubernetes, while avoiding security problems and common mistakes. Here are five tools you can try out.
GitLab
GitLab is one of the most popular all-in-one software delivery platforms. It includes source management and CI/CD functions with excellent Kubernetes integration.
Modern GitLab releases have transitioned to a pull-based workflow using the GitLab Agent for Kubernetes. The agent runs in your cluster and is authorized to access specific GitLab projects. You can then interact with Kubernetes from within your pipeline scripts. All communication flows securely through the agent.
GitHub Actions
GitHub Actions is GitHub’s CI/CD solution. You can use it to run automated tasks each time you change your code. Although the platform lacks a built-in Kubernetes integration, third-party plugins such as Azure’s Deploy to Kubernetes Cluster action can automate deployments and manage different rollout strategies.
Argo CD
Argo CD is a continuous delivery tool that’s purpose-built for Kubernetes. It’s maintained as a Cloud Native Computing Foundation (CNCF) project.
Argo is installed in your cluster as a Kubernetes controller. It monitors your applications and identifies any differences compared with your source control repository. When drift occurs, Argo will automatically resynchronize the state by pulling your code and updating the resources in your cluster. It facilitates fully automated Kubernetes deployments.
Read more about this tool in our ArgoCD — Practical Tutorial With Kubernetes article.
Kubectl and Helm
You can use existing Kubernetes CLIs such as Kubectl and Helm within your own CI/CD scripts. This will require you to supply credentials that let the CLIs authenticate with your cluster, creating a push-based workflow.
This method can be more approachable and easier to set up than using a dedicated tool, especially if you’re only experimenting with Kubernetes. It lets you use the same commands you run on your workstation to roll out changes within your pipelines.
Spacelift
Spacelift brings the benefits of CI/CD to infrastructure management. You can use Spacelift to deploy changes to Kubernetes with GitOps while benefiting from robust security policies and automated compliance checks. Spacelift also works with other infrastructure as code (IaC) providers, so you can use similar techniques to manage every component of your stack.
Key Points
Using CI/CD with Kubernetes allows you to automate deployments, rapidly scale your services, and be confident that all live code has passed your test suite. The two technologies pair well together if you follow the best practices we’ve explored.
When selecting a CI/CD pipeline solution for Kubernetes, you should assess how deep the integration goes and look at its security model, rollback mechanisms, and support for your source management system.
You can use CI/CD to apply infrastructure changes using IaC techniques, either to Kubernetes or other tools such as Terraform and Ansible. Check out Spacelift to collaborate on infrastructure with full control and flexibility. It includes role-based security policies, detailed usage insights, and full visibility into everything running in your cloud accounts.
Written by James Walker