As your team starts to deploy resources to Kubernetes regularly, it becomes necessary for you as a cluster administrator to maintain good standards and consistency of the Kubernetes resources. Be it, ensuring all the resources have set of labels, or ensuring you only pull images from your enterprise container registry. Gatekeeper is a well known policy enforcement tool using Open Policy Agent (OPA) – which is a opensource, Cloud Native Computing Foundation (CNCF) project.
But did you know you can validate policies on your Kubernetes manifests before you deploy them on to the cluster? In this post, we will see how we can govern our deployments using Conftest and OPA policy agent.
However, Gatekeeper is installed on the cluster and thus ensures no policy is broken at deployment time. This means that any validation of policies happen only when you are trying to deploy resources to cluster. While this ensures that no resource violates the policy, you would like to know about these policies much earlier in your CI/CD pipeline. Doing policy validations much to the left of your deployment pipeline ensures your deployment going smooth when necessary.
This is where Conftest helps. Conftest relies on OPA and policies are written using Rego – thus the policies you write for Gatekeeper will be compatible with Conftest. But more importantly with Conftest, you can validate your local manifests against OPA policies locally and ensure your resources are compliant before you deploy them.
Installation
Installation is really easy if you are on Mac – For other platforms refer to the documentation
brew install conftest
Folder structure
By default, Conftest expects you to maintain your policies under policy folder at the same location as your Kubernetes resources. If you prefer a different path, you will want to pass it using CLI or set environment variable CONFTEST_POLICY.
📂 src
📂 k8s
📄 deployment.yml
📄 service.yml
📁 policy
📄 replica.rego
📄 labels.rego
📂 app
📄 main.ts
📄 package.json
Writing Policies
As mentioned previously, policies are written in Rego. I struggled to write policies initially and constantly went back to documentation. However, once you write couple of policies, you will get a hang of it. Take a look at the simple policy to check every deployment has at least 2 replicas.
package main
deny_replicas[msg] {
input.kind == "Deployment" # check if it is a Deployment
input.spec.replicas < 2 # And the replicas are < 2
msg := "Deployments must have 2 or more replicas" # show the error message and fail the test
}
input is the complete yaml document from our deployment yaml (see below) and we we are checking if kind is equal to Deployment. If its deployment, we move to next line in the constraint and check if spec.replicas is less that 2.
apiVersion: apps/v1
kind: Deployment
metadata:
name: mynodeapi-dep
labels:
app: dep-k8s-nodejs-api
spec:
replicas: 1
selector:
...
You can write other policies similar to the one above to validate various aspects of Kubernetes resources. Let us see few examples.
Validate resources have recommended labels
This policy validates our resources have the required labels and fail if any labels from required_deployment_labels object are not found.
package main
import data.kubernetes
name = input.metadata.name
required_deployment_labels {
input.metadata.labels["app.kubernetes.io/name"]
input.metadata.labels["app.kubernetes.io/instance"]
input.metadata.labels["app.kubernetes.io/version"]
input.metadata.labels["app.kubernetes.io/component"]
input.metadata.labels["app.kubernetes.io/part-of"]
input.metadata.labels["app.kubernetes.io/managed-by"]
}
violation[msg] {
input.kind == "Deployment"
not required_deployment_labels
msg = "Must include Kubernetes recommended labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels"
}
Reference container images only from our enterprise Azure Container Registry
package main
deny[msg] {
input.kind == "Deployment"
some i
image := input.spec.template.spec.containers[i].image
not startswith(image, "myacr.azurecr.io") # validate images start with endpoint for our container registry
msg := sprintf("image '%v' comes from untrusted registry", [image])
}
As you can see rules can be very powerful.
Testing
The command to test resources using Conftest is conftest test <PATH>. Since I would like to test all resources under k8s folder, I pass folder path as below.
Running this you will see the output as below (ignore other errors as I have other policies). The test failed because we have set deny rule if spect.replicas < 2 and in our case our deployment yaml has replicas: 1 (see spec section in the deployment yaml above).
Using Conftest in GitHub Actions
Making Conftest work in your Continuous Integration (CI) process is simple. For demo purposes, I am using GitHub Actions in my repo here. If you run the tests, you will see the action fails with errors – see the output
My action workflow looks like below.
name: build
on:
push:
branches: [main]
pull_request:
branches: [main]
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: install conftest
run: |
wget https://github.com/open-policy-agent/conftest/releases/download/v0.30.0/conftest_0.30.0_Linux_x86_64.tar.gz
tar xzf conftest_0.30.0_Linux_x86_64.tar.gz
sudo mv conftest /usr/local/bin
rm -rf conftest_0.30.0_Linux_x86_64.tar.gz
- name: run conftest
run: |
conftest test $/k8s
Conclusion
As you can see, Conftest lets you validate and govern your Kubernetes resources efficiently and can easily be integrated with your CI workflows. This lets your team standardise the common practices, go through PR review process before eventually deploying to the cluster. Once deployed to cluster, you can use Gatekeeper to validate as well to full proof your workloads.
