How to Kustomize a Helm Chart?
Kustomize
Kustomize is a standalone tool to customize Kubernetes objects through a kustomization file. Kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.
Kustomize is a very opinionated tool. If you are interested to know more about it, you can listen/subscribe to this Kubernetes Podcast from Google by Adam Glick and Craig Box.
Let’s try to understand Kustomize using an example. Personally, Kustomize and Helm solves different problems and if you seriously think about patching the current kubernetes applications, Kustomize is the best tool. You can do the same with Helm and if you like lot of templating, Helm is what you need.
You can watch the below talk about limits of Helm.
Kustomize - Helm Chart
Lets Kustomize a Helm Chart. I took datadog helm chart for this blog. But you can take any helm chart that you want. Base idea will remain the same.
Here is my example repo for this blog Kustomization of datadog Helm Chart
[⎈ k8s ] tree -d
.
└── k8s
├── bases
│ └── vendor
│ └── stable
│ └── datadog-2.3.14
│ ├── charts
│ │ └── kube-state-metrics
│ │ └── templates
│ ├── ci
│ ├── docs
│ └── templates
└── overlays
├── dev
│ ├── patches
│ └── secrets
└── prod
├── patches
└── secrets
18 directories
Kustomize - Its Patching
There are 2 main directories that Kustomize uses i.e. bases and overlays.
bases will contain the base/raw form of Helm chart converted into a plain YAML file which can be easily understandable by kustomization.
overlays will contain the actual patches that you will put on bases. There will be different patches as per your needs. Here I took an example of patches for dev and prod environments.
Kustomize - bases dissection
bases contains the main helm chart for datadog.
├── Makefile ------------------> use to template the helm chart into yaml format
├── datadog.values.yaml -------> is the values files use to tempaltise datadog helm chart
├── datadog.yaml --------------> is the actual kustomize manifest file
├── kustomization.yaml --------> contains the templated helm chart manifests in yaml format
├── transformer-common.yaml ---> transformer used by kustomize to override images
├── transformer-images.yaml ---> transformer used by kustomize to add labels
├── transformer-patches.yaml --> transformer used by kustomize to patche k8s resources
└── vendor --------------------> actual helm chart here
#1. Makefile
Makefile have just 1 update command which will be used to create datadog manifest file again.
If a new version of datadog helm chart released, just fetch the helm chart in vendor/stable directory and update Makefile
[⎈ k8s ] cat Makefile
update:
helm3 template \
datadog \
--namespace=monitoring \
--values datadog.values.yaml \
vendor/stable/datadog-2.3.14/ \
> datadog.yaml
#2. datadog.values.yaml
The actual values for datadog helm chart are present in this file. These values can be default values from helm chart.
datadog.values.yaml will be applied everytime with Makefile update command which in turn run helm template command for the datadog helm chart.
All the common values that can be applied to helm chart should be put in datadog.values.yaml
datadog:
apiKey: ""
appKey: ""
clusterName: "eks-dev-cluster-1"
kubeStateMetricsEnabled: true
tags:
- "region: eu-east-1"
- "customer: xyz"
clusterAgent:
enabled: true
#3. datadog.yaml
This file is the result of Makefile update command.
Kustomize will be using it to create kubernetes resources.
---
# Source: datadog/templates/agent-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: datadog-cluster-agent
labels:
...
---
...
---
...
---
#4. kustomization.yaml
kustomization.yaml file has a specific syntax to create kubernetes manifests.
Under resources and transformers sections, need to add relative files location paths.
resources:
- ./datadog.yaml
transformers:
- ./transformer-common.yaml
- ./transformer-patches.yaml
- ./transformer-images.yaml
More details about kustomize docs
#5. transformer-common.yaml
The labels transformer adds labels to the metadata/labels field for all manifest files.
LabelTransformer code will look like below.
---
apiVersion: builtin
kind: LabelTransformer
metadata:
name: stub
labels:
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/instance: "datadog"
fieldSpecs:
- path: metadata/labels
create: true
As seen from above, following 2 labels will be added to all the manifests generated by kustomization.
labels:
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/instance: "datadog"
#6. transformer-images.yaml
The default images transformer updates the specified image key values found in paths that include containers and initcontainers sub-paths.
apiVersion: builtin
kind: ImageTagTransformer
metadata:
name: image-quay-coreos
imageTag:
name: quay.io/coreos/kube-state-metrics
newName: my-registry.servers.xyz.com/mirrors/quay.io/coreos/kube-state-metrics
newTag: v1.9.5
#7. transformer-patches.yaml
Patch Transformer can patch the k8s manifests and can replace/remove contents. Below PatchTransformer will replace the image for Deployment and remove the labels imposed by helm chart.
---
apiVersion: builtin
kind: PatchTransformer
metadata:
name: helm-cleaner-kube-state
target:
group: apps
version: v1
kind: Deployment
namespace: monitoring
name: datadog-kube-state-metrics
patch: |-
- op: replace
path: /spec/template/spec/containers/0/image
value: "my-registry.servers.xyz.com/mirrors/quay.io/coreos/kube-state-metrics:v1.9.5"
- op: remove
path: /spec/template/spec/containers/0/resources/limits/cpu
- op: remove
path: "/metadata/labels/helm.sh~1chart"
- op: remove
path: "/metadata/labels/app.kubernetes.io~1managed-by"
Its clear from code that our target is Deployment datadog-kube-state-metrics in monitoring namespace.
target:
group: apps
version: v1
kind: Deployment
namespace: monitoring
name: datadog-kube-state-metrics
PatchTransformer will replace first container’s image of the pod. Please lookout of numeral 0.
0 is first container as per spec definition
1 is second container as per spec definition and so on
- op: replace
path: /spec/template/spec/containers/0/image
value: "my-registry.servers.xyz.com/mirrors/quay.io/coreos/kube-state-metrics:v1.9.5"
PatchTransformer will remove first container’s resource limits CPU values.
- op: remove
path: /spec/template/spec/containers/0/resources/limits/cpu
PatchTransformer will remove below label from deployment
- op: remove
path: "/metadata/labels/helm.sh~1chart"
PatchTransformer will remove below label from deployment
- op: remove
path: "/metadata/labels/app.kubernetes.io~1managed-by"
#8. vendor
vendor directory will contain actual helm chart from upstream stable repository.
[⎈ k8s ] tree -d vendor/
vendor/
└── stable
└── datadog-2.3.14
├── charts
│ └── kube-state-metrics
│ └── templates
├── ci
├── docs
└── templates
Kustomize - overlays dissection
Overlays can be divided into environment based directories. Every environment contains patches relevant to it. Patching becomes easy and we can add manifest per environment very easily using a pipeline.
secrets can be added which later can become env variables or volumes in Pod using configmap/secret.
[⎈ k8s ] tree -d
.
├── dev
│ ├── patches
│ └── secrets
└── prod
├── patches
└── secrets
Kustomize - Run it
To run the kustomize, we change directory to overlays environment and apply it.
[⎈ k8s ] cd k8s/overlays/dev/
[⎈ k8s ] kustomize build .
[⎈ k8s ] kustomize build . | kubectl apply -f -
[⎈ k8s ]
[⎈ k8s ]
[⎈ k8s ] cd k8s/overlays/prod/
[⎈ k8s ] kustomize build .
[⎈ k8s ] kustomize build . | kubectl apply -f -
Kustomize - improvisation
- Use kapp tool with kustomize for deployment
- Use sops to encrypt secrets
- Use pipelines to install patches per environment of overlays
Reference lists