Kubernetes Manifests

The API interface to operate a Kubernetes (K8s) cluster are yaml documents, called manifests, which you can apply with kubectl apply.

Manifests, when applied, create or update Kubernetes API objects in the cluster (e.g. Namespaces, Pods, PersistentVolumeClaims, etc.). A Kubernetes manifest contains resource definitions that declare the desired state of these objects. A resource definition adheres to a specific form, i.e.

apiVersion: networking.k8s.io/v1
kind: Ingress
# Content adhering to the resource type `Ingress`.

where

• apiVersion is used for versioning and

  Details

  K8s uses mostly *.k8s.io/v1 for apiVersion for its native provided resource types such as:

  API Group	Stable Version	Resource Type (kind)
  apps	v1	Deployment, StatefulSet, DaemonSet
  batch	v1	Job, CronJob
  networking.k8s.io	v1	Ingress, NetworkPolicy
  storage.k8s.io	v1	StorageClass, VolumeAttachment
  authorization.k8s.io	v1	SubjectAccessReview
  certificates.k8s.io	v1	CertificateSigningRequest
  apiextensions.k8s.io	v1	CustomResourceDefinition

  For the complete list see here.

• kind is the resource type name the document describes for the K8s cluster.

Kubernetes supports operators. An operator, when applied in a cluster, can define/register a set of custom resource types (e.g. kind: MyResource) by installing custom resource definitions (a.k.a. CRDs). The operator is responsible to handle these custom resource definitions when they are applied to the cluster.

Example: Custom Resource Definition

The following defines a custom resource definition (CRD) to support a resource with kind: MyResource.

Custom Resource Definition

Define a new custom resource type MyResource by applying the following in the cluster:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myresource.ch.datascience.io
spec:
  group: ch.datascience.io
  scope: Namespaced
  names:
    plural: myresources
    singular: myresource
    kind: MyResource # <<< This defines the resource type.
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                value:
                  type: integer

Usage

Create a resource MyResource by applying the following in the cluster:

apiVersion: myresource.ch.datascience.io/v1
kind: MyResource
metadata:
  name: example
spec:
  value: 42

Writing Kubernetes Manifests

To deploy an application (a set of K8s objects e.g. a Service consisting of some Pods (containers)) one needs to write K8s manifest files. These files can quickly become pretty large. Normally, an application has different configurations depending on the environment it is deployed. Thus, the need to template manifests becomes essential to handle a certain degree of configurability depending on the deployed environment.

note

Imagine two environments e.g. development and a production environment where each has a K8s cluster. The application running in the development cluster runs with debug logging and other features enabled and the application in the production cluster runs without debug logs and with hardened container OCI images and other security features enabled.

A plethora of tools and processes have been developed to help in creating/templating K8s manifests. The below table summarizes these tools.

Tool	Approach	Language / Format	Best For	Key Strengths	Trade-Offs
Raw YAML	Static manifests	YAML	Small setups, learning	Simple, transparent, no tooling	No reuse, hard to scale
kustomize	Declarative patching	YAML	Simple environment overlays	Native to kubectl, no templating, deterministic	Limited logic, no loops/ifs
helm	Template rendering	YAML / Go Templating	App packaging, reuse	Large ecosystem, charts, versioning	Weak templ. typing, harder to reason
ytt	Data-driven template rendering	YAML / Custom + Starlark	Strong typing, safety	Templ. type safety, schema validation, single-purpose/focused/unix-philosophy tool	Smaller ecosystem, steeper learning
pulumi	Infrastructure as code	Go, TS, Python, etc.	Preferring code and full-stack infra	Real languages, strong typing, refactoring	Requires runtime, state backend
CDK8s	Infrastructure as code	Go, TS, Python, etc.	Preferring code	Real languages, strong typing, constructs	Extra abstraction, codegen step

Comparing different manifest rendering techniques in the following must consider secrets handling because it defines the flexibility of the workflow to a great deal.

Secrets & Manifests

Secrets in K8s are (normally) represented by Kubernetes native Secret (or ExternalSecret) objects in the cluster. How secrets interact with other objects depends on how you use them. Secrets will eventually be used used in containers (e.g. Pod). Secrets can be made available to containers by either environment variables or by mounting files into Pod containers (e.g. from ConfigMap K8s objects).

The following table gives an overview of different tools to handle secrets.

Method / Tool	Approach	Secret Storage	Integration	Pros	Cons / Trade-offs
Kubernetes Secrets	Native Kubernetes object	Base64-encoded in etcd	kubectl apply, Helm, Kustomize	Simple, native, easy to use	Stored in plaintext in etcd (base64 is not encryption), limited rotation
Sealed Secrets (Bitnami)	Controller + encrypted manifest	Encrypted YAML in Git (with Cluster's private key)	Controller decrypts at apply	GitOps-friendly, safe for repo, easy automation	Requires controller, extra CRD, vendor dependency
SOPS (Mozilla)	Encrypted secrets in YAML	Encrypted YAML in Git	Works with Git, Kustomize, Helm	Strong encryption, multiple key backends, Git-friendly	Requires decryption step before apply, tooling complexity
Secret Operator / Kubernetes Operator	Operator reconciles secrets	Can pull from vaults / KMS / other secret stores	Automatically populates Kubernetes Secrets	Automates secret rotation and syncing, integrates with vaults	Adds complexity, needs running operator, learning curve
HashiCorp Vault	External secret store + sync	Vault backend	Vault Agent / CSI driver / Operator	Centralized secret management, auditing, rotation, dynamic secrets	Extra infrastructure, requires authentication setup
External Secrets (Kubernetes External Secrets)	Operator/controller	External secret stores (AWS, GCP, Azure)	Syncs into K8s Secrets	GitOps-friendly, centralized control, multiple providers	Operator required, latency for sync, complexity
Helm secrets plugin	Template + encryption	Encrypted Helm values	Decrypts before rendering	GitOps-friendly, works with existing Helm workflow	Encryption limited to files, Helm-specific
sealed-env / sops + GitOps	Git repository encrypted secrets	Encrypted YAML in Git	Decrypt during CI/CD	Strong encryption, integrates with pipelines	Requires tooling in CI/CD, extra step

Any method above, will tie the Secret/ExternalSecret objects in templated manifests to secrets either committed & encrypted in Git (e.g. sops) or to secrets an external vault.

Template Workflows

Despite that helm and kustomize are widely used and established, the two tools are not recommended for new projects as they provide not much type-safety. Other tools like ytt, pulumi and cdk8s provide better alternatives for modern templating.

Workflow with ytt, sops & helm

Carvel has built ytt with the idea to do one thing and only one thing, which is a core strength and a weakness. The tool ytt only cares about rendering YAML from templates. It is not related to K8s manifest templating at all but is of course mostly used for that task. The input to ytt is always a schema.yaml and a bunch of template *.yamls which it renders into a final YAML.

The caveat of ytt becomes predominant when you want to combine it with some Helm templating and also having secrets in the game. Then you realize, that you need a script/orchestration to drive the templating workflow (e.g. encrypt secrets + render with ytt + include somehow also Helm charts).

The below diagram shows a manifest rendering workflow which is currently implemented in a quitsh manifest runner.

warning

This workflow is stable but also probably not the recommended approach for newer projects. The below workflow however serves as a point of inspiration when using more IaC-like tools like pulumi or cdk8s.

The above workflow works by performing two ytt rendering steps:

First ytt Rendering

The first ytt rendering produces a pre.yaml file from the following inputs:

• ❶ The schema schema.yaml, which defines the schema that all ytt data values must follow.

• ❷ The ytt-templated source files in src/.../*.yaml, which contain ytt syntax and define all Kubernetes objects for the application.

• ❸ Optional ytt-templated Helm objects, such as:

  apiVersion: ch.quitsh.io/v1
  kind: ManifestRunnerHelm
  # Targets the correct Chart.yaml
  target: custodian-mongodb-419fdcde-1d1f-46f2-a2b2-da76102da9fd
  # All values for the Helm rendering
  values: ...

  These objects encode data used in step ❽ to render additional Helm charts.

• ❹ ytt data values in deployment/<env>/.../*.yaml, which provide environment-specific configuration (for example, <env> = production).

• ❺ Optional sops-encrypted secret files, which are decrypted into plaintext as part of step ❻.

The output pre.yaml still contains the custom resource definition ManifestRunnerHelm, which is processed in the next step.

Second ytt Rendering

The second ytt rendering performs a combination of:

• Rendering all Helm charts defined by the extracted ManifestRunnerHelm CRDs from step ❽
• Combining those rendered charts with the filtered pre.yaml

The output is the final final.yaml, which can then be applied to a Kubernetes cluster using kubectl.

Summary

This process ensures that everything is driven and controlled through ytt templating. Helm values are defined via custom resource definitions (ManifestRunnerHelm) that target a specific Chart.yaml. As a result, the ytt data values validated against schema.yaml become the single source of truth for the entire workflow.

The workflow is reliable and stable but has the following caveats:

Caveats

• The secrets tooling over sops decrypt as shown above will render plain-text secrets in the final.yaml. This is mostly ok, since the final manifest is not stored.

• It is better to not allow plain-text secrets to be set as data-values specified by schema.yaml, e.g. the following data-value myapp.secret:

  #@data/values
  ---
  myapp:
    #@schema/validation min_len=1
    secret: ""

  should rather not be a direct plain-text secret but a reference to a Secret/ExternalSecret object which is provided somewhere in deployment/<env>/.../mysecret.yaml files and added to the ytt rendering in the first step.

  Better is the following schema.yaml:

  #@data/values
  ---
  myapp:
    #@schema/validation min_len=1
    secretRef: ""

  which lets the user set myapp.secretRef = "banana-secret" and then providing that Secret object in the deployment/<env>/.../banana-secret.yaml folder:

  apiVersion: v1
  kind: Secret
  type: Opaque
  metadata:
    name: "banana-secret"
  data:
    password.txt: "secret...."

  The above means that you would mount secrets always into containers (which is recommended) or use env: like:

  apiVersion: v1
  kind: Pod
  metadata:
  name: banana-pod
  spec:
    containers:
      - name: app
        image: awesome:1.0.0
  
        # 1. Secret mounted as a file `/run/secrets/password.txt` (mostly better).
        volumeMounts:
          - name: banana-secret-volume
            mountPath: /run/secrets
            readOnly: true
  
        # 2. Secret as environment variable.
        env:
          - name: SIGNATURE
            valueFrom:
              secretKeyRef:
                name: banana-secret
                key: password.txt
    volumes:
      - name: banana-secret-volume
        secret:
          secretName: banana-secret

• The above process is also bespoke and CD operators (GitOps driven) like kapp and argocd might be limited to run that workflow.

  ArgoCD provides config management plugins which enables to run a plugin inside a sidecar-container to produce the YAML output. Such a sidecar container has currently not been implemented yet for quitsh manifest runner.