Provision TLS certificates using Step-CA and Cert-Manager in Kubernetes

Blog

February 4, 2025

My Kubernetes Homelab is a place to experiment and to host stuff for private things.
Over the time there was many things in kubernetes that was shiny enough to catch my attention on it and one
thing got every time more out of scope and that was to implement a valid TLS for the applications.
The services on the lab are not exposed to the internet, so i need a sort of internal ca.
For me i could als not use the DNS01 Challenge Provider.
The soloution i have implemented is with step-ca as CA, step-issuer to issue the certificates and cert-manager to manage certificate requests.

the test system:

proxmox with two VMs
- one controller and one workload node
- debian(trixie) hosts, kubernetes v1.31 deployed with k3s
- cilium cni
- metalLB
- Ingress Nginx Controler

Install CA

step-cli

Install Helm Repository

We need to generate a configuration file, for that fetch and install the step package

wget https://dl.smallstep.com/cli/docs-cli-install/latest/step-cli_amd64.deb
sudo dpkg -i step-cli_amd64.deb

Generate a password file

echo -n "my-secret-ca-password" | base64 > password.txt

ℹ️

’echo -n’ to omit trailing newline

Add step-certificates helm repository

Add repository and update helm

helm repo add smallstep https://smallstep.github.io/helm-charts/
helm repo update

Install helm chart

helm install --set ca.dns="step-certificates.step-certificates.svc.cluster.local\,127.0.0.1" \
     --set service.targetPort=9000 --set inject.secrets.provisioner_password=$(cat password.txt) \
     --set inject.secrets.ca_password=$(cat password.txt) --set issuer.dnsNames="*.hq.c1.itknecht.de" \
     step-certificates smallstep/step-certificates --create-namespace --namespace step-certificates

cert-manger

Install cert-manager with helm

helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install \
     cert-manager jetstack/cert-manager \
     --namespace cert-manager \
     --create-namespace \
     --version v1.7.1 \
     --set installCRDs=true

step-issuer

Install step-issuer with helm, first we need some values from step-ca
fetch directly from the pod

identify pod

kubectl -n step-certificates get po -l app.kubernetes.io/instance=step-certificates

NAME                  READY   STATUS    RESTARTS     AGE
step-certificates-0   1/1     Running   1 (9d ago)   11d

CA Root certificate base64 encoded

kubectl -n step-certificates exec -it step-certificates-0 -- step ca root | base64 -w0

the CA URL

kubectl -n step-certificates get configmap step-certificates-config -o jsonpath="{.data['defaults\.json']}"  | jq -r '."ca-url"'

The “kid”

kubectl -n step-certificates get -o jsonpath="{.data['ca\.json']}" configmaps/step-certificates-config | jq .authority.provisioners[0]

The provisioner name

kubectl -n step-certificates get configmap step-certificates-config -o jsonpath="{.data['ca\.json']}"  | jq -r '.authority.provisioners[0].name'

put all together in a file ‘step-issuer-values.yaml’

stepClusterIssuer:
  create: true
  caUrl: "https://step-certificates.step-certificates.svc.cluster.local"
  caBundle: "LS0jhadinuahdahdpajHiuNPIUhpivztdUTd...ZUVRDD=="
  provisioner:
     name: "admin" 
     kid: "IUHUiuDSZRCESEtscztesczTRdtc" 
     passwordRef:
       name: "step-certificates-provisioner-password" 
       key: "password"
       namespace: "step-certificates"    <--- here
     enableCertificateChain: true

ℹ️

Important here the Namespace for passwordRef namespace: step-certificates

Then apply with helm

helm install step-issuer smallstep/step-issuer -f step-issuer-values.yaml --namespace step-certificates

Trust CA

last thing is to let your browser and system trust the new CA

fetch the root certificate and write out in a file

kubectl -n step-certificates exec -it step-certificates-0 -- step ca root > root.crt

to import into system is simple with

step certificate install root.crt
step certificate install --all root.crt

If your browser still not accept the import itroot CA directly into Browser

Firefox
Go to Settings → Privacy & Security → View Certificates → Authorities → Import → Select root.crt

Chrome
Open Chrome and go to chrome://settings/security → Manage certificates → Authorities → Import

Example Application

here is a simple example you could test the newly created certificate provider

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kuard
spec:
  selector:
    matchLabels:
      app: kuard
  replicas: 1
  template:
    metadata:
      labels:
        app: kuard
    spec:
      containers:
      - image: gcr.io/kuar-demo/kuard-amd64:1
        imagePullPolicy: Always
        name: kuard
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: kuard
spec:
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
    selector:
      app: kuard
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kuard
  annotations:
    cert-manager.io/issuer: step-issuer
    cert-manager.io/issuer-kind: StepClusterIssuer
    cert-manager.io/issuer-group: certmanager.step.sm
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - example.hq.c1.itknecht.de
    secretName: tls-example-app
  rules:
  - host: example.hq.c1.itknecht.de
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kuard
            port:
              number: 80

Ingress cluster issuer annotations

ℹ️

Ingress annotations has to be set correctly

I have to add this in ingress manifest otherwise cert-manager assumes false issuer

annotations:
  cert-manager.io/issuer: step-cluster-issuer
  cert-manager.io/issuer-kind: StepClusterIssuer
  cert-manager.io/issuer-group: certmanager.step.sm

soloution taken from here
https://github.com/cert-manager/cert-manager/issues/5872#issuecomment-1476087089

Links

Here some Links where i found useful information for my setup

Last updated on February 4, 2025

Simple and clever VM-Backups with Ansible, virtnbdbackup und Borg