Get 20€ credit with Hetzner using my link :)

Today i tried to setup a Kubernetes cluster on the Hetzner cloud using their loadbalancer to route my external traffic to the nginx ingress controller as well as issuing Letsencrypt certificates with cert-manager.
It was quite a ride but eventually i got a working version.
These are my steps working with K8S v.1.24.x (could be working on 1.25 but i have chosen 1.24 in order to support longhorn). Use my versions if you want a guaranteed success.

  1. Create a new project (or use existing)

2. Add your SSH key.

3. Create an internal network

4. Create 1 master server (e.g CX21. Do not use less than 2 CPUs) with Ubuntu 22.04 and attach it to your internal network.

5. Add as many worker nodes with Ubuntu 22.04 as you want with a bit more power (CX21 and above). You can automatically let them attach to the network.

Server Setup

Run this on all your servers

1. Update/Upgrade your Server

sudo apt update
sudo apt -y full-upgrade
[ -f /var/run/reboot-required ] && sudo reboot -f

2. Disable Swap:

swapoff -a
sed -i '/swap/d' /etc/fstab

3. Enable "br_netfilter" kernel module:

cat > /etc/modules-load.d/k8s.conf << EOF


modprobe br_netfilter

4. Enable routing:

cat > /etc/sysctl.d/kubernetes.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

sysctl --system

5. Import keys + add repos: (there could be an updated repo in the future but currently (December '22) there is only Xenial available)

sudo apt install curl apt-transport-https -y

curl -fsSL|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/k8s.gpg

curl -s | sudo apt-key add -

echo "deb kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

curl -fsSL | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] $(lsb_release -cs) stable"

6. Install Kubernetes packages: (im using a fixed version 1.24 here but you can use the default packages if you need a newer one).

apt update
apt install -y kubeadm=1.24.8-00 kubectl=1.24.8-00 kubelet=1.24.8-00
apt-mark hold kubelet kubeadm kubectl 
#this stops apt from upgrading to a newer version later on

7. Adjust containerd config

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sed -i -e "s/SystemdCgroup = false/SystemdCgroup = true/g" /etc/containerd/config.toml
systemctl restart containerd
systemctl restart kubelet

8. Configure "kublet" service to enable Hetzner Cloud Provider:

cat > /etc/systemd/system/kubelet.service.d/20-hcloud.conf << EOF

systemctl daemon-reload
systemctl restart kubelet

9. Edit /etc/hosts and add your servers. E.g: k8s-master k8s-worker-1 k8s-worker-3 k8s-worker-2

Initialize Kubernetes Cluster

  1. ONLY on the master node:
kubeadm init \
  --pod-network-cidr= \
  --apiserver-advertise-address= \
  --upload-certs \

2. Copy the kube config

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

3. Join your worker nodes with the master

kubeadm token create --print-join-command
This should output something like this:
Run this command on every worker/node
kubeadm join k8s-master:6443 --token xxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxx

All next steps are only on the master!

Network Controller

Install Flannel Network Controller:

kubectl apply -f
Note: When running "kubectl get nodes" now, the status of all nodes should be "Ready".
root@master:~# kubectl get nodes
NAME       STATUS   ROLES           AGE    VERSION
master     Ready    control-plane   103s   v1.24.8
worker-1   Ready    <none>          36s    v1.24.8
worker-2   Ready    <none>          35s    v1.24.8
worker-3   Ready    <none>          33s    v1.24.8

Hetzner Cloud Controller

1. Create Secrets: (create an API Token within hetzner cloud for this)

kubectl -n kube-system create secret generic hcloud \
  --from-literal=token=<$HETZNER_API_TOKEN> \
Note: To get  <$CLUSTER_NETWORK_ID> you need to check the URL on the Hetzner Cloud site and grab the ID there. E.g:

2. Install Hetzner Cloud Controller:

kubectl -n kube-system apply -f

CSI Driver (optional for volumes)

1. Create Secret:

kubectl -n kube-system create secret generic hcloud-csi \

2. Install CSI Driver:

kubectl apply -f

Ingress Controller

Install Nginx Ingress Controller as a DaemonSet:

wget -q -O /tmp/nginx.yaml

sed -i -e "s/kind: Deployment/kind: DaemonSet/g" /tmp/nginx.yaml
sed -i -e '/^kind: ConfigMap.*/i  \ \ compute-full-forwarded-for: \"true\"\n \ use-forwarded-headers: \"true\"\n \ use-proxy-protocol: \"true\"' /tmp/nginx.yaml

kubectl apply -f /tmp/nginx.yaml

Load Balancer

Create and connect Ingress Controller to Hetzner Load Balancer:

kubectl -n ingress-nginx annotate services ingress-nginx-controller \"k8s-lb" \"nbg1" \"true" \"true" \"<$CLUSTER_LB_HOSTNAME>"
Note:  <$CLUSTER_LB_HOSTNAME> needs to be a valid DNS record that points to the Load Balancers’ public IP address.

The controller should have created a Loadbalancer and attached all nodes to it (might take a minute).


1. Install cert-manager using helm (took a few mins to run for me):

curl | bash

helm repo add jetstack

helm repo update

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.10.1 \
  --set installCRDs=true

2. Create issuer:

Do this for every namespace you want to issue certificates from (replace the namespace in the end. e.g default). Change the email to your own.
kubectl create --edit -f -n default

3. Create an example service

kubectl apply -f
# expected output: deployment.extensions "kuard" created

kubectl apply -f
# expected output: service "kuard" created

4. Add ingress to example service

Replace all "" with your own domain. MUST BE pointing to your Loadbalancers IP!
kubectl create --edit -f

If anything is unclear or you want to add something please register an account and comment under this post. Im happy to help!



Kubernetes + Hetzner Cloud + Loadbalancer + Nginx Ingress + Cert-Manager