Kubernetes + Hetzner Cloud + Loadbalancer + Nginx Ingress + Cert-Manager

Get 20€ credit with Hetzner using my link :)
https://hetzner.cloud/?ref=Vi4UjGTcwywL

Today i tried to setup a Kubernetes cluster on the Hetzner cloud using their loadbalancer to route my external traffic to the nginx ingress controller as well as issuing Letsencrypt certificates with cert-manager.

It was quite a ride but eventually i got a working version.

These are my steps working with K8S v.1.24.x (could be working on 1.25 but i have chosen 1.24 in order to support longhorn). Use my versions if you want a guaranteed success.

1. Create a new project (or use existing)

2. Add your SSH key.

3. Create an internal network

4. Create 1 master server (e.g CX21. Do not use less than 2 CPUs) with Ubuntu 22.04 and attach it to your internal network.

5. Add as many worker nodes with Ubuntu 22.04 as you want with a bit more power (CX21 and above). You can automatically let them attach to the network.

Server Setup

Run this on all your servers

1. Update/Upgrade your Server

sudo apt update
sudo apt -y full-upgrade
[ -f /var/run/reboot-required ] && sudo reboot -f

2. Disable Swap:

swapoff -a
sed -i '/swap/d' /etc/fstab

3. Enable "br_netfilter" kernel module:

cat > /etc/modules-load.d/k8s.conf << EOF
br_netfilter

EOF

modprobe br_netfilter

4. Enable routing:

cat > /etc/sysctl.d/kubernetes.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

sysctl --system

5. Import keys + add repos: (there could be an updated repo in the future but currently (December '22) there is only Xenial available)

sudo apt install curl apt-transport-https -y

curl -fsSL  https://packages.cloud.google.com/apt/doc/apt-key.gpg|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/k8s.gpg

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

6. Install Kubernetes packages: (im using a fixed version 1.24 here but you can use the default packages if you need a newer one).

apt update
apt install -y containerd.io kubeadm=1.24.8-00 kubectl=1.24.8-00 kubelet=1.24.8-00
apt-mark hold kubelet kubeadm kubectl 
#this stops apt from upgrading to a newer version later on

7. Adjust containerd config

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sed -i -e "s/SystemdCgroup = false/SystemdCgroup = true/g" /etc/containerd/config.toml
systemctl restart containerd
systemctl restart kubelet

8. Configure "kublet" service to enable Hetzner Cloud Provider:

cat > /etc/systemd/system/kubelet.service.d/20-hcloud.conf << EOF
[Service]
Environment="KUBELET_EXTRA_ARGS=--cloud-provider=external"
EOF

systemctl daemon-reload
systemctl restart kubelet

9. Edit /etc/hosts and add your servers. E.g:

10.255.255.2 k8s-master
10.255.255.3 k8s-worker-1
10.255.255.4 k8s-worker-3
10.255.255.5 k8s-worker-2

Initialize Kubernetes Cluster

1. ONLY on the master node:

kubeadm init \
  --pod-network-cidr=10.244.0.0/16 \
  --apiserver-advertise-address=0.0.0.0 \
  --upload-certs \
  --control-plane-endpoint=k8s-master

2. Copy the kube config

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

3. Join your worker nodes with the master

kubeadm token create --print-join-command

This should output something like this:
Run this command on every worker/node

kubeadm join k8s-master:6443 --token xxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxx

All next steps are only on the master!

Network Controller

Install Flannel Network Controller:

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Note: When running "kubectl get nodes" now, the status of all nodes should be "Ready".

root@master:~# kubectl get nodes
NAME       STATUS   ROLES           AGE    VERSION
master     Ready    control-plane   103s   v1.24.8
worker-1   Ready    <none>          36s    v1.24.8
worker-2   Ready    <none>          35s    v1.24.8
worker-3   Ready    <none>          33s    v1.24.8

Hetzner Cloud Controller

1. Create Secrets: (create an API Token within hetzner cloud for this)

kubectl -n kube-system create secret generic hcloud \
  --from-literal=token=<$HETZNER_API_TOKEN> \
  --from-literal=network=<$CLUSTER_NETWORK_ID>

Note: To get  <$CLUSTER_NETWORK_ID> you need to check the URL on the Hetzner Cloud site and grab the ID there. E.g:

2. Install Hetzner Cloud Controller:

kubectl -n kube-system apply -f https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.13.2/ccm-networks.yaml

CSI Driver (optional for volumes)

1. Create Secret:

kubectl -n kube-system create secret generic hcloud-csi \
  --from-literal=token=<$HETZNER_API_TOKEN>

2. Install CSI Driver:

kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.1.0/deploy/kubernetes/hcloud-csi.yml

Ingress Controller

Install Nginx Ingress Controller as a DaemonSet:

wget -q https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml -O /tmp/nginx.yaml

sed -i -e "s/kind: Deployment/kind: DaemonSet/g" /tmp/nginx.yaml
sed -i -e '/^kind: ConfigMap.*/i  \ \ compute-full-forwarded-for: \"true\"\n \ use-forwarded-headers: \"true\"\n \ use-proxy-protocol: \"true\"' /tmp/nginx.yaml

kubectl apply -f /tmp/nginx.yaml

Load Balancer

Create and connect Ingress Controller to Hetzner Load Balancer:

kubectl -n ingress-nginx annotate services ingress-nginx-controller \
  load-balancer.hetzner.cloud/name="k8s-lb" \
  load-balancer.hetzner.cloud/location="nbg1" \
  load-balancer.hetzner.cloud/use-private-ip="true" \
  load-balancer.hetzner.cloud/uses-proxyprotocol="true" \
  load-balancer.hetzner.cloud/hostname="<$CLUSTER_LB_HOSTNAME>"

Note:  <$CLUSTER_LB_HOSTNAME> needs to be a valid DNS record that points to the Load Balancers’ public IP address.

The controller should have created a Loadbalancer and attached all nodes to it (might take a minute).

Cert-Manager

1. Install cert-manager using helm (took a few mins to run for me):

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

helm repo add jetstack https://charts.jetstack.io

helm repo update

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.10.1 \
  --set installCRDs=true

2. Create issuer:

Do this for every namespace you want to issue certificates from (replace the namespace in the end. e.g default). Change the email to your own.

kubectl create --edit -f https://raw.githubusercontent.com/cert-manager/website/master/content/docs/tutorials/acme/example/production-issuer.yaml -n default

3. Create an example service

kubectl apply -f https://raw.githubusercontent.com/cert-manager/website/master/content/docs/tutorials/acme/example/deployment.yaml
# expected output: deployment.extensions "kuard" created

kubectl apply -f https://raw.githubusercontent.com/cert-manager/website/master/content/docs/tutorials/acme/example/service.yaml
# expected output: service "kuard" created

4. Add ingress to example service

Replace all "example.example.com" with your own domain. MUST BE pointing to your Loadbalancers IP!

kubectl create --edit -f https://raw.githubusercontent.com/cert-manager/website/master/content/docs/tutorials/acme/example/ingress-tls-final.yaml

If anything is unclear or you want to add something please register an account and comment under this post. Im happy to help!

Credits:

- https://docs.j7k6.org/hetzner-cloud-kubernetes-cluster/
- https://computingforgeeks.com/install-kubernetes-cluster-ubuntu-jammy/