Kubernetes renew cluster certificates

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.

K8s is a nice to have for developers, security researchers but not limited to because of it’s simplicity to deploy a test container before adding to production.

The ugly part is when certificates are nearly to expire or worst when already expired. Today we will renew certificates of a K8s cluster (in production for more fun).

First of all we need to check expiration date:

# kubeadm alpha certs check-expiration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Sep 18, 2021 21:24 UTC   273d            no
apiserver                  Sep 18, 2021 21:24 UTC   273d            no
apiserver-etcd-client      Sep 18, 2021 21:24 UTC   273d            no
apiserver-kubelet-client   Sep 18, 2021 21:24 UTC   273d            no
controller-manager.conf    Sep 18, 2021 21:24 UTC   273d            no
etcd-healthcheck-client    Sep 18, 2021 21:24 UTC   273d            no
etcd-peer                  Sep 18, 2021 21:24 UTC   273d            no
etcd-server                Sep 18, 2021 21:24 UTC   273d            no
front-proxy-client         Sep 18, 2021 21:24 UTC   273d            no
scheduler.conf             Sep 18, 2021 21:24 UTC   273d            no

Now let’s back up the existing Kubernetes certificates, the existing configurtion files and home configuration:

mkdir -p $HOME/fcik8s-old-certs/pki
/bin/cp -p /etc/kubernetes/pki/*.* $HOME/fcik8s-old-certs/pki
/bin/cp -p /etc/kubernetes/*.conf $HOME/fcik8s-old-certs
mkdir -p $HOME/fcik8s-old-certs/.kube
/bin/cp -p ~/.kube/config $HOME/fcik8s-old-certs/.kube/.

Now it’s time to renew all the Kubernetes certificates:

kubeadm alpha certs renew all

Let’s confirm the certificates have been renewed and will expire in 364 days:

kubeadm alpha certs check-expiration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Dec 20, 2022 02:35 UTC   364d            no      
apiserver                  Dec 20, 2022 02:35 UTC   364d            no      
apiserver-etcd-client      Dec 20, 2022 02:35 UTC   364d            no      
apiserver-kubelet-client   Dec 20, 2022 02:35 UTC   364d            no      
controller-manager.conf    Dec 20, 2022 02:35 UTC   364d            no      
etcd-healthcheck-client    Dec 20, 2022 02:35 UTC   364d            no      
etcd-peer                  Dec 20, 2022 02:35 UTC   364d            no      
etcd-server                Dec 20, 2022 02:35 UTC   364d            no      
front-proxy-client         Dec 20, 2022 02:35 UTC   364d            no      
scheduler.conf             Dec 20, 2022 02:35 UTC   364d            no

Right now we are almost ready. Last step is to copy certificates to all masters:

scp -r /etc/kubernetes/ssl root@k8s_masters:/etc/kubernetes/

On all other master nodes we will delete old configuration and regenerate new configuration:

cd /etc/kubernetes && rm -rf admin.conf scheduler.conf controller-manager.conf kubelet.conf bootstrap-kubelet.conf
kubeadm init phase kubeconfig all --config "/etc/kubernetes/kubeadm-config.yaml"

Final step is to restart all K8s Related components:

docker ps |grep apiserver|grep -v pause|awk '{print $1}'|xargs docker kill -s HUP
docker ps |grep kube-scheduler|grep -v pause|awk '{print $1}'|xargs docker kill -s HUP
docker ps |grep kube-controller|grep -v pause|awk '{print $1}'|xargs docker kill -s HUP
systemctl restart kubelet

Need help managing or recovering Kubernetes clusters with expiring certificates?
→ Explore our Virtualization & Container Services