The Kubelet process receives the --rotate-certificates parameter, which determines whether Kubelet will automatically request new certificates when the currently used certificate is about to expire.
The kube-controller-manager process accepts the --cluster-signing-duration parameter (--experimental-cluster-signing-duration before version 1.19) to control the validity period of the issued certificate.
- experimental-cluster-signing-duration=87600h0m0s # kubeletThe client certificate issuance validity period is set to10surname Nian
- feature-gates=RotateKubeletServerCertificate=true # (computing) enable (a feature)serverAward of certificates
Note 1: The current environment is k8s 1.21.5 and Kubelet client certificate rotation is enabled by default.
3. Understanding Certificate Rotation Configuration
When a Kubelet starts, if configured as bootstrap (using the --bootstrap-kubeconfig parameter), the Kubelet connects to the Kubernetes API using its initial certificate and sends a request for certificate signing. The status of the certificate signing request can be viewed in the following ways:
kubectl get csr
Initially, a certificate signing request from a Kubelet on a node is in the Pending state. If the certificate signing request meets certain conditions, the controller manager automatically approves it and the request is placed in the Approved state. Next, the controller manager signs the certificate, which has a validity period specified by the --cluster-signing-duration parameter, and the signed certificate is appended to the certificate signing request.
Kubelet fetches the signed certificate from the Kubernetes API and writes it to disk at the storage location specified by the --cert-dir parameter. Kubelet then connects to the Kubernetes API using the new certificate.
When a signed certificate is about to expire, Kubelet automatically initiates a new certificate signing request using the Kubernetes API. This request occurs at any point in time between 30% and 10% of the certificate's validity time remaining. Again, the controller manager automatically approves the certificate request and attaches the signed certificate to the certificate signing request. The Kubelet retrieves the signed certificate from the Kubernetes API and writes it to disk. It then updates the connection to the Kubernetes API and reconnects to the Kubernetes API using the new certificate.
Note 1:Kubelet Bootstrap Patterns
In a secure configuration of a Kubernetes cluster, communication between components requires authentication and authorization to ensure the security of the cluster.Kubelet, as a key component on a node, needs to communicate securely with the Kubernetes API Server (API server). This communication is typically encrypted and authenticated through TLS certificates. However, during cluster initialization or when nodes join the cluster, Kubelet may not have been issued a valid TLS certificate. In this case, the Kubelet can use Bootstrap to obtain a certificate. Bootstrap allows a Kubelet to start up with a temporary, low-privilege certificate (or bootstrap token; or a kubeconfig file that contains configuration information for connecting to an API server, including certificates and keys) and request that the API server issue it a long-term, higher-privilege certificate.
When Kubelet starts, if it is configured to bootstrap (which is typically accomplished with the --bootstrap-kubeconfig parameter, which specifies a kubeconfig file containing a temporary certificate and the address of the API server), it uses the initial certificate in this file to connect to the Kubernetes API server. Once the connection is successful, Kubelet sends a Certificate Signing Request (CSR) to the API server. This CSR contains the Kubelet's identity and information about the certificate it wishes to obtain (e.g., expiration date, purpose, etc.).
After the API server receives this CSR, it creates a CSR resource object, and the controller processes it according to the cluster's certificate issuance policies (e.g., whether or not to allow automatic approval of CSRs from specific nodes). If the CSR is approved, the controller uses the cluster's Certificate Authority (CA) to sign the request, generates a new certificate, and returns this certificate to the Kubelet, which receives it and replaces the initial temporary certificate to enable higher privileges and security for subsequent communication with the API server. server with higher privileges and security.
Note 2:For detailed steps on issuing client certificates with Kubernetes using the CertificateSigningRequest method, see theKubernetes Client Authentication (III) - Kubernetes Issues Client Certificates Using the CertificateSigningRequest Method" This blog post.
4. Testing
kubectl get csr
Initially, a certificate signing request from a Kubelet on a node is in the Pending state. If the certificate signing request meets certain conditions, the controller manager automatically approves it and the request is placed in the Approved state. Next, the controller manager signs the certificate, which has a validity period specified by the --cluster-signing-duration parameter, and the signed certificate is appended to the certificate signing request.
Kubelet fetches the signed certificate from the Kubernetes API and writes it to disk at the storage location specified by the --cert-dir parameter. Kubelet then connects to the Kubernetes API using the new certificate.
When a signed certificate is about to expire, Kubelet automatically initiates a new certificate signing request using the Kubernetes API. This request occurs at any point in time between 30% and 10% of the certificate's validity time remaining. Again, the controller manager automatically approves the certificate request and attaches the signed certificate to the certificate signing request. The Kubelet retrieves the signed certificate from the Kubernetes API and writes it to disk. It then updates the connection to the Kubernetes API and reconnects to the Kubernetes API using the new certificate.
Note 1:Kubelet Bootstrap Patterns
In a secure configuration of a Kubernetes cluster, communication between components requires authentication and authorization to ensure the security of the cluster.Kubelet, as a key component on a node, needs to communicate securely with the Kubernetes API Server (API server). This communication is typically encrypted and authenticated through TLS certificates. However, during cluster initialization or when nodes join the cluster, Kubelet may not have been issued a valid TLS certificate. In this case, the Kubelet can use Bootstrap to obtain a certificate. Bootstrap allows a Kubelet to start up with a temporary, low-privilege certificate (or bootstrap token; or a kubeconfig file that contains configuration information for connecting to an API server, including certificates and keys) and request that the API server issue it a long-term, higher-privilege certificate.
When Kubelet starts, if it is configured to bootstrap (which is typically accomplished with the --bootstrap-kubeconfig parameter, which specifies a kubeconfig file containing a temporary certificate and the address of the API server), it uses the initial certificate in this file to connect to the Kubernetes API server. Once the connection is successful, Kubelet sends a Certificate Signing Request (CSR) to the API server. This CSR contains the Kubelet's identity and information about the certificate it wishes to obtain (e.g., expiration date, purpose, etc.).
After the API server receives this CSR, it creates a CSR resource object, and the controller processes it according to the cluster's certificate issuance policies (e.g., whether or not to allow automatic approval of CSRs from specific nodes). If the CSR is approved, the controller uses the cluster's Certificate Authority (CA) to sign the request, generates a new certificate, and returns this certificate to the Kubelet, which receives it and replaces the initial temporary certificate to enable higher privileges and security for subsequent communication with the API server. server with higher privileges and security.
Note 2:For detailed steps on issuing client certificates with Kubernetes using the CertificateSigningRequest method, see theKubernetes Client Authentication (III) - Kubernetes Issues Client Certificates Using the CertificateSigningRequest Method" This blog post.
(1) Find a testnodenode to view the validity of existing client certificates
[root@member-cluster1-worker1 ~]# cd /var/lib/kubelet/pki/
[root@member-cluster1-worker1 pki]# ls
[root@member-cluster1-worker1 pki]# openssl x509 -in -noout -dates
notBefore=Jul 1 03:55:22 2024 GMT
notAfter=Jun 29 03:55:22 2034 GMT
(2) Modify server time to simulatekubeletCertificates are about to expire
[root@member-cluster1-worker1 pki]# date 2024surname Nian 09moon 05date Thursdays 18:35:40 CST [root@member-cluster1-worker1 pki]# date -s "2034-6-22" 2034surname Nian 06moon 22date Thursdays 00:00:00 CST
（3）Reboot Kubeletservice
[root@member-cluster1-worker1 pki]# systemctl restart kubelet
（4）Check the validity of the certificate again
Note 1: KubeletStarting time of the certificateis composed of Kube-Controller-ManagerDetermined by the time of the server where it is located.
Note 2: Kubelet's certificate rotation feature is automatic, so restart the Kubelet service here to see the effect of certificate rotation right away.