Kubernetes kubeadm 证书到期，更新证书

2021年8月27日 1368点热度 0人点赞 0条评论

版本

服务	版本
CentOS	7.8
Kubernetes	1.18.x

证书问题

　　可能很多人在一开始学习 k8s 的时候，没有注意过证书的问题，在使用 kubeadm 安装 k8s 单机/集群的过程中就是一路往下，如果是学习或者测试使用，使用完毕之后就把虚拟机或者临时云服务器删除了，那也不会发现证书问题。如果这个 k8s 环境要使用 1 年以上，就会碰到这个问题，因为默认证书有效期为 1 年，CA 根证书是 10 年：

# 查看证书截止时间，该命令显示 /etc/kubernetes/pki 文件夹中的客户端证书以及 kubeadm 使用的 KUBECONFIG 文件中嵌入的客户端证书的到期时间/剩余时间。新版本命令：kubeadm certs check-expiration
[root@kubernetes ~]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Aug 26, 2022 10:47 UTC   364d                                    no
apiserver                  Aug 26, 2022 10:47 UTC   364d            ca                      no
apiserver-etcd-client      Aug 26, 2022 10:47 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Aug 26, 2022 10:47 UTC   364d            ca                      no
controller-manager.conf    Aug 26, 2022 10:47 UTC   364d                                    no
etcd-healthcheck-client    Aug 26, 2022 10:47 UTC   364d            etcd-ca                 no
etcd-peer                  Aug 26, 2022 10:47 UTC   364d            etcd-ca                 no
etcd-server                Aug 26, 2022 10:47 UTC   364d            etcd-ca                 no
front-proxy-client         Aug 26, 2022 10:47 UTC   364d            front-proxy-ca          no
scheduler.conf             Aug 26, 2022 10:47 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Aug 23, 2030 07:00 UTC   8y              no
etcd-ca                 Aug 23, 2030 07:00 UTC   8y              no
front-proxy-ca          Aug 23, 2030 07:00 UTC   8y              no

[root@kubernetes pki]# for i in $(ls *.crt); do echo "===== $i ====="; openssl x509 -in $i -text -noout | grep -A 3 'Validity' ; done
===== apiserver.crt =====
        Validity
            Not Before: Aug 25 07:00:38 2020 GMT
            Not After : Aug 25 07:00:38 2021 GMT
        Subject: CN=kube-apiserver
===== apiserver-etcd-client.crt =====
        Validity
            Not Before: Aug 25 07:00:39 2020 GMT
            Not After : Aug 25 07:00:40 2021 GMT
        Subject: O=system:masters, CN=kube-apiserver-etcd-client
===== apiserver-kubelet-client.crt =====
        Validity
            Not Before: Aug 25 07:00:38 2020 GMT
            Not After : Aug 25 07:00:39 2021 GMT
        Subject: O=system:masters, CN=kube-apiserver-kubelet-client
===== ca.crt =====
        Validity
            Not Before: Aug 25 07:00:38 2020 GMT
            Not After : Aug 23 07:00:38 2030 GMT
        Subject: CN=kubernetes
===== front-proxy-ca.crt =====
        Validity
            Not Before: Aug 25 07:00:39 2020 GMT
            Not After : Aug 23 07:00:39 2030 GMT
        Subject: CN=front-proxy-ca
===== front-proxy-client.crt =====
        Validity
            Not Before: Aug 25 07:00:39 2020 GMT
            Not After : Aug 25 07:00:39 2021 GMT
        Subject: CN=front-proxy-client

　　运行一年以后就会变成这样：

　　如果各个服务一直还都正常运行，那问题不大，也就是 kubectl 客户端连接连接不上了，比如：

　　而如果一些关键服务，比如 etcd，进行了重启，因为这时证书已经过期了，那证书就不好使了。就会导致 etcd 无法启动，进而导致 kube-apiserver，也启动不起来，最终就会崩盘，进而可能会导致业务服务也无法正常运行。

　　etcd 错误信息：

　　kube-apiserver 错误信息：

transport: authentication handshake failed: remote error: tls: bad certificate

error "tls: failed to verify client's certificate: x509: certificate has expired or is not yet valid", ServerName ""

error "remote error: tls: bad certificate", ServerName ""

生成新的证书

　　kubeadm alpha certs renew：可以使用 all 子命令来续订所有 Kubernetes 证书，也可以选择性地续订部分证书。详见参考。更多的相关细节，可参见手动续订证书。

# 查看证书命令下有哪些功能
[root@kubernetes ~]# kubeadm alpha certs -h
Commands related to handling kubernetes certificates

Usage:
  kubeadm alpha certs [command]

Aliases:
  certs, certificates

Available Commands:
  certificate-key  Generate certificate keys
  check-expiration Check certificates expiration for a Kubernetes cluster
  renew            Renew certificates for a Kubernetes cluster

Flags:
  -h, --help   help for certs

Global Flags:
      --add-dir-header           If true, adds the file directory to the header
      --log-file string          If non-empty, use this log file
      --log-file-max-size uint   Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --rootfs string            [EXPERIMENTAL] The path to the 'real' host root filesystem.
      --skip-headers             If true, avoid header prefixes in the log messages
      --skip-log-headers         If true, avoid headers when opening log files
  -v, --v Level                  number for the log level verbosity

Use "kubeadm alpha certs [command] --help" for more information about a command.

renew：Renew certificates for a Kubernetes cluster（更新 Kubernetes 集群的证书，注：不涉及 CA 根证书）。

　　要使用的就是 renew 命令，来进行重新生成集群证书。

# 查看重新生成证书下有哪些命令
[root@kubernetes etc]# kubeadm alpha certs -h
Commands related to handling kubernetes certificates

Usage:
  kubeadm alpha certs [command]

Aliases:
  certs, certificates

Available Commands:
  certificate-key  Generate certificate keys
  check-expiration Check certificates expiration for a Kubernetes cluster
  renew            Renew certificates for a Kubernetes cluster

Flags:
  -h, --help   help for certs

Global Flags:
      --add-dir-header           If true, adds the file directory to the header
      --log-file string          If non-empty, use this log file
      --log-file-max-size uint   Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --rootfs string            [EXPERIMENTAL] The path to the 'real' host root filesystem.
      --skip-headers             If true, avoid header prefixes in the log messages
      --skip-log-headers         If true, avoid headers when opening log files
  -v, --v Level                  number for the log level verbosity

Use "kubeadm alpha certs [command] --help" for more information about a command.
[root@kubernetes etc]# kubeadm alpha certs renew -h
This command is not meant to be run on its own. See list of available subcommands.

Usage:
  kubeadm alpha certs renew [flags]
  kubeadm alpha certs renew [command]

Available Commands:
  admin.conf               Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
  all                      Renew all available certificates
  apiserver                Renew the certificate for serving the Kubernetes API
  apiserver-etcd-client    Renew the certificate the apiserver uses to access etcd
  apiserver-kubelet-client Renew the certificate for the API server to connect to kubelet
  controller-manager.conf  Renew the certificate embedded in the kubeconfig file for the controller manager to use
  etcd-healthcheck-client  Renew the certificate for liveness probes to healthcheck etcd
  etcd-peer                Renew the certificate for etcd nodes to communicate with each other
  etcd-server              Renew the certificate for serving etcd
  front-proxy-client       Renew the certificate for the front proxy client
  scheduler.conf           Renew the certificate embedded in the kubeconfig file for the scheduler manager to use

Flags:
  -h, --help   help for renew

Global Flags:
      --add-dir-header           If true, adds the file directory to the header
      --log-file string          If non-empty, use this log file
      --log-file-max-size uint   Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --rootfs string            [EXPERIMENTAL] The path to the 'real' host root filesystem.
      --skip-headers             If true, avoid header prefixes in the log messages
      --skip-log-headers         If true, avoid headers when opening log files
  -v, --v Level                  number for the log level verbosity

Use "kubeadm alpha certs renew [command] --help" for more information about a command.

all：更新所有可用证书，一般可以直接使用这个更新所有。
apiserver：单独更新服务 Kubernetes API 的证书，相当于针对单个组件进行生成。
apiserver-etcd-client：更新 apiserver 用来访问 etcd 的证书。
其它都类似

　　生成新的证书之前最好备份一下数据：

[root@kubernetes etc]# cp -rp /etc/kubernetes /etc/kubernetes.bak
[root@kubernetes etc]# cp -rp /var/lib/etcd /var/lib/etcd.bak

　　生成新的证书：

[root@kubernetes etc]# kubeadm alpha certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

　　此时所有证书都已重新生成，在 /etc/kubernetes/pki 下。/etc/kubernetes/admin.conf、/etc/kubernetes/controller-manager.conf、/etc/kubernetes/scheduler.conf 也已经更新了。

警告： 如果你运行了一个 HA 集群，这个命令需要在所有控制面板节点上执行。

说明： certs renew 使用现有的证书作为属性 (Common Name、Organization、SAN 等) 的权威来源，而不是 kubeadm-config ConfigMap 。强烈建议使它们保持同步。

　　再次查看证书时间就延续了一年：

[root@kubernetes pki]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Aug 27, 2022 03:11 UTC   364d                                    no
apiserver                  Aug 27, 2022 01:44 UTC   364d            ca                      no
apiserver-etcd-client      Aug 27, 2022 01:44 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Aug 27, 2022 01:44 UTC   364d            ca                      no
controller-manager.conf    Aug 27, 2022 03:11 UTC   364d                                    no
etcd-healthcheck-client    Aug 27, 2022 01:44 UTC   364d            etcd-ca                 no
etcd-peer                  Aug 27, 2022 01:44 UTC   364d            etcd-ca                 no
etcd-server                Aug 27, 2022 01:44 UTC   364d            etcd-ca                 no
front-proxy-client         Aug 27, 2022 01:44 UTC   364d            front-proxy-ca          no
scheduler.conf             Aug 27, 2022 03:11 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Aug 23, 2030 07:00 UTC   8y              no
etcd-ca                 Aug 23, 2030 07:00 UTC   8y              no
front-proxy-ca          Aug 23, 2030 07:00 UTC   8y              no

　　如果容器服务使用的是 Docker，为了让证书生效，可以使用以下命令对涉及到证书使用的几个服务进行重启：

docker ps | grep -E 'k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd' | awk -F ' ' '{print $1}' | xargs docker restart

　　没有更新的文件就只有 /etc/kubernetes/kubelet.conf 了，因为 kubeadm 将 kubelet 配置为自动更新证书。轮换的证书位于目录 /var/lib/kubelet/pki。要修复过期的 kubelet 客户端证书，请参阅 kubelet 客户端证书轮换失败。当签署的证书即将到期时，kubelet 会使用 Kubernetes API，自动发起新的证书签名请求。该请求会发生在证书的有效时间剩下 30% 到 10% 之间的任意时间点。同样地，控制器管理器会自动批准证书请求，并将签署的证书附加到证书签名请求中。 Kubelet 会从 Kubernetes API 取回签署的证书，并将其写入磁盘。然后它会更新与 Kubernetes API 的连接，使用新的证书重新连接到 Kubernetes API。

警告：

在通过 kubeadm init 创建的节点上，在 kubeadm 1.17 版本之前有一个缺陷，该缺陷使得你必须手动修改 /etc/kubernetes/kubelet.conf 文件的内容。 kubeadm init 操作结束之后，你必须更新 kubelet.conf 文件将 client-certificate-data 和 client-key-data 改为如下所示的内容以便使用轮换后的 kubelet 客户端证书：
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

　　当然，如果是 1.17 以前的版本，也可以手动调用 kubeadm init phase kubeconfig kubelet 进行证书轮换。需要先把 /etc/kubernetes/kubelet.conf 删掉，使用 kubeadm init phase kubeconfig kubelet 进行重新生成。之后在 systemctl restart kubelet 进行重启。

kubeadm init phase kubeconfig：可以通过调用 all 子命令来创建所有必需的 kubeconfig 文件，或者分别调用它们。详见参考。

延长证书到 99 年

　　无论是最开始初始化或者后续进行证书轮换都是使用的 kubeadm 工具。只需把源码拉下来，修改证书年限，再打包成 kubeadm 即可。源码地址，Github 访问太慢了，所以在 Gitee 上复制了一份。

修改源码

获取源码

　　本身环境是 1.18.x，所以下载的源码也是 1.18 的。

[root@kubernetes ~]# git clone https://gitee.com/lynchj-kubernetes/kubernetes.git
[root@kubernetes ~]# cd kubernetes
[root@kubernetes kubernetes]# git checkout -b release-1.18 origin/release-1.18

CA 证书有效时间修改（默认10年）

[root@kubernetes kubernetes]# vim ./staging/src/k8s.io/client-go/util/cert/cert.go

......
// NewSelfSignedCACert creates a CA certificate
func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
        now := time.Now()
        tmpl := x509.Certificate{
                SerialNumber: new(big.Int).SetInt64(0),
                Subject: pkix.Name{
                        CommonName:   cfg.CommonName,
                        Organization: cfg.Organization,
                },
                NotBefore:             now.UTC(),
                    // CA 证书有效期
                // NotAfter:              now.Add(duration365d * 10).UTC(),
                NotAfter:              now.Add(duration365d * 100).UTC(),
                KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
                BasicConstraintsValid: true,
                IsCA:                  true,
        }

        certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key)
        if err != nil {
                return nil, err
        }
        return x509.ParseCertificate(certDERBytes)
}
......

证书有效期为 99 年（默认为 1 年）

[root@kubernetes kubernetes]# vim ./cmd/kubeadm/app/constants/constants.go

......
                // CertificateValidity defines the validity for all the signed certificates generated by kubeadm
        // CertificateValidity = time.Hour * 24 * 365
        CertificateValidity = time.Hour * 24 * 365 * 99
......

　　源码已经修改完毕，剩下就是打包成 kubeadm 可执行文件了，有两种方式：

使用官方提供的 Docker 镜像进行打包（推荐）。
自己把环境准备好，进行打包。

Docker 形式打包（推荐）

官方文档

　　官方提供的镜像打包工具：gcr.io/google-containers/kube-cross，源码中也有记录使用的镜像版本：

[root@kubernetes kubernetes]# cat ./build/build-image/cross/VERSION
v1.13.15-1

　　1.18 版本里面标识镜像打包工具 cross 的版本为 v1.13.15-1，但是在官方仓库中没有找到，最新也就是 v1.13.6-1：

　　这里就先使用 v1.13.6-1，由于不科学上网可能无法拉取镜像，这里通过一通周转，最终放到阿里云仓库，地址：registry.cn-hangzhou.aliyuncs.com/lynchj-google/kube-cross:v1.13.6-1。

[root@kubernetes kubernetes]# docker pull registry.cn-hangzhou.aliyuncs.com/lynchj-google/kube-cross:v1.13.6-1
[root@kubernetes kubernetes]# docker run -it --rm --name kube-cross -v /root/kubernetes:/go/src/k8s registry.cn-hangzhou.aliyuncs.com/lynchj-google/kube-cross:v1.13.6-1 bash

root@620e1e176e9e:/go# cd /go/src/k8s/
root@620e1e176e9e:/go/src/k8s# ls -l
total 184
lrwxrwxrwx.  1 root root    21 Aug 27 07:07 BUILD.bazel -> build/root/BUILD.root
drwxr-xr-x.  2 root root    62 Jun 16 13:49 CHANGELOG
lrwxrwxrwx.  1 root root    19 Aug 27 07:07 CHANGELOG.md -> CHANGELOG/README.md
-rw-r--r--.  1 root root   493 Jun 16 13:49 CONTRIBUTING.md
drwxr-xr-x.  2 root root    36 Jun 16 13:49 Godeps
-rw-r--r--.  1 root root 11358 Jun 16 13:49 LICENSE
lrwxrwxrwx.  1 root root    19 Aug 27 07:07 Makefile -> build/root/Makefile
lrwxrwxrwx.  1 root root    35 Aug 27 07:07 Makefile.generated_files -> build/root/Makefile.generated_files
-rw-r--r--.  1 root root   783 Jun 16 13:49 OWNERS
-rw-r--r--.  1 root root 10058 Jun 16 13:49 OWNERS_ALIASES
-rw-r--r--.  1 root root  3468 Jun 16 13:49 README.md
-rw-r--r--.  1 root root   563 Jun 16 13:49 SECURITY_CONTACTS
-rw-r--r--.  1 root root  1110 Jun 16 13:49 SUPPORT.md
lrwxrwxrwx.  1 root root    20 Aug 27 07:07 WORKSPACE -> build/root/WORKSPACE
drwxr-xr-x.  4 root root    57 Jun 16 13:49 api
drwxr-xr-x.  8 root root  4096 Jun 16 13:49 build
drwxr-xr-x. 11 root root  4096 Jun 16 13:49 cluster
drwxr-xr-x. 23 root root  4096 Jun 16 13:49 cmd
-rw-r--r--.  1 root root   148 Jun 16 13:49 code-of-conduct.md
drwxr-xr-x.  2 root root    51 Jun 16 13:49 docs
-rw-r--r--.  1 root root 39834 Jun 16 13:49 go.mod
-rw-r--r--.  1 root root 62140 Jun 16 13:49 go.sum
drwxr-xr-x. 11 root root  4096 Jun 16 13:49 hack
drwxr-xr-x.  2 root root  4096 Jun 16 13:49 logo
drwxr-xr-x. 33 root root  4096 Jun 16 13:49 pkg
drwxr-xr-x.  3 root root    44 Jun 16 13:49 plugin
drwxr-xr-x.  4 root root   106 Jun 16 13:49 staging
drwxr-xr-x. 17 root root   263 Jun 16 13:49 test
drwxr-xr-x.  7 root root   134 Jun 16 13:49 third_party
drwxr-xr-x.  4 root root    95 Jun 16 13:49 translations
drwxr-xr-x. 18 root root  4096 Jun 16 13:49 vendor

root@620e1e176e9e:/go/src/k8s# make all WHAT=cmd/kubeadm GOFLAGS=-v

# 编译 kubelet
# make all WHAT=cmd/kubelet GOFLAGS=-v

# 编译 kubectl
# make all WHAT=cmd/kubectl GOFLAGS=-v

注意：如果是在 Windows 环境下使用 Docker 运行， -v 参数可以挂在目录进容器，但是会导致目录中的 Makefile 软连接失效，进而导致打包报错：
Makefile:1: *** missing separator.  Stop.
可以启动容器的时候先别挂载，进入容器之后在容器里面进行 clone。

　　打包完毕之后会多出一个 _output 目录，把新的 kubeadm 替换掉：

root@aefcb94d4fb3:/go/src/k8s# ls _output/local/bin/linux/amd64/
conversion-gen  deepcopy-gen  defaulter-gen  go2make  go-bindata  kubeadm  openapi-gen

# 退出容器
root@aefcb94d4fb3:/go/src/k8s# exit
exit

# 替换原有的 kubeadm
[root@kubernetes kubernetes]# mv /usr/bin/kubeadm /usr/bin/kubeadm_backup
[root@kubernetes kubernetes]# cp _output/local/bin/linux/amd64/kubeadm /usr/bin/kubeadm

本地打包

官方文档

[root@kubernetes kubernetes]# yum -y install gcc make rsync jq

　　安装并配置好对应的 Golang：

[root@kubernetes ~]# wget https://dl.google.com/go/go1.13.15.linux-amd64.tar.gz
[root@kubernetes ~]# tar -xzvf go1.13.15.linux-amd64.tar.gz -C /usr/local

[root@kubernetes ~]# vim /etc/profile
......
# 文件最后增加
# Golang
export GOROOT=/usr/local/go

# Path
export PATH=$PATH:$GOROOT/bin
......

[root@kubernetes ~]# source /etc/profile
[root@kubernetes ~]# go version
go version go1.13.15 linux/amd64

# 进行编译打包
[root@kubernetes ~]# cd kubernetes
[root@kubernetes kubernetes]# make all WHAT=cmd/kubeadm GOFLAGS=-v

# 替换原有的 kubeadm 与 Docker 方式打包一样

续订证书到 99 年

# 事先查看
[root@kubernetes kubernetes]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Aug 27, 2022 03:11 UTC   364d                                    no
apiserver                  Aug 27, 2022 01:44 UTC   364d            ca                      no
apiserver-etcd-client      Aug 27, 2022 01:44 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Aug 27, 2022 01:44 UTC   364d            ca                      no
controller-manager.conf    Aug 27, 2022 03:11 UTC   364d                                    no
etcd-healthcheck-client    Aug 27, 2022 01:44 UTC   364d            etcd-ca                 no
etcd-peer                  Aug 27, 2022 01:44 UTC   364d            etcd-ca                 no
etcd-server                Aug 27, 2022 01:44 UTC   364d            etcd-ca                 no
front-proxy-client         Aug 27, 2022 01:44 UTC   364d            front-proxy-ca          no
scheduler.conf             Aug 27, 2022 03:11 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Aug 23, 2030 07:00 UTC   8y              no
etcd-ca                 Aug 23, 2030 07:00 UTC   8y              no
front-proxy-ca          Aug 23, 2030 07:00 UTC   8y              no

# 更新 99 年
[root@kubernetes kubernetes]# kubeadm alpha certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

# 再看
[root@kubernetes kubernetes]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Aug 03, 2120 08:20 UTC   98y                                     no
apiserver                  Aug 03, 2120 08:20 UTC   98y             ca                      no
apiserver-etcd-client      Aug 03, 2120 08:20 UTC   98y             etcd-ca                 no
apiserver-kubelet-client   Aug 03, 2120 08:20 UTC   98y             ca                      no
controller-manager.conf    Aug 03, 2120 08:20 UTC   98y                                     no
etcd-healthcheck-client    Aug 03, 2120 08:20 UTC   98y             etcd-ca                 no
etcd-peer                  Aug 03, 2120 08:20 UTC   98y             etcd-ca                 no
etcd-server                Aug 03, 2120 08:20 UTC   98y             etcd-ca                 no
front-proxy-client         Aug 03, 2120 08:20 UTC   98y             front-proxy-ca          no
scheduler.conf             Aug 03, 2120 08:20 UTC   98y                                     no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Aug 23, 2030 07:00 UTC   8y              no
etcd-ca                 Aug 23, 2030 07:00 UTC   8y              no
front-proxy-ca          Aug 23, 2030 07:00 UTC   8y              no