1
2
3
4
5
作者：李晓辉

微信联系：lxh_chat

联系邮箱: 939958092@qq.com
角色IP主机名操作系统平台软件版本硬件配置
k8s-master
etcd
haproxy
keepalived		主要IP：192.168.8.128
keepalived vip: 192.168.8.200		k8s-master01.xiaohui.cnRocky 9.4kubernetes 1.31.1
haproxy 3.0.5
Keepalived 2.3.1		CPU: 2核心以上
内存：4G以上
硬盘：100G以上
k8s-master
etcd
haproxy
keepalived		主要IP：192.168.8.129
keepalived vip: 192.168.8.200		k8s-master02.xiaohui.cnRocky 9.4kubernetes 1.31.1
haproxy 3.0.5
Keepalived 2.3.1		CPU: 2核心以上
内存：4G以上
硬盘：100G以上
k8s-master
etcd
haproxy
keepalived		主要IP：192.168.8.130
keepalived vip: 192.168.8.200		k8s-master03.xiaohui.cnRocky 9.4kubernetes 1.31.1
haproxy 3.0.5
Keepalived 2.3.1		CPU: 2核心以上
内存：4G以上
硬盘：100G以上
k8s-master
etcd
haproxy
keepalived		主要IP：192.168.8.131
keepalived vip: 192.168.8.200		k8s-master04.xiaohui.cnRocky 9.4kubernetes 1.31.1
haproxy 3.0.5
Keepalived 2.3.1		CPU: 2核心以上
内存：4G以上
硬盘：100G以上
k8s-worker主要IP：192.168.8.132k8s-worker01.xiaohui.cnRocky 9.4kubernetes 1.31.1CPU: 2核心以上
内存：2G以上
硬盘：100G以上
仅供参考，以负载为准

这里是k8s-master04只是用于演示在token过期后，还怎么将master加入到集群，所以你可以认为本文档提供的是3节点的控制平面部署指南

文档拓扑描述

流量走向：Client—>Keepalived vip—>Haproxy—>—>K8s-master

architecture-ha-k8s-cluster

添加域名解析

我的k8s-master04只是用于演示token过期后，如何加入更多节点而已，你集群的master最好是奇数个机器

1
2
3
4
5
6
7
cat << EOF >> /etc/hosts
192.168.8.128 k8s-master01.xiaohui.cn k8s-master01
192.168.8.129 k8s-master02.xiaohui.cn k8s-master02
192.168.8.130 k8s-master03.xiaohui.cn k8s-master03
192.168.8.131 k8s-master04.xiaohui.cn k8s-master04
192.168.8.132 k8s-worker01.xiaohui.cn k8s-worker01
EOF

分别给每个主机设置主机名，我这里以第一个master举例

1
hostnamectl hostname k8s-master01.xiaohui.cn

部署 Haproxy服务

Haproxy 在这里承担了K8S Master节点之间的负载均衡角色，需要在所有的控制平面机器上都安装

安装haproxy最新版

这里可以找到最新版的haproxy

1
https://github.com/haproxy/haproxy/tags

我写文章的时候，最新版是3.0.5，我用我自己的仓库做了加速，你用的时候在上面的链接找最新版

1
2
wget https://www.haproxy.org/download/3.0/src/haproxy-3.0.5.tar.gz
tar xf haproxy-3.0.5.tar.gz

编译安装的时候还需要lua支持，最新版可以在下面的网址找到最新版

1
https://www.lua.org/
1
2
wget https://www.lua.org/ftp/lua-5.4.7.tar.gz
tar xf lua-5.4.7.tar.gz

编译安装需要有make和gcc支持

1
2
3
yum clean all
yum makecache
yum install make gcc -y

编译安装lua

1
2
3
4
cd lua-5.4.7/
make
make install
cd

为haproxy准备编译先决条件

执行这个apt install有可能会跳出界面让你选择，你可以用上下键选择，用空格勾选所有，然后tab键选中ok并回车

1
2
cd haproxy-3.0.5/
yum install pcre2-devel openssl-devel -y

编译安装haproxy

1
2
3
make -j $(nproc) TARGET=linux-glibc \
USE_OPENSSL=1 USE_QUIC=1 USE_QUIC_OPENSSL_COMPAT=1 \
USE_LUA=1 USE_PCRE2=1
1
2
make install
cd

准备haproxy配置文件

这里要注意，你在文本最后将k8s-master04这台机器加入到k8s集群确认没问题后，将其他所有master机器上的haproxy配置文件都更新一下，包含新的控制节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
mkdir /etc/haproxy /var/lib/haproxy

cat << EOF > /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    log /dev/log local1 notice
    daemon
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          20s
    timeout server          20s
    timeout http-keep-alive 10s
    timeout check           10s
frontend k8s-api
    bind *:64433 #非并置的情况下，这里写6443和k8s更一致
    mode tcp
    option tcplog
    default_backend apiserver
backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        server k8s-master01 k8s-master01.xiaohui.cn:6443 check
        server k8s-master02 k8s-master02.xiaohui.cn:6443 check
        server k8s-master03 k8s-master03.xiaohui.cn:6443 check
        server k8s-master04 k8s-master04.xiaohui.cn:6443 check
EOF
1
2
3
4
5
6
7
8
9
10
11
cat << EOF > /etc/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target
[Service]
ExecStartPre=/usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/local/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/lib/haproxy/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID
[Install]
WantedBy=multi-user.target
EOF

启动haproxy服务

1
2
systemctl daemon-reload
systemctl enable haproxy --now

部署 Keepalived服务

keepalived提供了一个虚拟IP，需要在所有的控制平面都安装

安装依赖包

1
2
3
yum install epel-release -y
crb enable
yum makecache
1
2
3
4
yum install -y make autoconf automake openssl-devel libnl3-devel \
iptables-devel ipset-devel file-devel net-snmp-devel \
glib2-devel pcre2-devel libnftnl-devel libmnl-devel \
systemd-devel kmod-devel NetworkManager-libnm-devel python-sphinx

这里应该会更新systemd软件，建议重启一下所有机器

1
reboot

编译安装keepalived

1
2
3
4
yum install git -y
git clone https://github.com/acassen/keepalived.git
cd keepalived
./autogen.sh
1
./configure
1
2
3
make
make install
cd

准备keepalived配置文件

第一台

这是keepalived的检测脚本，主要是判断haproxy是否工作正常，这里需要注意你的网卡名称是否为ens160

1
2
3
4
5
6
7
8
9
10
11
cat <<- 'EOF' > /etc/keepalived/check_haproxy.sh
#!/bin/bash

# 检查 HAProxy 是否在运行
if systemctl is-active --quiet haproxy; then
    exit 0  # HAProxy 正常
else
    exit 1  # HAProxy 不正常
fi

EOF

以下为第一台keepalived配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
cat > /etc/keepalived/keepalived.conf <<- 'EOF'
global_defs {
   router_id K8S_VIP
}

vrrp_script check_haproxy {
    script "/etc/keepalived/check_haproxy.sh" #每个机器要一致
    interval 2
    weight 20 #这里的跨度要超越下面的priority，因为脚本返回0时，weight会直接加20
}

vrrp_instance VI_1 {
    state MASTER # 其他机器写成BACKUP
    interface ens160 # 注意每个机器的网卡名称
    virtual_router_id 51 #每个机器要一致
    priority 100 # 其他机器依次递减1就行
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111 #每个机器要一致
    }
    virtual_ipaddress {
        192.168.8.200 #每个机器要一致，这里是VIP
    }

    track_script {
        check_haproxy
    }

    accept
}
EOF
1
2
3
chmod +x /etc/keepalived/check_haproxy.sh
systemctl daemon-reload
systemctl enable keepalived --now

开通vrrp协议的放获取

1
2
firewall-cmd --add-protocol=vrrp --permanent
firewall-cmd --reload

第二台

这是keepalived的检测脚本，主要是判断haproxy是否工作正常，这里需要注意你的网卡名称是否为ens160

1
2
3
4
5
6
7
8
9
10
11
cat <<- 'EOF' > /etc/keepalived/check_haproxy.sh
#!/bin/bash

# 检查 HAProxy 是否在运行
if systemctl is-active --quiet haproxy; then
    exit 0  # HAProxy 正常
else
    exit 1  # HAProxy 不正常
fi

EOF

以下为第二台keepalived配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
cat > /etc/keepalived/keepalived.conf <<- 'EOF'
global_defs {
   router_id K8S_VIP
}

vrrp_script check_haproxy {
    script "/etc/keepalived/check_haproxy.sh" #每个机器要一致
    interval 2
    weight 20 #这里的跨度要超越下面的priority，因为脚本返回0时，weight会直接加20
}

vrrp_instance VI_1 {
    state BACKUP # 其他机器写成BACKUP
    interface ens160 # 注意每个机器的网卡名称
    virtual_router_id 51 #每个机器要一致
    priority 99 # 其他机器依次递减1就行
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111 #每个机器要一致
    }
    virtual_ipaddress {
        192.168.8.200 #每个机器要一致，这里是VIP
    }

    track_script {
        check_haproxy
    }

    accept
}

EOF
1
2
3
chmod +x /etc/keepalived/check_haproxy.sh
systemctl daemon-reload
systemctl enable keepalived --now

开通vrrp协议的放获取

1
2
firewall-cmd --add-protocol=vrrp --permanent
firewall-cmd --reload

第三台

这是keepalived的检测脚本，主要是判断haproxy是否工作正常，这里需要注意你的网卡名称是否为ens160

1
2
3
4
5
6
7
8
9
10
11
cat <<- 'EOF' > /etc/keepalived/check_haproxy.sh
#!/bin/bash

# 检查 HAProxy 是否在运行
if systemctl is-active --quiet haproxy; then
    exit 0  # HAProxy 正常
else
    exit 1  # HAProxy 不正常
fi

EOF

以下为第三台keepalived配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
cat > /etc/keepalived/keepalived.conf <<- 'EOF'
global_defs {
   router_id K8S_VIP
}

vrrp_script check_haproxy {
    script "/etc/keepalived/check_haproxy.sh" #每个机器要一致
    interval 2
    weight 20 #这里的跨度要超越下面的priority，因为脚本返回0时，weight会直接加20
}

vrrp_instance VI_1 {
    state BACKUP # 其他机器写成BACKUP
    interface ens160 # 注意每个机器的网卡名称
    virtual_router_id 51 #每个机器要一致
    priority 98 # 其他机器依次递减1就行
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111 #每个机器要一致
    }
    virtual_ipaddress {
        192.168.8.200 #每个机器要一致，这里是VIP
    }

    track_script {
        check_haproxy
    }

    accept
}

EOF
1
2
3
chmod +x /etc/keepalived/check_haproxy.sh
systemctl daemon-reload
systemctl enable keepalived --now

开通vrrp协议的防火墙

1
2
firewall-cmd --add-protocol=vrrp --permanent
firewall-cmd --reload

部署高可用Kubernetes

先决条件

先决条件的部分，在所有k8s节点都要完成

禁用交换分区

1
2
sed -ri 's/.*swap.*/#&/' /etc/fstab
swapoff -a

部署Docker和CRI-Docker

部署docker

1
2
3
4
5
6
7
wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.nju.edu.cn/docker-ce/linux/rhel/docker-ce.repo
sed -i 's|download.docker.com|mirrors.nju.edu.cn/docker-ce|g' /etc/yum.repos.d/docker-ce.repo
yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

systemctl daemon-reload
systemctl restart docker.service
systemctl enable docker.service

部署CRI-Docker

1
2
3
4
5
6
7
8
9
10
cat > /etc/yum.repos.d/fedora36.repo <<-'EOF'
[fedora36]
name=fedora36 repo
baseurl=https://mirrors.nju.edu.cn/fedora-archive/fedora/linux/releases/36/Everything/x86_64/os/
enabled=1
gpgcheck=0
EOF

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.15/cri-dockerd-0.3.15-3.fc36.x86_64.rpm
yum localinstall cri-dockerd-0.3.15-3.fc36.x86_64.rpm -y

将镜像指引到国内

1
2
3
4
5
sed -i 's|ExecStart=.*|ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10|' /lib/systemd/system/cri-docker.service

systemctl daemon-reload
systemctl restart cri-docker.service
systemctl enable cri-docker.service

安装kubeadm工具

1
2
3
4
5
6
7
8
9
10
cat > /etc/yum.repos.d/k8s.repo <<EOF
[k8s]
name=k8s repo
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.31/rpm
enabled=1
gpgcheck=0
EOF

yum install -y kubelet kubeadm kubectl socat
systemctl enable kubelet --now

添加命令自动补齐功能

1
2
3
4
kubectl completion bash > /etc/bash_completion.d/kubectl
kubeadm completion bash > /etc/bash_completion.d/kubeadm
source /etc/bash_completion.d/kubectl
source /etc/bash_completion.d/kubeadm

集成CRI-Docker

1
2
crictl config runtime-endpoint unix:///run/cri-dockerd.sock
crictl images

允许 iptables 检查桥接流量

1
2
3
4
5
6
7
8
9
10
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

开通Master节点防火墙

这里的防火墙是master所需要的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Kubernetes API 服务器
firewall-cmd --zone=public --add-port=6443/tcp --permanent

# etcd 服务器客户端 API
firewall-cmd --zone=public --add-port=2379-2380/tcp --permanent

# kubelet API
firewall-cmd --zone=public --add-port=10250/tcp --permanent

# kube-scheduler
firewall-cmd --zone=public --add-port=10259/tcp --permanent

# kube-controller-manager
firewall-cmd --zone=public --add-port=10257/tcp --permanent

# haproxy api-server
firewall-cmd --zone=public --add-port=64433/tcp --permanent

# calico
firewall-cmd --add-port=179/tcp --permanent
firewall-cmd --add-port=4789/udp --permanent
firewall-cmd --add-port=5473/tcp --permanent
firewall-cmd --add-port=9099/tcp --permanent

# 重新载入防火墙以应用更改
firewall-cmd --reload

集群部署

这个部署只在第一台k8s-master01完成，其他的master稍后以join的方式加入到集群中

1
2
3
4
5
6
7
8
9
10
kubeadm init \
--apiserver-advertise-address=192.168.8.128 \
--apiserver-bind-port=6443 \
--control-plane-endpoint=192.168.8.200:64433 \
--image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \
--kubernetes-version=v1.31.1 \
--service-cidr=10.96.0.0/12 \
--service-dns-domain=lixiaohui.cn \
--cri-socket unix:///var/run/cri-dockerd.sock \
--upload-certs

部分输出内容:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
...

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.8.200:64433 --token b8iv94.qg3nhbioxz6e0lzr \
        --discovery-token-ca-cert-hash sha256:2fd9a0461559cbb7a018ccb47350531d4d7ac4419458f4766694d9d05ba4af9c \
        --control-plane --certificate-key 7d99ab3cec720c9ac0c4fe1b929942acc0bad3bfbf5ad3a8bab068422ac7ea5c
...
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.8.200:64433 --token b8iv94.qg3nhbioxz6e0lzr \
        --discovery-token-ca-cert-hash sha256:2fd9a0461559cbb7a018ccb47350531d4d7ac4419458f4766694d9d05ba4af9c

加入更多Master

所有的Master节点都必须得完成本地hosts添加、Haproxy、Keepalived部署，以及k8s部署的所有先决条件，请注意在haproxy的后端添加包含新节点的所有后端，在keepalived配置文件中的优先级继续减1

第二台和第三台都输入下面的命令即可，下面的命令也是从第一台部署好之后的输出中复制粘贴的，注意加了–cri-socket参数

1
2
3
4
kubeadm join 192.168.8.200:64433 --token b8iv94.qg3nhbioxz6e0lzr \
        --discovery-token-ca-cert-hash sha256:2fd9a0461559cbb7a018ccb47350531d4d7ac4419458f4766694d9d05ba4af9c \
        --control-plane --certificate-key 7d99ab3cec720c9ac0c4fe1b929942acc0bad3bfbf5ad3a8bab068422ac7ea5c \
        --cri-socket unix:///var/run/cri-dockerd.sock

输出内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

        mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

这个加入过程可能会有一些警告，不过只要在授予管理权限以及部署完calico网络后，获取节点状态和kube-system下的pod状态没问题就可以忽略警告

在每台上，都授予自己管理权限

1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

部署Calico网络插件

这个在任何一台master上做都行，只需要做一次，这里会从国外下载容器镜像，需要自己解决镜像下载问题

1
kubectl create -f https://docs.projectcalico.org/manifests/calico.yaml

在这里稍等一会儿，等待容器镜像下载完毕，在此期间，你可以用下面的方式查看系统pod，比如我的，可以看到还在努力初始化中，继续等待即可

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[root@k8s-master01 ~]# kubectl get pod -A
NAMESPACE     NAME                                              READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-564b9d64dd-f9mnm          0/1     ContainerCreating   0          23s
kube-system   calico-node-4vwv8                                 0/1     Init:2/3            0          23s
kube-system   calico-node-gl7wr                                 0/1     Init:0/3            0          23s
kube-system   calico-node-pz9tb                                 0/1     Init:2/3            0          23s
kube-system   coredns-fcd6c9c4-7gvqd                            0/1     ContainerCreating   0          2m49s
kube-system   coredns-fcd6c9c4-9925q                            0/1     ContainerCreating   0          2m49s
kube-system   etcd-k8s-master01.xiaohui.cn                      1/1     Running             0          2m54s
kube-system   etcd-k8s-master02.xiaohui.cn                      1/1     Running             0          75s
kube-system   etcd-k8s-master03.xiaohui.cn                      1/1     Running             0          63s
kube-system   kube-apiserver-k8s-master01.xiaohui.cn            1/1     Running             0          2m54s
kube-system   kube-apiserver-k8s-master02.xiaohui.cn            1/1     Running             0          75s
kube-system   kube-apiserver-k8s-master03.xiaohui.cn            1/1     Running             0          63s
kube-system   kube-controller-manager-k8s-master01.xiaohui.cn   1/1     Running             0          2m54s
kube-system   kube-controller-manager-k8s-master02.xiaohui.cn   1/1     Running             0          75s
kube-system   kube-controller-manager-k8s-master03.xiaohui.cn   1/1     Running             0          63s
kube-system   kube-proxy-9dj5v                                  1/1     Running             0          70s
kube-system   kube-proxy-b675s                                  1/1     Running             0          76s
kube-system   kube-proxy-hsskk                                  1/1     Running             0          2m50s
kube-system   kube-scheduler-k8s-master01.xiaohui.cn            1/1     Running             0          2m54s
kube-system   kube-scheduler-k8s-master02.xiaohui.cn            1/1     Running             0          75s
kube-system   kube-scheduler-k8s-master03.xiaohui.cn            1/1     Running             0          63s

初始化好之后：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
root@k8s-master01:~# kubectl get nodes
NAME                      STATUS   ROLES           AGE     VERSION
k8s-master01.xiaohui.cn   Ready    control-plane   7m11s   v1.31.1
k8s-master02.xiaohui.cn   Ready    control-plane   5m32s   v1.31.1
k8s-master03.xiaohui.cn   Ready    control-plane   5m26s   v1.31.1

root@k8s-master01:~# kubectl get pod -A
NAMESPACE     NAME                                              READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-564b9d64dd-f9mnm          1/1     Running   0          4m3s
kube-system   calico-node-4vwv8                                 1/1     Running   0          4m3s
kube-system   calico-node-gl7wr                                 1/1     Running   0          4m3s
kube-system   calico-node-pz9tb                                 1/1     Running   0          4m3s
kube-system   coredns-fcd6c9c4-7gvqd                            1/1     Running   0          6m29s
kube-system   coredns-fcd6c9c4-9925q                            1/1     Running   0          6m29s
kube-system   etcd-k8s-master01.xiaohui.cn                      1/1     Running   0          6m34s
kube-system   etcd-k8s-master02.xiaohui.cn                      1/1     Running   0          4m55s
kube-system   etcd-k8s-master03.xiaohui.cn                      1/1     Running   0          4m43s
kube-system   kube-apiserver-k8s-master01.xiaohui.cn            1/1     Running   0          6m34s
kube-system   kube-apiserver-k8s-master02.xiaohui.cn            1/1     Running   0          4m55s
kube-system   kube-apiserver-k8s-master03.xiaohui.cn            1/1     Running   0          4m43s
kube-system   kube-controller-manager-k8s-master01.xiaohui.cn   1/1     Running   0          6m34s
kube-system   kube-controller-manager-k8s-master02.xiaohui.cn   1/1     Running   0          4m55s
kube-system   kube-controller-manager-k8s-master03.xiaohui.cn   1/1     Running   0          4m43s
kube-system   kube-proxy-9dj5v                                  1/1     Running   0          4m50s
kube-system   kube-proxy-b675s                                  1/1     Running   0          4m56s
kube-system   kube-proxy-hsskk                                  1/1     Running   0          6m30s
kube-system   kube-scheduler-k8s-master01.xiaohui.cn            1/1     Running   0          6m34s
kube-system   kube-scheduler-k8s-master02.xiaohui.cn            1/1     Running   0          4m55s
kube-system   kube-scheduler-k8s-master03.xiaohui.cn            1/1     Running   0          4m43s

重新生成加入master命令

第一台初始化好24小时后，token什么的就会失效，24小时后加入更多master或者worker节点可以用下面的方式，需要注意的是，新加入的所有节点，都必须完成所有关于部署k8s的先决条件

加入master除了需要token之外，还需要certificate key

在现有的master上，生成token和certificate key

1
2
3
4
5
6
7
8
[root@k8s-master01 ~]# kubeadm token create --print-join-command
kubeadm join 192.168.8.200:64433 --token 1bfxvf.x48fpyrbifxh39ot --discovery-token-ca-cert-hash sha256:2fd9a0461559cbb7a018ccb47350531d4d7ac4419458f4766694d9d05ba4af9c

[root@k8s-master01 ~]# kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
ab028cb7a36bcbb6a7a9df0b74ab208d5d4825e2bb70d81fde2d8ba519c80159

把上面两个拼接起来，然后额外加入–certificate-key和–control-plane就行了，由于我们安装的是docker和cri-docker，所以还得加一个–cri-socket

1
2
3
4
5
kubeadm join 192.168.8.200:64433 --token 1bfxvf.x48fpyrbifxh39ot \
--discovery-token-ca-cert-hash sha256:2fd9a0461559cbb7a018ccb47350531d4d7ac4419458f4766694d9d05ba4af9c \
--certificate-key ab028cb7a36bcbb6a7a9df0b74ab208d5d4825e2bb70d81fde2d8ba519c80159 \
--control-plane \
--cri-socket unix:///var/run/cri-dockerd.sock

授予自己管理权限

1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

查询节点

1
2
3
4
5
6
[root@k8s-master04 ~]# kubectl get nodes
NAME                      STATUS   ROLES           AGE   VERSION
k8s-master01.xiaohui.cn   Ready    control-plane   59m   v1.31.1
k8s-master02.xiaohui.cn   Ready    control-plane   57m   v1.31.1
k8s-master03.xiaohui.cn   Ready    control-plane   57m   v1.31.1
k8s-master04.xiaohui.cn   Ready    control-plane   37s   v1.31.1

加入worker节点

需要注意的是，每个需要加入的worker，也必须完成本地hosts添加和部署Kubernetes中的所有先决条件，不然不能加入

开通Worker防火墙

完成以上所有在worker上的任务后，为worker节点开通防火墙

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# kubelet API
firewall-cmd --zone=public --add-port=10250/tcp --permanent

# kube-proxy
firewall-cmd --zone=public --add-port=10256/tcp --permanent

# NodePort Services
firewall-cmd --zone=public --add-port=30000-32767/tcp --permanent

# calico
firewall-cmd --add-port=179/tcp --permanent
firewall-cmd --add-port=4789/udp --permanent
firewall-cmd --add-port=5473/tcp --permanent
firewall-cmd --add-port=9099/tcp --permanent

# 重新载入防火墙以应用更改
firewall-cmd --reload

在现有的master上，生成token，worker节点不需要certificate key

1
2
[root@k8s-master01 ~]# kubeadm token create --print-join-command
kubeadm join 192.168.8.200:64433 --token r00nr4.bvjdc1lor5kuv5hh --discovery-token-ca-cert-hash sha256:2fd9a0461559cbb7a018ccb47350531d4d7ac4419458f4766694d9d05ba4af9c

在新的worker节点加入

1
2
3
kubeadm join 192.168.8.200:64433 --token r00nr4.bvjdc1lor5kuv5hh \
--discovery-token-ca-cert-hash sha256:2fd9a0461559cbb7a018ccb47350531d4d7ac4419458f4766694d9d05ba4af9c \
--cri-socket unix:///var/run/cri-dockerd.sock

获取节点列表

给worker节点打上worker标签，并查询节点和pod

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
root@k8s-master01:~# kubectl label nodes k8s-worker01.xiaohui.cn node-role.kubernetes.io/Worker=
node/k8s-worker01.xiaohui.cn labeled

root@k8s-master01:~# kubectl get nodes
NAME                      STATUS   ROLES           AGE    VERSION
k8s-master01.xiaohui.cn   Ready    control-plane   87m    v1.31.1
k8s-master02.xiaohui.cn   Ready    control-plane   83m    v1.31.1
k8s-master03.xiaohui.cn   Ready    control-plane   83m    v1.31.1
k8s-master04.xiaohui.cn   Ready    control-plane   27m    v1.31.1
k8s-worker01.xiaohui.cn   Ready    Worker          104s   v1.31.1

root@k8s-master01:~# kubectl get pod -A
NAMESPACE     NAME                                              READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-564b9d64dd-gsdv6          1/1     Running   0          80m
kube-system   calico-node-8m4g7                                 1/1     Running   0          27m
kube-system   calico-node-hvn97                                 1/1     Running   0          80m
kube-system   calico-node-lh8q6                                 1/1     Running   0          80m
kube-system   calico-node-nrvzp                                 1/1     Running   0          80m
kube-system   calico-node-zwwfz                                 1/1     Running   0          109s
kube-system   coredns-fcd6c9c4-2qmdt                            1/1     Running   0          87m
kube-system   coredns-fcd6c9c4-6gxq5                            1/1     Running   0          87m
kube-system   etcd-k8s-master01.xiaohui.cn                      1/1     Running   0          87m
kube-system   etcd-k8s-master02.xiaohui.cn                      1/1     Running   0          83m
kube-system   etcd-k8s-master03.xiaohui.cn                      1/1     Running   0          83m
kube-system   etcd-k8s-master04.xiaohui.cn                      1/1     Running   0          10m
kube-system   kube-apiserver-k8s-master01.xiaohui.cn            1/1     Running   0          87m
kube-system   kube-apiserver-k8s-master02.xiaohui.cn            1/1     Running   0          83m
kube-system   kube-apiserver-k8s-master03.xiaohui.cn            1/1     Running   0          83m
kube-system   kube-apiserver-k8s-master04.xiaohui.cn            1/1     Running   0          10m
kube-system   kube-controller-manager-k8s-master01.xiaohui.cn   1/1     Running   0          87m
kube-system   kube-controller-manager-k8s-master02.xiaohui.cn   1/1     Running   0          83m
kube-system   kube-controller-manager-k8s-master03.xiaohui.cn   1/1     Running   0          83m
kube-system   kube-controller-manager-k8s-master04.xiaohui.cn   1/1     Running   0          10m
kube-system   kube-proxy-4vpqw                                  1/1     Running   0          83m
kube-system   kube-proxy-bfl4c                                  1/1     Running   0          27m
kube-system   kube-proxy-mx2nq                                  1/1     Running   0          109s
kube-system   kube-proxy-rwbvv                                  1/1     Running   0          87m
kube-system   kube-proxy-tst26                                  1/1     Running   0          83m
kube-system   kube-scheduler-k8s-master01.xiaohui.cn            1/1     Running   0          87m
kube-system   kube-scheduler-k8s-master02.xiaohui.cn            1/1     Running   0          83m
kube-system   kube-scheduler-k8s-master03.xiaohui.cn            1/1     Running   0          83m
kube-system   kube-scheduler-k8s-master04.xiaohui.cn            1/1     Running   0          10m