Kubernetes/部署文档/Rancher/Rke1部署Kubernetes集群.md

> 本文作者：丁辉

# Rke1部署Kubernetes集群

[RKE1文档](https://rke.docs.rancher.com/)

[Rancher中文文档](https://docs.rancher.cn/)

|       节点名称       |      IP      |      Kubernetes角色      |
| :------------------: | :----------: | :----------------------: |
| k8s-master-1,Rke管理 | 192.168.1.10 | controlplane,etcd,worker |
|     k8s-master-2     | 192.168.1.20 | controlplane,etcd,worker |
|     k8s-master-3     | 192.168.1.30 | controlplane,etcd,worker |

## 环境准备

> !!!每次部署都写挺麻烦的索性都放在一个文件内了请查看 [Kubernetes基础环境准备](https://gitee.com/offends/Kubernetes/blob/main/%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3/Kubernetes%E5%9F%BA%E7%A1%80%E7%8E%AF%E5%A2%83%E5%87%86%E5%A4%87.md) ,请按照此文档初始化环境

### 所有节点执行

1. 配置 SSH

   ```bash
   sed -i 's/#AllowTcpForwarding yes/AllowTcpForwarding yes/g' /etc/ssh/sshd_config
   ```

   重启 SSH

   ```bash
   systemctl restart sshd
   ```

2. 将用户添加到 docker 组

   ```bash
   groupadd docker
   useradd -m docker -g docker
   ```

   > 使用其他用户
   >
   > ```bash
   > useradd rke # 创建用户
   > usermod -aG docker rke  将rke用户加入docker组
   > ```

3. 配置 docker 用户免密登录

   ```bash
   mkdir -p /home/docker/.ssh/
   touch /home/docker/.ssh/authorized_keys
   chmod 700 /home/docker/.ssh/
   chown -R docker.docker /home/docker/.ssh/
   chmod 600 /home/docker/.ssh/authorized_keys
   ```

### Rke管理节点执行

1. 生成密钥

   ```bash
   ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa -q
   ```

2. 查看主节点密钥

   > 密钥需要到 RKE 初始化节点上获取，所有节点都是用此密钥

   ```bash
   cat ~/.ssh/id_rsa.pub
   ```


### 所有节点执行

1. 粘贴密钥内容到此文件内(提示：所有节点粘贴Rke管理节密钥)

   ```bash
   vi /home/docker/.ssh/authorized_keys
   ```

2. 验证是否可以免密登录

   ```bash
   ssh docker@192.168.1.10
   ```

## Docker安装

1. Docker安装

   ```bash
   curl https://releases.rancher.com/install-docker/20.10.sh | sh
   ```

   > 传递参数使用国内源
   >
   > ```bash
   > curl -fsSL https://releases.rancher.com/install-docker/20.10.sh | sh -s -- --mirror Aliyun
   > ```

2. 启动 Docker

   ```bash
   systemctl enable docker
   systemctl start docker
   ```

## 安装并初始化Rke

[RKE二进制文件](https://github.com/rancher/rke/releases/)

1. 下载 RKE 二进制文件，并添加到可执行路径下

   ```bash
   wget https://github.com/rancher/rke/releases/download/v1.4.10/rke_linux-amd64
   ```

2. 授权

   ```bash
   chmod 777 rke_linux-amd64 && mv rke_linux-amd64  /usr/local/bin/rke
   ```

### 方法一 (不推荐怪麻烦的请看"方法二")

> 如果 `Number of Hosts` 填的是多节点则会提示输入多次节点信息

```bash
rke config --name cluster.yml
```

```bash
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: #默认回车
[+] Number of Hosts [1]: #节点数量
[+] SSH Address of host (1) [none]: 192.168.1.10 #节点IP地址
[+] SSH Port of host (1) [22]: #默认回车
[+] SSH Private Key Path of host (192.168.1.10) [none]: #默认回车
[-] You have entered empty SSH key path, trying fetch from SSH key parameter
[+] SSH Private Key of host (192.168.1.10) [none]: #默认回车
[-] You have entered empty SSH key, defaulting to cluster level SSH key: ~/.ssh/id_rsa
[+] SSH User of host (192.168.1.10) [ubuntu]: docker #SSH用户
[+] Is host (192.168.1.10) a Control Plane host (y/n)? [y]: y #是否为控制节点
[+] Is host (192.168.1.10) a Worker host (y/n)? [n]: y #是否为计算节点
[+] Is host (192.168.1.10) an etcd host (y/n)? [n]: y #是否为etcd节点
[+] Override Hostname of host (192.168.1.10) [none]: #默认回车
[+] Internal IP of host (192.168.1.10) [none]: 192.168.1.10 #主机内部IP
[+] Docker socket path on host (192.168.1.10) [/var/run/docker.sock]: #默认回车
[+] Network Plugin Type (flannel, calico, weave, canal, aci) [canal]: flannel #选择网络插件类型
[+] Authentication Strategy [x509]: #默认回车
[+] Authorization Mode (rbac, none) [rbac]: #默认回车
[+] Kubernetes Docker image [rancher/hyperkube:v1.26.8-rancher1]: #选择 k8s 版本
[+] Cluster domain [cluster.local]: #集群域
[+] Service Cluster IP Range [10.43.0.0/16]: #服务集群IP范围
[+] Enable PodSecurityPolicy [n]: #默认回车
[+] Cluster Network CIDR [10.42.0.0/16]: #集群网络CIDR
[+] Cluster DNS Service IP [10.43.0.10]: #集群DNS服务IP
[+] Add addon manifest URLs or YAML files [no]: #默认回车
```

基础参数修改

```bash
sed -i '/^ingress:$/,/^  provider:/ s/provider: ""/provider: "none"/' cluster.yml
```

### 方法二

1. 生成初始文件

   ```bash
   rke config --empty --name cluster.yml
   ```

2. 按需要修改 address 参数

   > 多节点则写多个 `address` 并通过调整 role 指定节点属性

   ```yml
   nodes:
   - address: 192.168.1.10
     port: "22"
     internal_address: 192.168.1.10
     role:
     - controlplane #管理
     - worker #计算
     - etcd #etcd节点
     hostname_override: ""
     user: docker
     docker_socket: /var/run/docker.sock
     ssh_key: ""
     ssh_key_path: ~/.ssh/id_rsa
     ssh_cert: ""
     ssh_cert_path: ""
     labels: {}
     taints: []
   # 格式一样此处省略 20,30 节点配置 ...
   services:
   ...
   ```

3. 基础参数修改

   ```bash
   sed -i 's/service_cluster_ip_range: ""/service_cluster_ip_range: 10.43.0.0\/16/' cluster.yml
   sed -i 's/cluster_cidr: ""/cluster_cidr: 10.42.0.0\/16/' cluster.yml
   sed -i 's/cluster_domain: ""/cluster_domain: cluster.local/' cluster.yml
   sed -i 's/cluster_dns_server: ""/cluster_dns_server: 10.43.0.10/' cluster.yml
   sed -i 's/plugin: ""/plugin: flannel/' cluster.yml
   sed -i 's/strategy: ""/strategy: x509/' cluster.yml
   sed -i 's/^\s*mode: ""$/  mode: rbac/' cluster.yml
   sed -i '/^ingress:$/,/^  provider:/ s/provider: ""/provider: "none"/' cluster.yml
   sed -i '/^[^ ]/ s/ssh_key_path: ""/ssh_key_path: ~\/.ssh\/id_rsa/g' cluster.yml
   sed -i '0,/^\s*ssh_key_path: ""$/{s,^\s*ssh_key_path: ""$,  ssh_key_path: ~/.ssh/id_rsa,}' cluster.yml
   ```

## 初始化 Kubernetes 集群

```bash
rke up
```

> - 禁用 metrics-server 组件
>
>   ```bash
>   sed -i '/^monitoring:$/,/^  provider:/ s/provider: ""/provider: "none"/' cluster.yml
>   ```
>   
> - 调整节点端口范围
>
>   > 默认端口范围：30000-32767
>
>   ```bash
>   sed -i 's/service_node_port_range: ""/service_node_port_range: "10000-30000"/' cluster.yml
>   ```
>
> - 关闭 Docker 版本检测
>
>   ```bash
>   sed -i 's/ignore_docker_version: null/ignore_docker_version: true/' cluster.yml
>   ```
>
> - 调整部署版本
>
>   - 查看当前 RKE 支持的Kubernetes版本
>
>   ```bash
>   rke config --list-version --all
>   ```
>
>   - 替换版本
>
>   ```bash
>   sed -i 's/kubernetes_version: ""/kubernetes_version: "v1.24.17-rancher1-1"/' cluster.yml
>   ```
>
> - 更新集群
>
>   ```bash
>   rke up --update-only
>   ```

## 安装 kubectl

[Kubectl二进制文件](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)

1. 下载 kubectl

   ```bash
   curl -LO https://dl.k8s.io/release/v1.26.8/bin/linux/amd64/kubectl
   ```

2. 授权

   ```bash
   chmod 777 kubectl && mv kubectl /usr/local/bin/
   ```

3. 添加 kubctl 文件

   ```bash
   mkdir ~/.kube && cp kube_config_cluster.yml ~/.kube/config && chmod 600 ~/.kube/config
   ```

4. 验证

   ```bash
   kubectl get node
   ```

   > 本文中没有禁用 `monitoring` 所以也可以使用 `kubectl top node` 测试

## 卸载

1. 卸载 RKE 集群

   ```bash
   rke remove
   ```

2. 清理残余容器

   ```bash
   for i in $(docker ps -a | grep rancher | awk '{print $1}');do docker rm -f $i;done
   for i in $(docker ps -a | grep rke | awk '{print $1}');do docker rm -f $i;done
   ```

3. 清除 Docker 引擎的废弃资源和缓存

   ```bash
   docker system prune --all
   ```

4. 卸载挂载

   ```bash
   mount | grep /var/lib/kubelet/pods/ | awk '{print $1}' | xargs umount -l
   ```

5. 删除持久化目录

   ```bash
   rm -rf /var/lib/kubelet/
   rm -rf /run/flannel/
   ```

## 备份和恢复

> 非常重要，他奶奶的吃大亏了

### 创建一次性快照

> RKE 会将节点快照保存在 `/opt/rke/etcd-snapshots` 路径下

```bash
rke etcd snapshot-save --config cluster.yml --name <快照名称>
```

### 恢复集群

```bash
rke etcd snapshot-restore --config cluster.yml --name <快照名称>
```

## 恢复 Rke配置文件

> 准备依赖 `jq`
>
> - Centos
>
>   ```
>   yum install jq -y
>   ```
>
> - Ubuntu
>
>   ```bash
>   apt install jq -y
>   ```

- 恢复 Kubectl 配置文件

  > 修改 `--master-ip=` 为任意 K8S Master节点IP

  ```bash
  curl -sfL https://gitee.com/offends/Kubernetes/raw/main/File/Shell/restore-rkestate-kubeconfig.sh | bash -s -- --master-ip=<K8S Master节点IP>
  ```

- 恢复 rkestate 状态文件

  - 通过本地 kubectl 找回

    ```bash
    kubectl get configmap -n kube-system full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | jq -r . > cluster.rkestate
    ```

  - 通过 master 节点找回

    ```bash
    curl -sfL https://gitee.com/offends/Kubernetes/raw/main/File/Shell/restore-rkestate-config.sh | bash -s
    ```