Kubernetes/Helm/Helm部署Rook-Ceph.md

> 本文作者：丁辉

# Helm部署Rook-Ceph

## 介绍

**Rook-Ceph 是一个开源的云原生存储编排器，旨在简化 Ceph 存储集群在 Kubernetes 环境中的部署和管理**。Rook-Ceph 将复杂的 Ceph 部署流程简化为易于在 Kubernetes 上实施的操作，使得 Ceph 集群能够无缝地与云原生环境集成。它利用 Kubernetes 的资源管理和调度能力，提供了一种高效且可扩展的方式来部署和管理存储解决方案。

## 基础准备

[Rook官方主页](https://rook.io/)

|  节点名称   |      IP      |  存储盘  |
| :---------: | :----------: | :------: |
| ceph-node-1 | 192.168.1.10 | /dev/sdb |
| ceph-node-2 | 192.168.1.20 | /dev/sdb |
| ceph-node-3 | 192.168.1.30 | /dev/sdb |

> 添加仓库

```bash
helm repo add rook-release https://charts.rook.io/release
helm repo update
```

## 部署Rook-Ceph-Operator

1. 配置 values.yaml 文件

   ```bash
   vi rook-ceph-operator-values.yaml
   ```

   内容如下

   ```yaml
   # 配置镜像加速
   csi:
     cephcsi:
       repository: quay.dockerproxy.com/cephcsi/cephcsi
   
     registrar:
       repository: registry.aliyuncs.com/google_containers/csi-node-driver-registrar
   
     provisioner:
       repository: registry.aliyuncs.com/google_containers/csi-provisioner
   
     snapshotter:
       repository: registry.aliyuncs.com/google_containers/csi-snapshotter
   
     attacher:
       repository: registry.aliyuncs.com/google_containers/csi-attacher
   
     resizer:
       repository: registry.aliyuncs.com/google_containers/csi-resizer
   ```

2. 部署

   ```bash
   helm install rook-ceph rook-release/rook-ceph \
     --namespace rook-ceph --create-namespace \
     -f rook-ceph-operator-values.yaml
   ```

## 部署 Rook-Ceph-Cluster

1. 配置 values.yaml 文件

   ```bash
   vi rook-ceph-cluster-values.yaml
   ```

   内容如下

   ```yaml
   operatorNamespace: rook-ceph
   
   toolbox:
     enabled: true
     
   cephClusterSpec:
     storage:
       useAllNodes: false  # 关闭使用所有Node
       useAllDevices: false # 关闭使用所有设备
       # 指定存储节点和磁盘
       nodes: 
         - name: "192.168.1.10"
           devices: 
             - name: "sdb"
         - name: "192.168.1.20"
           devices: 
             - name: "sdb"
         - name: "192.168.1.20"
           devices: 
             - name: "sdb"
         #- name: "192.168.1.100"
           #deviceFilter: "^sd." # 过滤以 "sd." 开头的设备
         #- name: "nvme0"
           #config:
             #osdsPerDevice: "5" # 创建多个 OSD
         #- name: "/dev/disk/XXXX-XXXX" # 指定实际设备文件的路径
         
     # 开启监控面板
     dashboard:
       enabled: true
       ssl: false
         
     # 配置调度策略
     placement:
       all:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
               - matchExpressions:
                 - key: role
                   operator: In
                   values:
                   - storage-node
         podAffinity:
         podAntiAffinity:
         topologySpreadConstraints:
         tolerations:
         - key: storage-node
           operator: Exists
   ```

2. Ceph节点添加标签

   ```bash
   kubectl label nodes {ceph-node-1,ceph-node-2,ceph-node-3} role=storage-node
   ```

3. 部署

   ```bash
   helm install rook-ceph-cluster rook-release/rook-ceph-cluster \
     --namespace rook-ceph --create-namespace \
     -f rook-ceph-cluster-values.yaml
   ```

4. 查看状态

   ```bash
   kubectl -n rook-ceph exec -it $(kubectl get pod -l app=rook-ceph-tools -n rook-ceph | awk '{print $1}' | grep -v NAME) bash
   ```

   进入容器后查看 ceph 状态

   ```bash
   ceph -s
   ```

   > health: HEALTH_OK

5. 查看密码登录 Dashboard

   ```bash
   kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
   ```

   > 账户：admin

## 卸载

[集群清理文档](https://rook.io/docs/rook/latest-release/Storage-Configuration/ceph-teardown/#removing-the-cluster-crd-finalizer)

1. 卸载 Rook-Ceph-Cluster

   ```bash
   helm uninstall rook-ceph-cluster -n rook-ceph
   ```

2. 卸载 Rook-Ceph-Operator

   ```bash
   helm uninstall rook-ceph -n rook-ceph
   ```

3. 删除 CRD 资源

   ```bash
   for CRD in $(kubectl get crd -n rook-ceph | awk '/ceph.rook.io/ {print $1}'); do
       kubectl get -n rook-ceph "$CRD" -o name | \
       xargs -I {} kubectl patch -n rook-ceph {} --type merge -p '{"metadata":{"finalizers": []}}'
   done
   ```

4. 删除配置资源

   ```bash
   kubectl -n rook-ceph patch configmap rook-ceph-mon-endpoints --type merge -p '{"metadata":{"finalizers": []}}'
   kubectl -n rook-ceph patch secrets rook-ceph-mon --type merge -p '{"metadata":{"finalizers": []}}'
   ```

5. 删除命名空间

   ```bash
   kubectl delete ns rook-ceph
   ```

6. 删除持久化目录

   ```bash
   rm -rf /var/lib/rook
   ```