Kubernetes/Helm/Helm部署Rook-Ceph.md

本文作者：丁辉

Helm部署Rook-Ceph

介绍

Rook-Ceph 是一个开源的云原生存储编排器，旨在简化 Ceph 存储集群在 Kubernetes 环境中的部署和管理。Rook-Ceph 将复杂的 Ceph 部署流程简化为易于在 Kubernetes 上实施的操作，使得 Ceph 集群能够无缝地与云原生环境集成。它利用 Kubernetes 的资源管理和调度能力，提供了一种高效且可扩展的方式来部署和管理存储解决方案。

基础准备

Rook官方主页

节点名称	IP	存储盘
ceph-node-1	192.168.1.10	/dev/sdb
ceph-node-2	192.168.1.20	/dev/sdb
ceph-node-3	192.168.1.30	/dev/sdb

添加仓库

helm repo add rook-release https://charts.rook.io/release
helm repo update

部署Rook-Ceph-Operator

1. 配置 values.yaml 文件

   vi rook-ceph-operator-values.yaml
   

   内容如下

   # 配置镜像加速
   csi:
     cephcsi:
       repository: quay.dockerproxy.com/cephcsi/cephcsi
   
     registrar:
       repository: registry.aliyuncs.com/google_containers/csi-node-driver-registrar
   
     provisioner:
       repository: registry.aliyuncs.com/google_containers/csi-provisioner
   
     snapshotter:
       repository: registry.aliyuncs.com/google_containers/csi-snapshotter
   
     attacher:
       repository: registry.aliyuncs.com/google_containers/csi-attacher
   
     resizer:
       repository: registry.aliyuncs.com/google_containers/csi-resizer
   

2. 部署

   helm install rook-ceph rook-release/rook-ceph \
     --namespace rook-ceph --create-namespace \
     -f rook-ceph-operator-values.yaml
   

部署 Rook-Ceph-Cluster

1. 配置 values.yaml 文件

   vi rook-ceph-cluster-values.yaml
   

   内容如下

   operatorNamespace: rook-ceph
   
   toolbox:
     enabled: true
   
   cephClusterSpec:
     storage:
       useAllNodes: false  # 关闭使用所有Node
       useAllDevices: false # 关闭使用所有设备
       # 指定存储节点和磁盘
       nodes: 
         - name: "192.168.1.10"
           devices: 
             - name: "sdb"
         - name: "192.168.1.20"
           devices: 
             - name: "sdb"
         - name: "192.168.1.20"
           devices: 
             - name: "sdb"
         #- name: "192.168.1.100"
           #deviceFilter: "^sd." # 过滤以 "sd." 开头的设备
         #- name: "nvme0"
           #config:
             #osdsPerDevice: "5" # 创建多个 OSD
         #- name: "/dev/disk/XXXX-XXXX" # 指定实际设备文件的路径
   
     # 开启监控面板
     dashboard:
       enabled: true
       ssl: false
   
     # 配置调度策略
     placement:
       all:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
               - matchExpressions:
                 - key: role
                   operator: In
                   values:
                   - storage-node
         podAffinity:
         podAntiAffinity:
         topologySpreadConstraints:
         tolerations:
         - key: storage-node
           operator: Exists
   

2. Ceph节点添加标签

   kubectl label nodes {ceph-node-1,ceph-node-2,ceph-node-3} role=storage-node
   

3. 部署

   helm install rook-ceph-cluster rook-release/rook-ceph-cluster \
     --namespace rook-ceph --create-namespace \
     -f rook-ceph-cluster-values.yaml
   

4. 查看状态

   kubectl -n rook-ceph exec -it $(kubectl get pod -l app=rook-ceph-tools -n rook-ceph | awk '{print $1}' | grep -v NAME) bash
   

   进入容器后查看 ceph 状态

   ceph -s
   

   health: HEALTH_OK

5. 查看密码登录 Dashboard

   kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
   

   账户：admin

卸载

集群清理文档

1. 卸载 Rook-Ceph-Cluster

   helm uninstall rook-ceph-cluster -n rook-ceph
   

2. 卸载 Rook-Ceph-Operator

   helm uninstall rook-ceph -n rook-ceph
   

3. 删除 CRD 资源

   for CRD in $(kubectl get crd -n rook-ceph | awk '/ceph.rook.io/ {print $1}'); do
       kubectl get -n rook-ceph "$CRD" -o name | \
       xargs -I {} kubectl patch -n rook-ceph {} --type merge -p '{"metadata":{"finalizers": []}}'
   done
   

4. 删除配置资源

   kubectl -n rook-ceph patch configmap rook-ceph-mon-endpoints --type merge -p '{"metadata":{"finalizers": []}}'
   kubectl -n rook-ceph patch secrets rook-ceph-mon --type merge -p '{"metadata":{"finalizers": []}}'
   

5. 删除命名空间

   kubectl delete ns rook-ceph
   

6. 删除持久化目录

   rm -rf /var/lib/rook