208 lines
4.9 KiB
Markdown
208 lines
4.9 KiB
Markdown
> 本文作者：丁辉
|
||
|
||
# Helm部署Rook-Ceph
|
||
|
||
## 介绍
|
||
|
||
**Rook-Ceph 是一个开源的云原生存储编排器，旨在简化 Ceph 存储集群在 Kubernetes 环境中的部署和管理**。Rook-Ceph 将复杂的 Ceph 部署流程简化为易于在 Kubernetes 上实施的操作，使得 Ceph 集群能够无缝地与云原生环境集成。它利用 Kubernetes 的资源管理和调度能力，提供了一种高效且可扩展的方式来部署和管理存储解决方案。
|
||
|
||
## 基础准备
|
||
|
||
[Rook官方主页](https://rook.io/)
|
||
|
||
| 节点名称 | IP | 存储盘 |
|
||
| :---------: | :----------: | :------: |
|
||
| ceph-node-1 | 192.168.1.10 | /dev/sdb |
|
||
| ceph-node-2 | 192.168.1.20 | /dev/sdb |
|
||
| ceph-node-3 | 192.168.1.30 | /dev/sdb |
|
||
|
||
> 添加仓库
|
||
|
||
```bash
|
||
helm repo add rook-release https://charts.rook.io/release
|
||
helm repo update
|
||
```
|
||
|
||
## 部署Rook-Ceph-Operator
|
||
|
||
1. 配置 values.yaml 文件
|
||
|
||
```bash
|
||
vi rook-ceph-operator-values.yaml
|
||
```
|
||
|
||
内容如下
|
||
|
||
```yaml
|
||
# 配置镜像加速
|
||
csi:
|
||
cephcsi:
|
||
repository: quay.dockerproxy.com/cephcsi/cephcsi
|
||
|
||
registrar:
|
||
repository: registry.aliyuncs.com/google_containers/csi-node-driver-registrar
|
||
|
||
provisioner:
|
||
repository: registry.aliyuncs.com/google_containers/csi-provisioner
|
||
|
||
snapshotter:
|
||
repository: registry.aliyuncs.com/google_containers/csi-snapshotter
|
||
|
||
attacher:
|
||
repository: registry.aliyuncs.com/google_containers/csi-attacher
|
||
|
||
resizer:
|
||
repository: registry.aliyuncs.com/google_containers/csi-resizer
|
||
```
|
||
|
||
2. 部署
|
||
|
||
```bash
|
||
helm install rook-ceph rook-release/rook-ceph \
|
||
--namespace rook-ceph --create-namespace \
|
||
-f rook-ceph-operator-values.yaml
|
||
```
|
||
|
||
## 部署 Rook-Ceph-Cluster
|
||
|
||
1. 配置 values.yaml 文件
|
||
|
||
```bash
|
||
vi rook-ceph-cluster-values.yaml
|
||
```
|
||
|
||
内容如下
|
||
|
||
```yaml
|
||
operatorNamespace: rook-ceph
|
||
|
||
toolbox:
|
||
enabled: true
|
||
|
||
cephClusterSpec:
|
||
storage:
|
||
useAllNodes: false # 关闭使用所有Node
|
||
useAllDevices: false # 关闭使用所有设备
|
||
# 指定存储节点和磁盘
|
||
nodes:
|
||
- name: "192.168.1.10"
|
||
devices:
|
||
- name: "sdb"
|
||
- name: "192.168.1.20"
|
||
devices:
|
||
- name: "sdb"
|
||
- name: "192.168.1.20"
|
||
devices:
|
||
- name: "sdb"
|
||
#- name: "192.168.1.100"
|
||
#deviceFilter: "^sd." # 过滤以 "sd." 开头的设备
|
||
#- name: "nvme0"
|
||
#config:
|
||
#osdsPerDevice: "5" # 创建多个 OSD
|
||
#- name: "/dev/disk/XXXX-XXXX" # 指定实际设备文件的路径
|
||
|
||
# 开启监控面板
|
||
dashboard:
|
||
enabled: true
|
||
ssl: false
|
||
|
||
# 配置调度策略
|
||
placement:
|
||
all:
|
||
nodeAffinity:
|
||
requiredDuringSchedulingIgnoredDuringExecution:
|
||
nodeSelectorTerms:
|
||
- matchExpressions:
|
||
- key: role
|
||
operator: In
|
||
values:
|
||
- storage-node
|
||
podAffinity:
|
||
podAntiAffinity:
|
||
topologySpreadConstraints:
|
||
tolerations:
|
||
- key: storage-node
|
||
operator: Exists
|
||
```
|
||
|
||
2. Ceph节点添加标签
|
||
|
||
```bash
|
||
kubectl label nodes {ceph-node-1,ceph-node-2,ceph-node-3} role=storage-node
|
||
```
|
||
|
||
3. 部署
|
||
|
||
```bash
|
||
helm install rook-ceph-cluster rook-release/rook-ceph-cluster \
|
||
--namespace rook-ceph --create-namespace \
|
||
-f rook-ceph-cluster-values.yaml
|
||
```
|
||
|
||
4. 查看状态
|
||
|
||
```bash
|
||
kubectl -n rook-ceph exec -it $(kubectl get pod -l app=rook-ceph-tools -n rook-ceph | awk '{print $1}' | grep -v NAME) bash
|
||
```
|
||
|
||
进入容器后查看 ceph 状态
|
||
|
||
```bash
|
||
ceph -s
|
||
```
|
||
|
||
> health: HEALTH_OK
|
||
|
||
5. 查看密码登录 Dashboard
|
||
|
||
```bash
|
||
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
|
||
```
|
||
|
||
> 账户：admin
|
||
|
||
## 卸载
|
||
|
||
[集群清理文档](https://rook.io/docs/rook/latest-release/Storage-Configuration/ceph-teardown/#removing-the-cluster-crd-finalizer)
|
||
|
||
1. 卸载 Rook-Ceph-Cluster
|
||
|
||
```bash
|
||
helm uninstall rook-ceph-cluster -n rook-ceph
|
||
```
|
||
|
||
2. 卸载 Rook-Ceph-Operator
|
||
|
||
```bash
|
||
helm uninstall rook-ceph -n rook-ceph
|
||
```
|
||
|
||
3. 删除 CRD 资源
|
||
|
||
```bash
|
||
for CRD in $(kubectl get crd -n rook-ceph | awk '/ceph.rook.io/ {print $1}'); do
|
||
kubectl get -n rook-ceph "$CRD" -o name | \
|
||
xargs -I {} kubectl patch -n rook-ceph {} --type merge -p '{"metadata":{"finalizers": []}}'
|
||
done
|
||
```
|
||
|
||
4. 删除配置资源
|
||
|
||
```bash
|
||
kubectl -n rook-ceph patch configmap rook-ceph-mon-endpoints --type merge -p '{"metadata":{"finalizers": []}}'
|
||
kubectl -n rook-ceph patch secrets rook-ceph-mon --type merge -p '{"metadata":{"finalizers": []}}'
|
||
```
|
||
|
||
5. 删除命名空间
|
||
|
||
```bash
|
||
kubectl delete ns rook-ceph
|
||
```
|
||
|
||
6. 删除持久化目录
|
||
|
||
```bash
|
||
rm -rf /var/lib/rook
|
||
```
|
||
|