synchronization
This commit is contained in:
52
存储/Ceph/Ceph创建RBD块存储.md
Normal file
52
存储/Ceph/Ceph创建RBD块存储.md
Normal file
@@ -0,0 +1,52 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# Ceph创建RBD块存储
|
||||
|
||||
## 开始创建
|
||||
|
||||
1. 创建 Pool
|
||||
|
||||
```bash
|
||||
ceph osd pool create kubernetes
|
||||
```
|
||||
|
||||
2. 初始化 RBD
|
||||
|
||||
```bash
|
||||
rbd pool init kubernetes
|
||||
```
|
||||
|
||||
3. 创建访问块设备的认证用户
|
||||
|
||||
```bash
|
||||
ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes'
|
||||
```
|
||||
|
||||
|
||||
## 清理删除
|
||||
|
||||
1. 查找池中所有 RBD 镜像
|
||||
|
||||
```bash
|
||||
rbd ls -l kubernetes
|
||||
```
|
||||
|
||||
2. 删除 RBD 镜像
|
||||
|
||||
```bash
|
||||
rbd rm kubernetes/csi-vol-5a69b8e4-e8b5-4e74-8a41-1c7725f4c5f0
|
||||
```
|
||||
|
||||
3. 删除 Pool
|
||||
|
||||
```bash
|
||||
ceph osd pool rm kubernetes kubernetes --yes-i-really-really-mean-it
|
||||
```
|
||||
|
||||
4. 删除认证用户
|
||||
|
||||
```bash
|
||||
ceph auth del client.kubernetes
|
||||
```
|
||||
|
||||
|
66
存储/Ceph/Ceph创建文件系统.md
Normal file
66
存储/Ceph/Ceph创建文件系统.md
Normal file
@@ -0,0 +1,66 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# Ceph创建文件系统
|
||||
|
||||
## 开始创建
|
||||
|
||||
[官方文档](https://docs.ceph.com/en/reef/cephfs/createfs/)
|
||||
|
||||
1. 创建存储池
|
||||
|
||||
> 一个 ceph 文件系统需要至少两个 RADOS 存储池,一个用于存储数据,一个用于存储元数据
|
||||
|
||||
```bash
|
||||
ceph osd pool create cephfs_data
|
||||
ceph osd pool create cephfs_metadata
|
||||
```
|
||||
|
||||
检查
|
||||
|
||||
```bash
|
||||
ceph osd pool ls
|
||||
```
|
||||
|
||||
2. 创建 cephfs
|
||||
|
||||
```bash
|
||||
ceph fs new cephfs cephfs_metadata cephfs_data
|
||||
```
|
||||
|
||||
检查
|
||||
|
||||
```bash
|
||||
ceph fs ls
|
||||
```
|
||||
|
||||
3. 设置可用的元数据服务器 MDS 数量
|
||||
|
||||
查看 MDS 状态
|
||||
|
||||
```bash
|
||||
ceph mds stat
|
||||
```
|
||||
|
||||
> 结果为 `cephfs-1/1/1 up {0=rainbond=up:active}`
|
||||
|
||||
设置 MDS 数量
|
||||
|
||||
```bash
|
||||
ceph fs set cephfs max_mds 1
|
||||
```
|
||||
|
||||
4. 配置认证用户 cephfs
|
||||
|
||||
```bash
|
||||
ceph auth get-or-create client.cephfs mon 'allow r' mds 'allow rw' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'
|
||||
```
|
||||
|
||||
5. 查看 cephfs 用户密钥
|
||||
|
||||
```bash
|
||||
ceph auth get client.cephfs
|
||||
```
|
||||
|
||||
6. 本地挂载测试
|
||||
|
||||
请查看此篇文章 [Ceph挂载文件系统](https://gitee.com/offends/Kubernetes/blob/main/%E5%AD%98%E5%82%A8/Ceph/Ceph%E6%8C%82%E8%BD%BD%E6%96%87%E4%BB%B6%E7%B3%BB%E7%BB%9F.md) 尝试挂载
|
95
存储/Ceph/Ceph基础环境准备.md
Normal file
95
存储/Ceph/Ceph基础环境准备.md
Normal file
@@ -0,0 +1,95 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# Ceph基础环境准备
|
||||
|
||||
| 节点名称 | IP |
|
||||
| :---------: | :----------: |
|
||||
| ceph-node-1 | 192.168.1.10 |
|
||||
| ceph-node-2 | 192.168.1.20 |
|
||||
| ceph-node-3 | 192.168.1.30 |
|
||||
|
||||
1. 所有节点时间同步(必做)
|
||||
|
||||
[请查看此文章](https://gitee.com/offends/Linux/blob/main/Docs/NTP%E6%97%B6%E9%97%B4%E5%90%8C%E6%AD%A5.md)
|
||||
|
||||
2. 更改主机名
|
||||
|
||||
- ceph-node-1
|
||||
|
||||
```bash
|
||||
hostnamectl set-hostname ceph-node-1 && bash
|
||||
```
|
||||
|
||||
- ceph-node-2
|
||||
|
||||
```bash
|
||||
hostnamectl set-hostname ceph-node-2 && bash
|
||||
```
|
||||
|
||||
- ceph-node-3
|
||||
|
||||
```bash
|
||||
hostnamectl set-hostname ceph-node-3 && bash
|
||||
```
|
||||
|
||||
3. 配置 Hosts 文件
|
||||
|
||||
```bash
|
||||
cat >> /etc/hosts <<EOF
|
||||
192.168.1.10 ceph-node-1
|
||||
192.168.1.20 ceph-node-2
|
||||
192.168.1.30 ceph-node-3
|
||||
EOF
|
||||
```
|
||||
|
||||
4. 禁用 selinux
|
||||
|
||||
```bash
|
||||
sudo setenforce 0
|
||||
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config
|
||||
```
|
||||
|
||||
或如果启用了 SELinux, 请运行以下命令
|
||||
|
||||
```bash
|
||||
mkdir {/etc/ceph,/var/lib/ceph}
|
||||
chcon -Rt svirt_sandbox_file_t /etc/ceph
|
||||
chcon -Rt svirt_sandbox_file_t /var/lib/ceph
|
||||
```
|
||||
|
||||
5. 关闭防火墙
|
||||
|
||||
```bash
|
||||
systemctl stop firewalld
|
||||
systemctl disable firewalld
|
||||
```
|
||||
|
||||
6. 销毁磁盘上的分区表(用于清理旧盘)
|
||||
|
||||
> 如果遇到设备资源繁忙可查看此文档 [设备或资源繁忙问题解决](https://gitee.com/offends/Linux/blob/main/%E5%AD%98%E5%82%A8/%E9%97%AE%E9%A2%98%E8%AE%B0%E5%BD%95/%E8%AE%BE%E5%A4%87%E6%88%96%E8%B5%84%E6%BA%90%E7%B9%81%E5%BF%99.md)
|
||||
|
||||
- 官方清理方法
|
||||
|
||||
```bash
|
||||
docker run --rm --privileged=true \
|
||||
-v /dev/:/dev/ \
|
||||
-e OSD_DEVICE=/dev/sdb \
|
||||
quay.io/ceph/daemon:latest zap_device
|
||||
```
|
||||
|
||||
- [磁盘清理](https://gitee.com/offends/Linux/blob/main/%E5%AD%98%E5%82%A8/%E7%A3%81%E7%9B%98%E6%B8%85%E7%90%86.md)
|
||||
|
||||
- 下载脚本清理
|
||||
|
||||
```bash
|
||||
wget https://gitee.com/offends/Kubernetes/raw/main/File/Shell/clean-disk-ceph.sh
|
||||
```
|
||||
|
||||
> 通过修改 `DISK=""` 字段实现选择磁盘
|
||||
|
||||
7. 配置快捷命令
|
||||
|
||||
```bash
|
||||
echo 'alias ceph="docker exec ceph-mon ceph"' >> /etc/profile
|
||||
source /etc/profile
|
||||
```
|
9
存储/Ceph/Ceph常用配置文件参数解释.md
Normal file
9
存储/Ceph/Ceph常用配置文件参数解释.md
Normal file
@@ -0,0 +1,9 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# Ceph常用配置文件参数解释
|
||||
|
||||
| 参数名称 | 作用描述 |
|
||||
| ------------------------------ | ------------------------------------------------------------ |
|
||||
| `osd_pool_default_size` | 设置对象存储池(Object Storage Pool)的默认副本数量。该值定义了在Ceph集群中存储对象的默认副本数,即数据在集群中的冗余程度。增加副本数量可以提高数据的可靠性和容错性,但也会占用更多的存储空间。 |
|
||||
| `osd max object name len` | 设置对象名称的最大长度。Ceph中的对象(Object)可以通过名称进行标识,该参数指定了对象名称的最大长度限制。对象名称的长度不能超过该值,否则可能会导致错误或无法创建对象。 |
|
||||
| `osd max object namespace len` | 设置对象命名空间的最大长度。Ceph中的对象存储在命名空间(Namespace)下,该参数指定了对象命名空间的最大长度限制。对象命名空间的长度不能超过该值,否则可能会导致错误或无法创建对象。 |
|
32
存储/Ceph/Ceph挂载文件系统.md
Normal file
32
存储/Ceph/Ceph挂载文件系统.md
Normal file
@@ -0,0 +1,32 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# Ceph挂载文件系统
|
||||
|
||||
> 本文拿默认 cephfs 举例
|
||||
|
||||
1. 本地安装 ceph 客户端
|
||||
|
||||
- Centos
|
||||
|
||||
```bash
|
||||
yum install -y ceph-common
|
||||
```
|
||||
|
||||
- Ubuntu
|
||||
|
||||
```bash
|
||||
apt install -y ceph-common
|
||||
```
|
||||
|
||||
|
||||
2. 查看 cephfs 用户密钥
|
||||
|
||||
```bash
|
||||
ceph auth get client.cephfs
|
||||
```
|
||||
|
||||
3. 挂载文件系统到本地 /mnt/ 下测试
|
||||
|
||||
```bash
|
||||
mount -t ceph ceph-node-1:6789:/ /mnt/ -o name=cephfs,secret=AQAHjRdmdrPAAxAAuCtGNkM/W5XAVU2rWkUz+w==
|
||||
```
|
180
存储/Ceph/Docker单机部署Ceph集群(无裸盘).md
Normal file
180
存储/Ceph/Docker单机部署Ceph集群(无裸盘).md
Normal file
@@ -0,0 +1,180 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# Docker单机部署Ceph集群(无裸盘)
|
||||
|
||||
> 无压力 C+V 即可部署成功
|
||||
|
||||
1. 创建Ceph专用网络
|
||||
|
||||
```bash
|
||||
docker network create --driver bridge --subnet 172.20.0.0/16 ceph-network
|
||||
```
|
||||
|
||||
2. 拉取搭建用镜像
|
||||
|
||||
```bash
|
||||
docker pull ceph/daemon:latest
|
||||
```
|
||||
|
||||
3. 搭建mon节点
|
||||
|
||||
```bash
|
||||
docker run -d --name ceph-mon \
|
||||
--restart=always \
|
||||
--network ceph-network --ip 172.20.0.10 \
|
||||
-e MON_NAME=ceph-mon \
|
||||
-e MON_IP=172.20.0.10 \
|
||||
-e CEPH_PUBLIC_NETWORK=172.20.0.0/16 \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
ceph/daemon:latest mon
|
||||
```
|
||||
|
||||
4. 禁用不安全认证
|
||||
|
||||
> 这个参数控制是否允许在集群中通过全局 ID 回收机制(global ID reclaim)来认证用户身份
|
||||
|
||||
```bash
|
||||
docker exec ceph-mon ceph config set mon auth_allow_insecure_global_id_reclaim false
|
||||
```
|
||||
|
||||
5. 创建 osd 秘钥
|
||||
|
||||
```bash
|
||||
docker exec ceph-mon ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
|
||||
```
|
||||
|
||||
6. 搭建 osd 节点
|
||||
|
||||
```bash
|
||||
docker run -d --name ceph-osd-1 \
|
||||
--privileged=true \
|
||||
--restart=always \
|
||||
--network ceph-network --ip 172.20.0.11 \
|
||||
-e OSD_TYPE=directory \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /var/lib/ceph/osd/osd-1:/var/lib/ceph/osd \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
ceph/daemon:latest osd
|
||||
|
||||
docker run -d --name ceph-osd-2 \
|
||||
--privileged=true \
|
||||
--restart=always \
|
||||
--network ceph-network --ip 172.20.0.12 \
|
||||
-e OSD_TYPE=directory \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /var/lib/ceph/osd/osd-2:/var/lib/ceph/osd \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
ceph/daemon:latest osd
|
||||
|
||||
docker run -d --name ceph-osd-3 \
|
||||
--privileged=true \
|
||||
--restart=always \
|
||||
--network ceph-network --ip 172.20.0.13 \
|
||||
-e OSD_TYPE=directory \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /var/lib/ceph/osd/osd-3:/var/lib/ceph/osd \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
ceph/daemon:latest osd
|
||||
```
|
||||
|
||||
7. 搭建 mgr 节点
|
||||
|
||||
```bash
|
||||
docker run -d --name ceph-mgr \
|
||||
--restart=always \
|
||||
--network ceph-network --ip 172.20.0.14 \
|
||||
-p 7000:7000 \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
ceph/daemon:latest mgr
|
||||
```
|
||||
|
||||
8. 开启管理界面(查看此文档)
|
||||
|
||||
[MGR开启监控管理界面](https://gitee.com/offends/Kubernetes/blob/main/%E5%AD%98%E5%82%A8/Ceph/MGR%E5%BC%80%E5%90%AF%E7%9B%91%E6%8E%A7%E7%AE%A1%E7%90%86%E7%95%8C%E9%9D%A2.md)
|
||||
|
||||
9. 创建 rgw 秘钥
|
||||
|
||||
```bash
|
||||
docker exec ceph-mon ceph auth get client.bootstrap-rgw -o /var/lib/ceph/bootstrap-rgw/ceph.keyring
|
||||
```
|
||||
|
||||
10. 搭建 rgw 节点
|
||||
|
||||
```bash
|
||||
docker run -d --name ceph-rgw \
|
||||
--restart=always \
|
||||
--network ceph-network --ip 172.20.0.15 \
|
||||
-p 7480:7480 \
|
||||
-e RGW_NAME=ceph-rgw \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
ceph/daemon:latest rgw
|
||||
```
|
||||
|
||||
11. 搭建 mds 节点
|
||||
|
||||
```bash
|
||||
docker run -d --name ceph-mds \
|
||||
--restart=always \
|
||||
--network ceph-network --ip 172.20.0.16 \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
-e MDS_NAME=ceph-mds \
|
||||
-e CEPHFS_CREATE=1 \
|
||||
-e CEPHFS_NAME=cephfs \
|
||||
-e CEPHFS_DATA_POOL=cephfs_data \
|
||||
-e CEPHFS_DATA_POOL_PG=128 \
|
||||
-e CEPHFS_METADATA_POOL=cephfs_metadata \
|
||||
-e CEPHFS_METADATA_POOL_PG=64 \
|
||||
ceph/daemon:latest mds
|
||||
```
|
||||
|
||||
12. 检查Ceph状态
|
||||
|
||||
```bash
|
||||
docker exec ceph-mon ceph -s
|
||||
```
|
||||
|
||||
挂载地址为:172.20.0.10:6789
|
||||
|
||||
# 卸载清理
|
||||
|
||||
1. 清理容器
|
||||
|
||||
```bash
|
||||
docker rm -f ceph-mon
|
||||
docker rm -f ceph-osd-1
|
||||
docker rm -f ceph-osd-2
|
||||
docker rm -f ceph-osd-3
|
||||
docker rm -f ceph-mgr
|
||||
docker rm -f ceph-rgw
|
||||
docker rm -f ceph-mds
|
||||
```
|
||||
|
||||
2. 清理持久化目录
|
||||
|
||||
```bash
|
||||
rm -rf {/etc/ceph,/var/lib/ceph,/var/log/ceph}
|
||||
```
|
||||
|
||||
3. 清理 Docker 网络
|
||||
|
||||
```bash
|
||||
docker network rm ceph-network
|
||||
```
|
210
存储/Ceph/Docker部署高可用Ceph集群.md
Normal file
210
存储/Ceph/Docker部署高可用Ceph集群.md
Normal file
@@ -0,0 +1,210 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# Docker部署高可用Ceph集群
|
||||
|
||||
[CEPH版本](https://docs.ceph.com/en/latest/releases/) [CEPH下载](http://download.ceph.com/)
|
||||
|
||||
[DockerHub仓库(已废弃)](https://hub.docker.com/r/ceph/daemon) [Quay仓库](https://quay.io/repository/ceph/daemon)
|
||||
|
||||
| 节点名称 | IP | 存储盘 | 角色 |
|
||||
| :---------: | :----------: | :------: | :-----------------: |
|
||||
| ceph-node-1 | 192.168.1.10 | /dev/sdb | mon,mgr,osd,mds,rgw |
|
||||
| ceph-node-2 | 192.168.1.20 | /dev/sdb | mon,mgr,osd,mds,rgw |
|
||||
| ceph-node-3 | 192.168.1.30 | /dev/sdb | mon,mgr,osd,mds,rgw |
|
||||
|
||||
**组件介绍**
|
||||
|
||||
- OSD(Object Storage Daemon):负责管理磁盘上的数据块,执行数据的读写操作,并确保集群的高可用性。通常至少需要部署三个节点以保证系统的稳定运行。
|
||||
- MON(Monitor):负责维护 Ceph 集群的状态、配置和映射信息,保障集群元数据的一致性,协调节点间数据的分布和恢复。为确保集群的高可用性,一般需要至少部署三个节点。
|
||||
- MDS(Metadata Server):负责管理文件系统的目录结构和文件/目录的元数据信息,为 CephFS(Ceph 的分布式文件系统)提供元数据服务。需要注意的是,块存储和对象存储不需要部署 MDS。
|
||||
- MGR(Manager):负责收集 Ceph 集群的状态信息,包括 OSD、MON、MDS 的性能指标和健康状况,并提供可视化的仪表板(Ceph Dashboard)供用户查看。为确保集群的高可用性,通常至少需要部署两个节点。
|
||||
- RGW(Rados Gateway):提供了 RESTful API,允许用户通过 HTTP/HTTPS 请求访问和管理存储在 Ceph 集群中的数据。支持 Amazon S3 API 和 OpenStack Swift API。
|
||||
|
||||
**使用镜像介绍**
|
||||
|
||||
- `ceph/daemon` 是一个 Docker 镜像,它用于运行 Ceph 集群中的各种守护进程,如监控节点(MON)、管理节点(MGR)、元数据服务器(MDS)、对象存储守护进程(OSD)以及 RADOS 网关(RGW)。这个镜像包含了运行一个完整 Ceph 存储集群所需的所有组件,使得部署和管理变得更加容易和灵活,特别是在容器化环境中。
|
||||
|
||||
- `ceph/daemon` 镜像的优点是可以简化 Ceph 的部署和运维过程,因为它预配置了 Ceph 的各种服务和依赖,用户只需通过设置适当的环境变量和参数即可快速启动各种 Ceph 组件。这使得它在动态的云环境中特别有用,可以轻松地扩展或更新集群。
|
||||
|
||||
- 此外,使用 Docker 运行 Ceph 服务还有助于隔离不同服务的运行环境,提高系统的安全性和稳定性。
|
||||
|
||||
## 基础环境准备
|
||||
|
||||
[请查看此文章](https://gitee.com/offends/Kubernetes/blob/main/存储/Ceph/Ceph基础环境准备.md)
|
||||
|
||||
## 开始部署
|
||||
|
||||
### 部署监视器 MON
|
||||
|
||||
1. 节点一启动 mon
|
||||
|
||||
```bash
|
||||
docker run -d --net=host \
|
||||
--name ceph-mon \
|
||||
--restart=always \
|
||||
--privileged=true \
|
||||
-v /dev:/dev \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
-e MON_IP=192.168.1.10 \
|
||||
-e CEPH_PUBLIC_NETWORK=192.168.1.0/24 \
|
||||
ceph/daemon:latest mon
|
||||
```
|
||||
|
||||
2. 禁用不安全认证
|
||||
|
||||
> 这个参数控制是否允许在集群中通过全局 ID 回收机制(global ID reclaim)来认证用户身份
|
||||
|
||||
```bash
|
||||
ceph config set mon auth_allow_insecure_global_id_reclaim false
|
||||
```
|
||||
|
||||
3. 创建 OSD 秘钥
|
||||
|
||||
```bash
|
||||
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
|
||||
```
|
||||
|
||||
5. 拷贝节点一 ceph 目录, 传输至节点二和节点三, 传输完后按照第一步启动容器
|
||||
|
||||
> 注意修改 MON_IP
|
||||
|
||||
```bash
|
||||
# 节点二
|
||||
scp -r /etc/ceph/ root@ceph-node-2:/etc/
|
||||
scp -r /var/lib/ceph root@ceph-node-2:/var/lib/ceph
|
||||
# 节点三
|
||||
scp -r /etc/ceph root@ceph-node-3:/etc/ceph
|
||||
scp -r /var/lib/ceph root@ceph-node-3:/var/lib/ceph
|
||||
```
|
||||
|
||||
### 部署监控 MGR
|
||||
|
||||
> 为了确保集群的高可用性, 通常至少要部署两个节点
|
||||
|
||||
```bash
|
||||
docker run -d --net=host \
|
||||
--name ceph-mgr \
|
||||
--restart=always \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
ceph/daemon:latest mgr
|
||||
```
|
||||
|
||||
### 部署存储 OSD
|
||||
|
||||
1. 初始化块存储
|
||||
|
||||
```bash
|
||||
docker exec ceph-mon ceph-volume lvm prepare --data /dev/sdb
|
||||
```
|
||||
|
||||
ceph-volume **参数解释**
|
||||
|
||||
| 参数 | 解释 |
|
||||
| ---------------------- | ------------------------------------------------------------ |
|
||||
| `--bluestore` | 指定使用 Bluestore 存储引擎。Bluestore 是 Ceph 的一种存储格式,不依赖于文件系统,直接在裸设备上操作数据。 |
|
||||
| `--filestore` | 指定使用 Filestore 存储引擎。Filestore 是较旧的存储格式,它在底层文件系统(如 XFS)之上操作数据。 |
|
||||
| `--no-systemd` | 在创建时禁止生成 systemd 单元文件。通常用于不使用 systemd 的系统或者在特定的管理脚本中需要手动管理服务启动。 |
|
||||
| `--data` | 指定用于数据存储的设备或分区。 |
|
||||
| `--db` | 仅用于 Bluestore。指定用于存放数据库的设备或分区,可以提升性能。 |
|
||||
| `--wal` | 仅用于 Bluestore。指定用于写入前日志(Write-Ahead Log)的设备或分区,用于提升写入性能。 |
|
||||
| `--journal` | 仅用于 Filestore。指定日志存放的设备或分区,用于提升日志处理性能。 |
|
||||
| `--prepare` | 准备磁盘用于 Ceph 使用,但不启动 OSD。 |
|
||||
| `--activate` | 激活之前已准备好的 OSD。 |
|
||||
| `--crush-device-class` | 指定设备的 CRUSH 类别,用于 Ceph 集群的数据分布和复制策略中。 |
|
||||
|
||||
2. 部署 OSD
|
||||
|
||||
> 注意:根据自己情况修改 OSD_ID
|
||||
|
||||
```bash
|
||||
docker run -d --net=host --name=ceph-osd \
|
||||
--privileged=true \
|
||||
-e OSD_ID=0 \
|
||||
-v /dev/:/dev/ \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
ceph/daemon:latest osd_ceph_volume_activate
|
||||
```
|
||||
|
||||
>使用本地目录启动 OSD
|
||||
>
|
||||
>```bash
|
||||
>docker run -d --net=host --name=ceph-osd \
|
||||
> --privileged=true \
|
||||
> -v /etc/ceph:/etc/ceph \
|
||||
> -v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
> -v /var/log/ceph/:/var/log/ceph/ \
|
||||
> -v /var/lib/ceph/osd/osd-0:/var/lib/ceph/osd \
|
||||
> -v /etc/localtime:/etc/localtime:ro \
|
||||
> ceph/daemon:latest osd_directory
|
||||
>```
|
||||
|
||||
### 部署文件系统 MDS
|
||||
|
||||
```bash
|
||||
docker run -d --net=host \
|
||||
--name ceph-mds \
|
||||
--restart=always \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
-e CEPHFS_CREATE=1 \
|
||||
-e CEPHFS_NAME=cephfs \
|
||||
-e CEPHFS_DATA_POOL=cephfs_data \
|
||||
-e CEPHFS_DATA_POOL_PG=128 \
|
||||
-e CEPHFS_METADATA_POOL=cephfs_metadata \
|
||||
-e CEPHFS_METADATA_POOL_PG=64 \
|
||||
ceph/daemon:latest mds
|
||||
```
|
||||
|
||||
### 部署对象存储接 RGW
|
||||
|
||||
1. 创建 Rgw 密钥
|
||||
|
||||
```bash
|
||||
docker exec ceph-mon ceph auth get client.bootstrap-rgw -o /var/lib/ceph/bootstrap-rgw/ceph.keyring
|
||||
```
|
||||
|
||||
2. 启动
|
||||
|
||||
```bash
|
||||
docker run -d --net=host \
|
||||
--name=ceph-rgw \
|
||||
--restart=always \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /var/log/ceph/:/var/log/ceph/ \
|
||||
-v /etc/localtime:/etc/localtime:ro \
|
||||
ceph/daemon rgw
|
||||
```
|
||||
|
||||
### 开启管理界面
|
||||
|
||||
查看此文档 [MGR开启监控管理界面](https://gitee.com/offends/Kubernetes/blob/main/存储/Ceph/MGR开启监控管理界面.md)
|
||||
|
||||
## 卸载
|
||||
|
||||
1. 删除容器
|
||||
|
||||
```bash
|
||||
docker rm -f ceph-mon
|
||||
docker rm -f ceph-mgr
|
||||
docker rm -f ceph-osd
|
||||
docker rm -f ceph-mds
|
||||
docker rm -f ceph-rgw
|
||||
```
|
||||
|
||||
2. 删除持久化目录
|
||||
|
||||
```bash
|
||||
rm -rf {/etc/ceph,/var/lib/ceph,/var/log/ceph}
|
||||
```
|
51
存储/Ceph/MGR开启监控管理界面.md
Normal file
51
存储/Ceph/MGR开启监控管理界面.md
Normal file
@@ -0,0 +1,51 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# MGR开启监控管理界面
|
||||
|
||||
1. 开启 dashboard
|
||||
|
||||
```bash
|
||||
docker exec ceph-mgr ceph mgr module enable dashboard
|
||||
```
|
||||
|
||||
2. 创建证书
|
||||
|
||||
```bash
|
||||
docker exec ceph-mgr ceph dashboard create-self-signed-cert
|
||||
```
|
||||
|
||||
3. 创建登陆用户名和密码
|
||||
|
||||
```bash
|
||||
docker exec ceph-mgr sh -c 'echo "admin" > /password.txt'
|
||||
docker exec ceph-mgr ceph dashboard set-login-credentials admin -i /password.txt
|
||||
```
|
||||
|
||||
4. 配置外部访问端口
|
||||
|
||||
```bash
|
||||
docker exec ceph-mgr ceph config set mgr mgr/dashboard/server_port 7000
|
||||
```
|
||||
|
||||
5. 配置外部访问
|
||||
|
||||
```bash
|
||||
docker exec ceph-mgr ceph config set mgr mgr/dashboard/server_addr 192.168.1.10
|
||||
```
|
||||
|
||||
6. 关闭 HTTPS
|
||||
|
||||
```bash
|
||||
docker exec ceph-mgr ceph config set mgr mgr/dashboard/ssl false
|
||||
```
|
||||
|
||||
7. 重启 Mgr
|
||||
|
||||
```bash
|
||||
docker exec ceph-mgr ceph mgr module disable dashboard
|
||||
docker exec ceph-mgr ceph mgr module enable dashboard
|
||||
```
|
||||
|
||||
访问地址: `http://127.0.0.1:7000/`
|
||||
|
||||
账户密码: `admin/admin`
|
8
存储/Ceph/README.md
Normal file
8
存储/Ceph/README.md
Normal file
@@ -0,0 +1,8 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# CEPH对接Kubernetes-StorageClass
|
||||
|
||||
> 相关文档地址
|
||||
|
||||
- [Helm对接外部Ceph](https://gitee.com/offends/Kubernetes/blob/main/Helm/Helm%E5%AF%B9%E6%8E%A5%E5%A4%96%E9%83%A8Ceph.md)
|
||||
- [Helm部署Rook-Ceph](https://gitee.com/offends/Kubernetes/blob/main/Helm/Helm%E9%83%A8%E7%BD%B2Rook-Ceph.md)
|
9
存储/JuiceFS/README.md
Normal file
9
存储/JuiceFS/README.md
Normal file
@@ -0,0 +1,9 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# JuiceFS对接Kubernetes-StorageClass
|
||||
|
||||
> 相关文档地址
|
||||
|
||||
**查看此篇文章**
|
||||
|
||||
[Helm部署JuiceFS-CSI对接对象存储](https://gitee.com/offends/Kubernetes/blob/main/Helm/Helm%E9%83%A8%E7%BD%B2JuiceFS-CSI%E5%AF%B9%E6%8E%A5%E5%AF%B9%E8%B1%A1%E5%AD%98%E5%82%A8.md)
|
13
存储/NFS/README.md
Normal file
13
存储/NFS/README.md
Normal file
@@ -0,0 +1,13 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# 对接NFS相关文档
|
||||
|
||||
> 相关文档地址
|
||||
|
||||
- [Linux部署NFS存储](https://gitee.com/offends/Linux/blob/main/%E5%AD%98%E5%82%A8/NFS/Linux%E9%83%A8%E7%BD%B2NFS%E5%AD%98%E5%82%A8.md)
|
||||
- [Rsync常用参数解释](https://gitee.com/offends/Linux/blob/main/%E5%AD%98%E5%82%A8/NFS/Rsync%E5%B8%B8%E7%94%A8%E5%8F%82%E6%95%B0%E8%A7%A3%E9%87%8A.md)
|
||||
- [NFS服务优化](https://gitee.com/offends/Linux/blob/main/%E5%AD%98%E5%82%A8/NFS/NFS%E6%9C%8D%E5%8A%A1%E4%BC%98%E5%8C%96.md)
|
||||
- [Nfs高可用实现Rsync+Inotify](https://gitee.com/offends/Linux/blob/main/%E5%AD%98%E5%82%A8/NFS/Nfs%E9%AB%98%E5%8F%AF%E7%94%A8%E5%AE%9E%E7%8E%B0Rsync+Inotify.md)
|
||||
- [Nfs高可用实现Rsync+Sersync2](https://gitee.com/offends/Linux/blob/main/%E5%AD%98%E5%82%A8/NFS/Nfs%E9%AB%98%E5%8F%AF%E7%94%A8%E5%AE%9E%E7%8E%B0Rsync+Sersync2.md)
|
||||
- [Helm对接外部NFS存储](https://gitee.com/offends/Kubernetes/blob/main/Helm/Helm%E5%AF%B9%E6%8E%A5%E5%A4%96%E9%83%A8NFS%E5%AD%98%E5%82%A8.md)
|
||||
|
9
存储/OpenEBS/README.md
Normal file
9
存储/OpenEBS/README.md
Normal file
@@ -0,0 +1,9 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# OpenEBS对接Kubernetes-StorageClass
|
||||
|
||||
> 相关文档地址
|
||||
|
||||
**查看此篇文章**
|
||||
|
||||
[Helm部署OpenEBS储库](https://gitee.com/offends/Kubernetes/blob/main/Helm/Helm%E9%83%A8%E7%BD%B2OpenEBS%E5%AD%98%E5%82%A8.md)
|
9
存储/README.md
Normal file
9
存储/README.md
Normal file
@@ -0,0 +1,9 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# 磁盘读写性能测试
|
||||
|
||||
> 相关文档地址
|
||||
|
||||
**查看如下文档**
|
||||
|
||||
[磁盘读写性能测试](https://gitee.com/offends/Kubernetes/tree/main/Docker/Dockerfile/Fio/README.md)
|
40
存储/问题记录/Ceph问题记录.md
Normal file
40
存储/问题记录/Ceph问题记录.md
Normal file
@@ -0,0 +1,40 @@
|
||||
> 本文作者:丁辉
|
||||
|
||||
# Ceph问题记录
|
||||
|
||||
> 重启 Mon 后 `/opt/ceph-container/bin/entrypoint.sh: Existing mon, trying to rejoin cluster...`
|
||||
|
||||
- 该问题通过以下方式"解决": 在 Mon 持久化目录中 `rm -rf /var/lib/ceph/mon/<MONID>` , 其他 mon 将需要重复同样的修复方式。
|
||||
|
||||
或
|
||||
|
||||
- 修改启动文件修复
|
||||
|
||||
1. 将 mon 容器内的启动脚本复制到宿主机中。
|
||||
|
||||
```bash
|
||||
docker cp ceph-mon:/opt/ceph-container/bin/start_mon.sh .
|
||||
```
|
||||
|
||||
2. 修改启动脚本
|
||||
|
||||
```bash
|
||||
# 找到如下代码,注释掉
|
||||
v2v1=$(ceph-conf -c /etc/ceph/"${CLUSTER}".conf 'mon host' | tr ',' '\n' | grep -c "${MON_IP_NO_BRACKETS}")
|
||||
# 替换为如下代码
|
||||
v2v1=2
|
||||
```
|
||||
|
||||
3. 将修改后的启动脚本再复制到容器中
|
||||
|
||||
```bash
|
||||
docker cp start_mon.sh ceph-mon:/opt/ceph-container/bin/start_mon.sh
|
||||
```
|
||||
|
||||
4. 重新启动 mon 容器
|
||||
|
||||
```bash
|
||||
docker restart ceph-mon
|
||||
```
|
||||
|
||||
5. 再重启其他两个 mon 节点即可恢复
|
Reference in New Issue
Block a user