作者：李晓辉

联系方式：

微信：Lxh_Chat

邮箱：939958092@qq.com

这篇文章主要以将Cinder数据存储到本地硬盘为例，如果你需要了解如何将Cinder和Ceph集成，可以查看以下链接

https://www.linuxcenter.cn/linux/ceph-integrates-with-openstack.html

前言

在OpenStack环境中，Cinder块存储服务的高效运行离不开可靠的后端存储支持。虽然在之前的部署中，我们已经在控制节点上成功安装并配置了Cinder服务，但这些操作仅涉及管理层面的设置。真正的数据存储与管理，需要借助后端存储服务器来完成。后端存储服务器是Cinder服务的核心组成部分，它直接负责存储卷的创建、管理和维护，是确保虚拟机持久化存储的关键环节。

本文将聚焦于Cinder后端存储服务器的部署过程。我们将详细介绍如何选择合适的存储后端技术（如LVM、Ceph、NFS等），并根据实际需求进行安装、配置与优化。通过本文的指导，你将能够搭建起高效、稳定且可扩展的Cinder后端存储环境，为OpenStack云平台提供强大的存储支持，满足不同业务场景下的数据存储需求。

为数据存储到本地做LVM准备

在本地存储Cinder数据的文章中，由于我没有那么多机器，所以并置在了控制节点，节点最起码需要有一个额外的硬盘

我们这里将数据存储在本地的LVM中

准备相应软件包

在开始之前，需要安装LVM相关的软件包，以便在本地硬盘上创建和管理逻辑卷。

1	yum install lvm2 device-mapper-persistent-data -y

准备lvm卷组

我这里额外的盘是/dev/nvme0n2，你的也许是别的名称，用lsblk确认

1 2	pvcreate /dev/nvme0n2 vgcreate cinder-volumes /dev/nvme0n2

定义lvm扫描范围

默认情况下，LVM卷扫描工具会扫描/dev目录，查找包含卷的块存储设备。为了避免扫描到不必要的设备，需要重新配置LVM，让它只扫描包含cinder-volumes卷组的设备。

需要修改的配置文件是：/etc/lvm/lvm.conf

在devices中，添加了一个filter参数

每个过滤器组中的元素都以a开头，即为 accept，或以 r 开头，即为reject，并且包括一个设备名称的正则表达式规则。过滤器组必须以r/.*/结束，过滤所有保留设备。

1
2
3

devices {
filter = [ "a/nvme0n2/", "r/.*/"]
}

安装配置cinder存储组件

安装Cinder存储组件

在存储节点上安装Cinder和iSCSI目标服务。

1	yum install openstack-cinder targetcli -y

准备cinder配置文件

编辑Cinder配置文件/etc/cinder/cinder.conf，设置消息队列、身份认证、数据库连接以及LVM后端的相关参数。

/etc/cinder/cinder.conf 是cinder配置文件

crudini --set /etc/cinder/cinder.conf DEFAULT transport_url rabbit://openstack:LiXiaohui@controller
crudini --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone
crudini --set /etc/cinder/cinder.conf DEFAULT my_ip 192.168.8.10
crudini --set /etc/cinder/cinder.conf DEFAULT enabled_backends lvm
crudini --set /etc/cinder/cinder.conf DEFAULT glance_api_servers http://controller:9292
crudini --set /etc/cinder/cinder.conf database connection mysql+pymysql://cinder:LiXiaohui@controller/cinder
crudini --set /etc/cinder/cinder.conf keystone_authtoken www_authenticate_uri http://controller:5000
crudini --set /etc/cinder/cinder.conf keystone_authtoken auth_url http://controller:5000
crudini --set /etc/cinder/cinder.conf keystone_authtoken auth_uri http://controller:5000
crudini --set /etc/cinder/cinder.conf keystone_authtoken memcached_servers controller:11211
crudini --set /etc/cinder/cinder.conf keystone_authtoken auth_type password
crudini --set /etc/cinder/cinder.conf keystone_authtoken project_domain_name Default
crudini --set /etc/cinder/cinder.conf keystone_authtoken user_domain_name Default
crudini --set /etc/cinder/cinder.conf keystone_authtoken project_name service
crudini --set /etc/cinder/cinder.conf keystone_authtoken username cinder
crudini --set /etc/cinder/cinder.conf keystone_authtoken password LiXiaohui
crudini --set /etc/cinder/cinder.conf oslo_concurrency lock_path /var/lib/cinder/tmp
crudini --set /etc/cinder/cinder.conf lvm volume_driver cinder.volume.drivers.lvm.LVMVolumeDriver
crudini --set /etc/cinder/cinder.conf lvm volume_group cinder-volumes
crudini --set /etc/cinder/cinder.conf lvm target_helper lioadm
crudini --set /etc/cinder/cinder.conf lvm target_protocol iscsi

或者

[DEFAULT]
auth_strategy = keystone
transport_url = rabbit://openstack:LiXiaohui@controller
my_ip = 192.168.8.10
enabled_backends = lvm
glance_api_servers = http://controller:9292

[database]
connection = mysql+pymysql://cinder:LiXiaohui@controller/cinder

[keystone_authtoken]
www_authenticate_uri = http://controller:5000
auth_url = http://controller:5000
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = LiXiaohui

[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
target_protocol = iscsi
target_helper = lioadm

[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

启动cinder服务

在存储节点上，启动Cinder服务和iSCSI目标服务，并设置为开机自启。

1	systemctl enable openstack-cinder-volume.service target.service --now

在所有节点都要完成下列服务启动

1	systemctl enable iscsid --now

安装配置cinder备份服务

安装cinder组件

在控制节点上安装Cinder备份服务组件。

1	yum install openstack-cinder -y

准备配置文件

以下提供了备份到多个地方的配置，你只需要选一个就行

备份到本地

编辑Cinder配置文件/etc/cinder/cinder.conf，配置备份驱动为本地文件系统备份驱动，并设置备份文件的存储路径。

/etc/cinder/cinder.conf 是cinder的配置文件

crudini --set /etc/cinder/cinder.conf DEFAULT backup_driver cinder.backup.drivers.posix.PosixBackupDriver
crudini --set /etc/cinder/cinder.conf DEFAULT backup_posix_path /var/lib/cinder/backup
crudini --set /etc/cinder/cinder.conf DEFAULT backup_compression_algorithm zlib
crudini --set /etc/cinder/cinder.conf DEFAULT backup_enable_progress_timer True

或者

[DEFAULT]
backup_driver = cinder.backup.drivers.posix.PosixBackupDriver
backup_posix_path = /var/lib/cinder/backup
backup_compression_algorithm = zlib
backup_enable_progress_timer = True

创建出需要的目录

创建备份目录，并设置正确的权限。

1
2
3

mkdir -p /var/lib/cinder/backup
chown cinder:cinder /var/lib/cinder/backup
chmod 750 /var/lib/cinder/backup

同步数据库

1	cinder-manage db sync

重启服务

systemctl enable openstack-cinder-api.service \
openstack-cinder-backup.service \
openstack-cinder-scheduler.service \
openstack-cinder-volume.service

systemctl restart openstack-cinder-api.service \
openstack-cinder-backup.service \
openstack-cinder-scheduler.service \
openstack-cinder-volume.service

测试效果

创建一个测试卷并备份，验证备份功能是否正常。

[root@controller ~]# cd /var/lib/cinder/backup/
[root@controller backup]# ll
total 0
[root@controller backup]# openstack volume create --size 1 test-volume-lxh
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2024-05-30T13:36:09.774469           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | 987a3ed8-031e-46aa-a2dd-79399af522fe |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | test-volume-lxh                      |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 1                                    |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | __DEFAULT__                          |
| updated_at          | None                                 |
| user_id             | db56879a38d943ca8aacb93ffddb8f66     |
+---------------------+--------------------------------------+
[root@controller backup]# openstack volume backup create --name backupfile test-volume-lxh
+-----------+--------------------------------------+
| Field     | Value                                |
+-----------+--------------------------------------+
| id        | 57fbf893-7fb9-4609-94c2-f9fab222ede8 |
| name      | backupfile                           |
| volume_id | 987a3ed8-031e-46aa-a2dd-79399af522fe |
+-----------+--------------------------------------+
[root@controller backup]# ll
total 0
drwxr-xr-x. 3 cinder cinder 16 May 30 21:36 57

[root@controller backup]# ll -R /var/lib/cinder/backup/
/var/lib/cinder/backup/:
total 0
drwxr-xr-x. 3 cinder cinder 16 May 30 21:36 57

/var/lib/cinder/backup/57:
total 0
drwxr-xr-x. 3 cinder cinder 50 May 30 21:36 fb

/var/lib/cinder/backup/57/fb:
total 4
drwxrwx---. 2 cinder cinder 4096 May 30 21:36 57fbf893-7fb9-4609-94c2-f9fab222ede8

/var/lib/cinder/backup/57/fb/57fbf893-7fb9-4609-94c2-f9fab222ede8:
total 3332
-rw-rw----. 1 cinder cinder 1043644 May 30 21:36 volume_987a3ed8-031e-46aa-a2dd-79399af522fe_20240530133624_backup_57fbf893-7fb9-4609-94c2-f9fab222ede8-00001
-rw-rw----. 1 cinder cinder    2125 May 30 21:36 volume_987a3ed8-031e-46aa-a2dd-79399af522fe_20240530133624_backup_57fbf893-7fb9-4609-94c2-f9fab222ede8_metadata
-rw-rw----. 1 cinder cinder 2359578 May 30 21:36 volume_987a3ed8-031e-46aa-a2dd-79399af522fe_20240530133624_backup_57fbf893-7fb9-4609-94c2-f9fab222ede8_sha256file

备份到Ceph

准备配置文件

/etc/cinder/cinder.conf 是cinder的配置文件

需要先搞定ceph的账号和keyring

crudini --set /etc/cinder/cinder.conf DEFAULT backup_driver cinder.backup.drivers.ceph.CephBackupDriver
crudini --set /etc/cinder/cinder.conf DEFAULT backup_ceph_conf /etc/ceph/ceph.conf
crudini --set /etc/cinder/cinder.conf DEFAULT backup_ceph_user cinder-backup
crudini --set /etc/cinder/cinder.conf DEFAULT backup_ceph_chunk_size 134217728
crudini --set /etc/cinder/cinder.conf DEFAULT backup_ceph_pool backups
crudini --set /etc/cinder/cinder.conf DEFAULT backup_compression_algorithm zlib
crudini --set /etc/cinder/cinder.conf DEFAULT backup_enable_progress_timer True

或者

[DEFAULT]
backup_driver = cinder.backup.drivers.ceph
backup_ceph_conf = /etc/ceph/ceph.conf  # Ceph集群配置文件路径
backup_ceph_user = cinder-backup  # Ceph用户
backup_ceph_chunk_size = 134217728  # 分块大小（字节）
backup_ceph_pool = backups  # Ceph备份池的名称
backup_compression_algorithm = zlib  # 备份压缩算法
backup_enable_progress_timer = True  # 启用备份进度计时器

同步数据库

1	cinder-manage db sync

重启服务

systemctl enable openstack-cinder-api.service \
openstack-cinder-backup.service \
openstack-cinder-scheduler.service \
openstack-cinder-volume.service

systemctl restart openstack-cinder-api.service \
openstack-cinder-backup.service \
openstack-cinder-scheduler.service \
openstack-cinder-volume.service

测试效果

原来ceph池是空的

[root@ceph ~]# rados -p backups ls
rbd_directory
rbd_info
rbd_trash

创建个卷并备份

[root@controller cinder]# openstack volume create --size 1 ceph-test-lxh
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2024-05-30T14:03:12.482823           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | 4a6f25d5-93e2-49f7-9bf9-ac9913a7dba5 |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | ceph-test-lxh                        |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 1                                    |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | __DEFAULT__                          |
| updated_at          | None                                 |
| user_id             | 1a138f1f200b4d75a50b22ecd1a71903     |
+---------------------+--------------------------------------+
[root@controller cinder]# openstack volume backup create --name cephtest-lxh ceph-test-lxh
+-----------+--------------------------------------+
| Field     | Value                                |
+-----------+--------------------------------------+
| id        | 363b7645-eb5d-46b1-a209-e1348dc2d76a |
| name      | cephtest-lxh                         |
| volume_id | 4a6f25d5-93e2-49f7-9bf9-ac9913a7dba5 |
+-----------+--------------------------------------+

已经出现数据

[root@ceph ~]# rados -p backups ls
rbd_object_map.b6807bde1542.0000000000000006
rbd_object_map.b6807bde1542
rbd_directory
rbd_header.b6807bde1542
rbd_info
rbd_id.volume-4a6f25d5-93e2-49f7-9bf9-ac9913a7dba5.backup.363b7645-eb5d-46b1-a209-e1348dc2d76a
backup.363b7645-eb5d-46b1-a209-e1348dc2d76a.meta
rbd_trash

备份到NFS服务器

搭建NFS服务器

yum install nfs-utils -y
mkdir /backup
chmod 777 /backup -R
echo /backup *(rw,no_all_squash,no_root_squash) > /etc/exports
systemctl enable nfs-server --now
exportfs -rav

准备配置文件

/etc/cinder/cinder.conf 是cinder的配置文件

crudini --set /etc/cinder/cinder.conf DEFAULT backup_driver cinder.backup.drivers.nfs.NFSBackupDriver
crudini --set /etc/cinder/cinder.conf DEFAULT backup_mount_point_base /mnt/backup
crudini --set /etc/cinder/cinder.conf DEFAULT backup_share 192.168.8.10:/backup
crudini --set /etc/cinder/cinder.conf DEFAULT backup_file_size 209715200
crudini --set /etc/cinder/cinder.conf DEFAULT backup_sha_block_size_bytes 32768
crudini --set /etc/cinder/cinder.conf DEFAULT backup_compression_algorithm zlib
crudini --set /etc/cinder/cinder.conf DEFAULT backup_enable_progress_timer True

或者

[DEFAULT]
backup_driver = cinder.backup.drivers.nfs.NFSBackupDriver
backup_mount_point_base = /mnt/backup  # 备份挂载点基础路径
backup_share = 192.168.8.10:/backup  # NFS共享路径
backup_file_size = 209715200  # 备份文件大小限制（字节）
backup_sha_block_size_bytes = 32768  # SHA校验块大小（字节）
backup_enable_progress_timer = True  # 启用备份进度计时器

创建出挂载点

1
2
3

mkdir /mnt/backup
chown cinder:cinder /mnt/backup -R
chmod g+s /mnt/backup

同步数据库

1	cinder-manage db sync

重启服务

systemctl enable openstack-cinder-api.service \
openstack-cinder-backup.service \
openstack-cinder-scheduler.service \
openstack-cinder-volume.service

systemctl restart openstack-cinder-api.service \
openstack-cinder-backup.service \
openstack-cinder-scheduler.service \
openstack-cinder-volume.service

测试效果

[root@controller cinder]# openstack volume create --size 1 nfs-test
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2024-05-30T15:01:15.277805           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | 3a601d59-020c-4851-bfd6-5d983ac9627f |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | nfs-test                             |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 1                                    |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | __DEFAULT__                          |
| updated_at          | None                                 |
| user_id             | db56879a38d943ca8aacb93ffddb8f66     |
+---------------------+--------------------------------------+
[root@controller cinder]# openstack volume backup create --name nfstest nfs-test
+-----------+--------------------------------------+
| Field     | Value                                |
+-----------+--------------------------------------+
| id        | c30bff9d-e4cb-4fb8-a838-0acf30b3e619 |
| name      | nfstest                              |
| volume_id | 3a601d59-020c-4851-bfd6-5d983ac9627f |
+-----------+--------------------------------------+
[root@controller cinder]# ls /mnt/backup
49  c3
[root@controller cinder]# ls /mnt/backup
49/ c3/
[root@controller cinder]# ls /mnt/backup/c3/0b/c30bff9d-e4cb-4fb8-a838-0acf30b3e619/
volume_3a601d59-020c-4851-bfd6-5d983ac9627f_20240530150137_backup_c30bff9d-e4cb-4fb8-a838-0acf30b3e619-00001
volume_3a601d59-020c-4851-bfd6-5d983ac9627f_20240530150137_backup_c30bff9d-e4cb-4fb8-a838-0acf30b3e619-00002
volume_3a601d59-020c-4851-bfd6-5d983ac9627f_20240530150137_backup_c30bff9d-e4cb-4fb8-a838-0acf30b3e619-00003
volume_3a601d59-020c-4851-bfd6-5d983ac9627f_20240530150137_backup_c30bff9d-e4cb-4fb8-a838-0acf30b3e619-00004
volume_3a601d59-020c-4851-bfd6-5d983ac9627f_20240530150137_backup_c30bff9d-e4cb-4fb8-a838-0acf30b3e619-00005
volume_3a601d59-020c-4851-bfd6-5d983ac9627f_20240530150137_backup_c30bff9d-e4cb-4fb8-a838-0acf30b3e619-00006
volume_3a601d59-020c-4851-bfd6-5d983ac9627f_20240530150137_backup_c30bff9d-e4cb-4fb8-a838-0acf30b3e619_metadata
volume_3a601d59-020c-4851-bfd6-5d983ac9627f_20240530150137_backup_c30bff9d-e4cb-4fb8-a838-0acf30b3e619_sha256file

OpenStack部署(十一) 安装配置Cinder存储节点

前言

为数据存储到本地做LVM准备

准备相应软件包

准备lvm卷组

定义lvm扫描范围

安装配置cinder存储组件

安装Cinder存储组件

准备cinder配置文件

启动cinder服务

安装配置cinder备份服务

安装cinder组件

准备配置文件

备份到本地

创建出需要的目录

同步数据库

重启服务

测试效果

备份到Ceph

准备配置文件

同步数据库

重启服务

测试效果

备份到NFS服务器

搭建NFS服务器

准备配置文件

创建出挂载点

同步数据库

重启服务

测试效果