作者：李晓辉

联系方式：

1. 微信：Lxh_Chat

2. 邮箱：939958092@qq.com

对象存储简介

与文件系统中的文件不同，对象不会整理到由目录和子目录组成的树中，而是存储在扁平的命名空间中。

应用不会使用普通文件系统操作来访问对象数据，而是会访问 REST API 来发送和接收对象。红帽 Ceph 存储支持 Amazon S3（简单存储服务）和 OpenStack Swift（OpenStack 对象存储）两种常用的对象 API。

Amazon S3 将对象存储的扁平命名空间称为存储桶，而 OpenStack Swift 则将其称为容器。由于命名空间是扁平的，所以存储桶和容器都不能嵌套。Ceph 通常会使用存储桶一词

RADOS 网关简介

RADOS 网关（也称对象网关 (RGW)）是一种服务，支持客户端利用标准对象存储 API 来访问 Ceph 集群。

radosgw 是红帽 Ceph 存储的客户端，提供对其他客户端应用的对象访问权限。客户端应用使用标准 API 与 RADOS 网关通信，RADOS 网关则使用 librados 模块调用来与 Ceph 集群通信。

RADOS 网关提供 radosgw-admin 实用程序来创建网关用户。这些用户只能访问网关，不能像 cdephx 用户一样直接访问存储集群。提交 Amazon S3 或 OpenStack Swift API 请求时，RADOS 网关客户端会使用这些网关用户帐户进行身份验证。网关用户通过 RADOS 网关完成身份验证后，网关会使用 cephx 凭据向存储集群进行身份验证，以处理对象请求。

RADOS 网关会为默认区域创建多个池。

.rgw.root - 存储信息记录
.default.rgw.control - 用作控制池
.default.rgw.meta - 存储 user_keys 和其他关键元数据
.default.rgw.log - 包含所有存储桶/容器和对象操作（如创建、读取和删除）的日志
.default.rgw.buckets.index - 存储存储桶的索引
.default.rgw.buckets.data - 存储存储桶数据
.default.rgw.buckets.non-ec - 用于多部分对象元数据上传

您可使用自定义设置来手动创建池。红帽建议以区域名称为前缀来手动创建池，如 .<zone-name> .rgw.control 中所示。例如，如果区域名称是 us-east-1，则池名称可以是 .us-east-1.rgw.buckets.data。

实践：RADOS 网关部署

简单部署

这里将使用lixiaohui字符串作为service id，默认情况下，将会部署两个守护进程，从返回中，可以看到，默认在两个主机上部署了进程，服务提供在80端口

[root@serverc ~]# ceph orch apply rgw lixiaohui --port 80 --placement="2 serverc.lab.example.com"
Scheduled rgw.lixiaohui update...

[root@serverc ~]# ceph orch ls | grep rgw
rgw.lixiaohui                2/2  27s ago    41s  serverc.lab.example.com;count:2
[root@serverc ~]# ceph orch ps | grep rgw
rgw.lixiaohui.serverc.hrcyas        serverc.lab.example.com  running (43s)  30s ago    43s  *:80           16.2.0-117.el8cp  2142b60d7974  48acb2cd8ab9
rgw.lixiaohui.serverc.ynhwiz        serverc.lab.example.com  running (39s)  30s ago    39s  *:81           16.2.0-117.el8cp  2142b60d7974  0c3440b7b968

指定参数部署

先删除已有的rgw网关

1 2	[root@serverc ~]# ceph orch rm rgw.lixiaohui Removed service rgw.lixiaohui

指定运行网络、端口、主机、服务id

cat > rgw.yml <<EOF
service_type: rgw
service_id: lixiaohui
placement:
  hosts:
    - serverc.lab.example.com
  count_per_host: 2
networks:
- 172.25.250.0/24
spec:
  rgw_frontend_port: 80
EOF

部署一下

1 2	[root@serverc ~]# ceph orch apply -i rgw.yml Scheduled rgw.lixiaohui update...

再次查询，发现每个主机上按照我们的想法部署了两个，需要注意的是，只有第一个服务是按照我们想要的80端口运行，其后本机上的后续服务都是默认加1，可以在其上加一个负载均衡来统一接收服务请求

[root@serverc ~]# ceph orch ls | grep rgw
rgw.lixiaohui                2/2  27s ago    41s  serverc.lab.example.com;count-per-host:2
[root@serverc ~]# ceph orch ps | grep rgw
rgw.lixiaohui.serverc.ewfdsm        serverc.lab.example.com  running (44s)  32s ago    44s  172.25.250.12:80  16.2.0-117.el8cp  2142b60d7974  93e96ffa013c
rgw.lixiaohui.serverc.zsvleu        serverc.lab.example.com  running (40s)  32s ago    40s  172.25.250.12:81  16.2.0-117.el8cp  2142b60d7974  0cf27a128c73

启用SSL部署

生成root证书

openssl genrsa -out /etc/pki/tls/private/selfsignroot.key 4096
openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=CN/ST=Shanghai/L=Shanghai/O=Company/OU=SH/CN=Root" \
-key /etc/pki/tls/private/selfsignroot.key \
-out /etc/pki/ca-trust/source/anchors/selfsignroot.crt

信任根证书

1	update-ca-trust

生成服务器私钥以及证书请求文件

openssl genrsa -out /etc/pki/tls/private/rgw.key 4096
openssl req -sha512 -new \
-subj "/C=CN/ST=Shanghai/L=Shanghai/O=Company/OU=SH/CN=serverc.lab.example.com" \
-key /etc/pki/tls/private/rgw.key \
-out rgw.csr

生成openssl cnf扩展文件

cat > certs.cnf << EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = serverc.lab.example.com
EOF

签发证书

openssl x509 -req -in rgw.csr \
-CA /etc/pki/ca-trust/source/anchors/selfsignroot.crt \
-CAkey /etc/pki/tls/private/selfsignroot.key -CAcreateserial \
-out /etc/pki/tls/certs/rgw.crt \
-days 3650 -extensions v3_req -extfile certs.cnf

给网关添加SSL证书，可以先删除不安全的rgw

1 2	[root@serverc ~]# ceph orch rm rgw.lixiaohui Removed service rgw.lixiaohui

部署SSL服务

cat > rgw.yml <<EOF
service_type: rgw
service_id: lixiaohui
placement:
  hosts:
    - serverc.lab.example.com
  count_per_host: 2
networks:
- 172.25.250.0/24
spec:
  rgw_frontend_port: 443
  rgw_frontend_ssl_certificate: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIJKQIBAAKCAgEA36UVpl33a5Q3gQVvm7Q3PA92K9ZwZsZXDUKAUmjk8Kimv+f7
    xS1i0laN4zP01F8rjCZ99JZ5SSrl2c6J+M0yeBVch5ewfgJgpKF37AAZ7dskBdLc
    l6guPhBMZUqSCG4AJpypvQkH+dumIJcKmiuwP6BEwjNiwV+YnZnM7cgOv66L2xiY
    lxuG83TJVOmpw5RoEUO+BdEhf4KXny15Gy7XO4BuC1QzLXymyfD2W192CwKej6Lw
    rGXfTFNNL7MgiR6L13uZ0jzRFzpGSCln8owaI/0E/LDQNsmFKbahNUtB/4tSHgXX
    vr4vOcZMayNqNd4ieCYRUeQEtErxX23tG+NLeog1iEZEyPB9ymVNQj85KyRNDdJN
    regSWwTjze1SstU4zsRrMQnHa5U9K9CrTP4MzD6WhhHHsDc7AGaMX6ftInLYtcXk
    LdH5kc00oQFFkDw/08qVnUYZLas1c8g/dT6LyArUX1oR4TT4iNXw6a42IPEm4VIc
    lRj4Qr5KEy/EF4N5CRB0SQEMBPRx1JY6oAiZJLje80ydxdE8hal5WSO7J8jOt16z
    1Ge7zs1D+vuwt0dFGNzHieIpMvrwheUjjNgClARxNy/hdA/WBPMCaerIuar7VIvB
    1EieY7Ikee8XpEOxMO3Wls/S1sbKeKDhfEV28Mt1F57wj98iVVLygGPwo5cCAwEA
    AQKCAgEAsIuqXo25HmR/uEspQrnuDGyMNajvTEJwP9hqYiPJ7qzEDaQ83PeqFi8b
    jOrb77Y9M0LDlYwfrIG/tsPTqOObXq5GQAef3KlVz0Bj4Zpm3ZxrgGlnmBS8hGPr
    Wt6WDy9+0PxFBiQV9sjNCNcTGIc6d8+117Qm1k33tRnPaznNYsWQQ+HTux2Q16ku
    sehPCkSiLlbSr9baYcUemhqUir/cSm1k8W6Dg/nt1GZyo8jk72Ye83aGFSHKN39e
    5DO8kGHbtLwuvoch6/slthtxoke07wnyoOOXSlik1TwFYIPLUsaRiX/FQx7xOuNu
    OflYA7Qac8xhScr9Z7Htfw7Q6X3xewwimwRAKQUhZ62rhEi/QChfeyWSeDlx3dPe
    XbHTfQv/E8HUYECD+z7Uv+o6SLrOKRGsUaCJ3SajFfjBvU9guav5JZ/0KVbuna+1
    R/kUJvHuOmKzddkO+ljhVWJmqfD49Jx1/OBfFDWMRsHNtxbqusDfjAAmftVre5v9
    Y6/TweGU7c9YIECf+DlpC4fLo++hvOIEoPTXWdIqIEOWL5ADWajdrc6bQ/Daq7QL
    W9NOYKbuFnMZMicyNtod7e461n39rqU9DoaD0frO05SEPQcM7+LLHxnwSH/2IKgZ
    9ASj1s+ygqErwrSo6UUzk5MPPoK1zgpqyV6fkzdchiRR3RG9sMkCggEBAPSyH/BO
    n8Qu45bf3g9BimiZjMKk1N5koPygdaF/WlHkCzYJClVsLOl9ArY9GuNHwwCNJoCf
    Icgb5BLqtmEoHZwtYN6CkQXQ5yQkwvuMOqf/cXpHVzxQDjbRaF1TSftGg7wjeLGs
    SEtew2z5s3pNo4G0iwc0w3/NmHYR9KkbZocPlbgrLUUcAtx86MyAf7fHAewPeDn3
    i9HQbbGfseFiebikgzpboAzpGX020m+Ik3kW83OqlaRf6He4RVfCvSfWaZSxLf0p
    5Khp2RgztcRktbBhYvUc1sIpyzRpolY0RL1V/flSESBY6acD8LAjPmodk/R1HKKD
    XhXhV0yJ8MC3Qn0CggEBAOn6AKjrOUiFcpSYezhqlQhIZAtKyB/9coLNOfFBi34t
    5/nhJhThs3dnE+KT/G4TIroUGsvc0Hs+YX/qSr5f6QA9o83Lczn2M2y4/yEucglN
    c29liu5GI7dA9Ha+Z81QSy727DK3fRwhV84OlocwWIXlQwfHi8nG8m5L3VI+kllz
    X/nT8dLEkqLmBufKw4OYzrHxkxq5Me3x8+gLCqKAiP1M33//GlI61Rtd2dJghVwH
    g076W/r7hV2u3M8HQOQiOfg4eoQjtmxSgzIDGnv07n1jwWWPelmEUr+tfbfqKPPP
    mJ1Ol2NsQmmQsxsuO8nmWwriNmh3xlRNXtRMCsmp5qMCggEATtg6XFcpObCWGt5F
    kgrfSzb22Rz9ji9EhEI7xO7hLnPZfO7Kyp2RlyZ1wjzMNiHAXqQcnOMpom4CPmos
    m/+uEs24YdoWWpZmdAn4Xc9gcxNnACyOmlt9SQKOq0uUEgg1mGChmjZWERsWwz1u
    LGFp5vhCt+6zc7HOh3TfrcGrdqd56Z7X9mXBMkR5SrO484GHSAXwClWOUuvi/JHl
    Yic1613u/tXZK2/Mi8Ena1LJmgHgLgvwLcfNgw34IgAYrQZ0fLnehK2EBRQeJKM8
    1WxR8uReW3aQaH3JATlsWnR7Fbrom/ZqmE6t8ufdHRQuH0kqjGFT+bV0kn5Kf9oO
    njoujQKCAQAg5h03ozN/xfvRdwtEaIAiFQO0LsSq8tkqGS8/hAbYLB5FHWpcX0v7
    ywuZvVMuKxSj83W9GoRZ//B3qvrtf9DkTsZ8hlHiYYLjk7OT99LjCffkPvPlAwm8
    l66ID8fKr7KSnPejPfeif/G3sIdr5NldolnleyJlscqexZ4OFWxAlZ92Il4LU8aG
    jy+DYhhRpafSj3QuusGRgobYJs1NAiPA8hEvDzClRU53tZ+OHDJEOW42Ka/LP5iG
    DrNPjOaL7WAXTeG0OM1Kt+NoodUUfuEV0nP8EsuSUK/N8dgOhydQ+OytmcJhoXRJ
    IaZ9eOfBQNmmUx5xW/4QFnzx/us5ZeFfAoIBAQCnBYpLhPs8m93eg+Ys/8WoKq6/
    hY7vO+omOxAT3x0qNuj7EWLUKhA75qsMOf9FQqcwOSMDLml/XthCmjLS/8eH4wPQ
    N47HFR+rJJQWDx+jVGYC+D+djQXYUtP7VQ0IosMWOTjxTwVrX9EkLwCUj8FifLdJ
    e3HunydNVRgb9p/w1i6L3NW19FX3AzfhPUEocyuXAGTCPSqjFTccRO8Txb1Qwq7N
    +hVb1LgZwC4lKhLQeU9F6o5n9a9+wtKgay25FWzB+t9jEje5JaQsTwa3YiDMARMe
    x7UGFIsjHrsAMOnuHSctVTxURSuXjoIUZgegM55Lvhzyxo+lxlos1+pph1Iy
    -----END RSA PRIVATE KEY-----
    -----BEGIN CERTIFICATE-----
    MIIFoTCCA4mgAwIBAgIUfkq3tDXGftkLMEbAc24Qz65w/lAwDQYJKoZIhvcNAQEL
    BQAwYTELMAkGA1UEBhMCQ04xETAPBgNVBAgMCFNoYW5naGFpMREwDwYDVQQHDAhT
    aGFuZ2hhaTEQMA4GA1UECgwHQ29tcGFueTELMAkGA1UECwwCU0gxDTALBgNVBAMM
    BFJvb3QwHhcNMjMwOTA4MjIzMTA0WhcNMzMwOTA1MjIzMTA0WjB0MQswCQYDVQQG
    EwJDTjERMA8GA1UECAwIU2hhbmdoYWkxETAPBgNVBAcMCFNoYW5naGFpMRAwDgYD
    VQQKDAdDb21wYW55MQswCQYDVQQLDAJTSDEgMB4GA1UEAwwXc2VydmVyYy5sYWIu
    ZXhhbXBsZS5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDfpRWm
    XfdrlDeBBW+btDc8D3Yr1nBmxlcNQoBSaOTwqKa/5/vFLWLSVo3jM/TUXyuMJn30
    lnlJKuXZzon4zTJ4FVyHl7B+AmCkoXfsABnt2yQF0tyXqC4+EExlSpIIbgAmnKm9
    CQf526YglwqaK7A/oETCM2LBX5idmcztyA6/rovbGJiXG4bzdMlU6anDlGgRQ74F
    0SF/gpefLXkbLtc7gG4LVDMtfKbJ8PZbX3YLAp6PovCsZd9MU00vsyCJHovXe5nS
    PNEXOkZIKWfyjBoj/QT8sNA2yYUptqE1S0H/i1IeBde+vi85xkxrI2o13iJ4JhFR
    5AS0SvFfbe0b40t6iDWIRkTI8H3KZU1CPzkrJE0N0k2t6BJbBOPN7VKy1TjOxGsx
    CcdrlT0r0KtM/gzMPpaGEcewNzsAZoxfp+0icti1xeQt0fmRzTShAUWQPD/TypWd
    RhktqzVzyD91PovICtRfWhHhNPiI1fDprjYg8SbhUhyVGPhCvkoTL8QXg3kJEHRJ
    AQwE9HHUljqgCJkkuN7zTJ3F0TyFqXlZI7snyM63XrPUZ7vOzUP6+7C3R0UY3MeJ
    4iky+vCF5SOM2AKUBHE3L+F0D9YE8wJp6si5qvtUi8HUSJ5jsiR57xekQ7Ew7daW
    z9LWxsp4oOF8RXbwy3UXnvCP3yJVUvKAY/CjlwIDAQABoz4wPDAJBgNVHRMEAjAA
    MAsGA1UdDwQEAwIF4DAiBgNVHREEGzAZghdzZXJ2ZXJjLmxhYi5leGFtcGxlLmNv
    bTANBgkqhkiG9w0BAQsFAAOCAgEAHllKjfWjkruaoFM4AfNMVYFCgED9cX12WW+4
    2CaqNQiP2PL42UoRblTZ4BjplIk/ktD/yMDfESuxxNjsrw14b9LIEl6mtsUjM0HX
    MvK7yluMa0E6QXeq8I7bAchshY3QvFmtEPDRK6dn7UsIwDGe/0whRA2lo/lPew3U
    iMdsEzHO0jxq7R6Y4sbUi4Bf0Bv0v03fLfhBYFH47sXtMfhBVpFTfi/Ccc7CO2sD
    E2uyXsi4PNWF9AP4CHTtA38UyglxH+GUOJpsPxByk8p+TfJDWRYHNQ7XPMon6AUk
    V24aySk26fAKymHDJc+vb/4Yar3i3CU0hs8nISfgEjCENOZc4ao5t4qjHgF8or1O
    0EfSLLoYNmaMZQq/fhpI3CGjZMUtEAJs90vBiy0+cB4aL4oELbsF251G2CEQbhyI
    wS7GlqIFd2fqSNzF3NRYT1uv1G8EDMs4C1xP8DkhC2mj0GI8dEaEKe1Zr1wMZ3QO
    AoTHhBe1SzJBcdNODe8KronpNjfOKZa/UXw8Y7OpddSlnpir6J3/+6ZCKiIC8LtS
    +mPz+Av5n0ffZjJSzZgFzEPs9HnaFAtICVAQ9QnOS5/HJuXSNQUWa7xgZY5ssCcM
    1lJAuBx+tcT5ES0y6blPv3OJJJDJRvUQNRVyjGXnLHNfZArQ3dIur+6YgofiInsp
    /+qoCXE=
    -----END CERTIFICATE-----

  ssl: true 
EOF

1 2	[root@serverc ~]# ceph orch apply -i rgw.yml Scheduled rgw.lixiaohui update...

确认访问

1
2

[root@serverc ~]# curl https://serverc.lab.example.com
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

自定义服务配置

使用集群配置 client.rgw 部分 rgw_frontends 参数中的 port 选项来为 RADOS 网关配置 Beast 前端 Web 端口，SSL 时，需要在端口号的末尾添加 s 字符来定义这些端口，例如 port=443s。port 选项支持使用加号字符 (+) 进行双端口配置，以便用户可通过两个不同端口中的任一端口访问 RADOS 网关。

1
2
3

[root@serverc ~]# ceph config set client.rgw rgw_frontends port=80+443s
[root@serverc ~]# ceph config get client.rgw rgw_frontends
port=80+443s