RGW对象存储网关部署和使用
Rados GateWay简称rgw，底层采用亚马逊的S3作为对象存储实现。
参考链接：
https://docs.ceph.com/en/reef/radosgw/
https://docs.ceph.com/en/reef/radosgw/bucketpolicy/
https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/bucketnamingrules.html
https://www.s3express.com/help/help.html
01 部署rgw组件
1 部署之前查看集群状态
[root@ceph141 ~]# ceph -s
cluster:
id: 3cb12fba-5f6e-11ef-b412-9d303a22b70f
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 11m)
mgr: ceph141.cwgrgj(active, since 10m), standbys: ceph142.ymuzfe
mds: 1/1 daemons up, 1 standby
osd: 7 osds: 7 up (since 11m), 7 in (since 16h)
data:
volumes: 1/1 healthy
pools: 3 pools, 65 pgs
objects: 48 objects, 492 KiB
usage: 329 MiB used, 3.3 TiB / 3.3 TiB avail
pgs: 65 active+clean
2 创建一个服务
[root@ceph141 ~]# ceph orch apply rgw wenzhiyong
Scheduled rgw.wenzhiyong update...
[root@ceph141 ~]#
3 部署rgw组件
[root@ceph141 ~]# ceph orch daemon add rgw wenzhiyong ceph142
Deployed rgw.wenzhiyong.ceph141.csxaif on host 'ceph142'
[root@ceph141 ~]#
4 检查rgw组件是否部署成功
[root@ceph141 ~]# ceph -s
cluster:
id: 3cb12fba-5f6e-11ef-b412-9d303a22b70f
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 23m)
mgr: ceph141.cwgrgj(active, since 23m), standbys: ceph142.ymuzfe
mds: 1/1 daemons up, 1 standby
osd: 7 osds: 7 up (since 23m), 7 in (since 16h)
rgw: 1 daemon active (1 hosts, 1 zones) # Duang~不难发现，多了一个rgw组件！
data:
volumes: 1/1 healthy
pools: 7 pools, 193 pgs
objects: 274 objects, 499 KiB
usage: 430 MiB used, 3.3 TiB / 3.3 TiB avail
pgs: 193 active+clean
5 查看rgw默认创建的存储池信息
[root@ceph141 ~]# ceph osd pool ls
...
.rgw.root
default.rgw.log
default.rgw.control
default.rgw.meta
[root@ceph141 ~]#
[root@ceph141 ~]# radosgw-admin zone get --rgw-zone=default --rgw-zonegroup=default
{
"id": "10c61974-a41b-438d-ac2e-942b00e11d53",
"name": "default",
"domain_root": "default.rgw.meta:root",
"control_pool": "default.rgw.control",
"gc_pool": "default.rgw.log:gc",
"lc_pool": "default.rgw.log:lc",
"log_pool": "default.rgw.log",
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"roles_pool": "default.rgw.meta:roles",
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys",
"user_email_pool": "default.rgw.meta:users.email",
"user_swift_pool": "default.rgw.meta:users.swift",
"user_uid_pool": "default.rgw.meta:users.uid",
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"STANDARD": {
"data_pool": "default.rgw.buckets.data"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0,
"inline_data": true
}
}
],
"realm_id": "",
"notif_pool": "default.rgw.log:notif"
}
[root@ceph141 ~]#
6.查看ceph集群各组件部署信息
[root@ceph141~]# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 2m ago 2d count:1
ceph-exporter 3/3 2m ago 2d *
crash 3/3 2m ago 2d *
grafana ?:3000 1/1 2m ago 2d count:1
mds.zhiyong18-cephfs 2/2 2m ago 21h count:2
mgr 2/2 2m ago 2d count:2
mon 3/5 2m ago 2d count:5
node-exporter ?:9100 3/3 2m ago 2d *
osd 7 2m ago - <unmanaged>
prometheus ?:9095 1/1 2m ago 2d count:1
rgw.wenzhiyong ?:80 1/1 4s ago 10m ceph142
7 访问对象存储的WebUI
http://10.0.0.142/
可能遇到的故障：
- 重复添加142节点，导致提示80端口被占用
02 s3cmd工具安装
S3cmd 是一个用于与兼容 Amazon S3 的云存储服务交互的命令行工具。支持多种 S3 兼容的存储平台（如 MinIO、阿里云 OSS 等），可以用来执行常见的存储操作，例如上传、下载、同步文件和管理存储桶等
1 安装s3cmd工具
[root@ceph141 ~]# echo 10.0.0.142 www.wenzhiyong.com >> /etc/hosts
[root@ceph141 ~]#
[root@ceph141 ~]# apt -y install s3cmd
2 创建rgw账号
[root@ceph141 ~]# radosgw-admin user create --uid "wzy666" --display-name "文治勇"
{
"user_id": "wzy666",
"display_name": "文治勇",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "wzy666",
"access_key": "ZHOE7MVPLJFE5EIU738W", # 注意，别丢了，下面要用！
"secret_key": "VUNbdDwAGIq9AZv5f55e2gzptK1PUOnWg9nc44pE" # 注意，别丢了，下面要用！
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
3 运行s3cmd的运行环境，生成"/root/.s3cfg"配置文件
[root@ceph141 ~]# ll /root/.s3cfg
ls: cannot access '/root/.s3cfg': No such file or directory
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: ZHOE7MVPLJFE5EIU738W # rgw账号的access_key
Secret Key: VUNbdDwAGIq9AZv5f55e2gzptK1PUOnWg9nc44pE # rgw账号的secret_key
Default Region [US]: # 直接回车即可
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: www.wenzhiyong.com # 用于访问rgw的地址
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: www.wenzhiyong.com/%(bucket) # 设置DNS解析风格
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: # 文件不加密，直接回车即可
Path to GPG program [/usr/bin/gpg]: # 指定自定义的gpg程序路径，直接回车即可
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: No # 你的rgw是否是https，如果不是设置为No
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: # 代理服务器的地址，我并没有配置代理服务器，因此直接回车即可
New settings: # 注意，下面的信息是上面咱们填写时一个总的预览信息
Access Key: ZHOE7MVPLJFE5EIU738W
Secret Key: VUNbdDwAGIq9AZv5f55e2gzptK1PUOnWg9nc44pE
Default Region: US
S3 Endpoint: www.wenzhiyong.com
DNS-style bucket+hostname:port template for accessing a bucket: www.wenzhiyong.com/%(bucket)
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] Y # 如果确认上述信息没问题的话，则输入字母Y即可。
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y # 是否保存配置，我们输入y，默认是不保存配置的。
Configuration saved to '/root/.s3cfg'
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# ll /root/.s3cfg
-rw------- 1 root root 2269 Aug 23 09:59 /root/.s3cfg
[root@ceph141 ~]#
03 创建存储桶bucket
4 创建buckets
[root@ceph141 ~]# s3cmd mb s3://wenzhiyong-bucket
Bucket 's3://wenzhiyong-bucket/' created
[root@ceph141 ~]#
以下命名规则适用于通用存储桶
- 1存储桶名称必须介于 3（最少）到 63（最多）个字符之间。
- 2.存储桶名称只能由小写字母、数字、句点（.）和连字符（-）组成。
- 3.存储桶名称必须以字母或数字开头和结尾。
- 4.存储桶名称不得包含两个相邻的句点。
- 5.存储桶名称不得采用 IP 地址格式（例如，192.168.5.4）。
- 6.存储桶名称不得以前缀 xn-- 开头。
- 7.存储桶名称不得以前缀 sthree- 开头。
- 8.存储桶名称不得以前缀 sthree-configurator 开头。
- 9.存储桶名称不得以前缀 amzn-s3-demo- 开头。
- 10.存储桶名称不得以后缀 -s3alias 结尾。此后缀是为接入点别名预留的。有关更多信息，请参阅 为您的 S3 存储桶接入点使用存储桶式别名。
- 11.存储桶名称不得以后缀 --ol-s3 结尾。此后缀是为对象 Lambda 接入点别名预留的。有关更多信息，请参阅 如何为您的 S3 存储桶对象 Lambda 接入点使用存储桶式别名。
- 12.存储桶名称不得以后缀 .mrap 结尾。此后缀预留用于多区域接入点名称。有关更多信息，请参阅 命名 Amazon S3 多区域接入点的规则。
- 13.存储桶名称不得以后缀 --x-s3 结尾。此后缀预留用于目录存储桶。有关更多信息，请参阅 目录存储桶命名规则。
- 14.存储桶名称在分区内所有 AWS 区域中的所有 AWS 账户间必须是唯一的。分区是一组区域。AWS 目前有三个分区：aws（标准区域）、aws-cn（中国区域）和 aws-us-gov（AWS GovCloud (US)）。
- 15.存储桶名称不能被同一分区中的另一个 AWS 账户使用，直到存储桶被删除。
- 16.与 Amazon S3 Transfer Acceleration 一起使用的存储桶名称中不能有句点（.）。
04 上传视频到bucket
1.查看buckets
[root@ceph141 ~]# s3cmd ls
2024-08-23 02:03 s3://wenzhiyong-bucket
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# radosgw-admin buckets list
2.上传1个53MB的视频，如果文件体积足够大的话，才会有分片显示。下面的数字加起来正好是53MB
[root@ceph141~]# ll -h 01-MD5+简述.mp4
-rw-r--r-- 1 root root 53M Apr 13 2024 01-MD5+简述.mp4
[root@ceph141~]# s3cmd put 01-MD5+简述.mp4 s3://wenzhiyong-bucket
upload: '01-MD5+简述.mp4' -> 's3://wenzhiyong-bucket/01-MD5+简述.mp4' [part 1 of 4, 15MB] [1 of 1]
15728640 of 15728640 100% in 0s 40.55 MB/s done
upload: '01-MD5+简述.mp4' -> 's3://wenzhiyong-bucket/01-MD5+简述.mp4' [part 2 of 4, 15MB] [1 of 1]
15728640 of 15728640 100% in 0s 58.86 MB/s done
upload: '01-MD5+简述.mp4' -> 's3://wenzhiyong-bucket/01-MD5+简述.mp4' [part 3 of 4, 15MB] [1 of 1]
15728640 of 15728640 100% in 0s 51.43 MB/s done
upload: '01-MD5+简述.mp4' -> 's3://wenzhiyong-bucket/01-MD5+简述.mp4' [part 4 of 4, 7MB] [1 of 1]
8383328 of 8383328 100% in 0s 34.73 MB/s done
[root@ceph141~]# echo 15728640 + 15728640 + 15728640 + 8383328 | bc
55569248
[root@ceph141~]# echo 55569248 / 1024 /1024 | bc
52
3.使用s3cmd下载数据
[root@ceph141~]# s3cmd get s3://wenzhiyong-bucket/01-MD5+简述.mp4 /tmp/
download: 's3://wenzhiyong-bucket/01-MD5+简述.mp4' -> '/tmp/01-MD5+简述.mp4' [1 of 1]
55569248 of 55569248 100% in 0s 130.01 MB/s done
[root@ceph141~]# ll /tmp/01-MD5+简述.mp4
-rw-r--r-- 1 root root 55569248 Nov 7 11:15 /tmp/01-MD5+简述.mp4
05 创建授权策略以访问web页面
要通过web页面访问数据需要授权，否则就是下图这样显示权限拒绝：
1.创建策略
cat > wenzhiyong-anonymous-access-policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::wenzhiyong-bucket/*"
]
}]
}
EOF
2.应用策略
[root@ceph141~]# s3cmd setpolicy wenzhiyong-anonymous-access-policy.json s3://wenzhiyong-bucket
s3://wenzhiyong-bucket/: Policy updated
3.再次访问，http://10.0.0.142/wenzhiyong-bucket/01-MD5+简述.mp4。如果只有声音没有画面可能是浏览器问题，建议换成Chrome。
高可用提示：
于对象存储网关而言，需要将www.wenzhiyong.com解析到ceph141，ceph142,ceph143的任意一个节点上。
生产环境中，建议在rgw设备前加一个负载均衡器，以防止后端rgw宕机的情况，以减少单点故障的问题。
06 策略的删除
策略删除前后对比
[root@ceph141~]# s3cmd info s3://wenzhiyong-bucket
s3://wenzhiyong-bucket/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::wenzhiyong-bucket/*"
]
}]
}
CORS: none
ACL: 文治勇: FULL_CONTROL
[root@ceph141~]# s3cmd delpolicy s3://wenzhiyong-bucket
s3://wenzhiyong-bucket/: Policy deleted
[root@ceph141~]# s3cmd info s3://wenzhiyong-bucket
s3://wenzhiyong-bucket/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: none
CORS: none
ACL: 文治勇: FULL_CONTROL
再次访问web页面就会和之前一样提示权限拒绝
GAoBrduprZSwWqhuUAp4uxJekCEizwZCuG2QnxDB
7845VJXFP14XFBAL648B
wenzhiyong-bucket
nzhiyong-bucket
s3://wenzhiyong-bucket/: Policy deleted
[root@ceph141~]# s3cmd info s3://wenzhiyong-bucket
s3://wenzhiyong-bucket/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: none
CORS: none
ACL: 文治勇: FULL_CONTROL
再次访问web页面就会和之前一样提示权限拒绝
GAoBrduprZSwWqhuUAp4uxJekCEizwZCuG2QnxDB
7845VJXFP14XFBAL648B
wenzhiyong-bucket