确保集群健康
使用ceph -s命令查看检查集群健康情况，一般新建很多pool的时候集群容易出现异常，这样即使我们后面启动了网关，也无法使用。会报错libcurl doesn’t support curl_multi_wait()。
[zzq@localhost myceph]$ ceph -s
cluster 5862afaf-4f46-4d6f-9d30-cb911ce335fc
health HEALTH_WARN
1928 pgs degraded
1928 pgs stuck degraded
1670 pgs stuck inactive
2048 pgs stuck unclean
1928 pgs stuck undersized
1928 pgs undersized
19 requests are blocked > 32 sec
recovery 20/40 objects degraded (50.000%)
too many PGs per OSD (512 > max 300)
mon.ceph4 low disk space
monmap e3: 4 mons at {ceph1=192.168.199.81:6789/0,ceph2=192.168.199.82:6789/0,ceph3=192.168.199.83:6789/0,ceph4=192.168.199.84:6789/0}
election epoch 26, quorum 0,1,2,3 ceph1,ceph2,ceph3,ceph4
mdsmap e13: 1/1/1 up {0=ceph4=up:active}
osdmap e91: 4 osds: 4 up, 4 in; 120 remapped pgs
pgmap v916: 2048 pgs, 17 pools, 2297 bytes data, 20 objects
12589 MB used, 19418 MB / 32908 MB avail
20/40 objects degraded (50.000%)
1670 undersized+degraded+peered
258 active+undersized+degraded
120 active+remapped
[zzq@localhost myceph]$
具体解决调整参考
遇到问题—ceph—ceph的rbd命令没反应卡住
RGW配置
在规划作为网关的服务器/etc/ceph/ceph.conf上添加以下内容：
host需要与hostname对应，其他默认即可。
[client.radosgw.gateway]
rgw frontends=fastcgi socket_port=9000 socket_host=0.0.0.0
host=admin
keyring=/etc/ceph/ceph.client.radosgw.keyring
log file=/var/log/radosgw/client.radosgw.gateway.log
rgw socket path=/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
rgw print continue=false
rgw content length compat = true
新建日志目录
sudo mkdir -p /var/log/radosgw
sudo chown 777 /var/log/radosgw
sudo chown 777 /var/run/ceph
把配置分发到其他机子
ceph-deploy --overwrite-conf config push admin ceph1 ceph2 ceph3 ceph4
新建 Ceph 对象网关实例
在admin管理节点工作目录创建一个 Ceph rgw 实例，一旦对象网关开始运行，我们就可以通过 http://admin:7480 地址访问。
进入ceph-deploy的目录中，使用命令
cd /home/zzq/myceph
sudo ceph-deploy --overwrite-conf rgw create admin
正确输出如下:
[zzq@localhost myceph]$ sudo ceph-deploy --overwrite-conf rgw create admin
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.37): /usr/bin/ceph-deploy --overwrite-conf rgw create admin
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] rgw : [('admin', 'rgw.admin')]
[ceph_deploy.cli][INFO ] overwrite_conf : True
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x265af38>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function rgw at 0x25b8758>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts admin:rgw.admin
[admin][DEBUG ] connected to host: admin
[admin][DEBUG ] detect platform information from remote host
[admin][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO ] Distro info: CentOS 6.9 Final
[ceph_deploy.rgw][DEBUG ] remote host will use sysvinit
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to admin
[admin][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[admin][WARNIN] rgw keyring does not exist yet, creating one
[admin][DEBUG ] create a keyring file
[admin][DEBUG ] create path recursively if it doesn't exist
[admin][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.admin osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.admin/keyring
[admin][INFO ] Running command: service ceph-radosgw start
[admin][DEBUG ] Starting client.rgw.admin...
[admin][WARNIN] /usr/bin/dirname: extra operand `-n'
[admin][WARNIN] Try `/usr/bin/dirname --help' for more information.
[admin][DEBUG ] 2018-05-27 17:55:20.895832 7f76e44c5820 -1 WARNING: libcurl doesn't support curl_multi_wait()
[admin][DEBUG ] 2018-05-27 17:55:20.895832 7f76e44c5820 -1 WARNING: cross zone / region transfer performance may be affected
[admin][DEBUG ] [ OK ]
[admin][INFO ] Running command: chkconfig ceph-radosgw on
[ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host admin and default port 7480
Error in sys.exitfunc:
查看radosgw服务状态：
ps -ef | grep radosgw
sudo systemctl status ceph-radosgw.service #centos7
sudo service ceph-radosgw status #centos6
重启RGW使用命令
sudo systemctl restart ceph-radosgw.service #centos7
sudo service ceph-radosgw restart #centos6
如果报错ibcurl doesn’t support curl_multi_wait()说明没有权限认证或者需要的资源池没有创建好。
[zzq@localhost myceph]$ sudo service ceph-radosgw restart
Starting client.rgw.admin... [FAILED]
/usr/bin/dirname: extra operand `-n'
Try `/usr/bin/dirname --help' for more information.
2018-05-27 18:07:21.747065 7ff716144820 -1 WARNING: libcurl doesn't support curl_multi_wait()
2018-05-27 18:07:21.747066 7ff716144820 -1 WARNING: cross zone / region transfer performance may be affected
回到第二步第三步使用命令创建好key和资源池后重新执行create使用命令如下:
sudo ceph-deploy --overwrite-conf rgw create admin
如果确认RGW 已经运行起来了，我们来访问一下试下。
[zzq@localhost myceph]$ sudo service ceph-radosgw status
/usr/bin/radosgw is running.
使用命令
curl http://admin:7480
运行成功输出如下:
[zzq@localhost ~]$ curl http://admin:7480
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
[zzq@localhost ~]$
如果还是有异常则需要查看日志
cat /var/log/radosgw/ceph-client.rgw.admin.log
如果报错bind_and_listen: failed to bind the UNIX domain socket to ‘/var/run/ceph/ceph-client.rgw.admin.asok’: (17) File exists，
则需要确认是否指定 rgw socket path参数 |
