ceph-objectstore-tool工具

1. 简介

ceph提供的一个操作pg及pg里面的对象的工具。

2. 使用

2.1 通用示例

ceph-objectstore-tool --data-path <path to object store> --journal-path <path to journal> --type <filestore (default), memstore> --op <op type>

• --data-path是指osd的数据目录，比如：/var/lib/ceph/osd/ceph-0/
• --journal-path是指osd的journal路径，比如：/var/lib/ceph/osd/ceph-0/journal
• --type是指osd后端存储的存储引擎，现在有filestore和memstore两种，默认是filestore
• --op是指具体的操作类型，有info, log, remove, mkfs, fsck, fuse, export, import, list, fix-lost, list-pgs, rm-past-intervals, dump-journal, dump-super, meta-list, get-osdmap, set-osdmap, get-inc-osdmap, set-inc-osdmap, mark-complete这些操作。

2.2 具体示例

列举几个常用的示例

这里要注意下使用该工具操作某个osd时，必须要先停止该osd，否则会得到报错：OSD has the store locked

2.2.1 列出指定osd上的pg

[root@ceph01 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --journal-path /var/lib/ceph/osd/ceph-0/journal --type filestore --op list-pgs
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0.0
2.2b
0.3
2.f
2.38
0.5
0.6
1.2f
...

2.2.2 查看指定osd上指定pg的详细信息

[root@ceph01 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 0.0 --op info
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
{
    "pgid": "0.0",
    "last_update": "159'34",
    "last_complete": "159'34",
    "log_tail": "0'0",
    "last_user_version": 30,
    "last_backfill": "MAX",
    "last_backfill_bitwise": 0,
    "purged_snaps": "[1~3]",
    "history": {
        "epoch_created": 1,
        "last_epoch_started": 211,
        "last_epoch_clean": 211,
        "last_epoch_split": 0,
        "last_epoch_marked_full": 0,
        "same_up_since": 209,
        "same_interval_since": 210,
        "same_primary_since": 210,
        "last_scrub": "159'34",
        "last_scrub_stamp": "2018-07-25 06:28:32.362059",
        "last_deep_scrub": "159'34",
        "last_deep_scrub_stamp": "2018-07-19 01:26:45.229169",
        "last_clean_scrub_stamp": "2018-07-25 06:28:32.362059"
    },
    ···

2.2.3 列出指定osd上指定pg里面对象

[root@ceph01 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --journal-path /var/lib/ceph/osd/ceph-0/journal --type filestore  --pgid 0.0 --op list
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
["0.0",{"oid":"rbd_data.d3716b8b4567.0000000000000090","key":"","snapid":-2,"hash":2156491776,"max":0,"pool":0,"namespace":"","max":0}]
["0.0",{"oid":"rbd_data.d3716b8b4567.0000000000000149","key":"","snapid":-2,"hash":136847872,"max":0,"pool":0,"namespace":"","max":0}]
["0.0",{"oid":"rbd_data.d3716b8b4567.00000000000000c3","key":"","snapid":-2,"hash":542082304,"max":0,"pool":0,"namespace":"","max":0}]
["0.0",{"oid":"rbd_data.d3716b8b4567.000000000000024f","key":"","snapid":-2,"hash":4017789824,"max":0,"pool":0,"namespace":"","max":0}]
["0.0",{"oid":"rbd_data.d3716b8b4567.0000000000000197","key":"","snapid":-2,"hash":957173632,"max":0,"pool":0,"namespace":"","max":0}]

2.2.4 导出备份pg和导入pg

一般用在解决pg状态不对，比如incomplete，此时就会用到将pg导出之后再导入。

[root@ceph01 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 0.36 --op export --file 0.0
[root@ceph01 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --journal-path /var/lib/ceph/osd/ceph-0/journal --op import --file 0.0

2.2.5 把状态incomplete的pg，标记为complete

比如：pg 1.1是incomplete，先对比pg 1.1的主副本之间 pg里面的对象数 哪个对象数多 就把哪个pg export出来
然后import到对象数少的pg里面 然后再mark complete，一定要先export pg备份。

查看pg 1.1主副本里面的对象数，假设主本对象数多，则到主本所在osd节点执行

[root@ceph01 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 1.1 --op export --file /home/pg1.1

然后将/home/pg1.1 scp到副本所在节点（有多个副本，每个副本都要这么操作），然后到副本所在节点执行

[root@ceph02 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1/ --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 1.1 --op import --file /home/pg1.1

然后再makr complete

[root@ceph02 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1/ --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 1.1 --op mark-complete

2.2.6 删除pg

这个操作一般用在pg里面的某个对象的损坏了，比如元数据丢失，就会导致客户端读取请求卡住，此时可以删除pg之后，然后再repair pg来恢复。当然remove pg之前要记住使用export将pg导出备份。

[root@ceph01 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 0.0 --op remove