ceph代码大全

ceph命令总结一、集群1、启动一个ceph 进程2、查看机器的监控状态[查看集群健康状态细节]

ceph health [detail]3、查看ceph的实时运行状态(常用)

ceph -w4、检查信息状态信息(常用)

ceph -s5、查看ceph存储空间

ceph df10、查看ceph集群中的认证用户及相关的key(常用)

ceph auth list11、查看ceph log日志所在的目录

ceph-conf --name mon.node1 --show-config-value log_file二、mon1、查看mon的状态信息

#ceph mon stat

输出信息：

e2: 3 mons at {10.186.9.13=10.186.9.13:6789/0,10.186.9.14=10.186.9.14:6789/0,10.186.9.16=10.186.9.16:6789/0}, election epoch 46, leader 0 10.186.9.13, quorum 0,1,2 10.186.9.13,10.186.9.14,10.186.9.162、查看mon的选举状态

# ceph quorum_status

输出信息：

{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["10.186.9.13","10.186.9.14","10.186.9.16"],"quorum_leader_name":"10.186.9.13","monmap":{"epoch":2,"fsid":"e6e8accb-6f7d-460e-93fe-06d49b1fde83","modified":"2019-11-19 16:36:44.137334","created":"2019-11-19 16:36:30.935660","features":{"persistent":["kraken","luminous"],"optional":[]},"mons":[{"rank":0,"name":"10.186.9.13","addr":"10.186.9.13:6789/0","public_addr":"10.186.9.13:6789/0"},{"rank":1,"name":"10.186.9.14","addr":"10.186.9.14:6789/0","public_addr":"10.186.9.14:6789/0"},{"rank":2,"name":"10.186.9.16","addr":"10.186.9.16:6789/0","public_addr":"10.186.9.16:6789/0"}]}}3、查看mon的映射信息

# ceph mon dump

输出信息：

dumped monmap epoch 1 epoch 1 fsid c71f378b-e86b-4c49-8b54-a9d85a6122c2 last_changed 2020-08-03 16:50:50.647195 created 2020-08-03 16:50:50.647195 0: 10.5.29.54:6789/0 mon.10.5.29.544、删除一个mon节点

# ceph mon remove node15、获得一个正在运行的mon map，为二进制mon.bin

# ceph mon getmap -o mon.bin三、msd1、查看msd状态

# ceph mds stat

输出结果：

cephfs-1/1/1 up {0=openstack1=up:active}2、查看msd的映射信息

# ceph mds dump

输出结果：

dumped fsmap epoch 45552 fs_name cephfs epoch 45552 flags c created 2020-09-08 18:03:49.514279 modified 2020-09-25 09:25:56.099961 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 last_failure 0 last_failure_osd_epoch 210 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2} max_mds 1 in 0 up {0=2084098} failed damaged stopped data_pools [7] metadata_pool 8 inline_data disabled balancer standby_count_wanted 0 2084098: 10.5.29.54:6800/1252110594 'openstack1' mds.0.45540 up:active seq 233、删除一个mds节点

# ceph mds rm 0 mds.node1四、osd1、查看ceph osd运行状态

# ceph osd stat

输出信息：

1 osds: 1 up, 1 in2、查看osd映射信息

# ceph osd dump

输出信息：

epoch 218 fsid c71f378b-e86b-4c49-8b54-a9d85a6122c2 created 2020-08-03 16:50:50.839507 modified 2020-09-30 13:18:29.712934 flags sortbitwise,recovery_deletes,purged_snapdirs crush_version 7 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client jewel min_compat_client jewel require_osd_release luminous pool 1 'images' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 last_change 118 lfor 0/116 flags hashpspool stripe_width 0 application rbd removed_snaps [1~3,8~2] pool 2 'volumes' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 last_change 218 lfor 0/119 flags hashpspool stripe_width 0 application rbd removed_snaps [1~3] pool 3 'backups' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 last_change 200 lfor 0/125 flags hashpspool stripe_width 0 application rbd pool 4 'vms' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 last_change 124 lfor 0/122 flags hashpspool stripe_width 0 application rbd removed_snaps [1~3] pool 5 'k8s' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 60 flags hashpspool stripe_width 0 pool 6 '.rgw.root' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 132 flags hashpspool stripe_width 0 application rgw pool 7 'cephfs_data' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 last_change 173 flags hashpspool stripe_width 0 application cephfs pool 8 'cephfs_metadata' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 last_change 174 flags hashpspool stripe_width 0 application cephfs pool 9 'default.rgw.control' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 151 flags hashpspool stripe_width 0 application rgw pool 10 'default.rgw.meta' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 156 flags hashpspool stripe_width 0 application rgw pool 11 'default.rgw.log' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 160 flags hashpspool stripe_width 0 application rgw max_osd 1 osd.0 up in weight 1 up_from 212 up_thru 212 down_at 211 last_clean_interval [207,209) 10.5.29.54:6802/5668 10.5.29.54:6803/5668 10.5.29.54:6804/5668 10.5.29.54:6805/5668 exists,up 9fd52adc-c323-42a0-b130-a779bbdaf0b43、查看osd的目录树

# ceph osd tree

输出信息：

输出信息： ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.64000 root default -2 0 host 10.5.29.54 -5 0.64000 host openstack1 0 hdd 0.64000 osd.0 up 1.00000 1.000004、查看osd各硬盘使用率

# ceph osd df

输出信息：

输出信息： ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 0 hdd 0.64000 1.00000 600GiB 11.0GiB 589GiB 1.83 1.00 120 TOTAL 600GiB 11.0GiB 589GiB 1.83 MIN/MAX VAR: 1.00/1.00 STDDEV: 05、查看osd延时

# ceph osd perf

输出信息：

osd commit_latency(ms) apply_latency(ms) 0 0 0五、PG组1、查看pg组的映射信息

# ceph pg dump

输出信息：

dumped all version 607481 stamp 2020-10-09 11:16:32.865738 last_osdmap_epoch 0 last_pg_scan 0 full_ratio 0 nearfull_ratio 0 PG_STAT OBJECTS MISSING_ON_PRIMARY 11.3 22 10.2 0 9.1 1 3.b 0 2.a 3 1.9 86 4.c 7 8.0 4 11.2 26 10.3 0 9.0 1 3.a 0 2.b 1 1.8 86 4.d 5 8.1 0 3.5 0 1.7 81 2.4 1 4.2 3 5.3 0 6.0 1 7.1 0 3.4 0 1.6 82 2.5 1 4.3 3 5.2 0 6.1 0 7.0 0 3.7 0 2.6 2 1.5 86 4.0 6 5.1 0 6.2 1 7.3 0 3.6 0 2.7 3 1.4 90 4.1 5 5.0 0 6.3 0 7.2 0 3.1 0 1.3 73 2.0 2 4.6 2 5.7 0 6.4 0 7.5 0 3.0 0 1.2 76 2.1 2 4.7 6 5.6 0 6.5 0 7.4 0 3.3 0 1.1 77 2.2 2 4.4 4 5.5 0 6.6 0 7.7 0 3.2 0 1.0 74 2.3 4 4.5 3 5.4 0 6.7 2 7.6 0 11.0 26 10.1 0 9.2 0 3.8 0 2.9 3 1.a 77 4.f 6 8.3 3 11.1 32 10.0 0 9.3 1 3.9 0 2.8 5 1.b 79 4.e 3 8.2 0 11.6 24 10.7 0 9.4 1 2.f 0 1.c 77 3.e 0 4.9 8 8.5 4 11.7 23 10.6 0 9.5 2 2.e 1 1.d 81 3.f 0 4.8 6 8.4 5 11.4 29 10.5 0 9.6 1 3.c 0 2.d 5 1.e 79 4.b 1 8.7 4 11.5 25 10.4 0 9.7 1 3.d 0 2.c 2 1.f 76 4.a 9 8.6 1 11 207 0 0 0 0 1 1280 0 0 0 0 10534166580 2 37 0 0 0 0 70927032 3 0 0 0 0 0 0 4 77 0 0 0 0 103164205 5 0 0 0 0 0 0 6 4 0 0 0 0 1113 7 0 0 0 0 0 0 8 21 0 0 0 0 2246 9 8 0 0 0 0 0 10 0 0 0 0 0 0 sum 1634 0 0 0 0 10708261176 OSD_STAT USED AVAIL TOTAL 0 11.0GiB 589GiB 600GiB sum 11.0GiB 589GiB 600GiB2、查看一

输出信息： DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 0 0 0 0 0 1554 1554 active clean 2020-10-08 23:40:08.999691 218'65354 218:98261 [0] 0 [0] 0 218'63422 2020-10-08 23:40:08.999638 218'58648 2020-10-07 19:21:09.086796 0 0 0 0 0 0 0 0 active clean 2020-10-08 17:03:50.360386 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-08 17:03:50.360073 0'0 2020-10-03 16:33:37.680982 0 0 0 0 0 0 15 15 active clean 2020-10-08 08:10:13.634676 213'15 218:261552 [0] 0 [0] 0 213'15 2020-10-08 08:10:13.634540 213'15 2020-10-07 07:55:43.794747 0 0 0 0 0 0 0 0 active clean 2020-10-09 08:05:07.603764 0'0 218:441 [0] 0 [0] 0 0'0 2020-10-09 08:05:07.603570 0'0 2020-10-08 04:10:55.985889 0 0 0 0 0 117 696 696 active clean 2020-10-08 01:01:15.979950 218'696 218:124863 [0] 0 [0] 0 218'695 2020-10-08 01:01:15.979563 218'695 2020-10-05 09:23:23.610962 0 0 0 0 0 684376727 188 188 active clean 2020-10-08 08:27:35.916078 71'188 218:836 [0] 0 [0] 0 71'188 2020-10-08 08:27:35.915814 71'188 2020-10-08 08:27:35.915814 0 0 0 0 0 8402530 1563 1563 active clean 2020-10-09 05:48:38.892392 218'394363 218:672698 [0] 0 [0] 0 217'394362 2020-10-09 05:48:38.892240 217'394362 2020-10-08 04:31:51.649845 0 0 0 0 0 0 8 8 active clean 2020-10-09 05:11:56.978009 162'8 218:267 [0] 0 [0] 0 162'8 2020-10-09 05:11:56.977695 162'8 2020-10-05 12:27:57.351136 0 0 0 0 0 0 1594 1594 active clean 2020-10-08 12:58:06.268311 218'90994 218:136755 [0] 0 [0] 0 218'85762 2020-10-08 12:58:06.268206 218'52208 2020-10-02 13:31:54.916159 0 0 0 0 0 0 0 0 active clean 2020-10-09 00:13:54.479007 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-09 00:13:54.478738 0'0 2020-10-02 12:03:50.578100 0 0 0 0 0 0 15 15 active clean 2020-10-09 05:20:58.550242 213'15 218:268427 [0] 0 [0] 0 213'15 2020-10-09 05:20:58.550128 213'15 2020-10-07 21:28:01.376248 0 0 0 0 0 0 0 0 active clean 2020-10-08 18:27:30.668569 0'0 218:439 [0] 0 [0] 0 0'0 2020-10-08 18:27:30.668417 0'0 2020-10-07 16:04:00.976150 0 0 0 0 0 98 106 106 active clean 2020-10-08 10:05:35.206002 217'106 218:703 [0] 0 [0] 0 217'106 2020-10-08 10:05:35.205802 217'106 2020-10-06 03:38:00.975439 0 0 0 0 0 717931008 287 287 active clean 2020-10-08 07:06:10.779285 70'287 218:1569 [0] 0 [0] 0 70'287 2020-10-08 07:06:10.778936 70'287 2020-10-08 07:06:10.778936 0 0 0 0 0 16388608 1546 1546 active clean 2020-10-09 08:30:59.011121 218'624246 218:727723 [0] 0 [0] 0 217'624240 2020-10-09 08:30:59.010908 217'624240 2020-10-06 23:57:54.087248 0 0 0 0 0 0 0 0 active clean 2020-10-09 02:16:40.314110 0'0 218:249 [0] 0 [0] 0 0'0 2020-10-09 02:16:40.313909 0'0 2020-10-05 03:37:09.108887 0 0 0 0 0 0 0 0 active clean 2020-10-09 10:25:57.831498 0'0 218:426 [0] 0 [0] 0 0'0 2020-10-09 10:25:57.830926 0'0 2020-10-02 20:46:38.411226 0 0 0 0 0 679477248 418 418 active clean 2020-10-08 14:05:10.402235 66'418 218:1093 [0] 0 [0] 0 66'418 2020-10-08 14:05:10.402109 66'418 2020-10-07 06:01:39.461254 0 0 0 0 0 18 498 498 active clean 2020-10-08 20:12:13.357134 217'498 218:1247560 [0] 0 [0] 0 217'498 2020-10-08 20:12:13.356924 217'498 2020-10-05 00:45:28.059351 0 0 0 0 0 176180 1543 1543 active clean 2020-10-08 16:42:20.137802 217'298843 218:399234 [0] 0 [0] 0 217'298843 2020-10-08 16:42:20.137431 217'298843 2020-10-06 06:30:49.343340 0 0 0 0 0 0 0 0 active clean 2020-10-09 02:53:51.921079 0'0 218:329 [0] 0 [0] 0 0'0 2020-10-09 02:53:51.920938 0'0 2020-10-02 20:04:26.034336 0 0 0 0 0 688 1 1 active clean 2020-10-08 07:23:36.310287 142'1 218:277 [0] 0 [0] 0 142'1 2020-10-08 07:23:36.310122 142'1 2020-10-08 07:23:36.310122 0 0 0 0 0 0 0 0 active clean 2020-10-08 17:49:11.537675 0'0 218:243 [0] 0 [0] 0 0'0 2020-10-08 17:49:11.537376 0'0 2020-10-06 03:19:35.732187 0 0 0 0 0 0 0 0 active clean 2020-10-09 04:31:15.247127 0'0 218:435 [0] 0 [0] 0 0'0 2020-10-09 04:31:15.246848 0'0 2020-10-06 16:48:31.939261 0 0 0 0 0 679477283 310 310 active clean 2020-10-08 20:19:21.537754 70'310 218:1686 [0] 0 [0] 0 70'310 2020-10-08 20:19:21.537612 70'310 2020-10-08 20:19:21.537612 0 0 0 0 0 4055040 90 90 active clean 2020-10-09 04:23:31.750333 218'90 218:70691 [0] 0 [0] 0 217'60 2020-10-09 04:23:31.750105 217'60 2020-10-05 08:26:56.419549 0 0 0 0 0 188433 1531 1531 active clean 2020-10-08 12:08:22.190091 217'212431 218:920082 [0] 0 [0] 0 217'212431 2020-10-08 12:08:22.189777 217'212431 2020-10-08 12:08:22.189777 0 0 0 0 0 0 0 0 active clean 2020-10-09 09:28:43.292211 0'0 218:336 [0] 0 [0] 0 0'0 2020-10-09 09:28:43.292031 0'0 2020-10-05 19:37:54.878218 0 0 0 0 0 0 0 0 active clean 2020-10-08 13:36:13.932128 0'0 218:260 [0] 0 [0] 0 0'0 2020-10-08 13:36:13.931794 0'0 2020-10-08 13:36:13.931794 0 0 0 0 0 0 0 0 active clean 2020-10-09 09:29:09.307764 0'0 218:251 [0] 0 [0] 0 0'0 2020-10-09 09:29:09.307454 0'0 2020-10-09 09:29:09.307454 0 0 0 0 0 0 0 0 active clean 2020-10-08 14:08:17.318780 0'0 218:426 [0] 0 [0] 0 0'0 2020-10-08 14:08:17.318563 0'0 2020-10-04 18:27:13.385545 0 0 0 0 0 8376320 393 393 active clean 2020-10-09 01:22:42.643154 218'393 218:1367 [0] 0 [0] 0 217'337 2020-10-09 01:22:42.642792 217'337 2020-10-03 15:53:46.935028 0 0 0 0 0 707526946 167 167 active clean 2020-10-09 03:44:52.360481 63'167 218:63972 [0] 0 [0] 0 63'167 2020-10-09 03:44:52.360344 63'167 2020-10-08 02:00:07.024073 0 0 0 0 0 12559360 1562 1562 active clean 2020-10-08 19:01:16.152927 101'318862 218:828536 [0] 0 [0] 0 101'318862 2020-10-08 19:01:16.152716 101'318862 2020-10-07 08:36:13.213465 0 0 0 0 0 0 0 0 active clean 2020-10-09 00:29:55.218759 0'0 218:335 [0] 0 [0] 0 0'0 2020-10-09 00:29:55.218326 0'0 2020-10-05 17:39:25.346109 0 0 0 0 0 333 1 1 active clean 2020-10-08 14:37:19.490217 142'1 218:274 [0] 0 [0] 0 142'1 2020-10-08 14:37:19.490131 142'1 2020-10-08 14:37:19.490131 0 0 0 0 0 0 0 0 active clean 2020-10-08 20:58:04.469121 0'0 218:245 [0] 0 [0] 0 0'0 2020-10-08 20:58:04.468788 0'0 2020-10-07 13:04:26.017451 0 0 0 0 0 0 0 0 active clean 2020-10-09 04:07:18.496808 0'0 218:443 [0] 0 [0] 0 0'0 2020-10-09 04:07:18.496599 0'0 2020-10-02 17:40:58.371033 0 0 0 0 0 4194339 378 378 active clean 2020-10-08 03:23:28.591046 217'378 218:655744 [0] 0 [0] 0 217'378 2020-10-08 03:23:28.590692 217'378 2020-10-03 01:41:55.349009 0 0 0 0 0 723589246 1587 1587 active clean 2020-10-08 09:44:38.932913 71'2487 218:50536 [0] 0 [0] 0 71'2487 2020-10-08 09:44:38.932699 71'2487 2020-10-08 09:44:38.932699 0 0 0 0 0 4382772 1542 1542 active clean 2020-10-08 22:57:48.978911 217'352542 218:808146 [0] 0 [0] 0 217'352542 2020-10-08 22:57:48.978519 217'352542 2020-10-02 10:26:44.907168 0 0 0 0 0 0 0 0 active clean 2020-10-09 02:14:50.464730 0'0 218:332 [0] 0 [0] 0 0'0 2020-10-09 02:14:50.464550 0'0 2020-10-09 02:14:50.464550 0 0 0 0 0 0 0 0 active clean 2020-10-08 11:52:18.454136 0'0 218:257 [0] 0 [0] 0 0'0 2020-10-08 11:52:18.453892 0'0 2020-10-08 11:52:18.453892 0 0 0 0 0 0 0 0 active clean 2020-10-09 10:26:00.722055 0'0 218:253 [0] 0 [0] 0 0'0 2020-10-09 10:26:00.721822 0'0 2020-10-03 21:27:39.219344 0 0 0 0 0 0 0 0 active clean 2020-10-08 10:11:17.315356 0'0 218:434 [0] 0 [0] 0 0'0 2020-10-08 10:11:17.315070 0'0 2020-10-02 06:34:46.486690 0 0 0 0 0 603979792 1186 1186 active clean 2020-10-08 15:11:27.101874 70'1186 218:528365 [0] 0 [0] 0 70'1186 2020-10-08 15:11:27.101729 70'1186 2020-10-03 06:36:46.159828 0 0 0 0 0 4194304 161 161 active clean 2020-10-08 10:50:51.658886 218'161 218:3646 [0] 0 [0] 0 217'149 2020-10-08 10:50:51.658690 217'149 2020-10-02 05:54:00.944450 0 0 0 0 0 4055552 1593 1593 active clean 2020-10-08 18:25:57.203958 217'407603 218:519423 [0] 0 [0] 0 217'407603 2020-10-08 18:25:57.203816 217'407603 2020-10-02 14:27:00.120976 0 0 0 0 0 0 0 0 active clean 2020-10-09 02:07:56.120705 0'0 218:336 [0] 0 [0] 0 0'0 2020-10-09 02:07:56.120542 0'0 2020-10-09 02:07:56.120542 0 0 0 0 0 0 0 0 active clean 2020-10-08 15:30:06.812974 0'0 218:256 [0] 0 [0] 0 0'0 2020-10-08 15:30:06.812920 0'0 2020-10-07 05:18:20.563881 0 0 0 0 0 0 0 0 active clean 2020-10-08 22:01:37.445961 0'0 218:243 [0] 0 [0] 0 0'0 2020-10-08 22:01:37.445720 0'0 2020-10-07 10:34:22.205740 0 0 0 0 0 0 0 0 active clean 2020-10-08 19:33:48.443499 0'0 218:436 [0] 0 [0] 0 0'0 2020-10-08 19:33:48.443313 0'0 2020-10-08 19:33:48.443313 0 0 0 0 0 629145635 155 155 active clean 2020-10-08 01:56:32.707423 70'155 218:731 [0] 0 [0] 0 70'155 2020-10-08 01:56:32.707293 70'155 2020-10-02 10:30:10.099163 0 0 0 0 0 4194402 218 218 active clean 2020-10-09 04:11:49.268318 217'218 218:1281 [0] 0 [0] 0 217'218 2020-10-09 04:11:49.268031 217'218 2020-10-08 02:17:00.274874 0 0 0 0 0 4386868 1519 1519 active clean 2020-10-08 07:59:17.454869 218'154320 218:739719 [0] 0 [0] 0 217'154316 2020-10-08 07:59:17.454735 217'154316 2020-10-05 14:12:12.252121 0 0 0 0 0 0 0 0 active clean 2020-10-09 02:09:26.702351 0'0 218:325 [0] 0 [0] 0 0'0 2020-10-09 02:09:26.702215 0'0 2020-10-09 02:09:26.702215 0 0 0 0 0 0 0 0 active clean 2020-10-08 04:00:02.687366 0'0 218:258 [0] 0 [0] 0 0'0 2020-10-08 04:00:02.687219 0'0 2020-10-03 06:26:57.008192 0 0 0 0 0 0 0 0 active clean 2020-10-08 02:59:38.635329 0'0 218:243 [0] 0 [0] 0 0'0 2020-10-08 02:59:38.635023 0'0 2020-10-05 09:48:19.960110 0 0 0 0 0 0 0 0 active clean 2020-10-08 10:02:27.292244 0'0 218:436 [0] 0 [0] 0 0'0 2020-10-08 10:02:27.291922 0'0 2020-10-08 10:02:27.291922 0 0 0 0 0 645922816 188 188 active clean 2020-10-09 01:29:46.645099 71'188 218:812 [0] 0 [0] 0 71'188 2020-10-09 01:29:46.644840 71'188 2020-10-09 01:29:46.644840 0 0 0 0 0 8348160 732 732 active clean 2020-10-08 08:16:14.045553 218'732 218:185914 [0] 0 [0] 0 217'695 2020-10-08 08:16:14.045314 217'695 2020-10-07 02:53:34.753245 0 0 0 0 0 4018211 1556 1556 active clean 2020-10-08 06:18:49.134949 217'394356 218:532058 [0] 0 [0] 0 217'394356 2020-10-08 06:18:49.134707 217'394356 2020-10-01 20:05:17.730157 0 0 0 0 0 0 0 0 active clean 2020-10-09 07:02:06.450455 0'0 218:329 [0] 0 [0] 0 0'0 2020-10-09 07:02:06.450175 0'0 2020-10-02 19:14:45.702111 0 0 0 0 0 0 0 0 active clean 2020-10-09 01:51:28.416654 0'0 218:261 [0] 0 [0] 0 0'0 2020-10-09 01:51:28.416441 0'0 2020-10-09 01:51:28.416441 0 0 0 0 0 0 0 0 active clean 2020-10-08 09:34:00.752395 0'0 218:243 [0] 0 [0] 0 0'0 2020-10-08 09:34:00.752151 0'0 2020-10-05 23:30:16.410891 0 0 0 0 0 0 0 0 active clean 2020-10-08 21:30:12.649567 0'0 218:437 [0] 0 [0] 0 0'0 2020-10-08 21:30:12.649340 0'0 2020-10-02 11:24:48.064097 0 0 0 0 0 612368384 287 287 active clean 2020-10-08 12:10:51.317165 70'287 218:1574 [0] 0 [0] 0 70'287 2020-10-08 12:10:51.317042 70'287 2020-10-08 12:10:51.317042 0 0 0 0 0 8388804 119 119 active clean 2020-10-08 14:37:17.781382 218'119 218:1700 [0] 0 [0] 0 218'104 2020-10-08 14:37:17.781097 218'104 2020-10-08 14:37:17.781097 0 0 0 0 0 4198400 1532 1532 active clean 2020-10-08 23:00:49.156878 217'624232 218:708960 [0] 0 [0] 0 217'624232 2020-10-08 23:00:49.156523 217'624232 2020-10-04 02:26:30.123265 0 0 0 0 0 0 0 0 active clean 2020-10-08 18:14:55.237209 0'0 218:334 [0] 0 [0] 0 0'0 2020-10-08 18:14:55.236936 0'0 2020-10-07 14:58:11.211567 0 0 0 0 0 92 2 2 active clean 2020-10-09 05:05:07.463340 142'2 218:308 [0] 0 [0] 0 142'2 2020-10-09 05:05:07.463026 142'2 2020-10-04 07:44:47.516921 0 0 0 0 0 0 0 0 active clean 2020-10-09 09:26:40.918563 0'0 218:244 [0] 0 [0] 0 0'0 2020-10-09 09:26:40.918469 0'0 2020-10-09 09:26:40.918469 0 0 0 0 0 0 1580 1580 active clean 2020-10-08 06:59:28.073175 218'82480 218:124014 [0] 0 [0] 0 218'76498 2020-10-08 06:59:28.073021 218'61986 2020-10-05 10:44:27.787918 0 0 0 0 0 0 0 0 active clean 2020-10-08 08:31:48.712338 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-08 08:31:48.712031 0'0 2020-10-07 03:44:54.397047 0 0 0 0 0 0 0 0 active clean 2020-10-08 15:54:41.163026 0'0 218:158 [0] 0 [0] 0 0'0 2020-10-08 15:54:41.162795 0'0 2020-10-07 09:26:01.113031 0 0 0 0 0 0 0 0 active clean 2020-10-08 14:59:07.063854 0'0 218:431 [0] 0 [0] 0 0'0 2020-10-08 14:59:07.063526 0'0 2020-10-07 03:33:40.865941 0 0 0 0 0 12582912 229 229 active clean 2020-10-09 07:40:35.238749 218'229 218:1358 [0] 0 [0] 0 217'199 2020-10-09 07:40:35.238655 217'199 2020-10-04 09:17:39.275859 0 0 0 0 0 637534227 155 155 active clean 2020-10-08 22:51:26.423092 70'155 218:713 [0] 0 [0] 0 70'155 2020-10-08 22:51:26.422734 70'155 2020-10-08 22:51:26.422734 0 0 0 0 0 6488243 1508 1508 active clean 2020-10-09 05:49:19.373954 218'154309 218:739752 [0] 0 [0] 0 217'154308 2020-10-09 05:49:19.373283 217'154308 2020-10-06 14:40:37.175097 0 0 0 0 0 34 11 11 active clean 2020-10-08 21:36:08.224389 213'11 218:257 [0] 0 [0] 0 213'11 2020-10-08 21:36:08.224263 213'11 2020-10-08 21:36:08.224263 0 0 0 0 0 0 1556 1556 active clean 2020-10-08 06:48:18.371502 218'96556 218:145224 [0] 0 [0] 0 218'89476 2020-10-08 06:48:18.371360 218'89476 2020-10-08 06:48:18.371360 0 0 0 0 0 0 0 0 active clean 2020-10-08 19:00:45.314686 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-08 19:00:45.314460 0'0 2020-10-07 12:15:27.025212 0 0 0 0 0 0 15 15 active clean 2020-10-09 05:58:16.090892 213'15 218:266867 [0] 0 [0] 0 213'15 2020-10-09 05:58:16.090541 213'15 2020-10-02 12:43:51.931371 0 0 0 0 0 0 0 0 active clean 2020-10-08 13:57:56.740117 0'0 218:433 [0] 0 [0] 0 0'0 2020-10-08 13:57:56.739922 0'0 2020-10-02 09:09:43.302252 0 0 0 0 0 16580096 242 242 active clean 2020-10-08 20:43:07.343247 218'242 218:218351 [0] 0 [0] 0 218'161 2020-10-08 20:43:07.342942 218'161 2020-10-02 09:23:21.553739 0 0 0 0 0 629145650 1186 1186 active clean 2020-10-08 11:07:32.897503 70'1186 218:662508 [0] 0 [0] 0 70'1186 2020-10-08 11:07:32.897338 70'1186 2020-10-01 20:00:17.716690 0 0 0 0 0 176164 1507 1507 active clean 2020-10-08 09:48:44.386416 217'407617 218:534387 [0] 0 [0] 0 217'407617 2020-10-08 09:48:44.386260 217'407617 2020-10-08 09:48:44.386260 0 0 0 0 0 0 4 4 active clean 2020-10-08 12:11:18.403767 213'4 218:252 [0] 0 [0] 0 213'4 2020-10-08 12:11:18.403584 213'4 2020-10-08 12:11:18.403584 0 0 0 0 0 0 1554 1554 active clean 2020-10-08 01:28:07.112299 218'85554 218:128593 [0] 0 [0] 0 218'78106 2020-10-08 01:28:07.112159 218'71040 2020-10-06 17:16:30.513491 0 0 0 0 0 0 0 0 active clean 2020-10-09 02:35:31.561908 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-09 02:35:31.561519 0'0 2020-10-02 15:26:03.205788 0 0 0 0 0 0 15 15 active clean 2020-10-08 23:34:53.915915 213'15 218:261068 [0] 0 [0] 0 213'15 2020-10-08 23:34:53.915555 213'15 2020-10-05 10:08:15.797193 0 0 0 0 0 0 387 387 active clean 2020-10-08 15:39:20.897579 218'387 218:392027 [0] 0 [0] 0 217'374 2020-10-08 15:39:20.897417 217'374 2020-10-07 06:39:03.564104 0 0 0 0 0 629145890 1587 1587 active clean 2020-10-08 09:26:22.092018 71'2487 218:66700 [0] 0 [0] 0 71'2487 2020-10-08 09:26:22.091911 71'2487 2020-10-02 07:45:20.134932 0 0 0 0 0 0 0 0 active clean 2020-10-08 00:48:47.232535 0'0 218:442 [0] 0 [0] 0 0'0 2020-10-08 00:48:47.232458 0'0 2020-10-03 10:17:34.079407 0 0 0 0 0 12448901 1527 1527 active clean 2020-10-08 07:39:19.650861 217'352527 218:793189 [0] 0 [0] 0 217'352527 2020-10-08 07:39:19.650602 217'352527 2020-10-02 02:37:53.666969 0 0 0 0 0 136 6 6 active clean 2020-10-08 10:31:37.257934 162'6 218:262 [0] 0 [0] 0 162'6 2020-10-08 10:31:37.257782 162'6 2020-10-03 08:19:31.836694 0 0 0 0 0 0 1588 1588 active clean 2020-10-09 09:10:21.852553 218'73888 218:111064 [0] 0 [0] 0 218'73482 2020-10-09 09:10:21.852344 218'56354 2020-10-05 15:02:49.374283 0 0 0 0 0 0 0 0 active clean 2020-10-08 10:38:52.801017 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-08 10:38:52.800708 0'0 2020-10-07 09:15:18.381402 0 0 0 0 0 0 30 30 active clean 2020-10-08 18:12:26.845373 213'30 218:537512 [0] 0 [0] 0 213'30 2020-10-08 18:12:26.845071 213'30 2020-10-05 09:46:03.133136 0 0 0 0 0 0 335 335 active clean 2020-10-08 05:05:16.823255 217'335 218:268964 [0] 0 [0] 0 217'335 2020-10-08 05:05:16.823130 217'335 2020-10-04 07:47:15.933299 0 0 0 0 0 679477248 167 167 active clean 2020-10-09 08:47:39.232720 63'167 218:48251 [0] 0 [0] 0 63'167 2020-10-09 08:47:39.232597 63'167 2020-10-04 04:23:20.525375 0 0 0 0 0 0 0 0 active clean 2020-10-09 07:39:45.124798 0'0 218:437 [0] 0 [0] 0 0'0 2020-10-09 07:39:45.124460 0'0 2020-10-06 14:37:28.122757 0 0 0 0 0 8519750 1564 1564 active clean 2020-10-09 10:26:02.645356 218'318864 218:828555 [0] 0 [0] 0 218'318864 2020-10-09 10:26:02.645216 217'318863 2020-10-06 20:12:21.375773 0 0 0 0 0 1518 11 11 active clean 2020-10-08 23:57:36.427105 213'11 218:267 [0] 0 [0] 0 213'11 2020-10-08 23:57:36.426909 213'11 2020-10-06 06:56:40.729308 0 0 0 0 0 0 1506 1506 active clean 2020-10-08 20:23:25.016955 218'105706 218:158789 [0] 0 [0] 0 218'101664 2020-10-08 20:23:25.016836 218'84154 2020-10-06 03:55:52.470750 0 0 0 0 0 0 0 0 active clean 2020-10-09 05:36:32.329904 0'0 218:147 [0] 0 [0] 0 0'0 2020-10-09 05:36:32.329666 0'0 2020-10-09 05:36:32.329666 0 0 0 0 0 0 15 15 active clean 2020-10-08 07:32:40.729876 213'15 218:261128 [0] 0 [0] 0 213'15 2020-10-08 07:32:40.729567 213'15 2020-10-06 03:35:00.785216 0 0 0 0 0 0 0 0 active clean 2020-10-09 07:26:06.064060 0'0 218:439 [0] 0 [0] 0 0'0 2020-10-09 07:26:06.063823 0'0 2020-10-04 05:06:59.242164 0 0 0 0 0 12324 74 74 active clean 2020-10-08 08:42:00.640450 218'74 218:586741 [0] 0 [0] 0 217'69 2020-10-08 08:42:00.640281 217'69 2020-10-08 08:42:00.640281 0 0 0 0 0 654311424 310 310 active clean 2020-10-09 08:06:25.412511 70'310 218:1681 [0] 0 [0] 0 70'310 2020-10-09 08:06:25.412370 70'310 2020-10-09 08:06:25.412370 0 0 0 0 0 35 1524 1524 active clean 2020-10-09 09:17:12.386608 217'212424 218:905081 [0] 0 [0] 0 217'212424 2020-10-09 09:17:12.386265 217'212424 2020-10-09 09:17:12.386265 0 0 0 0 0 558 10 10 active clean 2020-10-08 10:59:31.054899 213'10 218:268 [0] 0 [0] 0 213'10 2020-10-08 10:59:31.054717 213'10 2020-10-04 11:57:57.741507 0 0 0 0 0 0 1506 1506 active clean 2020-10-09 00:35:25.756950 218'91706 218:137757 [0] 0 [0] 0 218'89194 2020-10-09 00:35:25.756819 218'89194 2020-10-09 00:35:25.756819 0 0 0 0 0 0 0 0 active clean 2020-10-08 13:00:40.455775 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-08 13:00:40.455565 0'0 2020-10-05 20:18:37.700719 0 0 0 0 0 0 15 15 active clean 2020-10-08 01:44:33.857684 213'15 218:266769 [0] 0 [0] 0 213'15 2020-10-08 01:44:33.856247 213'15 2020-10-05 14:22:35.612337 0 0 0 0 0 0 0 0 active clean 2020-10-08 14:37:18.523328 0'0 218:430 [0] 0 [0] 0 0'0 2020-10-08 14:37:18.523219 0'0 2020-10-04 15:16:33.537922 0 0 0 0 0 98 314 314 active clean 2020-10-08 12:17:04.319742 218'314 218:1013452 [0] 0 [0] 0 217'311 2020-10-08 12:17:04.319418 217'311 2020-10-03 15:08:22.893984 0 0 0 0 0 620757056 438 438 active clean 2020-10-09 06:58:50.424828 218'438 218:1222 [0] 0 [0] 0 217'434 2020-10-09 06:58:50.424527 217'434 2020-10-09 06:58:50.424527 0 0 0 0 0 16774198 1553 1553 active clean 2020-10-09 00:59:09.998945 217'298853 218:554641 [0] 0 [0] 0 217'298853 2020-10-09 00:59:09.998692 217'298853 2020-10-04 04:20:44.069459 0 0 0 0 0 0 6 6 active clean 2020-10-09 01:15:56.969147 213'6 218:252 [0] 0 [0] 0 213'6 2020-10-09 01:15:56.968950 213'6 2020-10-05 12:10:56.851219 0 0 12438 12438 8616 8616 4972 4972 0 0 24670 24670 0 0 4 4 0 0 56 56 120 120 0 0 50876 50876 HB_PEERS PG_SUM PRIMARY_PG_SUM [] 120 120 个PG的map

# ceph pg map 2.c

osdmap e218 pg 2.c (2.c) -> up [0] acting [0] 六、pool1、查看ceph集群中的pool数量

# ceph osd lspools

输出结果：

1 images,2 volumes,3 backups,4 vms,5 k8s,6 .rgw.root,7 cephfs_data,8 cephfs_metadata,9 default.rgw.control,10 default.rgw.meta,11 default.rgw.log, 2、在ceph集群中创建一个pool

ceph osd pool create jiayuan 100 #这里的100指的是PG组 3、为一个ceph pool配置配额

ceph osd pool set-quota data max_objects 10000 4、在集群中删除一个pool

ceph osd pool delete testpool testpool --yes-i-really-really-mean-it #集群名字需要重复两次

备注：删除pool需要在配置文件ceph.conf中 [mon]添加mon allow pool delete = true并重启mon服务, 如systemctl restart ceph-mon.target

5、显示集群中pool的详细信息# rados df
8、查看data池的pg数量
# ceph osd pool get volumes pg_num
输出信息：
pg_num: 16
9、设置data池的最大存储空间为100T（默认是1T)
# ceph osd pool set data target_max_bytes 100000000000000
10、设置data池的副本数是1
ceph osd pool set jiayuan size 1
11、设置data池能接受写操作的最小副本为1
ceph osd pool set jiayuan min_size 1
12、查看集群中所有pool的副本尺寸
ceph osd dump | grep 'replicated size'
13、设置一个pool的pg数量
ceph osd pool set jiayuan pg_num 128
14、设置一个pool的pgp数量
ceph osd pool set jiayuan pgp_num 128
15、查询public与cluster是否分开，在存储节点执行以下命令：
ps aux | grep osd
输出信息：
root      87153 39.3  3.1 6846900 4132380 ?     Sl   Jan09 155212:56 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 73
root      87293  8.0  2.8 6667388 3778780 ?     Sl   Jan09 31817:50 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 67
root      87478 10.8  2.7 6425832 3557612 ?     Sl   Jan09 42731:55 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 61
root      87660  9.8  2.7 7128020 3670144 ?     Sl   Jan09 38932:51 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 55
root      87834  7.8  2.8 6864492 3739852 ?     Sl   Jan09 30920:15 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 48
root      88074  8.0  2.6 6817916 3528300 ?     Sl   Jan09 31631:47 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 43
root      88247  9.6  2.7 6984468 3565332 ?     Sl   Jan09 38053:06 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 36
root      88384  7.5  2.7 6774176 3654044 ?     Sl   Jan09 29702:47 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 30
root      88594  8.1  2.6 6961384 3539420 ?     Sl   Jan09 32002:42 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 86
root      88812  8.4  2.8 7164612 3718620 ?     Sl   Jan09 33381:07 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 78
root      88988  8.3  2.8 6862320 3704392 ?     Sl   Jan09 32853:56 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 12
root      89164  8.3  2.7 6807364 3675788 ?     Sl   Jan09 32893:14 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 6
root      89300  9.4  2.7 6323416 3638680 ?     Sl   Jan09 37184:16 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 2
root     161819  0.0  0.0  50772  6912 ?        S    14:27   0:00 /usr/bin/python /usr/local/bin/zabbix-ceph get --target osd.num.in
root     161943  0.0  0.0 112716   956 pts/0    S    14:27   0:00 grep --color=auto osd
七、rados八、rbd1、查看ceph中一个pool里的所有镜像
# rbd ls volumes
输出信息：
volume-2203c1c8-64ea-4da2-934c-011ba9f99603
volume-6a5ea26a-377b-4bee-84c2-d8b95b58ca04
volume-b3b7e767-b63b-46ab-9f24-e52496551d6f
volume-c7c6642e-186d-4153-a2e1-8fdbd422c13c
volume-d9d935b8-74cd-4d7c-8c75-9cc2ef76051a
2 、查看ceph pool中一个镜像的信息
rbd info -p volumes --image volume-2203c1c8-64ea-4da2-934c-011ba9f99603
输出信息：
rbd image 'volume-2203c1c8-64ea-4da2-934c-011ba9f99603':
        size 1GiB in 256 objects
        order 22 (4MiB objects)
        block_name_prefix: rbd_data.2238386b8b4567
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        flags: 
        create_timestamp: Fri Oct  9 09:23:02 2020
3、在test池中创建一个命名为test2的10000M的镜像
# rbd create -p volumes --size 10000 test2
备注：有些操作系统需要强制指定特性, 加上--image-feature layering ：
# rbd create -p volumes --size 10000 test2 --image-feature layering
4、删除一个镜像
# rbd rm -p volumes test2
5、调整一个镜像的尺寸
# rbd resize -p volumes --size 20000 test2
6、镜像快照的创建、查询、删除
创建：
# rbd snap create volumes/test2@snap1
查询：
# rbd snap -p volumes ls test2
输出：
SNAPID NAME     SIZE TIMESTAMP                
     5 snap1 19.5GiB Fri Oct  9 14:51:14 2020
删除：
# rbd snap rm volumes/test2@snap1
删除一个镜像文件的所有快照：
# rbd snap purge -p volumes test2
7、导出/导入镜像
导出：
# rbd export -p volumes test2 /root/test2.raw
导入：
# rbd import /root/test2.raw -p volumes --image test-import