文章目录
A、pve 关于bcache的BUG现状
# pve7到目前为止,不能正常的通过web gui 创建基于bcache的设备作为osd存储设备,如下是bug完成状态记录
https://bugzilla.proxmox.com/show_bug.cgi?id=4679
# 创建osd(不行,提示没有设备)
pveceph osd create /dev/bcache0
pveceph osd create /dev/bcache1
pveceph osd create /dev/bcache2
root@2ceph01:~# pveceph osd create /dev/bcache0
unable to get device info for '/dev/bcache0'
# 这篇文章说了,ceph-volume lvm prepare 这样不行,可以用 ceph-volume lvm create 一步到位
https://forum.proxmox.com/threads/pve-gui-doesnt-recognize-kernel-bcache-device.109761/
B、pve安装bcache-tools
# 1、安装 bcache 管理工具,
apt install bcache-tools
# 2、查看内核
modprobe bcache
lsmod | grep bcache #当然创建了bcache设备后,开机会自动加载bcache内核模块
# 3、移植suse linux 的 bcache 、bcache-status 工具
wget https://qiniu.wsfnk.com/bokefiles/bcache -P /usr/sbin/ ; chmod +x /usr/sbin/bcache
wget https://qiniu.wsfnk.com/bokefiles/bcache-status -P /usr/sbin/ ; chmod +x /usr/sbin/bcache-status
# 注意,bcache 是二进制程序,bcache-status 是纯python3脚本,若是bcache无法正常运行,需要用 ldd 检查运行库依赖。
root@3-ceph15-1:~# ldd /usr/sbin/bcache
linux-vdso.so.1 (0x00007ffea1743000)
libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007fd2e881e000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007fd2e8815000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd2e8641000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd2e8887000)
推荐文档
https://support.huawei.com/enterprise/zh/doc/EDOC1100284237/c067e9ff
https://support.huawei.com/enterprise/zh/doc/EDOC1100180377/5510fed8
I、pve上使用bcache的实践经验
1、硬件准备
1、一定要保证充足的内存,bcache 对内存的消耗很大。
2、cache device 一定要带超级电容,且要求存储一定数据后,不掉速
1、硬件规划,ceph节点上有“1个1.92TiB 带超级电容的企业级SSD”、“3个普通的10T HDD”
# 将1.92T的SSD ,划分成4个分区,
其中3个分区,每个分区200G,用于作为后端普通HDD的cache device;
最后一个分区1.2TiB,用于存放osd的db和wal
parted -s /dev/vde mklabel gpt
2、一键创建backing 和 cache,并指定 block 为4KB(该值匹配backing device 的扇区大小,HDD一般是512B,SSD一般是4KB),bucket 为2MB(该值匹配cache device 的erase block size)
# 如果是一个4k sector的HDD和一个擦除block大小是2MB的SSD搭配
#make-bcache --wipe-bcache --writeback --block 4K --bucket 2M -C /dev/vde -B /dev/vdb /dev/vdc /dev/vdd
make-bcache --wipe-bcache --writeback --block 4K --bucket 2M -C /dev/vde1 -B /dev/vdb
make-bcache --wipe-bcache --writeback --block 4K --bucket 2M -C /dev/vde2 -B /dev/vdc
make-bcache --wipe-bcache --writeback --block 4K --bucket 2M -C /dev/vde3 -B /dev/vdd
# ceph-volume lvm create --bluestore 命令可用参数
--data /dev/bcache0 # 指定osd 数据存放路径
--block.db /dev/nvme0n1 # 指定osd db 和 wal 存放路径
--block.db-size=10G # 指定db的大小
--osd-id 8 # 指定osd-id 创建osd
--crush-device-class hdd # 指定osd类型创建osd
# 要做pve环境下,用这个命令,必须先用pveceph 创建一个osd,添加下ceph的keying ,然后您销毁该osd
# 或者 ln -s /var/lib/ceph/bootstrap-osd/ceph.keyring /etc/pve/priv/ceph.client.bootstrap-osd.keyring 执行这个应该也可以
# 需要注意的是,使用了bcache 后,需要注意的点
# 1、创建osd时您最好手动指定class type,不然都会设置成ssd
# 2、再怎么节约都要在 高速磁盘上 划分一个用于DB、WAL的专用空间,防止 bcache的错误策略导致,性能降低
# ceph-volume lvm create --bluestore --data /dev/bcache0 这种使用bcache 而不使用 独立非 bcache 的 db、wal的 坚决不推荐
ceph-volume lvm create --bluestore --data /dev/bcache0 --block.db /dev/nvme0n1
ceph-volume lvm create --bluestore --data /dev/bcache0 --block.db /dev/nvme0n1 --block.db-size=10G
ceph-volume lvm create --bluestore --data /dev/bcache0 --osd-id 8 --crush-device-class hdd --block.db /dev/nvme0n1 --block.db-size=10G
ceph-volume lvm activate --bluestore 0 2f6b54af-aec8-414e-a231-3cce47249463 这是单独激活的命令
3、创建osd(使用bcache的osd默认class type为ssd)
ceph-volume lvm create --bluestore --data /dev/bcache0 --block.db /dev/vde4 --block.db-size=30G
ceph-volume lvm create --bluestore --data /dev/bcache1 --block.db /dev/vde4 --block.db-size=30G
ceph-volume lvm create --bluestore --data /dev/bcache2 --block.db /dev/vde4 --block.db-size=30G
4、优化参数
echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
echo 0 > /sys/block/bcache1/bcache/sequential_cutoff
echo 0 > /sys/block/bcache2/bcache/sequential_cutoff
M、CEPH如何停用Bcache
1、停止OSD
2、解绑Bcache,不解绑当你再次创建osd时,可能会报错 --> RuntimeError: Device /dev/bcache0 has a filesystem.
3、观察Bcache缓存回写完为0时启用OSD
N、使用bcache与未使用bcache的性能对比
# 测试指令
rados bench -p volumes 60 write --no-cleanup
#测试完 清空脏数据
rados purge volumes --yes-i-really-really-mean-it
三个ceph盘,没有db 三个ceph盘,有db 三个ceph盘,有bcache,有db,wirteback 三个ceph盘,有bcache,有db,wirteback优化了顺序io
# 第一次测试数据
Total time run: 61.2611 Total time run: 61.5114 Total time run: 60.6557 Total time run: 60.7928
Total writes made: 498 Total writes made: 603 Total writes made: 905 Total writes made: 1254
Write size: 4194304 Write size: 4194304 Write size: 4194304 Write size: 4194304
Object size: 4194304 Object size: 4194304 Object size: 4194304 Object size: 4194304
Bandwidth (MB/sec): 32.5165 Bandwidth (MB/sec): 39.2123 Bandwidth (MB/sec): 59.6811 Bandwidth (MB/sec): 82.5097
Stddev Bandwidth: 10.978 Stddev Bandwidth: 8.70601 Stddev Bandwidth: 19.7928 Stddev Bandwidth: 20.6992
Max bandwidth (MB/sec): 56 Max bandwidth (MB/sec): 56 Max bandwidth (MB/sec): 100 Max bandwidth (MB/sec): 124
Min bandwidth (MB/sec): 0 Min bandwidth (MB/sec): 16 Min bandwidth (MB/sec): 0 Min bandwidth (MB/sec): 28
Average IOPS: 8 Average IOPS: 9 Average IOPS: 14 Average IOPS: 20
Stddev IOPS: 2.74449 Stddev IOPS: 2.1765 Stddev IOPS: 4.94821 Stddev IOPS: 5.1748
Max IOPS: 14 Max IOPS: 14 Max IOPS: 25 Max IOPS: 31
Min IOPS: 0 Min IOPS: 4 Min IOPS: 0 Min IOPS: 7
Average Latency(s): 1.94895 Average Latency(s): 1.61558 Average Latency(s): 1.06966 Average Latency(s): 0.774093
Stddev Latency(s): 1.16162 Stddev Latency(s): 1.09933 Stddev Latency(s): 0.370768 Stddev Latency(s): 0.321407
Max latency(s): 6.37118 Max latency(s): 6.87208 Max latency(s): 2.91389 Max latency(s): 1.88844
Min latency(s): 0.275261 Min latency(s): 0.204856 Min latency(s): 0.277915 Min latency(s): 0.0656808
# 第二次测试数据
Total time run: 61.3131 Total time run: 61.1041 Total time run: 60.551 Total time run: 60.2961
Total writes made: 554 Total writes made: 646 Total writes made: 721 Total writes made: 1248
Write size: 4194304 Write size: 4194304 Write size: 4194304 Write size: 4194304
Object size: 4194304 Object size: 4194304 Object size: 4194304 Object size: 4194304
Bandwidth (MB/sec): 36.1424 Bandwidth (MB/sec): 42.2885 Bandwidth (MB/sec): 47.6293 Bandwidth (MB/sec): 82.7914
Stddev Bandwidth: 10.9664 Stddev Bandwidth: 10.4169 Stddev Bandwidth: 17.8813 Stddev Bandwidth: 18.4918
Max bandwidth (MB/sec): 56 Max bandwidth (MB/sec): 64 Max bandwidth (MB/sec): 84 Max bandwidth (MB/sec): 132
Min bandwidth (MB/sec): 4 Min bandwidth (MB/sec): 8 Min bandwidth (MB/sec): 0 Min bandwidth (MB/sec): 36
Average IOPS: 9 Average IOPS: 10 Average IOPS: 11 Average IOPS: 20
Stddev IOPS: 2.7416 Stddev IOPS: 2.60422 Stddev IOPS: 4.47668 Stddev IOPS: 4.62296
Max IOPS: 14 Max IOPS: 16 Max IOPS: 21 Max IOPS: 33
Min IOPS: 1 Min IOPS: 2 Min IOPS: 0 Min IOPS: 9
Average Latency(s): 1.75992 Average Latency(s): 1.50196 Average Latency(s): 1.34148 Average Latency(s): 0.771946
Stddev Latency(s): 1.02127 Stddev Latency(s): 0.873313 Stddev Latency(s): 0.527519 Stddev Latency(s): 0.297788
Max latency(s): 7.92322 Max latency(s): 5.53811 Max latency(s): 3.18072 Max latency(s): 1.60706
Min latency(s): 0.286233 Min latency(s): 0.233545 Min latency(s): 0.230318 Min latency(s): 0.0546931
如果文章对你有帮助,欢迎点击上方按钮打赏作者
暂无评论