历史原因分析

问题一 ceph yaml文件执行失败

• 1.25后弃用PodSecurityPolicy

resource mapping not found for name: "rbd-csi-provisioner-psp" namespace: "ceph-csi" from "csi-provisioner-psp.yaml": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"

问题二 image拉取失败

• 1.20开始自研CRI，但是可以用docker-cri兼容
• 1.25后完全放弃docker，强制使用crictl工具对image、container进行处理

root@k8s-master01-41:/data/yaml/prd/csi-rbd# kubectl get  pod -n ceph-csi 
NAME                                         READY   STATUS             RESTARTS   AGE
csi-rbdplugin-5wxsn                          0/3     ErrImagePull       0          6m11s
csi-rbdplugin-6gz2l                          0/3     ErrImagePull       0          6m11s
csi-rbdplugin-kql8l                          0/3     ImagePullBackOff   0          6m11s
csi-rbdplugin-provisioner-548f4bd698-5xqpx   0/7     ImagePullBackOff   0          6m11s
csi-rbdplugin-provisioner-548f4bd698-78lsp   0/7     ImagePullBackOff   0          6m11s
csi-rbdplugin-provisioner-548f4bd698-pdrk2   0/7     ErrImagePull       0          6m11s

解决

问题一

• 删除yaml中的PodSecurityPolicy配即可

问题二

• 使用crictl组件拉取harbor的image
• 修改所有节点，配置私有仓库地址

## vim /etc/containerd/config.toml 末尾添加
cat  >>  /etc/containerd/config.toml  << EOF

#cri私有仓库
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.168.1.30"]
          endpoint = ["http://10.168.1.30"]
          [plugins."io.containerd.grpc.v1.cri".registry.configs."10.168.1.30".tls]
            insecure_skip_verify = true # 是否跳过证书认证
           # ca_file = "/etc/containerd/www.myharbor.com/ca.crt" # CA 证书
           # cert_file = "/etc/containerd/www.myharbor.com/www.myharbor.com.crt" # harbor 证书
           # key_file = "/etc/containerd/www.myharbor.com/www.myharbor.com.key" # harbor 私钥 
          [plugins."io.containerd.grpc.v1.cri".registry.configs."10.168.1.30".auth]
            username = "admin" # 在harbor里单独创建的用户，授权访问指定项目
            password = "admin"
EOF

• reload

systemctl daemon-reload && systemctl restart containerd.service

• 查看ceph资源状态

root@k8s-master01-41:/data/yaml/prd/csi-rbd# kubectl get po -n ceph-csi 
NAME                                         READY   STATUS    RESTARTS   AGE
csi-rbdplugin-gvph8                          3/3     Running   0          55s
csi-rbdplugin-m7b2d                          3/3     Running   0          55s
csi-rbdplugin-pgxk4                          3/3     Running   0          55s
csi-rbdplugin-provisioner-548f4bd698-dkpw4   7/7     Running   0          55s
csi-rbdplugin-provisioner-548f4bd698-dxpgr   7/7     Running   0          55s
csi-rbdplugin-provisioner-548f4bd698-rt7bh   7/7     Running   0          55s

root@k8s-master01-41:/data/yaml/prd/csi-rbd# kubectl  get  pv,pvc,pod |grep test
persistentvolume/pvc-cf3d7eb7-4b98-49ac-83ae-abd948edb379   10Gi       RWO            Retain           Bound    default/ceph-rbd-test   csi-rbd-sc              14m
persistentvolumeclaim/ceph-rbd-test   Bound    pvc-cf3d7eb7-4b98-49ac-83ae-abd948edb379   10Gi       RWO            csi-rbd-sc     14m
pod/ceph-rbd-test                        1/1     Running   0              14m

附

• pvc回滚失败

https://zhuanlan.zhihu.com/p/68601257

• 强制终止pvc的pod

删除顺序 pod>pvc>pv
如果删除PV则不影响已挂载的pod但影响新建的pod挂载
删除PVC需要看回收策略是否影响数据持久化

kubectl delete po ceph-rbd-test --force --grace-period=0

• ceph rbd方式供给的是块设备 类似于数据库的RAC 共享存储池 在ceph后台无法打开目录查看文件