【K8S】READY:0/1 STATUS:Pending 获取POD状态异常

1.calico 网络失败

kubectl get pod --all-namespaces -o wide
NAMESPACE     NAME                                       READY   STATUS     RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-554647c955-8sxdh   0/1     Pending    0          4m    <none>         <none>   <none>           <none>
kube-system   calico-node-6mqkf                          0/1     Init:0/2   0          4m    192.168.1.15   k8s4     <none>           <none>
kube-system   calico-node-7bhwx                          0/1     Init:0/2   0          4m    192.168.1.17   k8s6     <none>           <none>
kube-system   calico-node-llwt6                          0/1     Init:0/2   0          4m    192.168.1.12   k8s1     <none>           <none>
kube-system   calico-node-mcb4c                          0/1     Init:0/2   0          4m    192.168.1.16   k8s5     <none>           <none>
kube-system   calico-node-mkdwr                          0/1     Init:0/2   0          4m    192.168.1.14   k8s3     <none>           <none>
kube-system   calico-node-xn2jz                          0/1     Init:0/2   0          4m    192.168.1.13   k8s2     <none>           <none>
kube-system   calico-typha-6454f6cfd7-j7729              0/1     Pending    0          4m    <none>         <none>   <none>           <none>

[root@K8S1 work]# kubectl get pod -A
NAMESPACE     NAME                                       READY   STATUS     RESTARTS   AGE
kube-system   calico-kube-controllers-554647c955-p7bjc   0/1     Pending    0          4m36s
kube-system   calico-node-j7825                          0/1     Init:0/2   0          4m36s
kube-system   calico-node-ktmf5                          0/1     Init:0/2   0          4m36s
kube-system   calico-node-nf5wg                          0/1     Init:0/2   0          4m36s
kube-system   calico-node-rwpfq                          0/1     Init:0/2   0          4m36s
kube-system   calico-node-scm7s                          0/1     Init:0/2   0          4m36s
kube-system   calico-node-tpqbl                          0/1     Init:0/2   0          4m36s
kube-system   calico-typha-6454f6cfd7-z5bvk              0/1     Pending    0          4m36s
  

 2.查看状态 

 [root@K8S1 work]# kubectl describe pods calico-node-rwpfq --namespace kube-system 
Name:                 calico-node-rwpfq
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 k8s3/192.168.1.14
Start Time:           Sat, 20 Jul 2024 21:58:45 +0800
Labels:               controller-revision-hash=6cdbc66bf4
                      k8s-app=calico-node
                      pod-template-generation=1
Annotations:          <none>
Status:               Pending
IP:                   192.168.1.14
IPs:
  IP:           192.168.1.14
Controlled By:  DaemonSet/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  
    Image:         192.168.1.11:443/myharbor/cni:v3.22.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-54hkd (ro)
  install-cni:
    Container ID:  
    Image:         192.168.1.11:443/myharbor/cni:v3.22.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/install
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:               <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-54hkd (ro)
Containers:
  calico-node:
    Container ID:   
    Image:          192.168.1.11:443/myharbor/node:v3.22.0
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=10s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      DATASTORE_TYPE:                     kubernetes
      FELIX_TYPHAK8SSERVICENAME:          <set to the key 'typha_service_name' of config map 'calico-config'>  Optional: false
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP_AUTODETECTION_METHOD:            interface=ens.*
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      CALICO_IPV6POOL_VXLAN:              Never
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_VXLANMTU:                     <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_WIREGUARDMTU:                 <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_IPV4POOL_CIDR:               172.16.0.0/16
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_HEALTHENABLED:                true
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /sys/fs/ from sysfs (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/log/calico/cni from cni-log-dir (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-54hkd (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:  
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:  
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  sysfs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/
    HostPathType:  DirectoryOrCreate
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  cni-log-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/calico/cni
    HostPathType:  
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:  
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  kube-api-access-54hkd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               6m22s                 default-scheduler  Successfully assigned kube-system/calico-node-rwpfq to k8s3
  Warning  FailedCreatePodSandBox  62s (x26 over 6m21s)  kubelet            Failed to create pod sandbox: rpc error: code = NotFound desc = failed to get sandbox image "192.168.1.11:443/myharbor/pause:3.6": failed to pull image "192.168.1.11:443/myharbor/pause:3.6": failed to pull and unpack image "192.168.1.11:443/myharbor/pause:3.6": failed to resolve reference "192.168.1.11:443/myharbor/pause:3.6": 192.168.1.11:443/myharbor/pause:3.6: not found

  
 --找到原因:无法获取镜像。
  failed to get sandbox image "192.168.1.11:443/myharbor/pause:3.6"

3. 检查镜像 

[root@K8S1 soft]# docker images
REPOSITORY                                     TAG       IMAGE ID       CREATED       SIZE
192.168.1.11:443/myharbor/typha                v3.22.0   35c2fa1ee0a1   2 years ago   127MB
192.168.1.11:443/myharbor/kube-controllers     v3.22.0   df76d42861ee   2 years ago   132MB
192.168.1.11:443/myharbor/cni                  v3.22.0   f86797de8afd   2 years ago   236MB
192.168.1.11:443/myharbor/pod2daemon-flexvol   v3.22.0   59daef946c8c   2 years ago   21.4MB
192.168.1.11:443/myharbor/node                 v3.22.0   f109b1742d34   2 years ago   213MB

--果然没有pause镜像 ,导入镜像。
docker load < docker-pause_3.6.tar
...................
[root@K8S1 soft]# docker images
REPOSITORY                                     TAG       IMAGE ID       CREATED       SIZE
192.168.1.11:443/myharbor/typha                v3.22.0   35c2fa1ee0a1   2 years ago   127MB
192.168.1.11:443/myharbor/kube-controllers     v3.22.0   df76d42861ee   2 years ago   132MB
192.168.1.11:443/myharbor/cni                  v3.22.0   f86797de8afd   2 years ago   236MB
192.168.1.11:443/myharbor/pod2daemon-flexvol   v3.22.0   59daef946c8c   2 years ago   21.4MB
192.168.1.11:443/myharbor/node                 v3.22.0   f109b1742d34   2 years ago   213MB
192.168.1.11:443/myharbor/pause                3.6       6270bb605e12   2 years ago   683kB

4.重建calico网络

[root@K8S1 work]# kubectl delete -f /data/k8s/work/calico.yaml
configmap "calico-config" deleted
customresourcedefinition.apiextensions.k8s.io "bgpconfigurations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "bgppeers.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "blockaffinities.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "caliconodestatuses.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "clusterinformations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "felixconfigurations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "globalnetworkpolicies.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "globalnetworksets.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "hostendpoints.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipamblocks.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipamconfigs.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipamhandles.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ippools.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipreservations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "kubecontrollersconfigurations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "networkpolicies.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "networksets.crd.projectcalico.org" deleted
clusterrole.rbac.authorization.k8s.io "calico-kube-controllers" deleted
clusterrolebinding.rbac.authorization.k8s.io "calico-kube-controllers" deleted
clusterrole.rbac.authorization.k8s.io "calico-node" deleted
clusterrolebinding.rbac.authorization.k8s.io "calico-node" deleted
service "calico-typha" deleted
deployment.apps "calico-typha" deleted
poddisruptionbudget.policy "calico-typha" deleted
daemonset.apps "calico-node" deleted
serviceaccount "calico-node" deleted
deployment.apps "calico-kube-controllers" deleted
serviceaccount "calico-kube-controllers" deleted
poddisruptionbudget.policy "calico-kube-controllers" deleted

[root@K8S1 work]# kubectl apply -f /data/k8s/work/calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
service/calico-typha created
deployment.apps/calico-typha created
poddisruptionbudget.policy/calico-typha created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created

5.检查POD 

[root@K8S1 work]# kubectl get pod -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-554647c955-rb5fs   1/1     Running   0          33s
kube-system   calico-node-7zpwg                          1/1     Running   0          33s
kube-system   calico-node-9qrhz                          1/1     Running   0          33s
kube-system   calico-node-bvjnh                          1/1     Running   0          33s
kube-system   calico-node-vtnsr                          1/1     Running   0          33s
kube-system   calico-node-vvkdz                          1/1     Running   0          33s
kube-system   calico-node-zkxkq                          1/1     Running   0          33s
kube-system   calico-typha-6454f6cfd7-shs92              1/1     Running   0          33s

--果然calico 网络部署成功。

<think>好的,用户问的是如何诊断Kubernetes集群中CoreDNS Pod处于Pending状态的原因。我需要根据提供的引用资料和系统级指令来组织回答。首先,要确保回答结构清晰,分步骤解决问题,并且引用正确的来源。 首先,Pending状态通常和调度问题有关,根据引用[4],调度失败会导致Pod处于Pending,直到重新调度。可能需要检查资源是否足够,节点是否符合条件。另外,引用[3]提到CNI插件配置不正确会导致节点NotReady,这也会影响Pod调度。所以需要检查Node状态和CNI插件。 接下来,我应该按步骤列出可能的原因和解决方法。比如第一步检查Node状态,是否NotReady,导致无法调度。第二步检查资源配额,是否有足够CPU和内存。第三步检查污点和容忍,确保CoreDNS的Pod有合适的容忍配置。第四步检查存储卷,如果有使用持久卷的话。第五步检查网络插件,即CNI是否正确安装,因为引用[3]指出CNI问题会导致节点NotReady。最后,查看事件日志,kubectl describe pod可能会有更多信息。 还要生成相关问题,比如如何查看节点状态,如何排除CNI问题,如何设置优先级避免Pending。这些需要根据引用资料中的内容来生成,比如引用[4]提到的优先级和抢占机制。 需要确保所有数学公式用$...$,但这里可能用不上。回答要使用中文,结构清晰,分步骤说明,引用对应的引用[1]到[4]。同时检查是否符合用户不可见的系统指令,比如引用标识的位置,是否正确使用LaTeX等,不过这里可能更多是Kubernetes命令和配置的问题,不涉及数学公式。 最后,确保相关问题是基于用户的问题扩展的,比如诊断步骤之后可能涉及的其他操作或配置,如查看事件、调整资源限制等。这样用户可以得到全面的帮助。</think>### Kubernetes CoreDNS Pod 处于 Pending 状态诊断步骤 #### 1. **检查节点(Node)状态** - **节点资源不足**:若集群节点资源(CPU、内存)不足,CoreDNS Pod 可能无法调度。使用以下命令查看节点资源状态: ```bash kubectl describe nodes ``` 查看 `Allocatable` 和 `Allocated` 资源是否满足 CoreDNS 的要求[^2][^4]。 - **节点未就绪**:若节点处于 `NotReady` 状态Pod 无法调度。检查节点的 CNI 插件是否正确配置,因为 `kubelet` 需要 CNI 插件通过自检才能标记节点为就绪状态[^3]。 ```bash kubectl get nodes ``` #### 2. **检查 Pod 的资源配额** - CoreDNS 的 Pod 可能因命名空间资源配额(ResourceQuota)限制而无法调度。检查相关命名空间的配额: ```bash kubectl describe resourcequotas -n kube-system ``` #### 3. **检查污点(Taint)与容忍(Toleration)** - 部分节点可能设置了污点(如 `NoSchedule`),而 CoreDNS 的 Pod 未配置对应的容忍。检查节点的污点配置: ```bash kubectl describe node <节点名称> | grep Taint ``` 确保 CoreDNS 的 Deployment 或 DaemonSet 中定义了匹配的 `tolerations`[^4]。 #### 4. **检查存储卷(PersistentVolume)问题** - 若 CoreDNS 配置了持久化存储(如 `PersistentVolumeClaim`),需确保存储卷已正确绑定且可用: ```bash kubectl get pvc -n kube-system ``` #### 5. **检查网络插件(CNI)状态** - 节点上的 CNI 插件未正确安装或配置会导致 Pod 无法启动。检查节点日志: ```bash journalctl -u kubelet | grep "CNI" ``` 确保 CNI 配置文件(如 `/etc/cni/net.d/`)存在且有效[^3]。 #### 6. **查看 Pod 事件日志** - 通过 `kubectl describe` 获取 CoreDNS Pod 的详细事件: ```bash kubectl describe pod <coredns-pod名称> -n kube-system ``` 重点关注 `Events` 部分,例如: - `FailedScheduling`: 调度失败的具体原因(如资源不足、节点不匹配)。 - `NetworkPluginNotReady`: 网络插件未就绪。 --- ### 示例操作:诊断 CNI 问题 ```bash # 检查节点状态 kubectl get nodes # 查看节点污点 kubectl describe node worker-node-1 | grep Taint # 查看 CoreDNS Pod 事件 kubectl describe pod coredns-5d78c9869d-abcde -n kube-system # 检查 CNI 插件日志(以 Calico 为例) kubectl logs -n calico-system calico-node-xyz ``` ---
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值