由于docker pull image失败,导致k8s pod卡在ContainerCreating状态

由于国内有一些公有云的服务器,访问docker.io非常不稳定, 导致pull image一直是失败的,

nginx.yaml

# cat nginx.yaml
apiVersion: v1
kind: ReplicationController
metadata:name: myweb
spec:replicas: 2        selector:app: mywebtemplate:metadata:labels:app: mywebspec:containers:- name: myweb#image: registry.cn-shenzhen.aliyuncs.com/yansongda/nginx:latest  #访问稳定image: nginx  # 访问不稳定的imageports:- containerPort: 80

查看pod的events, 最后一行, 一直提示 pulling image "nginx"

# kubectl describe pod myweb-fq
Name:           myweb-fqhxm
Namespace:      default
Node:           test.novalocal/172.16.0.138
Start Time:     Thu, 09 Aug 2018 17:10:58 +0800
Labels:         app=myweb
Annotations:    <none>
Status:         Pending
IP:
Controlled By:  ReplicationController/myweb
Containers:myweb:Container ID:   Image:          nginxImage ID:       Port:           80/TCPHost Port:      0/TCPState:          WaitingReason:       ContainerCreatingReady:          FalseRestart Count:  0Environment:    <none>Mounts:/var/run/secrets/kubernetes.io/serviceaccount from default-token-s7722 (ro)
Conditions:Type           StatusInitialized    True Ready          False PodScheduled   True
Volumes:default-token-s7722:Type:        Secret (a volume populated by a Secret)SecretName:  default-token-s7722Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300snode.kubernetes.io/unreachable:NoExecute for 300s
Events:Type     Reason                 Age              From                     Message----     ------                 ----             ----                     -------Normal   Scheduled              10s              default-scheduler        Successfully assigned myweb-fqhxm to test.novalocalNormal   SuccessfulMountVolume  9s               kubelet, test.novalocal  MountVolume.SetUp succeeded for volume "default-token-s7722"Warning  MissingClusterDNS      9s (x2 over 9s)  kubelet, test.novalocal  pod: "myweb-fqhxm_default(21410469-9bb4-11e8-bdef-fa163e875cc8)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.Normal   Pulling                9s               kubelet, test.novalocal  pulling image "nginx"

在nginx镜像拉下来之前, pod一直是ContainerCreating

# kubectl get pods,svc
NAME              READY     STATUS              RESTARTS   AGE
pod/myweb-fqhxm   0/1       ContainerCreating   0          4m
pod/myweb-xn89g   0/1       ContainerCreating   0          4mNAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.222.0.1     <none>        443/TCP        1h
service/myweb        NodePort    10.222.9.108   <none>        80:30001/TCP   4m

如果使用国内的镜像

image: registry.cn-shenzhen.aliyuncs.com/yansongda/nginx:latest  #访问稳定

仍然出来 ContainerCreating , 可以查询一下kubelet的状态及日志 ,

# systemctl status kubelet.service
● kubelet.service - Kubernetes API ServerLoaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)Active: active (running) since Thu 2018-08-09 17:05:51 CST; 14min agoDocs: https://kubernetes.io/docMain PID: 21582 (kubelet)Tasks: 0Memory: 8.3MCGroup: /system.slice/kubelet.service‣ 21582 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubeconfig.yaml --logtostderr=false --log-dir=/var/log/kubernetes --v=2 --cgroup-driver=systemd --runtime-cgro
ups=/systemd/system.slice --kubelet-cgroups=/systemd/system.sliceAug 09 17:10:54 test.novalocal kubelet[21582]: E0809 17:10:54.014150   21582 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /var/lib/
docker/overlay2/a09ba0411b8d5460276094e64a6e70ce92182d027ea72c9031c20eca8096d9e2/diff with output stdout: , stderr: du: cannot access ‘/var/lib/docker/overlay2/a09ba0411b8d546
0276094e64a6e70ce92182d027ea72c9031c20eca8096d9e2/diff’: No such file or directory
Aug 09 17:10:54 test.novalocal kubelet[21582]: - exit status 1, rootInodeErr: cmd [find /var/lib/docker/overlay2/a09ba0411b8d5460276094e64a6e70ce92182d027ea72c9031c20eca8096d9
e2/diff -xdev -printf .] failed. stderr: find: ‘/var/lib/docker/overlay2/a09ba0411b8d5460276094e64a6e70ce92182d027ea72c9031c20eca8096d9e2/diff’: No such file or directory
Aug 09 17:10:54 test.novalocal kubelet[21582]: ; err: exit status 1, extraDiskErr: du command failed on /var/lib/docker/containers/49913bfa94a2725fad871fd7c389443a7d1bed26881f
96097dcaa8f183ebc57e with output stdout: , stderr: du: cannot access ‘/var/lib/docker/containers/49913bfa94a2725fad871fd7c389443a7d1bed26881f96097dcaa8f183ebc57e’: No such fil
e or directory
Aug 09 17:10:54 test.novalocal kubelet[21582]: - exit status 1
Aug 09 17:17:43 test.novalocal kubelet[21582]: E0809 17:17:43.632628   21582 kubelet_pods.go:163] Mount cannot be satisfied for container "myweb", because the volume is missin
g or the volume mounter is nil: {Name:default-token-s7722 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil>}
Aug 09 17:17:43 test.novalocal kubelet[21582]: E0809 17:17:43.632730   21582 kuberuntime_manager.go:733] container start failed: CreateContainerConfigError: cannot find volume"default-token-s7722" to mount into container "myweb"
Aug 09 17:17:43 test.novalocal kubelet[21582]: E0809 17:17:43.632864   21582 pod_workers.go:186] Error syncing pod 213ff9c6-9bb4-11e8-bdef-fa163e875cc8 ("myweb-xn89g_default(2
13ff9c6-9bb4-11e8-bdef-fa163e875cc8)"), skipping: failed to "StartContainer" for "myweb" with CreateContainerConfigError: "cannot find volume \"default-token-s7722\" to mount
into container \"myweb\""
Aug 09 17:17:48 test.novalocal kubelet[21582]: E0809 17:17:48.819683   21582 kubelet_pods.go:163] Mount cannot be satisfied for container "myweb", because the volume is missin
g or the volume mounter is nil: {Name:default-token-s7722 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil>}
Aug 09 17:17:48 test.novalocal kubelet[21582]: E0809 17:17:48.819880   21582 kuberuntime_manager.go:733] container start failed: CreateContainerConfigError: cannot find volume"default-token-s7722" to mount into container "myweb"
Aug 09 17:17:48 test.novalocal kubelet[21582]: E0809 17:17:48.819968   21582 pod_workers.go:186] Error syncing pod 21410469-9bb4-11e8-bdef-fa163e875cc8 ("myweb-fqhxm_default(2
1410469-9bb4-11e8-bdef-fa163e875cc8)"), skipping: failed to "StartContainer" for "myweb" with CreateContainerConfigError: "cannot find volume \"default-token-s7722\" to mount
into container \"myweb\""

然而一般情况下, kubelet的状态是这样的

# systemctl status kubelet.service
● kubelet.service - Kubernetes API ServerLoaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)Active: active (running) since Thu 2018-08-09 17:22:19 CST; 20s agoDocs: https://kubernetes.io/docMain PID: 25756 (kubelet)Tasks: 0Memory: 26.4MCGroup: /system.slice/kubelet.service‣ 25756 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubeconfig.yaml --logtostderr=false --log-dir=/var/log/kubernetes --v=2 --cgroup-driver=systemd --runtime-c...Aug 09 17:22:19 test.novalocal systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Aug 09 17:22:19 test.novalocal systemd[1]: Unit kubelet.service entered failed state.
Aug 09 17:22:19 test.novalocal systemd[1]: kubelet.service failed.
Aug 09 17:22:19 test.novalocal systemd[1]: Started Kubernetes API Server.
Aug 09 17:22:19 test.novalocal systemd[1]: Starting Kubernetes API Server...
Aug 09 17:22:19 test.novalocal kubelet[25756]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet...formation.
Aug 09 17:22:19 test.novalocal kubelet[25756]: Flag --kubelet-cgroups has been deprecated, This parameter should be set via the config file specified by the Kubel...formation.
Aug 09 17:22:19 test.novalocal kubelet[25756]: E0809 17:22:19.782359   25756 kubelet.go:1282] Image garbage collection failed once. Stats initialization may not h...ontainer /
Aug 09 17:22:22 test.novalocal kubelet[25756]: Starting Device Plugin manager
Hint: Some lines were ellipsized, use -l to show in full.

最近在看了一部电影,道恩·强森主演的 <摩天营救>, 里面有一句经典台词 "重启试试"

systemctl restart kubelet.service

小生文笔不好, 只做了一下记录 :-)

由于docker pull image失败,导致k8s pod卡在ContainerCreating状态相关推荐

排查Pod卡在Terminating状态
排查Pod卡在Terminating状态参考: 终极好文章 https://www.jianshu.com/p/817f11912253 https://www.cnblogs.com/rongfe ...
kubernetes创建yaml，pod服务一直处于 ContainerCreating状态的原因查找与解决
最近刚刚入手研究kubernetes,运行容器的时候,发现一直处于ContainerCreating状态,悲了个催,刚入手就遇到了点麻烦,下面来讲讲如何查找问题及解决的运行容器命令: kubectl ...
docker pull 镜像失败
1.question 1 ERROR: Get https://*/v2/: http: server gave HTTP response to HTTPS client 配置 /etc/hosts ...
docker pull报错：Timeout exceeded while awaiting headers解决思路
问题使用fluntd的时候,docker pull镜像失败,原因是镜像在墙外 $docker pull k8s.gcr.io/fluentd-elasticsearch:1.20 Error res ...
docker pull拉取镜像_docker拉取镜像失败解决
今天在虚拟机上安装了docker-ce之后,从docker官网上去拉取镜像时,报如下错误 [root@localhost util-linux-2.27]# docker pull mysql:8.0 ...
k8s 拉取镜像失败_[k8s问题笔录] 记一次 kubectl 部署时，pull image 报错，拉取不到镜像...
部署一个简单的 demo 返现一个很奇妙的错误,yaml 文件内容在下方,镜像仓库是内网自建的 harbor-1.9.1 单节点仓库,测试就没有整高可用了. 用户名:admin 密码:admin123 ...
k8s pod 详解
https://www.cnblogs.com/kevingrace/p/11309409.html 一.什么是Pod kubernetes中的一切都可以理解为是一种资源对象,pod,rc,servi ...
K8S --- Pod 基础概念
文章目录 1. 资源限制 2. Pod 的两种使用方式 3. Pod 资源共享 4. 底层容器 Pause 4.1 Pause共享资源 4.1.1 网络 4.1.2 存储 4.1.3 总结 4.2 P ...
k8s pod详细讲解
文章目录一.pod介绍 1.结构 2.定义二.pod生命周期 1.创建和终止 2.初始化容器 3.钩子函数 4.容器探测 5.重启策略三.Pod调度 1.定向调度 2.亲和性调度 3.污点和容忍 ...

由于docker pull image失败,导致k8s pod卡在ContainerCreating状态

由于docker pull image失败,导致k8s pod卡在ContainerCreating状态相关推荐

最新文章

热门文章