【报错】

隔壁部门研发环境k8s集群（单master）的master上 kube-flannel 和 kube-proxy 一直pending，找我们帮忙排查。
describe 查看pod 报错说挂载问题，一个节点上所有pod挂载不上，大概率是kubelet会报错（当然也可以从journalctl命令过滤错误日志）。
kubelet 报错如下：

Redirecting to /bin/systemctl status  -l kubelet.service
● kubelet.service - kubelet: The Kubernetes Node AgentLoaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)Drop-In: /usr/lib/systemd/system/kubelet.service.d└─10-kubeadm.confActive: active (running) since 五 2021-10-08 12:35:54 CST; 53min agoDocs: https://kubernetes.io/docs/Main PID: 51270 (kubelet)Tasks: 16Memory: 48.3MCGroup: /system.slice/kubelet.service└─51270 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.110月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.379424   51270 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/secret/e2f8f8ed-6386-4f0b-b7ef-ebde580ba880-flannel-token-bfxdm\" (\"e2f8f8ed-6386-4f0b-b7ef-ebde580ba880\")" failed. No retries permitted until 2021-10-08 13:31:53.37940227 +0800 CST m=+3359.052988748 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"flannel-token-bfxdm\" (UniqueName: \"kubernetes.io/secret/e2f8f8ed-6386-4f0b-b7ef-ebde580ba880-flannel-token-bfxdm\") pod \"kube-flannel-ds-amd64-wh59m\" (UID: \"e2f8f8ed-6386-4f0b-b7ef-ebde580ba880\") : couldn't propagate object cache: timed out waiting for the condition"
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.382424   51270 reflector.go:125] object-"ingress-nginx"/"ingress-nginx-token-pnlcd": Failed to list *v1.Secret: secrets "ingress-nginx-token-pnlcd" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "ingress-nginx": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.580367   51270 reflector.go:125] object-"ingress-nginx"/"ingress-nginx-admission": Failed to list *v1.Secret: secrets "ingress-nginx-admission" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "ingress-nginx": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.780370   51270 reflector.go:125] object-"kube-system"/"flannel-token-bfxdm": Failed to list *v1.Secret: secrets "flannel-token-bfxdm" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: I1008 13:29:51.980323   51270 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.980714   51270 reflector.go:125] object-"kube-system"/"kube-flannel-cfg": Failed to list *v1.ConfigMap: configmaps "kube-flannel-cfg" is forbidden: User "system:node:XL-master" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: I1008 13:29:51.986133   51270 kubelet_node_status.go:72] Attempting to register node xl-master
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.380341   51270 reflector.go:125] object-"kube-system"/"kube-proxy-token-74txd": Failed to list *v1.Secret: secrets "kube-proxy-token-74txd" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.580340   51270 reflector.go:125] object-"kube-system"/"kube-proxy": Failed to list *v1.ConfigMap: configmaps "kube-proxy" is forbidden: User "system:node:XL-master" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.780279   51270 reflector.go:125] object-"kubesphere-monitoring-system"/"qingcloud": Failed to list *v1.Secret: secrets "qingcloud" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kubesphere-monitoring-system": no relationship found between node "XL-master" and this object

【分析】

重点是这里

User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object

如上可知 system:node:XL-master用户没有该命名空间的相关权限

配置文件
查看 /etc/kubernetes/kubelet.conf文件，kubelet 确实使用的是 system:node:XL-master用户。
具当事部门运维人员说并没有更换过用户，权限绑定也没有人删除过。好吧我们是友军，不深究了。

【解决】

创建一个clusterrolebinding给system:node:XL-master用户绑定cluster-admin权限

kubectl create clusterrolebinding 10-18-XL-master --clusterrole=cluster-admin --user=system:node:XL-master

说明：10-18-XL-master是 clusterrolebinding 的名字，随便起的。

master上启动的容器一直pending相关推荐

linux运行容器,容器与云|如何在 Windows 上运行 Linux 容器
1.概述现在能够在 Windows 10 和 Windows 服务器上运行 Docker 容器了,它是以 Ubuntu 作为宿主基础的. 想象一下,使用你喜欢的 Linux 发行版--比如 Ubun ...
centos8启动docker-mysql8容器
[README] 本文记录了 centos8 安装,启动mysql8的docker容器的步骤: [1]安装mysql8 docker容器步骤1, 查看mysql8 docker镜像版本 : 最简单的 ...
centos8上docker tomcat容器访问报404解决方法
目录 [README] [1]docker安装tomcat [2]启动多个tomcat容器 [README] 1.本文记录了访问docker tomcat容器报404的解决方法: 2.附带安装tom ...
在同一台机器上启动多个JBoss A-MQ JVM
因此,我最近没有写过博客-只是很忙. 但是,我新年的决心是写一些博客,分享一些我经常遇到的事情,即使这些简单的事情可能会帮助一些人. 因此,对于本条目,我将展示一种在单台机器上启动多个JBoss A- ...
一款跑在云上的定制容器专属 OS 来了——LifseaOS | 龙蜥技术
简介:如果可以把运维 API 化,那我们是不是可以把 OS 也作为一个 K8S 可以管理的资源,让 K8S 像管理容器一样管理OS? 引言在 2021 年 10 月的云栖大会上,为云原生而生的 OS ...
Microsoft 推出在AzureApp Service上支持Windows容器的公开预览版
微软宣布推出了在Azure App Service上支持Windows容器的公开预览版,该预览版对安装到App Service环境中的内容提供了更多的控制.这个声明扩展了容器服务的web应用程序的功能 ...
Windows Server 2016上具有Docker容器SQL Server
In this article let's see how to install a SQL Server Instance using a Docker container in Windows s ...
Linux上SQL Server 2019和Ubuntu上的Docker容器
In the first article of the series on SQL Server 2019 and Ubuntu, we prepared the virtual machine en ...
linux启动一个进程吗,你知道，当你在 Linux 上启动一个进程时会发生什么嘛？
原标题:你知道,当你在 Linux 上启动一个进程时会发生什么嘛? 本文是关于 fork 和 exec 是如何在 Unix 上工作的.你或许已经知道,也有人还不知道.几年前当我了解到这些时,我惊叹不已 ...

master上启动的容器一直pending

文章目录

【报错】

【分析】

【解决】

master上启动的容器一直pending相关推荐

最新文章

热门文章