安装前准备

virtualbox安装三个Ubuntu虚拟机

k8s-master 和 k8s-slave1、k8s-slave2 为了上网，都是双网卡（NAT和Host-only）

关闭防火墙

sudo ufw disable

关闭swap

sudo swapoff -a

安装Docker

卸载旧docker

sudo apt-get remove docker docker-engine docker.io

彻底删除docker-ce和docker-cli

sudo apt-get autoremove docker-ce

安装依赖

sudo apt-get install \apt-transport-https \ca-certificates \curl \software-properties-common

添加docker的gpg key

为了使用https访问，使用阿里云加速

curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -

设置docker镜像源

sudo add-apt-repository \"deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu \$(lsb_release -cs) \stable"

更新源

sudo apt update && sudo apt-get update

安装docker 18.06

查看docker-ce版本

apt-cache madison docker-ce

安装指定版本

sudo apt-get install -y docker-ce=18.06.3~ce~3-0~ubuntu

如果因为libltdl版本低，可以安装新版本 wget http://launchpadlibrarian.net/236916213/libltdl7_2.4.6-0.1_amd64.deb && sudo dkpg -i libltdl7_2.4.6-0.1_amd64.deb

默认启动

sudo systemctl enable docker && sudo systemctl start docker

修改docker启动参数，进行加速

$ sudo tee /etc/docker/daemon.json <<-'EOF'
{"registry-mirrors": ["https://xxxxxxxx.mirror.aliyuncs.com"]
}
EOF

将xxxxxx替换为在阿里云申请的路径

安装kubernetes

更新apt源

sudo apt-get update && sudo apt-get install -y apt-transport-https curlsudo curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -sudo tee /etc/apt/sources.list.d/kubernetes.list <<-'EOF'
deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main
EOFsudo apt-get update

安装kubectl kubeadm kubelet

在master和node都安装

sudo apt-get install -y kubelet=1.12.8-00 kubeadm=1.12.8-00 kubectl=1.12.8-00
sudo apt-mark hold kubelet=1.12.8-00 kubeadm=1.12.8-00 kubectl=1.12.8-00

启动kubelet

sudo systemctl enable kubelet && sudo systemctl start kubelet

初始化集群

sudo kubeadm init --apiserver-advertise-address 192.168.56.112 --pod-network-cidr=10.244.0.0/16 --kubernetes-version=1.12.8

安装网络插件

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

加入node

sudo kubeadm join 192.168.56.112:6443 --token xxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxx

将之前kubeadm init后，输出的加入node指令拷贝到node执行。

遇到的问题

无法启动kubelet

journalctl -u kubelet 看到报错： failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/ku 这个不影响，不用关系，启动了kubernetes后就会正常

kubectl无法连上服务

执行kubectl看到报错 The connection to the server localhost:8080 was refused - did you specify the right host or port? 这是由于kubernetes默认不开放http 8080访问。

可以在/etc/kubernetes/manifests/kube-apiserver.yaml看到开启了https 6443访问。 也可以通过sudo netstat -apn | grep tcp | grep LISTEN看到开的的端口

可以使用sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf api-versions访问，admin.conf中有端口以及key

使用阿里云apt源导致无法安装

备份/etc/apt/sources.list后替换为以下内容

deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse

执行sudo apt update && sudo apt-get update更新apt源。但是导致了 package xxx has no installation candidate，使用Ubuntu官方源就OK

pod状态异常

安装完集群后，生成一个pod，pod一直是pending状态。查看pod的描述

sudo kubectl describe pod helloworld

pod描述信息

.......
Events:Type     Reason            Age                From               Message----     ------            ----               ----               -------Warning  FailedScheduling  32s (x2 over 32s)  default-scheduler  0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.

执行kubectl taint nodes --all node-role.kubernetes.io/master-，node正常使用参与调度

node状态异常

service无法访问，通查看node状态

sudo kubectl get pod -o yaml

存储node网络异常情况

......
conditions:- lastHeartbeatTime: 2019-05-21T14:39:22ZlastTransitionTime: 2019-05-21T13:08:37Zmessage: kubelet has sufficient disk space availablereason: KubeletHasSufficientDiskstatus: "False"type: OutOfDisk- lastHeartbeatTime: 2019-05-21T14:39:22ZlastTransitionTime: 2019-05-21T13:08:37Zmessage: kubelet has sufficient memory availablereason: KubeletHasSufficientMemorystatus: "False"type: MemoryPressure- lastHeartbeatTime: 2019-05-21T14:39:22ZlastTransitionTime: 2019-05-21T13:08:37Zmessage: kubelet has no disk pressurereason: KubeletHasNoDiskPressurestatus: "False"type: DiskPressure- lastHeartbeatTime: 2019-05-21T14:39:22ZlastTransitionTime: 2019-05-21T13:08:37Zmessage: kubelet has sufficient PID availablereason: KubeletHasSufficientPIDstatus: "False"type: PIDPressure- lastHeartbeatTime: 2019-05-21T14:39:22ZlastTransitionTime: 2019-05-21T13:08:37Zmessage: 'runtime network not ready: NetworkReady=false reason:NetworkPluginNotReadymessage:docker: network plugin is not ready: cni config uninitialized'reason: KubeletNotReadystatus: "False"type: Ready
......

第一次出现问题是因为没有安装flannel网络插件。

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

安装完毕网络插件后，提示cni config uninitialized。

先关闭node上的kubelet； 然后使用kubeadm reset|| kubeadm join ..., 将node重新加入集群

系统宕机后重启服务

1. 关闭防火墙
2. 关闭swap
3. 重启docker
4. 重启kubelet
5. 然后使用docker先重启pause容器，然后在重启其他容器
6. 也可以使用kubeadm reset重置kubernetes，然后重新初始化集群
7. 注意使用journal查看docker和kubelet日志，另外要主要docker和kubelet的systemd启动参数

参考资料

基于 Ubuntu 18.04 安装部署K8s集群 Ubuntu-18.04使用kubeadm安装kubernetes-1.12.0 kubeadm安装k8s集群 apt-get 软件源文件格式以及ubuntu 18.04更换阿里源 [cni config uninitialized解决方案](cni config uninitialized "cni config uninitialized解决方案") cni config uninitialized