aws eks 快速启动和配置

经常使用eks集群进行大量的测试工作，但是通过控制台启动集群需要做很多配置，这里整理出快速启动和配置eks集群的方式

配置工具

aws-cli

yum install unzip -y
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
./aws/install

eksctl

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin

kubectl

curl -o kubectl https://s3.cn-north-1.amazonaws.com.cn/amazon-eks/1.23.7/2022-06-29/bin/linux/amd64/kubectl
chmod +x ./kubectl
mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin
echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
kubectl version --short --client

completion

source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
. <(eksctl completion bash)
complete -C '/usr/local/bin/aws_completer' aws

makefile

SHELL := /bin/bash # Use bash syntaxinstall-awscli:curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"unzip -o -q awscliv2.zipsudo ./aws/install --updaterm -r ./awsrm -r awscliv2.zipaws --versioninstall-eksctl:$(eval EKSCTL_VERSION:=v0.111.0)curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/$(EKSCTL_VERSION)/eksctl_Linux_amd64.tar.gz" | tar xz -C /tmpsudo mv /tmp/eksctl /usr/local/bineksctl versioninstall-kubectl:$(eval KUBECTL_VERSION:=v1.25.0)curl -LO "https://dl.k8s.io/release/$(KUBECTL_VERSION)/bin/linux/amd64/kubectl"sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectlrm kubectlkubectl version --clientinstall-kustomize:$(eval KUSTOMIZE_VERSION:=3.2.0)wget https://github.com/kubernetes-sigs/kustomize/releases/download/v$(KUSTOMIZE_VERSION)/kustomize_$(KUSTOMIZE_VERSION)_linux_amd64chmod +x kustomize_$(KUSTOMIZE_VERSION)_linux_amd64sudo mv kustomize_$(KUSTOMIZE_VERSION)_linux_amd64 /usr/local/bin/kustomizekustomize versioninstall-yq:$(eval YQ_VERSION:=v4.26.1)wget https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_amd64.tar.gz -O - | tar xzsudo mv yq_linux_amd64 /usr/bin/yqrm install-man-page.shrm yq.1yq --versioninstall-jq:$(eval JQ_VERSION:=1.5+dfsg-2)sudo apt-get install jq=$(JQ_VERSION) -yinstall-terraform:$(eval TERRAFORM_VERSION:=1.2.7)curl "https://releases.hashicorp.com/terraform/$(TERRAFORM_VERSION)/terraform_$(TERRAFORM_VERSION)_linux_amd64.zip" -o "terraform.zip"unzip -o -q terraform.zipsudo install -o root -g root -m 0755 terraform /usr/local/bin/terraformrm terraform.ziprm terraformterraform --versioninstall-helm:curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bashhelm versioninstall-python:sudo apt install -q python3.8 -ysudo apt install -q python3-pip -ypython3.8 -m pip install --upgrade pipinstall-python-packages:python3.8 -m pip install -r tests/e2e/requirements.txtinstall-tools: install-awscli install-eksctl install-kubectl install-kustomize install-yq install-jq install-terraform install-helm install-python install-python-packages

创建资源

eks集群手动配置较为繁琐，需要满足权限配置（集群和节点），vpc和子网的要求（ip数量/终端节点），集群安全组配置，还有节点组本身的配置（类型/大小/存储/子网/安全组标签/启动模板/ami/启动脚本等），可以使用工具简化这一过程。

创建集群

eksctl的参数说明

创建一个完全托管的集群，会创建包括vpc/子网/role等等资源在内的所有资源，一键完成，想要使用已有的基础设置配置集群则需要配置文件的方式

eksctl create cluster <cluter-name>

使用已有的vpc和子网创建集群，可以按照官方文档配置参数

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:name: <cluster-name>region: cn-north-1version: "1.23"
iam:withOIDC: true
vpc:clusterEndpoints:publicAccess:  trueprivateAccess: trueid: "vpc-xxxxxxxxxxxxxx"subnets:private:private1:id: "subnet-xxxxxxxxxxxxxx"private2:id: "subnet-xxxxxxxxxxxxxx"public:public1:id: "subnet-xxxxxxxxxxxxxx"public2:id: "subnet-xxxxxxxxxxxxxx"addons:- name: vpc-cniversion: latest- name: corednsversion: latest- name: kube-proxyversion: latestfargateProfiles:- name: eksctl-fargateselectors:- namespace: fargate

创建节点组

增加单个节点IP数量，需要确保使用的vpc cni插件版本大于1.9.0 或 1.10.1。之后可以为节点组指定单个节点的启动pod数量超过实例ip数量的限制，最高110个，但是可以手动调整节点kubelet为更大数量

创建的节点组实际上是一个autoscaling group，不指定ami和启动模板时会选择默认配置启动节点，如果指定了ami需要配置overrideBootstrapCommand，可以指定启动模板

kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

使用配置文件创建托管节点组

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfigmetadata:name: <cluster-name>region: <region>managedNodeGroups:- name: ng-1-workerslabels: { role: workers }instanceType: m5.xlargedesiredCapacity: 10volumeSize: 80privateNetworking: true- name: ng-2-builderslabels: { role: builders }instanceType: m5.2xlargedesiredCapacity: 2volumeSize: 100privateNetworking: true

配置文件中只需要指定cluster名称即可

eksctl create nodegroup -f xxxx.yaml

在私有子网创建节点组，并且修改单个节点pod为60

eksctl create nodegroup \--cluster <cluster-name> \--node-type t3.medium \--max-pods-per-node 60 \--nodes 1 --nodes-max 5 --nodes-min 0 \--node-volume-size 30 --node-volume-type gp3 \--ssh-access --ssh true \--ssh-public-key <key-name> \--node-private-networking false \--enable-ssm

在公有子网创建托管节点组，指定最大pod数量为60，允许ssm和ssh访问

eksctl create nodegroup \--cluster testtemp1 \--node-type t3.small \--nodes 1 --nodes-max 5 --nodes-min 0 \--node-volume-size 30 --node-volume-type gp3  \--max-pods-per-node 60 \--ssh-access  --ssh-public-key cluster-key   \--enable-ssm --dry-run

在公有子网创建资管节点组

eksctl create nodegroup \--cluster xxxxxxx \--node-type t3.medium \--max-pods-per-node 60 \--nodes 1 --nodes-max 3 --nodes-min 0 \--node-volume-size 30 --node-volume-type gp3  \--ssh-access  --ssh-public-key cluster-key   \--enable-ssm  \--managed=false \# -P //私有子网开启

添加插件

$ eksctl create addon --cluster xxxxxxx --name vpc-cni
$ eksctl create addon --cluster xxxxxxx --name coredns
$ eksctl create addon --cluster xxxxxxx --name kube-proxy
$ eksctl get addon  --cluster xxxxxxx
2022-10-22 02:19:41 [ℹ]  Kubernetes version "1.23" in use by cluster "xxxxxxx"
2022-10-22 02:19:41 [ℹ]  getting all addons
2022-10-22 02:19:42 [ℹ]  to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>`
NAME            VERSION                 STATUS  ISSUES  IAMROLE UPDATE AVAILABLE
coredns         v1.8.7-eksbuild.3       ACTIVE  0
kube-proxy      v1.23.7-eksbuild.1      ACTIVE  0               v1.23.8-eksbuild.2
vpc-cni         v1.10.4-eksbuild.1      ACTIVE  0               v1.11.4-eksbuild.1,v1.11.3-eksbuild.1,v1.11.2-eksbuild.1,v1.11.0-eksbuild.1

添加fargate配置

$ eksctl create fargateprofile \--cluster my-cluster \--name my-fargate-profile \--namespace my-kubernetes-namespace \--labels key=value

添加新用户

实际上是向eks集群中名为aws-auth的config map 中增加映射关系

$ eksctl get iamidentitymapping --cluster worklearn
ARN                                                                                                     USERNAME                               GROUPS                                                   ACCOUNT
arn:aws-cn:iam::037047667284:role/eksctl-worklearn-nodegroup-ng-322-NodeInstanceRole-1TM9E1DSSFSRS      system:node:{{EC2PrivateDNSName}}      system:bootstrappers,system:nodes$ eksctl create iamidentitymapping \--cluster xxxxxxx \--region=cn-north-1 \--arn arn:aws-cn:iam::xxxxxxx:role/xxxxxxxx \--group system:masters \--no-duplicate-arns

添加service account

会创建一个cloudformation模板，创建角色并和eks集群中的角色关联起来

eksctl create iamserviceaccount \--cluster xxxxxx \--name xxxxxxx \--namespace kube-system \--attach-policy-arn xxxxxx \--override-existing-serviceaccounts \--region cn-north-1 \--approve

部署驱动

部署ebs-csi

创建ebs-csi-controller-sa

eksctl create iamserviceaccount \--name ebs-csi-controller-sa \--namespace kube-system \--cluster xxxxx \--attach-policy-arn arn:aws-cn:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \--approve \# --role-only \# --role-name AmazonEKS_EBS_CSI_DriverRole

添加ebs csi

eksctl create addon --name aws-ebs-csi-driver \--cluster my-cluster \--service-account-role-arn xxxxxxxxxxxxxxxxx \--force

aws-ebs-csi-driver的使用demo

部署efs-csi

创建sa

curl -o iam-policy-example.json https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/docs/iam-policy-example.jsonaws iam create-policy \--policy-name AmazonEKS_EFS_CSI_Driver_Policy \--policy-document file://iam-policy-example.jsoneksctl create iamserviceaccount \--cluster xxxxxxx \--namespace kube-system \--name efs-csi-controller-sa \--attach-policy-arn arn:aws:iam::111122223333:policy/AmazonEKS_EFS_CSI_Driver_Policy \--approve \--region cn-north-1

下载清单和i修改

kubectl kustomize \"github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/ecr?ref=release-1.4" > efs-driver.yaml

中国区 Amazon 容器镜像注册表

Amazon EKS 会将镜像复制到 AWS 区域支持的每个 Amazon EKS 中的存储库。您的节点可以通过互联网从以下任何注册表中提取容器镜像

cn-north-1 918309763551.dkr.ecr.cn-north-1.amazonaws.com.cn

cn-northwest-1 961992271922.dkr.ecr.cn-northwest-1.amazonaws.com.cn

cn-north-1	918309763551.dkr.ecr.cn-north-1.amazonaws.com.cn
cn-northwest-1	961992271922.dkr.ecr.cn-northwest-1.amazonaws.com.cn

修改

sed -i.bak -e 's|us-west-2|cn-north-1|' driver.yaml
sed -i.bak -e 's|602401143452|918309763551|' driver.yaml

!!! 删除清单driver中名为efs-csi-controller-sa的ServiceAccount，并部署

kubectl apply -f driver.yaml

部署ingress-controller

alb ingress controller流程和组件关系

控制器观察来自 API 服务器的进站事件。如果发现 Ingress 资源满足要求，则将开始创建 AWS 资源。
为 Ingress 资源创建 ALB。
为 Ingress 资源中指定的每个后端创建目标组。
为 Ingress 资源注释中指定的每个端口创建侦听器。如果未指定端口，则将使用合理的默认值（80 或 443）。
为 Ingress 资源中指定的每个路径创建规则。这将确保指向特定路径的流量将被路由至所创建的正确目标组

下载策略

curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.3/docs/install/iam_policy.jsonsed -i.bak -e 's|arn:aws:|arn:aws-cn:|' iam_policy.jsonaws iam create-policy \--policy-name AWSLoadBalancerControllerIAMPolicy \--policy-document file://iam_policy.json

创建sa

eksctl create iamserviceaccount \--cluster worklearn \--namespace kube-system \--name aws-load-balancer-controller \--attach-policy-arn arn:aws-cn:iam::xxxxxx:policy/AWSLoadBalancerControllerIAMPolicy \--override-existing-serviceaccounts \--region cn-north-1 \--approve

注意：helm中的image版本和yaml的版本不一致一个叫loadbalance-controller，另一个叫alb-controller

helm部署

helm repo add eks https://aws.github.io/eks-charts
kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"
helm repo update
helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \-n kube-system \--set clusterName=george-c1 \--set serviceAccount.create=false \--set serviceAccount.name=aws-load-balancer-controller \--set image.repository=918309763551.dkr.ecr.cn-north-1.amazonaws.com.cn/amazon/aws-alb-ingress-controller

手动部署

下载cert-manager

curl -Lo cert-manager.yaml https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml
kubectl apply --validate=false -f cert-manager.yaml

部署controller

curl -Lo ingress-controller.yaml https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/download/v2.4.3/v2_4_3_full.yamlsed -i.bak -e 's|your-cluster-name|<cluster-name>|' ./ingress-controller.yamlkubectl apply -f controller.yaml