我的k8s随笔：Kubernetes部署-问题篇
2024-05-24 22:01:51
本文集中记录k8s集群部署过程的问题。由于各人环境不同，限于经验，本文仅供参考。
注：本文会不定时更新。
源、key问题
使用国内中科大源：
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
EOF
更新：
apt-get update
但出错：
Ign:7 http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial/main amd64 Packages
Get:7 http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial/main amd64 Packages [31.3 kB]
Err:7 http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial/main amd64 PackagesHash Sum mismatch
Fetched 38.9 kB in 1s (20.2 kB/s)
Reading package lists... Done
E: Failed to fetch http://mirrors.ustc.edu.cn/kubernetes/apt/dists/kubernetes-xenial/main/binary-amd64/Packages.gz Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
原因及解决：
添加key:
gpg --keyserver keyserver.ubuntu.com --recv-keys 6A030B21BA07F4FB
gpg --export --armor 6A030B21BA07F4FB | sudo apt-key add -
结果：失败。
使用k8s官方提供的国外源：
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
更新apt-get update，会卡住，失败。
使用阿里云源：
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
添加key：
cat https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
如果不成功，先通过一些方法下载：https://packages.cloud.google.com/apt/doc/apt-key.gpg， 保存到当前目录。
再执行：
cat apt-key.gpg | sudo apt-key add -
再执行更新apt-get update，成功。
如果不添加 key，更新阿里云源出错：
W: GPG error: https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6A030B21BA07F4FB
W: The repository 'https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
查询k8s相应配置包：
W1214 08:46:14.303158 8461 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1214 08:46:14.303772 8461 version.go:102] falling back to the local client version: v1.17.0
W1214 08:46:14.304223 8461 validation.go:28] Cannot validate kube-proxy config - no validator is available
W1214 08:46:14.304609 8461 validation.go:28] Cannot validate kubelet config - no validator is available
原因及解决：
外网无法访问。不用理会，因为会根据执行的版本使用默认的版本。
脚本执行
pullk8s.sh: 3: pullk8s.sh: Syntax error: "(" unexpected
原因及解决：
脚本开头需为#!/bin/bash，如非，则用 bash pullk8s.sh执行。
初始化环境 kubeadm init
提示：
[ERROR Swap]: running with swap on is not supported. Please disable swap
原因及解决：
不支持 swap 需要禁止。
提示：
[ERROR Port-10250]: Port 10250 is in use
需要停止 kubelet 的运行：systemctl stop kubelet。
提示WARNING IsDockerSystemdCheck。
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
error execution phase preflight: [preflight] Some fatal errors occurred:[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
）
原因及解决：
docker使用cgroupfs，与k8s不一致。先查看：
# docker info | grep -i cgroup
Cgroup Driver: cgroupfs // !!! 此处为cgroupfs
WARNING: No swap limit support
需要修改，先停止docker：
systemctl stop docker
更改 /etc/docker/daemon.json，添加：
"exec-opts": ["native.cgroupdriver=systemd"]
重启docker:
systemctl start docker
查看 cgroup：
# docker info | grep -i cgroup
Cgroup Driver: systemd
已改。
（！！！！！！
注：
修改kubeadm配置文件：
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
在 Environment 后再新加：
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
另一个指定的pod源的：
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"
重启：
systemctl daemon-reload
systemctl restart kubelet
此方法实践不成功
！！！！！！）
提示ERROR NumCPU：
error execution phase preflight: [preflight] Some fatal errors occurred:[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...
原因解决：CPU要双核以上，改虚拟机cpu为2个核心或以上即可。
运行时
查看状态：
kubectl get pods -n kube-system
出错：
The connection to the server localhost:8080 was refused - did you specify the right host or port?
原因及解决：
没有执行：
mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config
执行后，成功。
The connection to the server 192.168.0.102:6443 was refused - did you specify the right host or port?
# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6955765f44-j7lvd 0/1 CrashLoopBackOff 14 51m
coredns-6955765f44-kmhfc 0/1 CrashLoopBackOff 14 51m
etcd-ubuntu 1/1 Running 0 52m
kube-apiserver-ubuntu 1/1 Running 0 52m
kube-controller-manager-ubuntu 1/1 Running 0 52m
kube-proxy-qlhfs 1/1 Running 0 51m
kube-scheduler-ubuntu 1/1 Running 0 52m
也可用 kubectl get pod --all-namespaces 查看所有命名空间。
如果不设置网络，则coredns为Pending状态。
部署flannel：
kubectl apply -f kube-flannel.yml
提示：
error: unable to recognize "kube-flannel-aliyun-0.11.0.yml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
换calico，也是一样的。应该是命名的问题。
解决：找master对应的文件。
https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml
kube-flannel-aliyun.yml的mster和其它tag使用了"extensions/v1beta1"。kube-flannel.yml的Tag使用了，但master又恢复了。
未部署flannel前，
[FATAL] plugin/loop: Loop (127.0.0.1:60825 -> :53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 7805087528265218508.4857814245207702505."
部署flannel后，
1、coredns出现 CrashLoopBackOff，kube-flannel出现 Init:ImagePullBackOff，
# kubectl logs kube-flannel-ds-amd64-n55rf -n kube-system
Error from server (BadRequest): container "kube-flannel" in pod "kube-flannel-ds-amd64-n55rf" is waiting to start: PodInitializing
使用 kubectl describe pod 查看：
# kubectl describe pod kube-flannel-ds-amd64-n55rf -n kube-system
...Normal Scheduled 13m default-scheduler Successfully assigned kube-system/kube-flannel-ds-amd64-n55rf to ubuntuNormal Pulling 4m21s (x4 over 13m) kubelet, ubuntu Pulling image "quay.io/coreos/flannel:v0.11.0-amd64"Warning Failed 3m6s (x4 over 10m) kubelet, ubuntu Failed to pull image "quay.io/coreos/flannel:v0.11.0-amd64": rpc error: code = Unknown desc = context canceledWarning Failed 3m6s (x4 over 10m) kubelet, ubuntu Error: ErrImagePullNormal BackOff 2m38s (x7 over 10m) kubelet, ubuntu Back-off pulling image "quay.io/coreos/flannel:v0.11.0-amd64"Warning Failed 2m27s (x8 over 10m) kubelet, ubuntu Error: ImagePullBackOff
原因：flannel:v0.11.0-amd64 无法下载，通过其它方式下载。注意，名称 quay.io/coreos/flannel:v0.11.0-amd64 一定要对。
下载flannel后，有2种情况：
2、
coredns 变成 ContainerCreating 状态：
# kubectl logs coredns-6955765f44-4csvn -n kube-system
Error from server (BadRequest): container "coredns" in pod "coredns-6955765f44-r96qk" is waiting to start: ContainerCreating
3、
coredns 变成 CrashLoopBackOff 状态：
# kubectl logs coredns-6955765f44-4csvn -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.5
linux/amd64, go1.13.4, c2fd1b2
[FATAL] plugin/loop: Loop (127.0.0.1:41252 -> :53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 1746539958269975925.3391392736060997773."
查看详细信息：
# kubectl describe pod coredns-6955765f44-4csvn -n kube-system
Name: coredns-6955765f44-r96qk
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: ubuntu/192.168.0.102
Start Time: Sun, 15 Dec 2019 22:45:15 +0800
Labels: k8s-app=kube-dnspod-template-hash=6955765f44
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/coredns-6955765f44
Containers:coredns:Container ID: Image: k8s.gcr.io/coredns:1.6.5Image ID: Ports: 53/UDP, 53/TCP, 9153/TCPHost Ports: 0/UDP, 0/TCP, 0/TCPArgs:-conf/etc/coredns/CorefileState: WaitingReason: ContainerCreatingReady: FalseRestart Count: 0Limits:memory: 170MiRequests:cpu: 100mmemory: 70MiLiveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3Environment: <none>Mounts:/etc/coredns from config-volume (ro)/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-qq7qf (ro)
Conditions:Type StatusInitialized True Ready False ContainersReady False PodScheduled True
Volumes:config-volume:Type: ConfigMap (a volume populated by a ConfigMap)Name: corednsOptional: falsecoredns-token-qq7qf:Type: Secret (a volume populated by a Secret)SecretName: coredns-token-qq7qfOptional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
Tolerations: CriticalAddonsOnlynode-role.kubernetes.io/master:NoSchedulenode.kubernetes.io/not-ready:NoExecute for 300snode.kubernetes.io/unreachable:NoExecute for 300s
Events:Type Reason Age From Message---- ------ ---- ---- -------Warning FailedScheduling 7m21s (x3 over 8m32s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.Normal Scheduled 6m55s default-scheduler Successfully assigned kube-system/coredns-6955765f44-r96qk to ubuntuWarning FailedCreatePodSandBox 6m52s kubelet, ubuntu Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9a2d45536097d22cc6b10f338b47f1789869f45f4b12f8a202aa898295dc80a4" network for pod "coredns-6955765f44-r96qk": networkPlugin cni failed to set up pod "coredns-6955765f44-r96qk_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.0.1/24
安装flannel后，删除出问题的pod：
kubectl delete pod coredns-6955765f44-4csvn -n kube-system
会自动重启一个新的pod，但问题依然。查看 ifconfig，发现有 cni0 。
网上解决方法：
#在master节点之外的节点进行操作
kubeadm reset
systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1
##重启kubelet
systemctl restart kubelet
##重启docker
systemctl restart docker
尝试，失败！
又一次部署的提示：
Warning FailedScheduling 77s (x5 over 5m53s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.Normal Scheduled 76s default-scheduler Successfully assigned kube-system/coredns-9d85f5447-4jwf2 to ubuntuWarning FailedCreatePodSandBox 73s kubelet, ubuntu Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5c109baa51b8d97e75c6b35edf108ca4f2f56680b629140c8b477b9a8a03d97c" network for pod "coredns-9d85f5447-4jwf2": networkPlugin cni failed to set up pod "coredns-9d85f5447-4jwf2_kube-system" network: open /run/flannel/subnet.env: no such file or directoryWarning FailedCreatePodSandBox 71s kubelet, ubuntu Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3f8c5b704fb1dc4584a2903b2ecff329e717e5c2558c9f761501fab909d32133" network for pod "coredns-9d85f5447-4jwf2": networkPlugin cni failed to set up pod "coredns-9d85f5447-4jwf2_kube-system" network: open /run/flannel/subnet.env: no such file or directoryNormal SandboxChanged 70s (x2 over 72s) kubelet, ubuntu Pod sandbox changed, it will be killed and re-created.Normal Pulled 29s (x4 over 69s) kubelet, ubuntu Container image "registry.aliyuncs.com/google_containers/coredns:1.6.5" already present on machineNormal Created 29s (x4 over 69s) kubelet, ubuntu Created container corednsNormal Started 29s (x4 over 69s) kubelet, ubuntu Started container corednsWarning BackOff 10s (x9 over 67s) kubelet, ubuntu Back-off restarting failed container
原因及解决：
网上说初始化时要添加 --pod-network-cidr=10.244.0.0/16，但已添加。注：稍等片刻即产生此文件。
Warning FailedScheduling 56m (x5 over 60m) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.Normal Scheduled 56m default-scheduler Successfully assigned kube-system/coredns-9d85f5447-4jwf2 to ubuntuWarning FailedCreatePodSandBox 56m kubelet, ubuntu Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5c109baa51b8d97e75c6b35edf108ca4f2f56680b629140c8b477b9a8a03d97c" network for pod "coredns-9d85f5447-4jwf2": networkPlugin cni failed to set up pod "coredns-9d85f5447-4jwf2_kube-system" network: open /run/flannel/subnet.env: no such file or directoryWarning FailedCreatePodSandBox 55m kubelet, ubuntu Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3f8c5b704fb1dc4584a2903b2ecff329e717e5c2558c9f761501fab909d32133" network for pod "coredns-9d85f5447-4jwf2": networkPlugin cni failed to set up pod "coredns-9d85f5447-4jwf2_kube-system" network: open /run/flannel/subnet.env: no such file or directoryNormal SandboxChanged 55m (x2 over 55m) kubelet, ubuntu Pod sandbox changed, it will be killed and re-created.Normal Pulled 55m (x4 over 55m) kubelet, ubuntu Container image "registry.aliyuncs.com/google_containers/coredns:1.6.5" already present on machineNormal Created 55m (x4 over 55m) kubelet, ubuntu Created container corednsNormal Started 55m (x4 over 55m) kubelet, ubuntu Started container corednsWarning BackOff 59s (x270 over 55m) kubelet, ubuntu Back-off restarting failed container
log信息：
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.5
linux/amd64, go1.13.4, c2fd1b2
[FATAL] plugin/loop: Loop (127.0.0.1:48100 -> :53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 639535139534040434.6569166625322327450."
原因及解决：
ConfigMap里使用了 /etc/resolv.conf，里面的DNS为127.0.1.1，此导致问题。
执行：
kubectl edit cm coredns -n kube-system
删除 loop 字段，保存，退出（vim编辑器）。
删除出问题的所有的 coredns：
kubectl delete pod coredns-9d85f5447-4jwf2 -n kube-system
coredns ConfigMap内容如下：
apiVersion: v1
data:Corefile: |.:53 {errorshealth {lameduck 5s}readykubernetes cluster.local in-addr.arpa ip6.arpa {pods insecurefallthrough in-addr.arpa ip6.arpattl 30}prometheus :9153forward . /etc/resolv.confcache 30loopreloadloadbalance}
kind: ConfigMap
metadata:creationTimestamp: "2019-12-21T09:50:31Z"name: corednsnamespace: kube-systemresourceVersion: "171"selfLink: /api/v1/namespaces/kube-system/configmaps/corednsuid: 62485b55-3de6-4dee-b24a-8440052bdb66
注：理论上修改 /etc/resolv.conf 为8.8.8.8 应该能解决，但该文件手动修改重启后恢复为127网段，无效。删除 loop 字段可解决问题。
加入集群失败
[preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
原因及解决：
猜测可能是主机名与 master 一致导致的，但没有实证。
网上收集的
WARNING FileExisting-socat
socat是一个网络工具， k8s 使用它来进行 pod 的数据交互，出现这个问题直接安装socat即可：
apt-get install socat
工作节点加入失败
在子节点执行kubeadm join命令后返回超时错误，如下：
root@worker2:~# kubeadm join 192.168.56.11:6443 --token wbryr0.am1n476fgjsno6wa --discovery-token-ca-cert-hash sha256:7640582747efefe7c2d537655e428faa6275dbaff631de37822eb8fd4c054807
[preflight] Running pre-flight checks
error execution phase preflight: couldn't validate the identity of the API Server: abort connecting to API servers after timeout of 5m0s
在master节点上执行 kubeadm token create --print-join-command 重新生成加入命令，并使用输出的新命令在工作节点上重新执行即可。
master节点的token 24小时过期后，可以通过命令产生新的token：
kubeadm token list
创建永不过期的token
kubeadm token create --ttl 0
master节点上运行命令，可查询discovery-token-ca-cert-hash值：
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
重新加入节点
kubeadm join 192.168.124.195:6443 --token 8xwg8u.lkj382k9ox58qkw9 \
--discovery-token-ca-cert-hash sha256:86291bed442dd1dcd6c26f2213208e10cab0f87763f44e0edf01fa670cd9e8b
我的k8s随笔：Kubernetes部署-问题篇相关推荐
- 《Kubernetes部署篇：基于docker使用kubespray工具部署高可用K8S集群(国内互联网方案四)》
文章目录 一.部署背景简介 二.部署工具介绍 三.部署方案介绍 四.部署环境信息 五.部署资源下载 六.部署准备工作 6.1.系统内核升级 6.2.设置主机名 6.3.环境初始化 6.4.ssh多机互 ...
- 《Kubernetes部署篇：基于docker使用kubespray工具离线部署高可用K8S集群(国内专网方案)》
文章目录 一.部署背景简介 二.部署工具介绍 三.部署方案介绍 四.部署环境信息 五.部署资源下载 六.部署准备工作 6.1.系统内核升级 6.2.设置主机名 6.3.环境初始化 6.4.ssh多机互 ...
- 《Kubernetes部署篇：基于docker使用kubespray工具部署高可用K8S集群(国内互联网方案三)》
文章目录 一.部署背景简介 二.部署工具介绍 三.部署方案介绍 四.部署环境信息 五.部署资源下载 六.部署准备工作 6.1.系统内核升级 6.2.设置主机名 6.3.环境初始化 6.4.ssh多机互 ...
- 我的k8s随笔：Kubernetes 1.17.0 部署
k8s集群部署过程实践笔记共两种版本:一为专注部署操作,一为涉及部署操作讲解.本文为前者. 本文介绍了如何在两台 ubuntu 16.04 64 bit 双核 CPU 虚拟机上使用 kubeadm 部 ...
- Kubernetes(k8s)集群部署（k8s企业级Docker容器集群管理）系列目录
0.目录 整体架构目录:ASP.NET Core分布式项目实战-目录 k8s架构目录:Kubernetes(k8s)集群部署(k8s企业级Docker容器集群管理)系列目录 一.感谢 在此感谢.net ...
- 第二篇：kubernetes部署calico网络插件
说明: 总的目标是在k8s集群部署gitlab.jenkins,并且在本地提交代码到gitlab后jenkin流水线可以自动编译打包成为docker镜像然后部署到k8s中并实现客户端外部域名访问,在文 ...
- 第五篇：kubernetes部署dashboard(图形化界面)
说明: 总的目标是在k8s集群部署gitlab.jenkins,并且在本地提交代码到gitlab后jenkin流水线可以自动编译打包成为docker镜像然后部署到k8s中并实现客户端外部域名访问,在文 ...
- k8s与CICD--将drone部署到kubernetes中，实现agent动态收缩
前言 本文主要讲如何把drone部署到k8s集群当中,本身drone这种基于容 器的pipeline方式,和k8s是相当契合的.这样的好处有: k8s集群守护drone-server 和drone-a ...
- k8s部署jar包_使用Kubernetes部署Springboot或Nginx的详细教程
1 前言 经过<Maven一键部署Springboot到Docker仓库,为自动化做准备>,Springboot的Docker镜像已经准备好,也能在Docker上成功运行了,是时候放上Ku ...
最新文章
- 新晋院士：直到硕士毕业前都想做公务员，现在只对科研感兴趣
- java拆分excel_Jexcel实现按一定规则分割excel文件的方法
- BCH代币化方案讨论大爆发，你的观点是什么？
- yii required 字段去除*号
- 反思代码能力提升方法：重构 多写 知识面
- Incompatible JavaHL library loaded. Subversion 1.8.x required.
- JMX和Spring –第3部分
- 【LeetCode - 227】基本计算器 II（栈）
- python中下拉菜单大小_自动化测试——Selenium+Python之下拉菜单的定位
- Python高能小技巧：不要在for与while循环后面写else块
- PCI、PCI-X、PCI-E AGP区别
- pyspark连接MySQL数据库，执行SQL语句，返回数据查询结果
- win10如何做服务器双系统,双系统怎么做？Win7+Win10 双系统安装图文教程
- Android人脸识别的初步学习
- Hello Juejin
- IIS MIME设置
- 开发、测试、测试开发
- rarlinux(rar\unrar在linux使用）
- 国际数据保护日，数据安全谁来守护？
- 转速闭环控制直流调速系统的仿真 matlab