k8s集群下创建高可用postgresql数据库集群实践

K8s 安装 pg集群服务器

主要的目的是做到自动灾备切换，利用kubernetes 集群做到高可用的数据库服务。
三个节点， 2节点部署 proxy，sentinel，3节点keeper （DB）

本文选着stolon，

详细解决方案，原理请看：项目地址 https://github.com/sorintlab/stolon

总体架构架构

下面是步骤

1. 创建临时目录下载项目

mkdir stolon

git clone https://github.com/sorintlab/stolon.git

或者自己手工下载zip包，解压到stolon目录

进入/examples/kubernetes

这里保存k8s的脚本

  kubernetes git:(master) ✗ ll
total 104
-rw-r--r--  1 moqiyi  staff   8.4K Apr 18 20:58 README.md
drwxr-xr-x  3 moqiyi  staff    96B Apr 18 21:41 image
-rw-r--r--  1 moqiyi  staff   3.7K Apr 18 20:58 postgresql_upgrade.md
-rw-r--r--  1 moqiyi  staff   255B Apr 18 20:58 role-binding.yaml
-rw-r--r--  1 moqiyi  staff   571B Apr 18 20:58 role.yaml
-rw-r--r--  1 moqiyi  staff   136B Apr 18 21:51 secret.yaml
-rw-r--r--  1 moqiyi  staff   3.2K Apr 18 22:05 stolon-keeper.yaml
-rw-r--r--  1 moqiyi  staff   234B Apr 18 23:34 stolon-proxy-service.yaml
-rw-r--r--  1 moqiyi  staff   1.5K Apr 18 21:56 stolon-proxy.yaml
-rw-r--r--  1 moqiyi  staff   1.3K Apr 18 20:58 stolon-sentinel.yaml

一共有7个yaml文件

2. 创建pv数据盘（nfs）

git上默认是利用了本地存储，这里我们使用nfs去保存数据，一面服务挂了可以恢复。更加好的是使用分布式存取服务rook ceph.（https://rook.io/, cloud native storage）

在nsf服务器上创建pv1，pv2，pv3 目录

创建pv1, pv2,pv3 的yaml

#  pv1.yaml
apiVersion: v1
kind: PersistentVolume
metadata:name: pg-pv1
spec:capacity:storage: 2GiaccessModes:- ReadWriteManystorageClassName: nfsnfs:path: /home/nfs/pv1   <--这个一定要先在服务器上创建server: k8s-nfs       <--服务器地址

kubectl apply -f pv1.yaml

重复以上依次创建 2，3作为冗余备份数据

原则上一个pv就对应一个pvc，有兴趣可以查询

如何创建nfs服务器，请参看本人的上一篇文章（NFS服务器部分）

https://blog.csdn.net/davidqymo/article/details/105473358

3. 修改密码文件

修改密码文件secret.yaml

---
apiVersion: v1
kind: Secret
metadata:name: stolon
type: Opaque
data:
#    echo -n 'yourpassword' |base64password: eW91cnBhc3N3b3Jk

由于secret里面只能保存Base64的字符，请自行转换，在linux/mac上用 echo -n ‘你的密码’ ｜ base64 来转换base64的密码。或者自己着base64转换网站自行转换。比如

➜  stolon echo -n 'yourpassword' |base64
eW91cnBhc3N3b3Jk

4. 修改stolon-keeper deployment文件

修改stolon-keeper.yaml的pg的pg容器环境变量的初始化用户名字

...- name: STKEEPER_PG_SU_USERNAMEvalue: "postgres"...volumeClaimTemplates:- metadata:name: data
#        annotations:
#          volume.alpha.kubernetes.io/storage-class: standardspec:accessModes: ["ReadWriteMany"]resources:requests:storage: 512MistorageClassName: nfs

5. 修改stolon-proxy-service.yaml

这里开放NodePort的方式开放服务，集群众的任意节点都可以通过这个端口访问数据库

apiVersion: v1
kind: Service
metadata:name: stolon-proxy-service
spec:ports:- port: 5432targetPort: 5432nodePort: 30010type: NodePortselector:component: stolon-proxystolon-cluster: kube-stolon

其他的不需要修改

执行目录下所有的yaml文件

kubectl apply -f .

6. 初始化数据库

按照git上面的提示（个人感觉应该在做成side car方式去启动，否者pod挂了之后不能自动恢复）

kubectl run -i -t stolonctl --image=sorintlab/stolon:master-pg10 --restart=Never --rm -- /usr/local/bin/stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init

7. 测试数据库通过

工具连接 k8s-master可以是你任意节点的ip，这里是虚拟集群用，这些节点的ip都加入了hosts文件中。如果有改动就改hosts文件即可

jdbc:postgresql://k8s-master:30010/postgres

输入用户名postgres,密码yourpassword （第三部创建，你也可以去dash board UI去secret去查看）

8. 灾备模拟

创建测试数据库

create database testDB;创建表CREATE TABLE Persons
(PersonID int,LastName varchar(255),FirstName varchar(255),Address varchar(255),City varchar(255)
)INSERT INTO "public"."persons" ("personid", "lastname", "firstname", "address", "city") VALUES (111, 'david', 'test', 'china', 'gz');

查看数据库节点

stolon kubectl get pods -o wide
NAME                               READY   STATUS    RESTARTS   AGE    IP            NODE        NOMINATED NODE   READINESS GATES
stolon-keeper-0                    1/1     Running   0          69m    10.244.2.11   k8s-node1   <none>           <none>
stolon-keeper-1                    1/1     Running   0          64m    10.244.1.17   k8s-node2   <none>           <none>
stolon-proxy-5c996dbbcb-8wfhl      1/1     Running   0          3h3m   10.244.1.15   k8s-node2   <none>           <none>
stolon-proxy-5c996dbbcb-c58lg      1/1     Running   0          3h3m   10.244.3.15   k8s-node3   <none>           <none>
stolon-sentinel-8455d466f5-bzdxk   1/1     Running   0          3h2m   10.244.3.16   k8s-node3   <none>           <none>
stolon-sentinel-8455d466f5-gsbmj   1/1     Running   0          3h2m   10.244.2.10   k8s-node1   <none>           <none>

模拟删除 pods，服务，代理

kubectl delete -f stolon-keeper.yamlkubectl delete -f stolon-proxy.yaml kubectl delete -f stolon-sentinel.yamlkubectl delete -f stolon-proxy-service.yaml

查看数据库无法连接

重新建立pod，service

kubectl create -f stolon-keeper.yamlkubectl create -f stolon-proxy.yaml kubectl create -f stolon-sentinel.yamlkubectl create -f stolon-proxy-service.yaml

查看服务正常

➜  stolon kubectl get pod,svc -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP            NODE        NOMINATED NODE   READINESS GATES
pod/stolon-keeper-0                    1/1     Running   0          2m26s   10.244.2.12   k8s-node1   <none>           <none>
pod/stolon-keeper-1                    1/1     Running   0          2m25s   10.244.1.19   k8s-node2   <none>           <none>
pod/stolon-proxy-5c996dbbcb-5lhcr      1/1     Running   0          2m7s    10.244.2.13   k8s-node1   <none>           <none>
pod/stolon-proxy-5c996dbbcb-j7n2n      1/1     Running   0          2m7s    10.244.3.17   k8s-node3   <none>           <none>
pod/stolon-sentinel-8455d466f5-76tkp   1/1     Running   0          106s    10.244.3.18   k8s-node3   <none>           <none>
pod/stolon-sentinel-8455d466f5-wfsqq   1/1     Running   0          106s    10.244.1.20   k8s-node2   <none>           <none>NAME                           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE    SELECTOR
service/kubernetes             ClusterIP   10.1.0.1       <none>        443/TCP          4d9h   <none>
service/stolon-proxy-service   NodePort    10.1.214.229   <none>        5432:30010/TCP   39s    component=stolon-proxy,stolon-cluster=kube-stolon
➜  stolon

客户端连接数据库，查看数据库表，数据，都在

模拟删除 pod/stolon-keeper-0

kubectl delete pod stolon-keeper-0pod "stolon-keeper-0" deleted

连接数据库表，还在，内容也正确。

发现单机的速度太快，删除完之后系统自动重新创建 stolon-keeper-0，系统还没来得及主备切换

调整策略

➜  stolon kubectl get pod,svc -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP            NODE        NOMINATED NODE   READINESS GATES
pod/stolon-keeper-0                    1/1     Running   0          3m19s   10.244.2.17   k8s-node1   <none>           <none>
pod/stolon-keeper-1                    1/1     Running   0          27m     10.244.1.19   k8s-node2   <none>           <none>
pod/stolon-keeper-2                    1/1     Running   0          14m     10.244.3.19   k8s-node3   <none>           <none>
pod/stolon-proxy-5c996dbbcb-5lhcr      1/1     Running   0          26m     10.244.2.13   k8s-node1   <none>           <none>
pod/stolon-proxy-5c996dbbcb-j7n2n      1/1     Running   0          26m     10.244.3.17   k8s-node3   <none>           <none>
pod/stolon-sentinel-8455d466f5-76tkp   1/1     Running   0          26m     10.244.3.18   k8s-node3   <none>           <none>
pod/stolon-sentinel-8455d466f5-wfsqq   1/1     Running   0          26m     10.244.1.20   k8s-node2   <none>           <none>NAME                           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE     SELECTOR
service/kubernetes             ClusterIP   10.1.0.1       <none>        443/TCP          4d10h   <none>
service/stolon-proxy-service   NodePort    10.1.214.229   <none>        5432:30010/TCP   25m     component=stolon-proxy,stolon-cluster=kube-stolon

可以看到 stolon-keeper-0 运行在 k8s-node1，其他各自在node2， node3，为了避免 stolon-keeper-0 再自动启动，现在把 k8s-node1 虚拟机关闭

查看变化 stolon-keeper-1 的log，发现他被选举成master完成主备切换

2020-04-19T01:40:42.055Z INFO    cmd/keeper.go:1526  already standby
2020-04-19T01:40:42.076Z    INFO    cmd/keeper.go:1626  postgres parameters not changed
2020-04-19T01:40:42.077Z    INFO    cmd/keeper.go:1653  postgres hba entries not changed
2020-04-19T01:40:47.125Z    INFO    cmd/keeper.go:1457  our db requested role is master
2020-04-19T01:40:47.128Z    INFO    cmd/keeper.go:1486  promoting to master
2020-04-19T01:40:47.128Z    INFO    postgresql/postgresql.go:532    promoting database
2020-04-19 01:40:47.136 UTC [4152] LOG:  received promote request
2020-04-19 01:40:47.136 UTC [4156] FATAL:  terminating walreceiver process due to administrator command
2020-04-19 01:40:47.140 UTC [4152] LOG:  invalid record length at 0/5000370: wanted 24, got 0
2020-04-19 01:40:47.140 UTC [4152] LOG:  redo done at 0/5000338
2020-04-19 01:40:47.155 UTC [4152] LOG:  selected new timeline ID: 2
2020-04-19 01:40:47.431 UTC [4152] LOG:  archive recovery complete
2020-04-19 01:40:47.469 UTC [4151] LOG:  database system is ready to accept connections
waiting for server to promote.... done
server promoted
2020-04-19T01:40:47.556Z    INFO    cmd/keeper.go:1626  postgres parameters not changed
2020-04-19T01:40:47.556Z    INFO    cmd/keeper.go:1653  postgres hba entries not changed
2020-04-19T01:40:52.613Z    INFO    cmd/keeper.go:1457  our db requested role is master
2020-04-19T01:40:52.615Z    INFO    cmd/keeper.go:1493  already master

再重启node1。查看他的logs，发现他成为了standby

2020-04-19T01:50:33.113Z INFO    cmd/keeper.go:1526  already standby
2020-04-19T01:50:33.133Z    INFO    cmd/keeper.go:1626  postgres parameters not changed
2020-04-19T01:50:33.133Z    INFO    cmd/keeper.go:1653  postgres hba entries not changed