一、前言

在《IPVS概览》一文中提到IPVS的Direct Routing模式是一个扩展性比较好的负载均衡方式。但是在默认的情况下，IPVS的Direct Routing模式需要指定一个Director，这个director负责绑定并对外暴露VIP，这个director所在的node上的进程是无法访问VIP获取后端real service的服务的。

在一般系统中，这可能不是问题，因为IPVS director作为单独的节点进行部署；但是在kubernetes环境中，IPVS被做成POD和service之后，再给IPVS单独指定worker节点，就会造成节点资源的浪费。本文就是基于这个目的，探究director上无法访问VIP的原因及其解决方案，以便于基于kubernetes部署IPVS服务。

转载自https://blog.csdn.net/cloudvtech

二、无法从IPVS director访问VIP服务

2.1 机器配置

IPVS director： 192.168.166.102/
IPVS real server：192.168.166.103/
VIP：192.168.166.111

real server上的服务是80端口的HTTP服务

2.2 IPVS director的设置

   ipvsadm -Cipvsadm -A -t 192.168.166.111 -s rripvsadm -a -t  192.168.166.111:80 -r 192.168.166.103:80 -w 1 -gifconfig  ens33:0 192.168.166.111 broadcast 192.168.166.255 netmask 255.255.255.0 uproute add -host 192.168.166.111  dev ens33:0

/etc/sysctl.conf

net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

2.3 IPVS real server的设置

/etc/sysctl.conf

net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.ens33.arp_ignore = 1
net.ipv4.conf.ens33.arp_announce = 2
ifconfig lo:0 192.168.166.111 broadcast 192.168.166.255 netmask 255.255.255.255 up
route add -host 192.168.166.111  dev lo:0

并且启动http服务

2.4 从外部可以访问VIP并且获得后端real server HTTP服务的返回

2.5 从director(192.168.166.102)访问VIP失败

curl 192.168.166.111

超时

IPVS的状态信息

[root@k8s-node1 ~]#  ipvsadm -L -n --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes-> RemoteAddress:Port
TCP  192.168.166.111:80                  1        4        0      240        0-> 192.168.166.103:80                  1        4        0      240        0

director上的tcpdump

09:57:42.919671 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 783029 ecr 0,nop,wscale 7], length 0
09:57:42.920165 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 779155 ecr 783029,nop,wscale 7], length 0
09:57:43.921996 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 784032 ecr 0,nop,wscale 7], length 0
09:57:43.922565 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 780157 ecr 783029,nop,wscale 7], length 0
09:57:45.124180 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 781359 ecr 783029,nop,wscale 7], length 0
09:57:45.925794 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 786036 ecr 0,nop,wscale 7], length 0
09:57:45.926194 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 782161 ecr 783029,nop,wscale 7], length 0
09:57:48.127567 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 784363 ecr 783029,nop,wscale 7], length 0
09:57:49.937754 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 790048 ecr 0,nop,wscale 7], length 0
09:57:49.938057 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 786174 ecr 783029,nop,wscale 7], length 0
09:57:54.137227 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 790373 ecr 783029,nop,wscale 7], length 0
09:58:02.151601 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 798389 ecr 783029,nop,wscale 7], length 0
09:58:18.173820 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 814413 ecr 783029,nop,wscale 7], length 0

real server上的tcpdump

09:57:42.943819 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 783029 ecr 0,nop,wscale 7], length 0
09:57:42.943892 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 779155 ecr 783029,nop,wscale 7], length 0
09:57:43.946328 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 784032 ecr 0,nop,wscale 7], length 0
09:57:43.946385 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 780157 ecr 783029,nop,wscale 7], length 0
09:57:45.148027 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 781359 ecr 783029,nop,wscale 7], length 0
09:57:45.950287 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 786036 ecr 0,nop,wscale 7], length 0
09:57:45.950333 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 782161 ecr 783029,nop,wscale 7], length 0
09:57:48.151818 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 784363 ecr 783029,nop,wscale 7], length 0
09:57:49.962678 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 790048 ecr 0,nop,wscale 7], length 0
09:57:49.962719 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 786174 ecr 783029,nop,wscale 7], length 0
09:57:54.162245 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 790373 ecr 783029,nop,wscale 7], length 0
09:58:02.177762 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 798389 ecr 783029,nop,wscale 7], length 0
09:58:18.201944 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 814413 ecr 783029,nop,wscale 7], length 0

2.6 分析

看起来IPVS director以及顺利将请求转发到后端real server，并且real server也向源地址(director所在的192.168.166.102)返回了响应，这个响应也顺利到达了director，但是没有被上层应用接收到。

分析最后的数据包也未见异常：

Frame 2: 76 bytes on wire (608 bits), 76 bytes captured (608 bits)Encapsulation type: Linux cooked-mode capture (25)Arrival Time: Apr 30, 2018 23:13:02.824544000 CST[Time shift for this packet: 0.000000000 seconds]Epoch Time: 1525101182.824544000 seconds[Time delta from previous captured frame: 0.000468000 seconds][Time delta from previous displayed frame: 0.000468000 seconds][Time since reference or first frame: 0.000468000 seconds]Frame Number: 2Frame Length: 76 bytes (608 bits)Capture Length: 76 bytes (608 bits)[Frame is marked: False][Frame is ignored: False][Protocols in frame: sll:ethertype:ip:tcp][Coloring Rule Name: HTTP][Coloring Rule String: http || tcp.port == 80 || http2]
Linux cooked capturePacket type: Unicast to us (0)Link-layer address type: 1Link-layer address length: 6Source: Vmware_75:24:37 (00:0c:29:75:24:37)Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.168.166.111, Dst: 192.168.166.1020100 .... = Version: 4.... 0101 = Header Length: 20 bytesDifferentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)0000 00.. = Differentiated Services Codepoint: Default (0).... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)Total Length: 60Identification: 0x0000 (0)Flags: 0x02 (Don't Fragment)0... .... = Reserved bit: Not set.1.. .... = Don't fragment: Set..0. .... = More fragments: Not setFragment offset: 0Time to live: 64Protocol: TCP (6)Header checksum: 0x6c95 [validation disabled][Good: False][Bad: False]Source: 192.168.166.111Destination: 192.168.166.102[Source GeoIP: Unknown][Destination GeoIP: Unknown]
Transmission Control Protocol, Src Port: 80 (80), Dst Port: 39852 (39852), Seq: 0, Ack: 1, Len: 0Source Port: 80Destination Port: 39852[Stream index: 0][TCP Segment Len: 0]Sequence number: 0    (relative sequence number)Acknowledgment number: 1    (relative ack number)Header Length: 40 bytesFlags: 0x012 (SYN, ACK)Window size value: 28960[Calculated window size: 28960]Checksum: 0x2589 [validation disabled]Urgent pointer: 0Options: (20 bytes), Maximum segment size, SACK permitted, Timestamps, No-Operation (NOP), Window scaleMaximum segment size: 1460 bytesTCP SACK Permitted Option: TrueTimestamps: TSval 5297287, TSecr 5302934No-Operation (NOP)Window scale: 7 (multiply by 128)[SEQ/ACK analysis][This is an ACK to the segment in frame: 1][The RTT to ACK the segment was: 0.000468000 seconds]

转载自https://blog.csdn.net/cloudvtech

三、使用iptables进行debug

根据文章《iptables概览》：

mangle表是在所有五个chain里面都有check point的表，所以可以在mangle表加入LOG target进行包的跟踪。

3.1 在director和real server的iptables插入debug LOG target如下

iptables -t mangle -A PREROUTING -s 192.168.166.102 -d 192.168.166.111 -j LOG --log-prefix "[PREROUTING|mangle] "
iptables -t mangle -A PREROUTING -d 192.168.166.102 -s 192.168.166.111 -j LOG --log-prefix "[PREROUTING|mangle] "
iptables -t mangle -A INPUT -s 192.168.166.102 -d 192.168.166.111 -j LOG --log-prefix "[INPUT|mangle] "
iptables -t mangle -A INPUT -d 192.168.166.102 -s 192.168.166.111 -j LOG --log-prefix "[INPUT|mangle] "
iptables -t mangle -A FORWARD -s 192.168.166.102 -d 192.168.166.111 -j LOG --log-prefix "[FORWARD|mangle] "
iptables -t mangle -A FORWARD -d 192.168.166.102 -s 192.168.166.111 -j LOG --log-prefix "[FORWARD|mangle] "
iptables -t mangle -A OUTPUT -s 192.168.166.102 -d 192.168.166.111 -j LOG --log-prefix "[OUTPUT|mangle] "
iptables -t mangle -A OUTPUT -d 192.168.166.102 -s 192.168.166.111 -j LOG --log-prefix "[OUTPUT|mangle] "
iptables -t mangle -A POSTROUTING -s 192.168.166.102 -d 192.168.166.111 -j LOG --log-prefix "[POSTROUTING|mangle] "
iptables -t mangle -A POSTROUTING -d 192.168.166.102 -s 192.168.166.111 -j LOG --log-prefix "[POSTROUTING|mangle] "

3.2 curl VIP:80

超时

3.3 查看director的iptables的LOG输出如下

Apr 30 18:32:19 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=lo SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0
Apr 30 18:32:19 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0
Apr 30 18:32:19 k8s-node1 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0
Apr 30 18:32:19 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0
Apr 30 18:32:19 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:19 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:20 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:20 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:22 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:22 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:26 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:26 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:35 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:35 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:51 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:51 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0

3.4 查看real server的iptables的LOG输出如下

Apr 30 18:32:17 k8s-node2 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0
Apr 30 18:32:17 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0
Apr 30 18:32:17 k8s-node2 kernel: [PREROUTING|nat] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0
Apr 30 18:32:17 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0
Apr 30 18:32:17 k8s-node2 kernel: INPUT|filter1IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0
Apr 30 18:32:17 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:17 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:17 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:17 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:18 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:18 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:18 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:18 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:21 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:21 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:21 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:21 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:25 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:25 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:25 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:25 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:33 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:33 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:33 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:33 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:49 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:49 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:49 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 18:32:49 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0

3.5 分析

看起来数据包在进入director的mangel表PREROUTING链之后就丢失了。

查阅网络上相关问题，最后发现这个问题( https://www.linuxquestions.org/questions/linux-networking-3/packets-lost-after-mangle-prerouting-chain-4175437227/)与本文的问题类似，是关于地址验证的宽容度rp_filter设置的。
linux内核参数列表( https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)里面这样描述这个参数的： 
rp_filter - INTEGER0 - No source validation.1 - Strict mode as defined in RFC3704 Strict Reverse PathEach incoming packet is tested against the FIB and if the interfaceis not the best reverse path the packet check will fail.By default failed packets are discarded.2 - Loose mode as defined in RFC3704 Loose Reverse PathEach incoming packet's source address is also tested against the FIBand if the source address is not reachable via any interfacethe packet check will fail.Current recommended practice in RFC3704 is to enable strict modeto prevent IP spoofing from DDos attacks. If using asymmetric routingor other complicated routing, then loose mode is recommended.The max value from conf/{all,interface}/rp_filter is usedwhen doing source validation on the {interface}.Default value is 0. Note that some distributions enable itin startup scripts.

rp_filter参数有三个值，0、1、2，具体含义：
0：不开启源地址校验。
1：开启严格的反向路径校验。对每个进来的数据包，校验其反向路径是否是最佳路径。如果反向路径不是最佳路径，则直接丢弃该数据包。
2：开启松散的反向路径校验。对每个进来的数据包，校验其源地址是否可达，即反向路径是否能通（通过任意网口），如果反向路径不同，则直接丢弃该数据包。

所以看看将rp_filter设置成0，不进行任何源地址校验，是否可以让数据包往上层走。

转载自https://blog.csdn.net/cloudvtech

四、设置rp_filter为0继续进行测试

4.1 在sysctl.conf中加入如下配置

net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.ens33.rp_filter = 0

运行sysctl -p

4.2 curl VIP:80可以访问都后端HTTP服务

4.3 查看director的iptables的LOG输出如下

Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58279 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58279 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58279 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58280 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58280 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58280 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=58281 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK PSH URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=58281 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK PSH URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=58281 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK PSH URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2183 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2183 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=5201 TOS=0x00 PREC=0x00 TTL=64 ID=2184 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK PSH URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=5201 TOS=0x00 PREC=0x00 TTL=64 ID=2184 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK PSH URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58282 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58282 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58282 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58283 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK FIN URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58283 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK FIN URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58283 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK FIN URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2188 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK FIN URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2188 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK FIN URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58284 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58284 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58284 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0

4.4 查看real server的iptables的LOG输出如下

Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58279 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58279 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58280 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58280 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=58281 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK PSH URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=58281 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK PSH URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2183 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2183 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=4396 TOS=0x00 PREC=0x00 TTL=64 ID=2184 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=4396 TOS=0x00 PREC=0x00 TTL=64 ID=2184 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=857 TOS=0x00 PREC=0x00 TTL=64 ID=2187 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK PSH URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=857 TOS=0x00 PREC=0x00 TTL=64 ID=2187 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK PSH URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58282 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58282 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58283 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK FIN URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58283 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK FIN URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2188 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK FIN URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2188 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK FIN URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58284 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58284 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0

转载自https://blog.csdn.net/cloudvtech

五、references

https://bugzilla.redhat.com/show_bug.cgi?id=1261410

http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.lvs_clients_on_realservers.html

容器云负载均衡之二：从IPVS DR模式下director不能访问VIP问题的探究相关推荐

  1. 企业级负载均衡集群——lvs的DR模式（直接路由模式）详细说明

    1.DR模式的原理 其实就是在一台主机上面搭建lvs服务器,设置lvs的工作模式是DR模式,lvs仅仅是一个调度器,它会把客户端的请求转发给后备服务器 DR模式直接由后备服务器把数据返回给客户端,不需 ...

  2. 容器云负载均衡之一：容器云平台负载均衡解决方案的一些思考

    一.前言 在典型的网络服务模型中,请求流量/响应流量(Req/Res)比是很小的一个值:在直播.高清视频等视频多媒体服务中,请求响应流量比将更小(几十KB的请求V.S.几十MB的返回).对于这样的应用 ...

  3. Linux系统（五）负载均衡LVS集群之DR模式

    序言 DR模式是lvs集群中三种负载均衡模式的其中一种,那么上一篇中我写啦关于NAT模式的搭建与原理,为什么还要有DR模式与IP隧道模式呢? 首先我们来看3张图.LVS/NAT模式如下图: LVS/I ...

  4. 新功能：阿里云负载均衡SLB支持HTTP访问强制跳转HTTPS

    摘要: 很高兴的告诉大家,阿里云负载均衡SLB已经在澳大利亚(悉尼).日本(东京).阿联酋(迪拜).美国 (弗吉尼亚).美国(硅谷).马来西亚(吉隆坡).德国(法兰克福).新加坡.印度尼西亚(雅加达) ...

  5. 新功能：阿里云负载均衡SLB支持HTTP/HTTPS超时时间自定义功能

    2019独角兽企业重金招聘Python工程师标准>>> 摘要: 大家好,很高兴的告诉大家,阿里云负载均衡SLB已经在新加坡.澳大利亚(悉尼).马来西亚(吉隆坡).日本(东京).美国( ...

  6. 阿里云负载均衡白名单自动修改脚本

    一.应用场景 公司服务器使用阿里云,有很多后台管理系统如kibana.jenkins.Apollo配置中心等应用.为了安全,这些系统是无法从互联网访问的,为了方便管理,使用了阿里云的 负载均衡+白名单 ...

  7. 【minIO集群 配置负载均衡（二）】

    minIO集群 配置负载均衡(二) minIO集群搭建参考:https://blog.csdn.net/qq_38066812/article/details/122477030 配置负载均衡 使用n ...

  8. 阿里云 负载均衡 HTTP转HTTPS

    一.相关文档 1.证书服务 2.简单路由-HTTP 协议变为 HTTPS 协议 二.阿里云操作界面 1.云盾证书服务管理控制台(查询CA证书服务) 2.负载均衡管理控制台 三.相关文档 1.Syman ...

  9. 阿里云负载均衡SLB配置步骤

    阿里云负载均衡--SLB,是将访问流量根据转发策略分发到后端多台云服务器(ECS实例)的流量分发控制服务.包含两种含义:一是通过流量分发,扩展应用系统的服务能力:二是消除单点故障,提高应用系统的可用性 ...

最新文章

  1. 华为鸿蒙2.0打游戏,网友Mate X2升级鸿蒙2.0：部分游戏体验比EMUI更好 功耗却更低...
  2. 【合并单元格】纵向合并单元格之前对数组处理【针对饿了么element的table的span-method合并行或列的计算方法】
  3. Python的scrapy之爬取顶点小说网的所有小说
  4. Apache Kafka-Spring Kafka生产消费@KafkaListener源码解析
  5. 测试回收站测试回收站2测试回收站2测试回收站2测试回收站2测试回收站2测试回收站2测试回收站2测试回收站2测试回收站2测试回收站2测试回收站2测试回收站2测试回收站2测试回收站2测试回收站2测试回收站
  6. PHP函数调用的新的用法
  7. 问题六十八：着色模型（shading model）（2）——光照模型（Light model）
  8. [Ant]Note of develop java with Ant
  9. 三体归零者和盘龙鸿蒙,《三体》中归零者这样的大神级文明已经脱离黑暗森林和猜疑链了吗，为什么？...
  10. Chapter 2 (Discrete Random Variables): Probability mass functions (PMF 分布列)
  11. linux crontab 每30秒,crontab 每 30 秒自動執行
  12. 2021年电工（初级）考试题库及电工（初级）模拟考试
  13. VBA—压缩文件夹成一个rar压缩包
  14. CentOS7.9下安装Oracle19c
  15. DayDayUp：心灵鸡汤之天空飘来五个字~那都不是事(一问一答告诉你什么是高层次的认知)
  16. 「GoTeam 招聘时间」星汉未来 Golang研发高级工程师
  17. C# 微信扫码授权登录
  18. shell脚本——sed编辑器
  19. 解决GitHub release下载慢、clone和push慢的问题
  20. 通过运行命令services.msc快速打开服务和注册

热门文章

  1. Rides持久化问题
  2. oracle数据库中的注释
  3. Java程序员：一整个项目的具体开发流程介绍
  4. 工作8年，跳槽字节跳动，月薪从1w涨到7w，网友：本科的话有点牛逼了
  5. emqttd 0.9.0版本的会话(Session)、消息队列(MQueue)、飞行窗口(Inflight Window)设计
  6. Thymeleaf教程（10分钟入门）
  7. 【如何用python计算利率互换收盘曲线的即期利率】
  8. 漫画城之引导页+主页
  9. linux megaraid恢复raid,MegaCli修改RAID级别
  10. Quartz+ssm注解方式的最最最最简单使用