kubernetes 运维 k8s svc通过externalIPs并结合Keepalived对外暴露服务

UG多轴编程理论知识运维 2024-01-06 16 0

Kubemetes 的 Service需要对集群外暴露，可以通过NodePort、Ingress和LoadBalancer，其中 NodePort：需要预先知道k8s集群节点的IP地址，在使用时需指定node的ip（这样配置存在单点故障）； Ingress：实现的是HTTP(S)负载均衡器，只能代理七层； LoadBalancer：需要通过云服务商提供的负载均衡器将服务暴露到集群外部。在非公有云环境的k8s集群上，ClusterIp类型的Service可通过externalIPs设置一个外部的 IP 地址，并且将流量导入到集群内部。externalIps(外部IP)要求是至少能路由到一个k8s节点上。即如果有外部IP可以路由到一个或多个k8s节点上，就可以把k8s的Service暴露在这个外部IP上，通过访问外部IP+Service的端口将流量接入到集群内。这个IP再通过Keepalived配置为VIP，浮动于多个node节点上，即可避免NodePort单点故障问题。以下是实现步骤

一、基础环境

一个自建的k8s集群，测试环境配置如下

kubectl get node -owide

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME

192.168.5.124 Ready,SchedulingDisabled master 3h13m v1.21.12 192.168.5.124 Ubuntu 20.04.3 LTS 5.4.0-121-generic docker://19.3.12

192.168.5.134 Ready node 3h11m v1.21.12 192.168.5.134 Ubuntu 20.04.3 LTS 5.4.0-81-generic docker://19.3.12

192.168.5.144 Ready node 3h11m v1.21.12 192.168.5.144 Ubuntu 20.04.3 LTS 5.4.0-81-generic docker://19.3.12

master节点网卡信息

ifconfig

ens32: flags=4163 mtu 1500

inet 192.168.5.124 netmask 255.255.255.0 broadcast 192.168.5.255

node1节点网卡信息

ifconfig

ens32: flags=4163 mtu 1500

inet 192.168.5.134 netmask 255.255.255.0 broadcast 192.168.5.255

node2节点网卡信息

ens32: flags=4163 mtu 1500

inet 192.168.5.144 netmask 255.255.255.0 broadcast 192.168.5.255

二、Keepalived生成externalIPs

在node1 和 node2上安装配置Keepalived node1 Keepalived配置

cat /etc/keepalived/keepalived.conf

global_defs {

notification_email {

root@localhost

}

notification_email_from keepalived@localhost

smtp_server 127.0.0.1

smtp_connect_timeout 30

router_id node1

vrrp_skip_check_adv_addr

#vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

vrrp_mcast_group4 224.0.0.18

}

vrrp_instance EIP {

state MASTER

interface ens32

virtual_router_id 66

priority 100

advert_int 1

authentication {

auth_type PASS #预共享密钥认证，同一个虚拟路由器的keepalived节点必须一样

auth_pass 12345678

}

virtual_ipaddress {

192.168.5.200 dev ens32 label ens32:0

}

node2 Keepalived配置

cat /etc/keepalived/keepalived.conf

global_defs {

notification_email {

root@localhost

}

notification_email_from keepalived@localhost

smtp_server 127.0.0.1

smtp_connect_timeout 30

router_id node2

vrrp_skip_check_adv_addr

#vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

vrrp_mcast_group4 224.0.0.18

}

vrrp_instance VI_1 {

state BACKUP

interface ens32

virtual_router_id 66

priority 80

advert_int 1

authentication {

auth_type PASS #预共享密钥认证，同一个虚拟路由器的keepalived节点必须一样

auth_pass 12345678

}

virtual_ipaddress {

192.168.5.200 dev ens32 label ens32:0

}

启动Keepalived并验证

systemctl daemon-reload

systemctl start keepalived.service

systemctl enable keepalived.service

systemctl status keepalived.service

● keepalived.service - Keepalive Daemon (LVS and VRRP)

Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)

Active: active (running) since Wed 2023-01-04 14:26:40 CST; 1h 12min ago

Main PID: 156590 (keepalived)

Tasks: 2 (limit: 9413)

Memory: 1.8M

CGroup: /system.slice/keepalived.service

├─156590 /usr/sbin/keepalived --dont-fork

└─156603 /usr/sbin/keepalived --dont-fork

查看VIP信息

root@node1:~# ifconfig ens32:0

ens32:0: flags=4163 mtu 1500

inet 192.168.5.200 netmask 255.255.255.255 broadcast 0.0.0.0

root@node2:~# ping -c4 192.168.5.200

PING 192.168.5.200 (192.168.5.200) 56(84) bytes of data.

64 bytes from 192.168.5.200: icmp_seq=1 ttl=64 time=0.061 ms

64 bytes from 192.168.5.200: icmp_seq=2 ttl=64 time=0.074 ms

64 bytes from 192.168.5.200: icmp_seq=3 ttl=64 time=0.071 ms

64 bytes from 192.168.5.200: icmp_seq=4 ttl=64 time=0.068 ms

--- 192.168.5.200 ping statistics ---

4 packets transmitted, 4 received, 0% packet loss, time 3060ms

rtt min/avg/max/mdev = 0.061/0.068/0.074/0.004 ms

三、创建svc并指定externalIPs

借用大佬的镜像创建一个deployment，docker hub地址：https://hub.docker.com/r/ikubernetes/demoapp

cat nginx.yaml

---

apiVersion: apps/v1

kind: Deployment

metadata:

name: nginx-deploy

labels:

app: nginx

spec:

replicas: 2

selector:

matchLabels:

app: nginx

template:

metadata:

labels:

app: nginx

spec:

containers:

- name: nginx

image: ikubernetes/demoapp:v1.0

---

apiVersion: v1

kind: Service

metadata:

labels:

app: nginx-svc

name: nginx-svc

namespace: default

spec:

ports:

- name: http

port: 80

targetPort: 80

protocol: TCP

selector:

app: nginx

externalIPs:

- 192.168.5.200 # 设置 externalIPs 为VIP

创建资源

kubectl apply -f nginx.yaml

kubectl get pod

NAME READY STATUS RESTARTS AGE

nginx-deploy-7788867569-hdm4g 1/1 Running 0 64m

nginx-deploy-7788867569-wcqhp 1/1 Running 0 64m

kubectl get svc

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

nginx-svc ClusterIP 10.96.225.101 192.168.5.200 80/TCP 70m

访问测试

while true; do curl 192.168.5.200;sleep 1; done