NGINX Ingress 控制器返回 502，后端应用程序 pod 中没有日志

如何解决NGINX Ingress 控制器返回 502，后端应用程序 pod 中没有日志

我已经在我的 kubernetes 集群（所有 vagrant 虚拟机）上部署了 ECK。集群有以下配置。

NAME       STATUS   ROLES                       AGE   VERSION 
kmaster1   Ready    control-plane,master        27d     v1.21.1 
kworker1   Ready    <none>                      27d     v1.21.1
kworker2   Ready    <none>                      27d     v1.21.1

我还使用 HAProxy 设置了负载均衡器。负载均衡器配置如下（创建我自己的私有证书）

frontend http_front
  bind *:80
  stats uri /haproxy?stats
  default_backend http_back

frontend https_front
  bind *:443 ssl crt /etc/ssl/private/mydomain.pem
  stats uri /haproxy?stats
  default_backend https_back


backend http_back
  balance roundrobin
  server kworker1 172.16.16.201:31953
  server kworker2 172.16.16.202:31953
 

backend https_back
  balance roundrobin
  server kworker1 172.16.16.201:31503 check-ssl ssl verify none
  server kworker2 172.16.16.202:31503 check-ssl ssl verify none

我还部署了一个 Nginx 入口控制器和 31953是Nginx控制器的http端口 31503是Nginx控制器的https端口

nginx-ingress    nginx-ingress-controller-service   NodePort    10.103.189.197   <none>        80:31953/TCP,443:31503/TCP   8d    app=nginx-ingress

我正在尝试通过 https 使 kibana 仪表板在集群外部可用。它工作正常，我可以在集群内访问它。但是我无法通过负载均衡器访问它。

Kibana Pod：

default          quickstart-kb-f74c666b9-nnn27              1/1     Running   4          27d   192.168.41.145   kworker1   <none>           <none>

我已将负载均衡器映射到主机

172.16.16.100   elastic.kubekluster.com

对 https://elastic.kubekluster.com 的任何请求都会导致以下错误（来自 Nginx 入口控制器 pod 的日志）

 10.0.2.15 - - [20/Jun/2021:17:38:14 +0000] "GET / HTTP/1.1" 502 157 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0" "-"
2021/06/20 17:38:14 [error] 178#178: *566 upstream prematurely closed connection while reading response header from upstream,client: 10.0.2.15,server: elastic.kubekluster.com,request: "GET / H
TTP/1.1",upstream: "http://192.168.41.145:5601/",host: "elastic.kubekluster.com"

HAproxy 日志如下

Jun 20 18:11:45 loadbalancer haproxy[18285]: 172.16.16.1:48662 [20/Jun/2021:18:11:45.782] https_front~ https_back/kworker2 0/0/0/4/4 502 294 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

入口如下

apiVersion: networking.k8s.io/v1
kind: Ingress
Metadata:
  name: kubekluster-elastic-ingress
  annotations:
    kubernetes.io/ingress.class: Nginx
    Nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    Nginx.ingress.kubernetes.io/default-backend: quickstart-kb-http
    Nginx.ingress.kubernetes.io/rewrite-target: /
    Nginx.ingress.kubernetes.io/proxy-connect-timeout: "600s"
    Nginx.ingress.kubernetes.io/proxy-read-timeout: "600s"
    Nginx.ingress.kubernetes.io/proxy-send-timeout: "600s"
    Nginx.ingress.kubernetes.io/proxy-body-size: 20m
spec:
  tls:
    - hosts:
      - elastic.kubekluster.com
  rules:
  - host: elastic.kubekluster.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: quickstart-kb-http
            port: 
              number: 5601

我认为请求没有到达 kibana pod，因为我在 pod 中没有看到任何日志。另外我不明白为什么 Haproxy 将请求作为 HTTP 而不是 HTTPS 发送。 你能指出我的配置有什么问题吗？

解决方法

我希望这会有所帮助...以下是我使用 nginx 设置“LoadBalancer”并将流量转发到 HTTPS 服务的方法：

 kubectl get nodes -o wide 
NAME           STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP      OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
asd-master-1   Ready    master   72d   v1.19.8   192.168.1.163   213.95.154.199   Ubuntu 20.04.2 LTS   5.8.0-45-generic   docker://20.10.6
asd-node-1     Ready    <none>   72d   v1.19.8   192.168.1.101   <none>           Ubuntu 20.04.1 LTS   5.8.0-45-generic   docker://19.3.15
asd-node-2     Ready    <none>   72d   v1.19.8   192.168.0.5     <none>           Ubuntu 20.04.1 LTS   5.8.0-45-generic   docker://19.3.15
asd-node-3     Ready    <none>   15d   v1.19.8   192.168.2.190   <none>           Ubuntu 20.04.1 LTS   5.8.0-45-generic   docker://19.3.15

这是nginx的服务：

# kubectl get service -n ingress-nginx
NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.101.161.113   <none>        80:30337/TCP,443:31996/TCP   72d

这是负载均衡器配置：

# cat /etc/nginx/nginx.conf
... trimmed ...
stream {
    upstream nginx_http {
        least_conn;
        server asd-master-1:30337 max_fails=3 fail_timeout=5s;
        server asd-node-1:30337 max_fails=3 fail_timeout=5s;
        server asd-node-2:30337 max_fails=3 fail_timeout=5s;
    }
    server {
        listen 80;
        proxy_pass nginx_http;
        proxy_protocol on;
    }

    upstream nginx_https {
        least_conn;
        server 192.168.1.163:31996 max_fails=3 fail_timeout=5s;
        server 192.168.1.101:31996 max_fails=3 fail_timeout=5s;
        server 192.168.0.5:31996 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     443;
        proxy_pass nginx_https;
        proxy_protocol on;
    }

}

相关部分是我正在发送代理协议。您需要配置 nginx 入口（在配置映射中）以接受这一点，并且可能将正确的语法添加到 haproxy 配置中。

这可能类似于：

backend https_back
  balance roundrobin
  server kworker1 172.16.16.201:31503 check-ssl ssl verify none send-proxy-v2
  server kworker2 172.16.16.202:31503 check-ssl ssl verify none send-proxy-v2

Nginx Ingress 配置应该是：

# kubectl get configmap -n ingress-nginx  nginx-configuration -o yaml
apiVersion: v1
data:
  use-proxy-protocol: "true"
kind: ConfigMap
metadata:
...

我希望这能让你走上正轨。