如何解决为什么 EMQX Persistence 在本地 kubernetes 上工作时不能在 azure kubernetes 上工作？
在本地机器上使用 kubernetes(minikube) statefulset 时，EMQX 规则持续存在，因为相同的 pod IP 被分配给 emqx 节点，例如 /opt/emqx/data/mnesia /emqx@172.17.0.9。即使我在新 pod 启动时删除了 pod，它也会被分配与以前相同的 IP。一切正常。
但是当我使用 aks(azure kubernetes) 在使用 azure 文件的 aks 集群上部署 EMQX 时，pod IP 每次都不同。例如，如果 /opt/emqx/data/mnesia/emqx@10.1.1.10 被分配给 EMQX 节点，那么如果我尝试删除 pod，则 /opt/emqx/data/ mnesia/emqx@10.1.1.11 可能已分配给它。
所以，没有什么是持久的。
本地代码
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
Metadata:
name: local-storage5
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
Metadata:
name: emqx-pv5
spec:
capacity:
storage: 300Mi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage5
local:
path: /opt/emqx/data/mnesia
nodeAffinity:
required:
nodeselectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- minikube
---
apiVersion: v1
kind: Service
Metadata:
name: emqx-headless
spec:
type: ClusterIP
clusterIP: None
selector:
app: emqx
ports:
- name: mqtt
port: 1883
protocol: TCP
targetPort: 1883
- name: mqttssl
port: 8883
protocol: TCP
targetPort: 8883
- name: mgmt
port: 8081
protocol: TCP
targetPort: 8081
- name: websocket
port: 8083
protocol: TCP
targetPort: 8083
- name: wss
port: 8084
protocol: TCP
targetPort: 8084
- name: dashboard
port: 18083
protocol: TCP
targetPort: 18083
---
apiVersion: apps/v1
kind: StatefulSet
Metadata:
name: emqx-statefulset
labels:
app: emqx
spec:
replicas: 1
serviceName: emqx-headless
selector:
matchLabels:
app: emqx
template:
Metadata:
labels:
app: emqx
spec:
containers:
- name: emqx
image: emqx/emqx:4.2.7
ports:
- name: emqx-dashboard
containerPort: 18083
- name: ssl-port
containerPort: 8883
- name: emqx-port
containerPort: 1883
- name: ssl-dashboard
containerPort: 18084
env:
- name: EMQX_LOADED_PLUGINS
value: emqx_management,emqx_recon,emqx_retainer,emqx_dashboard,emqx_rule_engine,emqx_auth_username
- name: EMQX_CLUSTER__disCOVERY
value: k8s
- name: EMQX_NAME
value: emqx
- name: EMQX_CLUSTER__K8S__APISERVER
value: https://kubernetes.default:443
- name: EMQX_CLUSTER__K8S__SERVICE_NAME
value: emqx
- name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
value: ip
- name: EMQX_CLUSTER__K8S__APP_NAME
value: emqx
- name: EMQX_ALLOW_ANONYMOUS
value: "false"
- name: EMQX_LISTENER__SSL__EXTERNAL__MAX_CONNECTIONS
value: "1024000"
- name: EMQX_AUTH__USER__PASSWORD_HASH
value: sha256
- name: EMQX_AUTH__USER__1__USERNAME
value:
- name: EMQX_AUTH__USER__1__PASSWORD
value:
- name: EMQX_DASHBOARD__DEFAULT_USER__LOGIN
value:
- name: EMQX_DASHBOARD__DEFAULT_USER__PASSWORD
value:
- name: EMQX_DASHBOARD__LISTENER__HTTPS
value: "18084"
- name: MQX_DASHBOARD__LISTENER__HTTPS__ACCEPTORS
value: "4"
- name: EMQX_DASHBOARD__LISTENER__HTTPS__MAX_CLIENTS
value: "512"
tty: true
volumeMounts:
- name: emqx-mnesia
mountPath: "/opt/emqx/data/mnesia"
volumeClaimTemplates:
- Metadata:
name: emqx-mnesia
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "local-storage5"
resources:
requests:
storage: 300Mi
Azure Kubernetes 代码
apiVersion: v1
kind: ServiceAccount
Metadata:
name: emqx
namespace: emqx-test
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
Metadata:
name: emqx
subjects:
- kind: ServiceAccount
name: emqx
namespace: emqx-test
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
Metadata:
name: emqx-files
provisioner: kubernetes.io/azure-file
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=0
- gid=0
- mfsymlinks
- cache=strict
- actimeo=30
parameters:
skuName: Standard_lrs
---
apiVersion: v1
kind: PersistentVolumeClaim
Metadata:
name: emqx-pvc
namespace: emqx-test
spec:
accessModes:
- ReadWriteMany
storageClassName: emqx-files
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Service
Metadata:
name: emqx
namespace: emqx-test
spec:
ports:
- name: emqx-dashboard
port: 80
targetPort: 18083
protocol: TCP
- name: ssl-port
port: 8883
targetPort: ssl-port
protocol: TCP
- name: emqx-port
port: 1883
targetPort: emqx-port
protocol: TCP
- name: ssl-dashboard
port: 443
targetPort: 18084
protocol: TCP
selector:
app: emqx
type: LoadBalancer
---
apiVersion: apps/v1
kind: StatefulSet
Metadata:
name: emqx
labels:
app: emqx
namespace: emqx-test
spec:
serviceName: "emqx"
selector:
matchLabels:
app: emqx
replicas: 1
template:
Metadata:
labels:
app: emqx
spec:
containers:
- name: emqx
image: emqx/emqx:4.2.7
ports:
- name: emqx-dashboard
containerPort: 18083
- name: ssl-port
containerPort: 8883
- name: emqx-port
containerPort: 1883
- name: ssl-dashboard
containerPort: 18084
env:
- name: EMQX_LOADED_PLUGINS
value: emqx_management,emqx_auth_username
- name: EMQX_CLUSTER__disCOVERY
value: k8s
- name: EMQX_NAME
value: emqx
- name: EMQX_CLUSTER__K8S__APISERVER
value: https://kubernetes.default:443
- name: EMQX_CLUSTER__K8S__NAMESPACE
value: emqx-test
- name: EMQX_CLUSTER__K8S__SERVICE_NAME
value: emqx
- name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
value: ip
- name: EMQX_CLUSTER__K8S__APP_NAME
value: emqx
- name: EMQX_ALLOW_ANONYMOUS
value: "false"
- name: EMQX_LISTENER__SSL__EXTERNAL__MAX_CONNECTIONS
value: "1024000"
- name: EMQX_AUTH__USER__PASSWORD_HASH
value: sha256
- name: EMQX_AUTH__USER__1__USERNAME
value:
- name: EMQX_AUTH__USER__1__PASSWORD
value:
- name: EMQX_DASHBOARD__DEFAULT_USER__LOGIN
value:
- name: EMQX_DASHBOARD__DEFAULT_USER__PASSWORD
value:
- name: EMQX_DASHBOARD__LISTENER__HTTPS
value: "18084"
- name: MQX_DASHBOARD__LISTENER__HTTPS__ACCEPTORS
value: "4"
- name: EMQX_DASHBOARD__LISTENER__HTTPS__MAX_CLIENTS
value: "512"
volumeMounts:
- name: emqx-data
mountPath: "/opt/emqx/data/mnesia"
tty: true
volumes:
- name: emqx-data
persistentVolumeClaim:
claimName: emqx-pvc
解决方法
在 StatefulSet Basics 上的 k8s 文档中，您阅读了：
Pod 的序号、主机名、SRV 记录和 A 记录名称具有 未更改，但与 Pod 关联的 IP 地址可能已更改 改变了。在本教程使用的集群中，他们有。 这就是为什么 重要的是不要配置其他应用程序来连接到 Pod 在按 IP 地址的 StatefulSet 中。
这是预期的，正如您所看到的，文档中提到了这种行为。
但是为什么你在 minikube 上看到不同的行为，在 azure 上看到不同的行为？ IP 地址由 CNI 分配。在 minikube 默认 CNI 上它是 docker-bridge，在 azure 上是它的 Azure CNI，所以它是由 CNI 分配的地址。
最好始终假设您不能依赖 pod IP 地址来保持静态。使用 DNS for statefulsets 和 for other pods and services 进行通信，切勿直接使用硬编码的 pod ip 地址。
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。
