如何解决在kafka中设置ACL后无法生产
我在本地使用 wurstmeister kafka 和 zookeeper docker 镜像来测试 kafka 中的 SASL 和 ACL。
我的 docker-compose.yml 是 -
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
hostname: zookeeper
container_name: zookeeper
volumes:
- ./zookeeper/zookeeper.sasl.jaas.config:/etc/kafka/zookeeper_server_jaas.conf
- ./zk/data:/var/lib/zookeeper/data
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_SET_ACL: 'true'
KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/zookeeper_server_jaas.conf
-Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
-Dzookeeper.allowSaslFailedClients=false
-Dzookeeper.requireClientAuthScheme=sasl
broker:
image: wurstmeister/kafka:2.13-2.6.0
hostname: broker
container_name: broker
depends_on:
- zookeeper
ports:
- "9092:9092"
volumes:
- ./kafka/kafka.jaas.conf:/etc/kafka/kafka_server_jaas.conf
- ./kfk/data:/kafka
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_Security_PROTOCOL_MAP: EXTERNAL:SASL_PLAINTEXT
KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
KAFKA_AUTO_CREATE_TOPIC: 'true'
KAFKA_LISTENERS: EXTERNAL://:9092
KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:9092
KAFKA_ADVERTISED_PORT: 9092
KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
KAFKA_LISTENER_NAME_EXTERNAL_SASL_ENABLED_MECHANISMS: PLAIN
KAFKA_LISTENER_NAME_EXTERNAL_PLAIN_SASL_JAAS_CONfig: |
org.apache.kafka.common.security.plain.PlainLoginModule required \
username="broker" \
password="broker" \
user_broker="broker" \
user_client="client-secret" \
user_alice="alice-secret";
KAFKA_SASL_MECHANISM_INTER_broKER_PROTOCOL: PLAIN
KAFKA_INTER_broKER_LISTENER_NAME: EXTERNAL
以下是zookeeper和kafka的jaas文件-
zookeeper.sasl.jaas.config -
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_kafka="kafka";
};
kafka.jaas.config -
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="kafka"
password="kafka";
};
我创建了 zookeeper 和 kafka 容器并在 kafka 容器中运行命令 -
/opt/kafka_2.13-2.6.0/bin # ./kafka-acls.sh --authorizer-properties zookeeper.connect=zookeeper:2181 --add --allow-principal User:alice --producer --topic testtopic
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC,name=testtopic,patternType=LIteraL)`:
(principal=User:alice,host=*,operation=DESCRIBE,permissionType=ALLOW)
(principal=User:alice,operation=WRITE,operation=CREATE,permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC,permissionType=ALLOW)
但是当我尝试从我的 go 代码(使用 Sarama)中生成事件时 - 它给出了错误
kafka server: In the middle of a leadership election,there is currently no leader for this partition and hence it is unavailable for writes.
我的代码是 -
package main
import "github.com/Shopify/Sarama"
var brokers = []string{"127.0.0.1:9092"}
func newProducer() (Sarama.SyncProducer,error) {
config := Sarama.NewConfig()
config.Producer.Partitioner = Sarama.NewRandomPartitioner
config.Producer.requiredAcks = Sarama.WaitForAll
config.Producer.Return.Successes = true
config.Net.SASL.User = "alice"
config.Net.SASL.Password = "alice-secret"
config.Net.SASL.Handshake = true
config.Net.SASL.Enable = true
producer,err := Sarama.NewSyncProducer(brokers,config)
return producer,err
}
func prepareMessage(topic,message string) *Sarama.ProducerMessage {
msg := &Sarama.ProducerMessage{
Topic: topic,Partition: -1,Value: Sarama.StringEncoder(message),}
return msg
}
func panicOnError(err error) {
if err != nil {
panic(err)
}
}
func main() {
producer,err := newProducer()
panicOnError(err)
msg := prepareMessage("testtopic",`{"key":"value"}`)
_,_,err = producer.SendMessage(msg)
panicOnError(err)
}
我也尝试了带有 --bootstrap-server (command - ./kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:alice --producer --topic testtopic
) 参数的 kafka-acls.sh,但是脚本会卡住,我可以在 kafka docker 日志中观察到身份验证错误 -
[2021-05-29 16:27:46,288] INFO [SocketServer brokerId=1002] Failed authentication with /127.0.0.1 (Unexpected Kafka request of type MetaDATA during SASL handshake.) (org.apache.kafka.common.network.Selector)
PS:如果我只使用 SASL(没有 ACL),一切都正常
现在我被困在 acl 部分。任何人都知道我缺少什么(可能在 zookeeper 或 kafka 配置中)?
感谢任何帮助。提前致谢。
解决方法
对于您的第一个问题,我会尝试以下建议 https://github.com/Shopify/sarama/issues/272
对于第二个问题,您应该在命令行中添加 --command-config /path/cmd.cfg
指示管理客户端属性以连接您的代理,例如 mechainsem SASL 等...
KAFKA_OPTS 设置 jaas 文件
jaas 文件应该包含 KafkaClient 和用户,密码以使用 PLAIN 身份验证方法连接到您的代理
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。