无法使用Apache Kafka启动带有MongoDb插件的Kafka Connect

如何解决无法使用Apache Kafka启动带有MongoDb插件的Kafka Connect

我是Kafka的新手，我想看看是否可以使用Kafka将MongoDb数据与另一个系统同步。

我的设置：

我正在运行AWS MSK Cluster，并且已经使用Kafka客户端手动创建了EC2实例。
我已将MongoDB Kafka Connect插件添加到/usr/local/share/kafka/plugins。
我正在运行Kafka connect，可以看到它已加载插件

./bin/connect-standalone.sh ./config/connect-standalone.properties /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/MongoSourceConnector.properties

[2020-10-17 13:57:22,304] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:264)
[2020-10-17 13:57:22,305] INFO Added plugin 'com.mongodb.kafka.connect.MongoSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-10-17 13:57:22,305] INFO Added plugin 'com.mongodb.kafka.connect.MongoSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)

解压后的插件具有这种结构

Archive:  mongodb-kafka-connect-mongodb-1.3.0.zip
   creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/
   creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/
  inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/MongoSourceConnector.properties  
  inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/MongoSinkConnector.properties  
   creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/doc/
  inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/doc/README.md  
  inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/doc/LICENSE.txt  
  inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/manifest.json  
   creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/lib/
  inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/lib/mongo-kafka-1.3.0-all.jar  
   creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/assets/
  inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/assets/mongodb-leaf.png  
  inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/assets/mongodb-logo.png

此插件来自融合页面，我也尝试过从Maven页面下载它。问题是我运行Kafka Connect时失败，因为插件缺少Java依赖项。

[2020-10-17 13:57:24,898] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:121)
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/avro/Schema
    at com.mongodb.kafka.connect.source.MongoSourceConfig.createConfigDef(MongoSourceConfig.java:591)
    at com.mongodb.kafka.connect.source.MongoSourceConfig.<clinit>(MongoSourceConfig.java:293)
    at com.mongodb.kafka.connect.MongoSourceConnector.config(MongoSourceConnector.java:91)
    at org.apache.kafka.connect.connector.Connector.validate(Connector.java:129)
    at com.mongodb.kafka.connect.MongoSourceConnector.validate(MongoSourceConnector.java:51)
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:313)
    at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:192)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:115)
Caused by: java.lang.NoClassDefFoundError: org/apache/avro/Schema
    ... 8 more
Caused by: java.lang.classNotFoundException: org.apache.avro.Schema
    at java.net.urlclassloader.findClass(urlclassloader.java:382)
    at java.lang.classLoader.loadClass(ClassLoader.java:418)
    at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
    at java.lang.classLoader.loadClass(ClassLoader.java:351)
    ... 8 more

我的印象是，插件应该在jar文件/usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/lib/mongo-kafka-1.3.0-all.jar中而不是Java SDK中寻找依赖项。

此设置中我缺少什么？

解决方法

快速浏览一下应该可以告诉您错误是否正确...

jar -tf  mongo-kafka-1.3.0-all.jar | grep avro

如果该JAR自身未捆绑Avro，则MSK很可能不像Confluent Platform那样包括Avro（我假设Mongo主要将其连接器捆绑了）。至少，Avro不是Apache Kafka的依赖项，因此可以解释该错误。

您将需要下载Avro JAR并将其放在Kafka Connect Classpath中（或至少在该lib文件夹中）

我在本地运行时遇到了同样的问题。我从不再提供超级 jar 的融合平台下载了 jar(mongo-kafka-connect-1.6.0-confluent.jar)。所以我搜索了 uber jar 并找到了下面的站点，我可以从那里下载 uber jar（在下载下拉列表中选择全部）并解决了问题。

https://search.maven.org/search?q=a:mongo-kafka-connect