如何解决Fluentd Sidecar无法将日志发送到Elasticsearch:类型删除已弃用在批量请求中指定类型
我首先使用以下配置测试了fleunt.conf
:
<source>
@type forward
bind "127.0.0.1"
port 24224
<parse>
@type json
</parse>
</source>
<match app.default>
@type stdout
</match>
当我使用fluentd
将kubectl
容器的日志拖到pod中时,效果很好,我可以看到JSON格式的应用日志。
现在,我正在尝试将日志发送到elasticsearch
。这是我的fluent.conf
,表示相同:
<source>
@type forward
bind "127.0.0.1"
port 24224
<parse>
@type json
</parse>
</source>
<match app.default>
@type elasticsearch
host "elasticsearch_host"
port 9200
index_name "app-log"
user "log_user"
password xxxxxx
</match>
流利的容器在pod内运行良好,但是我无法定义在Kibana的fluetn.conf
中配置的索引。看来索引模式没有加载到elasticsearch
中。
fluentd
容器显示以下警告:
2020-10-22 12:31:10 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-10-22 12:31:10 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.0'
2020-10-22 12:31:10 +0000 [info]: gem 'fluentd' version '1.10.4'
2020-10-22 12:31:12 +0000 [info]: using configuration file: <ROOT>
<source>
@type forward
bind "127.0.0.1"
port 24224
<parse>
@type json
</parse>
</source>
<match app.default>
@type elasticsearch
host "elasticsearch_host"
port 9200
index_name "app-log"
user "log_user"
password xxxxxx
</match>
</ROOT>
2020-10-22 12:31:12 +0000 [info]: starting fluentd-1.10.4 pid=8 ruby="2.5.8"
2020-10-22 12:31:12 +0000 [info]: spawn command to main: cmdline=["/usr/bin/ruby","-Eascii-8bit:ascii-8bit","/usr/bin/fluentd","-c","/fluentd/etc/fluent.conf","-p","/fluentd/plugins","--under-supervisor"]
2020-10-22 12:31:16 +0000 [info]: adding match pattern="app.default" type="elasticsearch"
2020-10-22 12:31:17 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2020-10-22 12:31:17 +0000 [info]: adding source type="forward"
2020-10-22 12:31:17 +0000 [warn]: section <parse> is not used in <source> of forward plugin
2020-10-22 12:31:17 +0000 [info]: #0 starting fluentd worker pid=22 ppid=8 worker=0
2020-10-22 12:31:17 +0000 [info]: #0 listening port port=24224 bind="127.0.0.1"
2020-10-22 12:31:17 +0000 [info]: #0 fluentd worker is Now running worker=0
warning: 299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Specifying types in bulk requests is deprecated."
warning: 299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Specifying types in bulk requests is deprecated."
warning: 299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Specifying types in bulk requests is deprecated."
如何摆脱此警告"[types removal] Specifying types in bulk requests is deprecated."
? fluent.conf
是否有任何其他配置可以帮助我克服这些困难?
我尝试在配置中使用suppress_type_name true
,但没有用。
解决方法
仅在插件的版本4.0.10 https://github.com/uken/fluent-plugin-elasticsearch/blob/master/History.md#4010中添加了对suppress_type_name
true的支持。
根据日志,您正在运行4.0.0,因此升级和设置config中的标志应摆脱警告。该警告不应影响批量请求。
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。