如何解决如何在索引时停止在内容中存储特殊字符
这是一个示例文档,具有以下几点: 制药业 营销学 建筑物– 责任。 马萨诸塞州-2020年8月13日-
如何在索引时从内容中删除特殊字符或非ASCII Unicode字符?我正在使用ES 7.x和Storm搜寻器1.17
解决方法
看起来像是对字符集的错误检测。您可以在写入索引之前通过写custom parse filter并删除那里不需要的字符来规范内容。
,如果编写自定义的解析过滤器和规范化对您来说似乎很困难。您只需在分析器定义中添加asciifolding token filter,即可将非ascii字符转换为其ascii字符,如下所示
POST http:// {{hostname}}:{{port}} / _analyze
{
"tokenizer": "standard","filter": [
"asciifolding"
],"text": "Pharmaceutical Marketing Building â responsibilities.  Mass. â Aug. 13,2020 âÂ"
}
并为您的文本生成令牌。
{
"tokens": [
{
"token": "Pharmaceutical","start_offset": 0,"end_offset": 14,"type": "<ALPHANUM>","position": 0
},{
"token": "Marketing","start_offset": 15,"end_offset": 24,"position": 1
},{
"token": "Building","start_offset": 25,"end_offset": 33,"position": 2
},{
"token": "a","start_offset": 34,"end_offset": 35,"position": 3
},{
"token": "responsibilities.A","start_offset": 36,"end_offset": 54,"position": 4
},{
"token": "A","start_offset": 55,"end_offset": 56,"position": 5
},{
"token": "Mass","start_offset": 57,"end_offset": 61,"position": 6
},"start_offset": 63,"end_offset": 64,"position": 7
},{
"token": "Aug","start_offset": 65,"end_offset": 68,"position": 8
},{
"token": "13","start_offset": 70,"end_offset": 72,"type": "<NUM>","position": 9
},{
"token": "2020","start_offset": 74,"end_offset": 78,"position": 10
},{
"token": "aA","start_offset": 79,"end_offset": 81,"position": 11
}
]
}
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。