如何解决Spacy BILOU格式转换为spacy json格式
我正试图将我的spacy版本升级到每晚,尤其是对于使用spacy转换器的情况
所以我转换了格式像这样的伪造简单火车数据集
td = [["Who is Shaka Khan?",{"entities": [(7,17,"FRIENDS")]}],["I like London.",13,"LOC")]}],]
高于
[[{"head": 0,"dep": "","tag": "","orth": "Who","ner": "O","id": 0},{"head": 0,"orth": "is","id": 1},"orth": "Shaka","ner": "B-FRIENDS","id": 2},"orth": "Khan","ner": "L-FRIENDS","id": 3},"orth": "?","id": 4}],[{"head": 0,"orth": "I","orth": "like","orth": "London","ner": "U-LOC","orth": ".","id": 3}]]
使用以下脚本
sentences = []
for t in td:
doc = nlp(t[0])
tags = offsets_to_biluo_tags(doc,t[1]['entities'])
ner_info = list(zip(doc,tags))
tokens = []
for n,i in enumerate(ner_info):
token = {"head" : 0,"dep" : "","tag" : "","orth" : i[0].orth_,"ner" : i[1],"id" : n}
tokens.append(token)
sentences.append(tokens)
with open("train_data.json","w") as js:
json.dump(sentences,js)```
then i tried to convert this train_data.json using
spacy's convert command
```python -m spacy convert train_data.json converted/```
but the result in converted folder is
```✔ Generated output file (0 documents): converted/train_data.spacy```
which means it doesn't created dataset
can anybody help on what i am missing
i am trying to do this with spacy-nightly
解决方法
您可以跳过JSON中间步骤,并将注释直接转换为DocBin
。
import spacy
from spacy.training import Example
from spacy.tokens import DocBin
td = [["Who is Shaka Khan?",{"entities": [(7,17,"FRIENDS")]}],["I like London.",13,"LOC")]}],]
nlp = spacy.blank("en")
db = DocBin()
for text,annotations in td:
example = Example.from_dict(nlp.make_doc(text),annotations)
db.add(example.reference)
db.to_disk("td.spacy")
请参阅:https://nightly.spacy.io/usage/v3#migrating-training-python
(如果您确实想使用中间JSON格式,则请遵循以下规范:https://spacy.io/api/annotation#json-input。您只需在orth
中包含ner
和tokens
并保留其他功能,但您需要使用paragraphs
,raw
和sentences
的结构。此处是一个示例:https://github.com/explosion/spaCy/blob/45c9a688285081cd69faa0627d9bcaf1f5e799a1/examples/training/training-data.json)
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。