如何在 pytorch 中为机器翻译任务加载 torchtext 数据集？

如何解决如何在 pytorch 中为机器翻译任务加载 torchtext 数据集？

我是 torchtext 的新手我一直在使用 Multi30k 数据集来学习基础知识。在学习基础知识的过程中，我想使用其他数据集，例如 IWSLT2017。我阅读了文档，他们向我展示了如何加载数据。

这就是我加载 Multi30k 数据集的方式

# creating the fields

SRC = data.Field(
    tokenize = tokenize_de,lower= True,init_token = "<sos>",eos_token = "<eos>"
)
TRG = data.Field(
    tokenize = tokenize_en,eos_token = "<eos>"
)

### Splitting the sets
train_data,valid_data,test_data = datasets.Multi30k.splits(
    exts=('.de','.en'),fields = (SRC,TRG)
)

当我运行这个时：

print(vars(train_data.examples[0]))

我明白了：

{'src': ['zwei','junge','weiße','männer','sind','im','freien','in','der','nähe','vieler','büsche','.'],'trg': ['two','young',','white','males','are','outside','near','many','bushes','.']}

我的问题是如何在调用 IWSLT2017 时加载 print(vars(train_data.examples[0])) 以获得类似的结果。

这是我尝试过的：

from torchtext.datasets import IWSLT2017
train_iter,valid_iter,test_iter = IWSLT2017(
    root='.data',split=('train','valid','test'),language_pair=('it','en')
)
src_sentence,tgt_sentence = next(train_iter)

它返回一个元组，如下所示：

('Sono impressionato da questa conferenza,e voglio ringraziare tutti voi per i tanti,lusinghieri commenti,anche perché... Ne ho bisogno!!!\n','I have been blown away by this conference,and I want to thank all of you for the many nice comments\n')

我的问题是我怎样才能从这一步移动到获得这样的东西的步骤：

{'src': ['zwei','.']}

任何帮助输入将不胜感激。

解决方法

为此，您可以使用例如 spacy 的 processing_pipeline。 一个示例如下所示：

import spacy
from torchtext.data.utils import get_tokenizer
from torchtext.datasets import IWSLT2017

train_iter,valid_iter,test_iter = IWSLT2017(root='.data',split=('train','valid','test'),language_pair=('it','en'))

src_sentence,tgt_sentence = next(train_iter)
print(src_sentence,tgt_sentence)

nlp = spacy.load("it_core_news_sm")
for doc in nlp.pipe([src_sentence]):
    # Do something with the doc here
    print([(ent.text) for ent in doc])

nlp = spacy.load("en_core_web_sm")
for doc in nlp.pipe([tgt_sentence]):
    # Do something with the doc here
    print([(ent.text) for ent in doc])

第一个例句的输出：

Grazie mille,Chris. E’ veramente un grande onore venire su questo palco due volte. Vi sono estremamente grato.
Thank you so much,Chris. And it's truly a great honor to have the opportunity to come to this stage twice; I'm extremely grateful.

标记化句子的输出：

['Grazie','mille',','Chris','.','E','’','veramente','un','grande','onore','venire','su','questo','palco','due','volte','Vi','sono','estremamente','grato','\n']
['Thank','you','so','much','And','it',"'s",'truly','a','great','honor','to','have','the','opportunity','come','this','stage','twice',';','I',"'m",'extremely','grateful','\n']