如何使用句子转换器预训练模型来使用paraphrase_mining

如何解决如何使用句子转换器预训练模型来使用paraphrase_mining

我正在尝试使用预训练的句子变形模型来寻找句子之间的相似性。我正在尝试遵循此处的代码-https://www.sbert.net/docs/usage/paraphrase_mining.html

在试用版中，我运行了2个for循环，在其中尝试查找给定句子与其他句子的相似性。这是该代码-

from sentence_transformers import SentenceTransformer,util
model = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')


# Single list of sentences
sentences = ['The cat sits outside','A man is playing guitar','The new movie is awesome','Do you like pizza?']

#Compute embeddings
embeddings = model.encode(sentences,convert_to_tensor=True)

#Compute cosine-similarities for each sentence with each other sentence
cosine_scores = util.pytorch_cos_sim(embeddings,embeddings)

#Find the pairs with the highest cosine similarity scores
pairs = []
for i in range(len(cosine_scores)-1):
    for j in range(i+1,len(cosine_scores)):
        pairs.append({'index': [i,j],'score': cosine_scores[i][j]})

#Sort scores in decreasing order
pairs = sorted(pairs,key=lambda x: x['score'],reverse=True)

print(len(pairs))
6

for pair in pairs[0:10]:
    i,j = pair['index']
    print("{} \t\t {} \t\t score: {:.4f}".format(sentences[i],sentences[j],pair['score']))

A man is playing guitar          Do you like pizza?          score: 0.1080
The new movie is awesome         Do you like pizza?          score: 0.0829
A man is playing guitar          The new movie is awesome        score: 0.0652
The cat sits outside         Do you like pizza?          score: 0.0523
The cat sits outside         The new movie is awesome        score: -0.0270
The cat sits outside         A man is playing guitar         score: -0.0530

这可以按预期工作，因为在4个句子的组合之间可以有6个相似度得分组合。他们在文档页面上指出，由于二次复杂度，这种方法无法很好地扩展，因此建议使用paraphrase_mining()方法。

但是当我尝试使用该方法时，我没有得到6种组合，而是得到了5种。为什么会这样？

这是我尝试使用paraphrase_mining()方法的示例代码-

# Single list of sentences
sentences = ['The cat sits outside','Do you like pizza?']


paraphrases = util.paraphrase_mining(model,sentences)
print(len(paraphrases))
5

k = 0
for paraphrase in paraphrases:
    print(k)
    score,i,j = paraphrase
    print("{} \t\t {} \t\t score: {:.4f}".format(sentences[i],score))
    print()
    k = k + 1

0
A man is playing guitar          Do you like pizza?          score: 0.1080

1
The new movie is awesome         Do you like pizza?          score: 0.0829

2
A man is playing guitar          The new movie is awesome        score: 0.0652

3
The cat sits outside         Do you like pizza?          score: 0.0523

4
The cat sits outside         The new movie is awesome        score: -0.0270

paraphrase_mining()的工作方式是否有区别？

解决方法

感谢您指出这一点。

当句子列表很小时，paraphrase_mining函数中存在一个小错误。它没有计算所有组合，而是仅为每个句子计算n-1个组合。对于较大的句子列表，如果，但对于您的特定示例，它忽略了最不相关的组合并返回了超出预期的查看器对，则没有问题。

它在存储库中已修复，将成为下一版本的一部分。

PS：您也可以在Github上发布您的问题： https://github.com/UKPLab/sentence-transformers/issues

我收到一封电子邮件通知，可以更快地回复。