python – Gensim.Similarity添加文档或实时培训

关于这个项目的一点背景.我有带有标识符和文本的副本,例如{name：“sports-football”,text：“与足球运动相关的内容”}.

我需要在这个语料库中找到给定文本输入的正确匹配.
但是,我能够在某种程度上使用Gensim.与LDA和LSI模型的相似性.

如何使用新文档更新Genism.Similarity Index.这里的想法是在现场阶段继续训练模型.

这是我遵循的步骤.

QueryText =“瓜迪奥拉将莱昂内尔·梅西转移到9号角色,这样他就不必深入了解我认为阿圭罗经常会回到更深的位置.”

注意：有些代码只是外行

使用创建索引

`similarities.Similarity(indexpath, model,topics)`

>创建字典

dictionary = Dictionary(QueryText)
>创建语料库

corpus = Corpus(QueryText,字典)
>创建LDA模型

Ldamodel = ldamodel(语料库,字典)

更新现有字典,模型和索引

更新现有字典

existing_dictionary.add_document(dictionary)

更新现有的LDA模型

existing_lda_model.update(corpus)

更新现有的相似性指数

existing_index.add_dcoument(Ldamodel[corpus])

除了以下警告更新似乎工作.

gensim\models\ldamodel.py:535: RuntimeWarning: overflow encountered in exp2 perwordbound, np.exp2(-perwordbound), len(chunk), corpus_words

让我们运行查询文本的相似性

vec_bow = dictionary.doc2bow(QueryText) 
vec_model = existing_lda_model[vec_bow] 
sims = existing_index[vec_model]

但是,它失败了以下错误.

Similarity index with 723 documents in 1 shards (stored under \Files\models\lda_model)
Similarity index with 725 documents in 0 shards (stored under \Files\models\lda_model)
\lib\site-packages\gensim\models\ldamodel.py:535: RuntimeWarning: overflow encountered in exp2
  perwordbound, np.exp2(-perwordbound), len(chunk), corpus_words
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-3-8fe711724367> in <module>()
     45 trigram = Trigram.apply_trigram_model(queryText, bigram, trigram)
     46 vec_bow = dictionry.doc2bow(trigram)
---> 47 vec_model =  lda_model[vec_bow]
     48 print(vec_model)
     49 

~\Anaconda3\envs\lf\lib\site-packages\gensim\models\ldamodel.py in __getitem__(self, bow, eps)
   1103             `(topic_id, topic_probability)` 2-tuples.
   1104         """
-> 1105         return self.get_document_topics(bow, eps, self.minimum_phi_value, self.per_word_topics)
   1106 
   1107     def save(self, fname, ignore=('state', 'dispatcher'), separately=None, *args, **kwargs):

~\Anaconda3\envs\lf\lib\site-packages\gensim\models\ldamodel.py in get_document_topics(self, bow, minimum_probability, minimum_phi_value, per_word_topics)
    944             return self._apply(corpus, **kwargs)
    945 
--> 946         gamma, phis = self.inference([bow], collect_sstats=per_word_topics)
    947         topic_dist = gamma[0] / sum(gamma[0])  # normalize distribution
    948 

~\Anaconda3\envs\lf\lib\site-packages\gensim\models\ldamodel.py in inference(self, chunk, collect_sstats)
    442             Elogthetad = Elogtheta[d, :]
    443             expElogthetad = expElogtheta[d, :]
--> 444             expElogbetad = self.expElogbeta[:, ids]
    445 
    446             # The optimal phi_{dwk} is proportional to expElogthetad_k * expElogbetad_w.

IndexError: index 718 is out of bounds for axis 1 with size 713

我真的很感激,帮助我.
期待很棒的回复.

解决方法:

后来的错误(断言错误：稀疏矩阵中提供的和计算的非零数之间的不匹配)很可能来自警告所建议的问题 – perwordbound溢出和使用其未定义值计算的矩阵使更新失败.

我建议用更大的批次更新模型(不是单个查询).可能存在不成比例的单词数量,模型中您尝试使用相对较少的单词更新的单词数.对于花车,这可能会导致subtle errors.

同样,请尝试使用与模型源数据成比例的批量更新模型(例如,其大小的1/10,1/20).

修订,基于this thread：

Melissa Roemmele wrote:

FYI, I also got this error when I tried to create an LSI index for a
corpus on a bag-of-words corpus without first transforming it into
tf-idf. I Could build the LSI model on the bag-of-words but building
the index for it gave me the error.

在将QueryText传递给模型之前,您可能想先尝试使用tf-idf.

python – Gensim.Similarity添加文档或实时培训

相关推荐