如何在pyspark LDA模型中获取主题中的单词

如何解决如何在pyspark LDA模型中获取主题中的单词

我已经在Spark 2.4中使用word2vec训练了我的LDA模型。我的声带大小为10，并且我尝试从文本数据中找出3个主题。但是我不知道该模型学习的主题中最常用的单词。这是我的代码-

# remove stopwords
from pyspark.ml.feature import StopWordsRemover
swr = StopWordsRemover(inputCol = 'text_token',outputCol = 'text_sw_removed')
df_sw_cleaned = swr.transform(df_regex_tokenized)
df_sw_cleaned.show(3,truncate=False)


# implement word2vec model
from pyspark.ml.feature import Word2Vec
#create an average word vector for each document (works well according to Zeyu & Shu)
word2vec = Word2Vec(vectorSize = 10,minCount = 1,inputCol = 'text_sw_removed',outputCol = 'features')
word2vec_model = word2vec.fit(df_sw_cleaned)
df_word2_vec = word2vec_model.transform(df_sw_cleaned)

#test similarity between words
synonyms = word2vec_model.findSynonyms("school",5)   
synonyms.show(5)


# start topic modeling
from pyspark.ml.linalg import Vectors,SparseVector
from pyspark.ml.clustering import LDA

# create LDA object 
lda = LDA(k=3,seed=1,optimizer="em")
lda.setMaxIter(10)

# Fit LDA model
lda_model = lda.fit(df_word2_vec.select("features"))
lda_model.transform(df_word2_vec.select("features")).show()

当我做lda_model.transform(df_word2_vec.select("features")).show()时，我得到如下数据框-

20/10/28 19:04:41 WARN TaskSetManager: Stage 1768 contains a task of very large size (155 KB). The maximum recommended task size is 100 KB.
+--------------------+--------------------+
|            features|   topicDistribution|
+--------------------+--------------------+
|[0.02595538695808...|[0.33430867975113...|
|[0.03099112078886...|[0.33368134622986...|
|[0.00664915300072...|[0.33361193170264...|
|[0.05927112722850...|[0.33437258288643...|
|[0.06410298396700...|[0.33565417329573...|
|[-0.0101503337183...|[0.33328074245659...|
|[-0.0739752339820...|[0.33187141503696...|
|[0.05009885969000...|[0.33344419642461...|
|[-0.0355422796215...|[0.33358112450240...|
|[-0.0989323664795...|[0.33297312392442...|
|[-0.0213827120772...|[0.33633261990846...|
|[-0.0449083298444...|[0.33610590534864...|
|[0.04275943336210...|[0.33274010181046...|
|[0.03544508706887...|[0.33524942705743...|
|[-0.0253385022515...|[0.33425766949656...|
|[0.15776347978548...|[0.33269623147833...|
|[0.03685717905561...|[0.33304748588742...|
|[0.04340785674139...|[0.33366007496412...|
|[0.05481550529260...|[0.33390206213966...|
|[0.04185649161429...|[0.33370378552855...|
+--------------------+--------------------+
only showing top 20 rows

这是topics matrix

array([[  53.21365581,54.91929856,55.11689835],[  67.15857681,68.79563631,68.42798562],[-101.68002051,-101.17864047,-97.2594664 ],[  92.34440882,98.02719048,91.41366965],[ 153.97213627,155.5668838,153.3548064 ],[ 143.10394679,140.10223081,139.23135738],[   6.72004202,4.14344254,7.56754843],[ -20.04475822,-24.12710322,-20.66088877],[ -82.9174468,-85.23527773,-84.42102787],[ 113.90200616,114.74137255,112.94434583]])

如何为每个主题打印前N个单词？