如何解决在R中同时输出有单字和双字的文本
我正在尝试弄清楚如何在R中的文本中识别单字组和双字组,然后根据阈值将两者都保留在最终输出中。我已经使用gensim的Phraser模型在Python中完成了此操作,但是还没有弄清楚如何在R中执行此操作。
例如:
strings <- data.frame(text = 'This is a great movie from yesterday','I went to the movies','Great movie time at the theater','I went to the theater yesterday')
#Pseudocode below
bigs <- tokenize_uni_bi(strings,n = 1:2,threshold = 2)
print(bigs)
[['this','great_movie','yesterday'],['went','movies'],['great_movie','theater'],'theater','yesterday']]
谢谢!
解决方法
您可以为此使用Quanteda框架:
library(quanteda)
# tokenize,tolower,remove stopwords and create ngrams
my_toks <- tokens(strings$text)
my_toks <- tokens_tolower(my_toks)
my_toks <- tokens_remove(my_toks,stopwords("english"))
bigs <- tokens_ngrams(my_toks,n = 1:2)
# turn into document feature matrix and filter on minimum frequency of 2 and more
my_dfm <- dfm(bigs)
dfm_trim(my_dfm,min_termfreq = 2)
Document-feature matrix of: 4 documents,6 features (50.0% sparse).
features
docs great movie yesterday great_movie went theater
text1 1 1 1 1 0 0
text2 0 0 0 0 1 0
text3 1 1 0 1 0 1
text4 0 0 1 0 1 1
# use convert function to turn this into a data.frame
或者,您可以使用tidytext包,tm,tokenizers等。这全都取决于您期望的输出。
使用tidytext / dplyr的示例如下:
library(tidytext)
library(dplyr)
strings %>%
unnest_ngrams(bigs,text,n = 2,n_min = 1,ngram_delim = "_",stopwords = stopwords::stopwords()) %>%
count(bigs) %>%
filter(n >= 2)
bigs n
1 great 2
2 great_movie 2
3 movie 2
4 theater 2
5 went 2
6 yesterday 2
quanteda和tidytext都有大量在线帮助。参见使用cran上的两个软件包的小插图。
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。