dbplyr 和 corrr 两个变量之间的分组相关性

如何解决dbplyr 和 corrr 两个变量之间的分组相关性

我与黑斑羚有关

con <- DBI::dbConnect(odbc::odbc(),"impala connector",schema = "some_schema")        
library(dplyr)
library(dbplyr) #I have to load both of them,if not tbl won't work
table <- tbl(con,'serverTable')

我想使用 Pearson 的 R 来及时跟踪度量的变化，作为快速而肮脏的预测模型。

在语言环境中，它工作得很好，但我在服务器上实现它时遇到了问题。 代码如下：

library(corrr)
table %>%
  filter(!is.na(VAR) | VAR > -10 | VAR < -32) %>%
#VAR is the measure,and values over -10 or under -32 are already out of the threshold,I wanna intercept the subjects before that
  mutate(num_date = as.numeric(as.POSIXct(date))) %>%
#to convert the date string into the number of seconds since 1970
  group_by(id) %>%
#the measure is taken daily for varIoUs subjects,I am interested in isolating the subjects approaching the thresholds
 mutate(corr = corrr::correlate(VAR,num_date)) %>%
 ungroup() %>%
#here I calculare Pearson's R,I must specify corrr:: if not I get an error
  filter(abs(corr) > 0.9) %>%
#in locale I found out that a value of 0.9 is good for isolating the subjects whose measure is approaching the thresholds
  select(id) %>%
  collect()

如果我运行这个，我会得到错误：

corrr::correlate(VAR,num_date) 中的错误：未找到对象“VAR”。

所以我试着用

代替那行

mutate(corr = corrr::correlate(.$VAR,.$num_date)) %>%

像这样我得到错误

stats::cor(x = x,y = y,use = use,method = method) 中的错误：同时提供 'x' 和 'y' 或类似矩阵的 'x'

如果我尝试使用来自 stats 的 cor，cor(VAR,num_date)，我会收到错误

new_result(connection@ptr,statement,immediate)中的错误：nanodbc/nanodbc.cpp:1412: HY000: [Cloudera][ImpalaODBC] (370) 查询执行期间发生查询分析错误：[HY000] : AnalysisException: some_schema .cor() 未知

像 dbplyr 不能将 cor 转换为 sql（如果我运行 show_query() 而不是 collect() ，我会看到它）

编辑， 我使用 sql 解决了这个问题：

SELECT id,cor
FROM(
SELECT id,((tot_sum - (VAR_sum * date_sum / _count)) / sqrt((VAR_sq - pow(VAR_sum,2.0) / _count) * (date_sq - pow(date_sum,2.0) / _count))) AS cor
FROM (
SELECT id,sum(VAR) AS VAR_sum,sum(CAST(CAST(date AS TIMESTAMP) AS DOUBLE)) AS date_sum,sum(VAR * VAR) AS VAR_sq,sum(CAST(CAST(date AS TIMESTAMP) AS DOUBLE) * CAST(CAST(date AS TIMESTAMP) AS DOUBLE)) AS date_sq,sum(VAR * CAST(CAST(date_push AS TIMESTAMP) AS DOUBLE)) AS tot_sum,count(*) as _count
FROM (
SELECT id,VAR,date
FROM (
SELECT id,date
FROM schema
WHERE VAR IS NOT NULL) AS a
WHERE VAR < -10 OR VAR > -32) AS b
GROUP BY idur) AS c) AS d
WHERE ABS(cor) > 0.9 AND ABS(cor) <= 1

感谢这篇文章： https://chartio.com/learn/postgresql/correlation-coefficient-pearson/

解决方法

cor 不在 dplyr 可以翻译的函数列表中 - 请参阅此处：https://dbplyr.tidyverse.org/articles/sql-translation.html#known-functions

您可以在代码中尝试以下操作：

mutate(corr = translate_sql(corr(VAR,num_date)))

这应该直接转换为 CORR(VAR,num_date)。这些翻译不适用于所有数据库类型。如果在您的情况下无法实现此功能，您可能别无选择，只能在尝试运行不可翻译的函数之前收集数据。