如何解决将雪花任务结果复制到stage并下载到csv
基本上我需要在雪花任务中自动执行以下所有操作
- 在 SNowflake 中创建/替换 csv 文件格式和阶段
- 运行任务查询(每隔几天运行一次以获取一些统计信息)
- 每次运行到 Stage csv 时将查询结果卸载
- 将 stage csv 的内容下载到我机器上的本地文件
我做对的是copY INTO阶段,每次运行的时候怎么把任务的结果卸载到stage中?
我不知道在 FROM 语句中放什么 - 无法识别 TITANLOADSUCCESSVSFAIL 但这是任务的名称
copY INTO @TitanLoadStage/unload/ FROM TITANLOADSUCCESSVSFAIL FILE_FORMAT = TitanLoadSevendays
第一次使用 stage,并使用 SF 在本地下载,因此请感谢有关如何启动和运行它的任何建议!
谢谢, 尼克
完整代码:
-- create a csv file format
CREATE OR REPLACE FILE FORMAT TitanLoadSevendays
type = 'CSV'
field_delimiter = '|';
--create a sNowflake staging table using the csv
CREATE OR REPLACE STAGE TitanLoadStage
file_format = TitanLoadSevendays;
CREATE TASK IF NOT EXISTS TitanLoadSuccessVsFail
WAREHOUSE = ITSM_LWH
SCHEDULE = 'USING CRON 1 * * * * Australia/Canberra' --every minute for testing purposes
COMMENT = 'Last 7 days of Titan game success vs fail load %'
AS
WITH SUCCESSCTE AS (
SELECT CLIENTNAME,COUNT(EVENTTYPE) AS SuccessLoad --count success load events for that game
FROM vw_fact_gameload60
WHERE EVENTTYPE = 103 --success load events
AND USERTYPE = 1 --real users
AND APPID = 2 --titan games
AND EVENTARRIVALDATE >= DATEADD(DAY,-7,CAST(GETDATE() AS DATE)) --only looking at the last week
GROUP BY CLIENTNAME
),FAILCTE AS ( --same as above but for Failed loads
SELECT CLIENTNAME,COUNT(EVENTTYPE) AS FailedLoads -- count Failed load events for that game
FROM vw_fact_gameload60
WHERE EVENTTYPE = 106 -- Failed load events
AND USERTYPE = 1 -- real users
AND APPID = 2 -- Titan games
AND EVENTARRIVALDATE >= DATEADD(DAY,CAST(GETDATE() AS DATE)) -- last 7 days
--AND FACTEVENTARRIVALDATE BETWEEN DATEADD(DAY,GETDATE())AND GETDATE() -- last 7 days
GROUP BY CLIENTNAME
)
SELECT COALESCE(s.CLIENTNAME,f.CLIENTNAME) AS ClientName,ZEROIFNULL(s.SuccessLoad) + ZEROIFNULL(f.FailedLoads) AS TotalLoads --sum the success and Failed loads found for 103,106 events only,calculated in CTEs,ZEROIFNULL(s.SuccessLoad) AS Cnt_SuccessLoad --count from success cte,ZEROIFNULL(f.FailedLoads) AS Cnt_FailedLoads --count from fail cte,CONCAT(ZEROIFNULL(ROUND(s.SuccessLoad * 100.0 / TotalLoads,2)),'%') As Pct_Success --percentage of SuccessLoads against total,CONCAT(ZEROIFNULL(ROUND(f.FailedLoads * 100.0 / TotalLoads,'%') AS Pct_Fail---percentage of FailedLoads against total
FROM SUCCESSCTE s
FULL OUTER JOIN FAILCTE f -- outer join in the fail CTE by game name,outer required because some titan games sucess or fail events are NULL
ON s.CLIENTNAME = f.Clientname
ORDER BY CLIENTNAME ASC
--copy the results from the query to the sNowflake staging table created above
copY INTO @TitanLoadStage/unload/ FROM TITANLOADSUCCESSVSFAIL FILE_FORMAT = TitanLoadSevendays
-- export the stage data to csv located in common folder
GET @TitanLoadStage/unload/data_0_0_0.csv.gz file:\\itsm\group\ITS%20Management\Common\All%20Staff\SMD\Games\SNowflake%20and%20GamesDNA\SNowflake\SNowflakeCSV\TitanLoad.csv
-- start the task
ALTER TASK IF EXISTS TitanLoadSuccessVsFail RESUME
解决方法
如果您想获得通过任务运行的查询结果,您需要将所述查询的结果具体化到一个表中。
你现在拥有的:
CREATE TASK mytask_minute
WAREHOUSE = mywh
SCHEDULE = '5 MINUTE'
AS
SELECT 1 x;
COPY INTO @TitanLoadStage/unload/
FROM mytask_minute;
(mytask_minute
不是表格,因此您无法从中选择)
你应该做什么:
CREATE TASK mytask_minute
WAREHOUSE = mywh
SCHEDULE = '5 MINUTE'
AS
CREATE OR REPLACE TABLE task_results_table
AS
SELECT 1 x;
COPY INTO @TitanLoadStage/unload/
SELECT *
FROM task_results_table;
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。