Redshift删除重复记录但保持最新

如何解决Redshift删除重复记录但保持最新

当前表

select * from currentTable;

select * from PG_TABLE_DEF where tablename='currenttable';
 schemaname |  tablename   |        column         |         type          | encoding | distkey | sortkey | notnull
------------+--------------+-----------------------+-----------------------+----------+---------+---------+---------
 public     | currenttable | kafkaoffset           | integer               | az64     | f       |       0 | t
 public     | currenttable | operation             | character varying(25) | lzo      | f       |       0 | t
 public     | currenttable | othertablepk          | integer               | az64     | f       |       0 | t
 public     | currenttable | othertableorderstatus | character varying(25) | lzo      | f       |       0 | t

| kafkaOffset | operation | otherTablePK  | otherTableOrderStatus |
|:------------|----------:|:-------------:|-----------------------:
| 1024        | CREATE    |  23           | Cooking
| 1025        | UPDATE    |  23           | Shipped
| 1026        | UPDATE    |  23           | Delivered
| 1027        | CREATE    |  51           | Cooking
| 1028        | UPDATE    |  51           | Shipped
| 1029        | CREATE    |  52           | Cooking

我想对当前表进行重复数据删除，以仅保留基于kafkaOffset的最新记录（otherTablePk）。

重复表（预期结果）

select * from currentTable;

| kafkaOffset | operation | otherTablePK  | otherTableOrderStatus |
|:------------|----------:|:-------------:|-----------------------:
| 1026        | UPDATE    |  23           | Delivered
| 1028        | UPDATE    |  51           | Shipped
| 1029        | CREATE    |  52           | Cooking

解决方案1：使用InnerJoin和max

使用inner join和max在redshift中进行类似MysqL的查询。 More Info.

DELETE
FROM currentTable
INNER JOIN
  (SELECT max(kafkaOffset) AS lastOffset,otherTablePk AS otherTablePkID
   FROM currentTable
   WHERE otherTablePkID IN
       (SELECT otherTablePk
        FROM currentTable
        GROUP BY otherTablePk
        HAVING count(*) > 1)
   GROUP BY otherTablePk) lastTable ON lastTable.otherTablePkID = currentTable.otherTablePkID
WHERE current_table.kafkaOffset < lastTable.lastOffset;

解决方案2：使用USING并进行自我联接。

DELETE from currentTable t1 
JOIN currentTable t2 USING (otherTablePK) 
WHERE t1.kafkaOffset < t2.kafkaOffset

解决方案3：使用TEMP表和手术删除

如本blog和this answer所述，但此处的用例几乎没有什么不同。我们需要删除所有内容，但保持最新状态。执行max会使查询变慢。

以上所有解决方案在Redshift中都是缓慢的，因为它是柱状存储。请提出在Redshift中执行此操作最快的方法是什么？

解决方法

请包括表DDL。否则都是猜测。

关于猜测，请尝试以下未经测试的查询（我没有Redshift）：

DELETE FROM currentTable
WHERE kafkaOffset IN (
    SELECT kafkaOffset
    FROM (
        SELECT kafkaOffset,row_number() OVER (PARTITION BY otherTablePK ORDER BY kafkaOffset DESC) rn
        FROM currentTable
    ) t
    WHERE rn > 1
);