Spark结构化流中的指标以获取总批处理时间

如何解决Spark结构化流中的指标以获取总批处理时间

默认情况下，在Spark结构化流式Spark中，打印指标，但根据文档 triggerExecution ，必须是执行整个批次所花费的总时间。

triggerExecution = batch_time = (addBatch + getBatch + getEndOffset + queryPlanning  + setoffsetRange + triggerExecution + walCommit)

不是这种情况，总批处理时间也不完全等于 triggerExecution 时间。 批处理的总执行时间与

batch_time = triggerExecution + addBatch

指标

"timestamp" : "2020-10-06T09:45:41.721Z","batchId" : 12609,"numInputRows" : 199800,"inputRowsPerSecond" : 651.0624570275969,"processedRowsPerSecond" : 517.4758161639968,"durationMs" : {
    "addBatch" : 357041,"getBatch" : 0,"getEndOffset" : 1,"queryPlanning" : 29,"setoffsetRange" : 3012,"triggerExecution" : 386105,"walCommit" : 12949
  }

参考

• https://www.waitingforcode.com/apache-spark-structured-streaming/query-metrics-apache-spark-structured-streaming/read
• https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-ProgressReporter.html