如何使用 ScalaMock 来评估使用特定 Spark Dataframe 参数调用的函数并具有有用的输出

如何解决如何使用 ScalaMock 来评估使用特定 Spark Dataframe 参数调用的函数并具有有用的输出

我一直在看：

但还没有得到我想要的结果，但基本上我进行了这个测试

 scenario("myFunction reads parquet and writes to db") {
          var mockUtil: UtilitiesService = stub[UtilitiesService]
          val service = new myService(mockUtil)
          
          val expectedParquetDf = Seq(
              (999,"testData")
          ).toDF("number","word")

          (mockUtil.getDataFrameFromParquet _).when("myParquetPath") returns Right(expectedParquetDf)
          service.publishToDatabase()
          (mockUtil.insertDataFrameIntoDb_).verify(expectedParquetDf,"myTable").once()        
      }

但是如果该测试失败（由于数据帧不匹配），则输出并不理想，只需说

[info]   Expected:
[info]   inAnyOrder {
[info]     <stub-4> UtilitiesService.getDataFrameFromParquet(path) any number of times (called once)
[info]     <stub-4> UtilitiesService.insertDataFrameIntoPostgres[number: int,word: string] once (never called - UNSATISFIED)
[info]   }
[info]   
[info]   Actual:
[info]     <stub-4> UtilitiesService.getDataFrameFromParquet(oath)
[info]     <stub-4> UtilitiesService.insertDataFrameIntoPostgres([number: int,word: string],"myTable" (myFile.scala:28)

字符串部分是正确的，但是数据帧部分；仅在说一列被删除时才有用，如果有一个坏行等等，那么有用。有没有一个很好的方法来改进这个？

目前我的兔子洞让我进入了下面，这仍然不起作用，并且返回 true 以使“&&”部分工作的“assert”函数都感觉必须更好的方法。我可以在标准验证中覆盖一些比较器功能吗？？ :

  def assertStringsAreEqual(expectedpath:String,actualPath:String) : Boolean = {
          assert(actualPath == expectedpath)
          true
      }

      def assertDataFramesAreEqual(expected: DataFrame,actual: DataFrame) : Boolean = {
          AssertHelpers.assertDataEqual(expected,actual) //verbos info,asserts on each row etc
          true
      }


      scenario("myFunction reads parquet and writes to db") {
          var mockUtil: UtilitiesService = stub[UtilitiesService]
          val service = new myService(mockUtil)
          val expectedParquetDf = Seq(
              (999,"testData"),(898,"wrongData"),(999,"extraRow")
          ).toDF("number","word")

          val incorrectExample = Seq(
              (999,"word")

          (mockUtil.getDataFrameFromParquet _).when("myParquetPath") returns Right(incorrectExample) //forced to incorrect for Now
          (_mockUtilService.insertDataFrameIntoPostgres _).
              expects(where { {
                      (actualDf,path) => assertDataFramesAreEqual(expectedParquetDf,actualDf) && assertStringsAreEqual(path,"ExpectedTable")
              }  })
              .once()

          service.publishToDb()

      }

作为参考，我的目标是在某处弹出这样的东西：

Expected:
Dataframe:
[number,word]
[999,"testData"]
[898,"wrongData"]
[999,"extraRow"]

Actual:
Dataframe
[number,"testData"]
[999,"testData"]

解决方法

所以这仍然不理想，但使用“expects.onCall”我可以获得我想要的输出

scenario("myFunction reads parquet and writes to db") {
      var mockUtil: UtilitiesService = mock[UtilitiesService]
      val service = new myService(mockUtil)
      val expectedParquetDf = Seq(
          (999,"testData"),(898,"wrongData"),(999,"extraRow")
      ).toDF("number","word")

      val incorrectExample = Seq(
          (999,"testData")
      ).toDF("number","word")

     //set up expectations
      (mockUtilService.insertDataFrameIntoPostgres _).expects(*,"ExpectedTable").onCall( { (df: DataFrame,path:String) =>
        AssertHelpers.assertDataEqual(df,expectedParquetDf)
        Right(sxDbData)
        })


      (mockUtil.getDataFrameFromParquet _).when("myParquetPath") returns Right(incorrectExample) //forced to incorrect for now

      service.publishToDb()

  }

希望有人对此有更清晰的解决方案