Describe the bug
aws Databricks 7.3 runtime environment tests failed last night with:
[2021-07-30T11:55:22.517Z] Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.(Lorg/apache/parquet/schema/MessageType;ZZZZIZ)V
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.shims.spark301.SparkBaseShims.getParquetFilters(SparkBaseShims.scala:95)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.filterBlocks(GpuParquetScan.scala:272)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.MultiFileCloudParquetPartitionReader$ReadBatchRunner.call(GpuParquetScan.scala:1016)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.MultiFileCloudParquetPartitionReader$ReadBatchRunner.call(GpuParquetScan.scala:997)
[2021-07-30T11:55:22.517Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-07-30T11:55:22.517Z] ... 3 more
It looks like perhaps they deployed a change to ParquetFilters that likely matches spark 3.0.4, need to investigate further
Describe the bug
aws Databricks 7.3 runtime environment tests failed last night with:
[2021-07-30T11:55:22.517Z] Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.(Lorg/apache/parquet/schema/MessageType;ZZZZIZ)V
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.shims.spark301.SparkBaseShims.getParquetFilters(SparkBaseShims.scala:95)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.filterBlocks(GpuParquetScan.scala:272)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.MultiFileCloudParquetPartitionReader$ReadBatchRunner.call(GpuParquetScan.scala:1016)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.MultiFileCloudParquetPartitionReader$ReadBatchRunner.call(GpuParquetScan.scala:997)
[2021-07-30T11:55:22.517Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-07-30T11:55:22.517Z] ... 3 more
It looks like perhaps they deployed a change to ParquetFilters that likely matches spark 3.0.4, need to investigate further