Skip to content

[BUG] Databricks parquetFilters build failure #3098

@tgravescs

Description

@tgravescs

Describe the bug
aws Databricks 7.3 runtime environment tests failed last night with:

[2021-07-30T11:55:22.517Z] Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.(Lorg/apache/parquet/schema/MessageType;ZZZZIZ)V
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.shims.spark301.SparkBaseShims.getParquetFilters(SparkBaseShims.scala:95)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.filterBlocks(GpuParquetScan.scala:272)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.MultiFileCloudParquetPartitionReader$ReadBatchRunner.call(GpuParquetScan.scala:1016)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.MultiFileCloudParquetPartitionReader$ReadBatchRunner.call(GpuParquetScan.scala:997)
[2021-07-30T11:55:22.517Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-07-30T11:55:22.517Z] ... 3 more

It looks like perhaps they deployed a change to ParquetFilters that likely matches spark 3.0.4, need to investigate further

Metadata

Metadata

Assignees

Labels

P0Must have for releasebugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions