Skip to content

Support fast merge in the storage plugin interface. #60

@jealous

Description

@jealous

Allow the plugin developer to implement a fast merge which will be invoked when:

  • Encryption is disabled.
  • Compression is disabled or the compression codec supports concatenation of serialized streams.
  • The spark.shuffle.unsafe.fastMergeEnabled option is true.
  • The plugin supports fast merge.

Note the performance of this fast merge function could seriously impact the performance of Spark SQL joins with multiple spills.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions