Skip to content

[Epic] Extract catalog functionality from the core to make it more modular #10782

@alamb

Description

@alamb

Is your feature request related to a problem or challenge?

As @goldmedal started trying to move the DynamicFileProvider so others could use it in #10745 I think it is clear that there is not a good way to add additional catalog support in the core without everything being intertwined.

Thus I think we should try and extract the different catalog providers out of datafusion core so it it easier

Describe the solution you'd like

I suggest the following final layout:

  1. traits like CatalogProvider, SchemaProvider, etc in a new crate datafusion-catalog (since these traits rely on table provider, etc I think this can't be in datafusion-common or datafusion-expr)
  2. The built in Memory* providers are in datafusion-catalog
  3. The bult in InformationSchema providers are in datafusion-catalog
  4. The newly proposed DynamicFileCatalog in datafusion-catalog
  5. (eventually) the LIstingTableProvider (which is by far the most complicated) moved to its own crate datafusion-catalog-listing

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions