Skip to content

[FEATURE]: migrate jdbc tables to Lakehouse Federation #2186

@pohlposition

Description

@pohlposition

Is there an existing issue for this?

  • I have searched the existing issues

Problem statement

Some customers register jdbc tables in their HMS. These tables need to be registered as Lakehouse Federation 'FOREIGN' catalogs within UC (with the appropriate connection). This functionality has not been built into UCX yet. Lakehouse Federation is now Public Preview.

Proposed Solution

Assessment

  1. Highlight all tables that are jdbc tables.
  2. Nice to have: parse out the URL to show the database server and database that each table maps to.
  3. Nice to have: do a test against each table to verify if the underlying table still exists or not.

Mapping Tables

  1. Allow user to map each jdbc table to a new home in UC
  2. In UC all of the tables for a particular database will need to reside within a "FOREIGN CATALOG"
  3. It is possible that the same physical jdbc table is registered multiple times within a HMS and will need to be mapped to a single object within UC

Upgrading Tables

  1. Manage the database credentials securely into a UC Connection Object
  2. There could be multiple credentials used in an HMS to connect via JDBC (with varying levels of access). Need to pick one.
  3. Need to make sure that jobs, queries, and views reference the newly consolidated UC objects in their code

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    migrate/externalgo/uc/upgrade SYNC EXTERNAL TABLES step

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions