Is there an existing issue for this?

• I have searched the existing issues

Category of feature request

Reconcile

Problem statement

Currently required Reconcile secret keys are hard-coded. For example, Oracle requires the secret keys user, password, host, port, database stored in some secret scope.

This is not flexible enough to adhere to all user security policies. In particular, some users require that all secret keys adhere to a specified pattern that gives information as to their use. It is not sufficient enough to allow flexibility at the scope level, as is currently done, as scopes might have their own different policies.

Thinking ahead, one might wish to allow other secret providers different than Databricks, for example a customer may have a policy to keep all credentials for system X in provider Y. For example, AWS secrets manager might be the location for all Snowflake credentials.

Proposed Solution

Manage this similar to MLFlow, in that allow a user to override the defaults by creating a list of dictionaries similar to the following:

auth_config = [
  {
    "provider": "databricks",  (Optional, if this is all that will ever be supported)
    "connector": "oracle",
    "scope": "<scope_name>"
    "user_key": "<oracle_user_key_name>",
    "password_key": "<oracle_password_key_name>",
  },
{
   "provider": "databricks",
   "connector": "snowflake",
   "scope": "<scope name>"
   "sfUser_key": "<sfUser_key_name>"
]

Additional Context

No response