Profiler CLI #1623

sundarshankar89 · 2025-06-03T20:19:25Z

Changes

What does this PR do?

This introduces the intial version of CLI

Relevant implementation details

Caveats/things to watch out for when reviewing:

Linked issues

~ ❯ databricks labs lakebridge execute-database-profiler --help                                                                                                                                                 26s
Profile the source system database

Usage:
  databricks labs lakebridge execute-database-profiler [flags]

Flags:
  -h, --help                 help for execute-database-profiler
      --source-tech string   (Optional) The technology/platform of the sources to Profile

Global Flags:
      --debug            enable debug logging
  -o, --output type      output type: text or json (default text)
  -p, --profile string   ~/.databrickscfg profile
  -t, --target string    bundle target to use (if applicable)

~ ❯ databricks labs lakebridge execute-database-profiler --source-tech mssql
12:24:59    ERROR [d.labs.lakebridge] Only following source system is supported ['synapse']
12:24:59    ERROR [d.l.lakebridge.execute-database-profiler] ValueError: Invalid source technology mssql


~ ❯ databricks labs lakebridge execute-database-profiler --source-tech Synapse
[UPGRADE ADVISED] Newer lakebridge version was released 12 days ago. Please run `databricks labs upgrade lakebridge` to upgrade: feature/profiler_entry_point -> v0.10.11
12:25:15    ERROR [d.l.lakebridge.execute-database-profiler] ValueError: Connection details not found. Please run `databricks labs lakebridge configure-database-profiler` to set up connection details for synapse.
~ ❯

~ ❯ databricks labs lakebridge execute-database-profiler
Select the source technology
[0] synapse
Enter a number between 0 and 0: 0
12:32:50    ERROR [d.l.lakebridge.execute-database-profiler] ValueError: Connection details not found. Please run `databricks labs lakebridge configure-database-profiler` to set up connection details for synapse.

Functionality

added relevant user documentation
added new CLI command
modified existing command: databricks labs remorph ...
... +add your own

Tests

manually tested
added unit tests
added integration tests

…into feature/synapse_data_collector

…collector

…into feature/synapse_data_collector

…collector

Copilot

Pull Request Overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

src/databricks/labs/lakebridge/cli.py

Copilot

Pull Request Overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

src/databricks/labs/lakebridge/cli.py

Copilot

Pull Request Overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

src/databricks/labs/lakebridge/cli.py

gueniai

We should also add the command to deploy the profiler dashboard

gueniai · 2025-10-19T04:27:30Z

labs.yml

+  - name: execute-database-profiler
+    description: Profile the source system database
+    flags:
+      - name: source-tech


What's the advantage of having this as optional?

We can prompt if it is not passed in CLI for interactive operations.

sundarshankar89 · 2025-07-18T07:23:41Z

src/databricks/labs/lakebridge/assessments/profiler.py

+    def _execute(self, platform: str, config_path: str, extractor=None):
+        try:
+            config_full_path = self._locate_config(config_path)
+            config = Profiler.path_modifier(config_full_path)


will adjust this, make pathmodifier as class method

src/databricks/labs/remorph/assessments/pipeline.py

src/databricks/labs/lakebridge/resources/assessments/synapse/common/connector.py

sundarshankar89 · 2025-10-22T03:04:00Z

labs.yml

+  - name: execute-database-profiler
+    description: Profile the source system database
+    flags:
+      - name: source-tech


We can prompt if it is not passed in CLI for interactive operations.

codecov · 2025-10-22T03:06:16Z

Codecov Report

❌ Patch coverage is 24.24242% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.85%. Comparing base (3002743) to head (fca09b9).

Files with missing lines	Patch %	Lines
src/databricks/labs/lakebridge/cli.py	21.87%	25 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1623      +/-   ##
==========================================
+ Coverage   64.92%   65.85%   +0.93%     
==========================================
  Files          96       96              
  Lines        7908     7934      +26     
  Branches      822      825       +3     
==========================================
+ Hits         5134     5225      +91     
+ Misses       2595     2530      -65     
  Partials      179      179

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

goodwillpunning · 2025-10-27T12:27:15Z

src/databricks/labs/lakebridge/cli.py

+    """Deploys a profiler summary as a Databricks dashboard."""
+    ctx = ApplicationContext(w)
+    ctx.add_user_agent_extra("cmd", "create-profiler-dashboard")
+    ctx.dashboard_manager.create_profiler_summary_dashboard(extract_file, source_tech, catalog_name, schema_name)


Dependent upon the dashboard_manager Class introduced in PR#2000.

@goodwillpunning we need this branch working so I m removing this for now, you stack the changes on top of this PR as stacked PR

gueniai

LGTM

sundarshankar89 and others added 30 commits March 21, 2025 19:39

pipeline extension

e61dd3e

Init commit

bfa8490

Merge branch 'main' into feature/synapse_data_collector

65bd96f

Merge branch 'main' into feature/synapse_data_collector

e578178

Merge branch 'main' into feature/pipeline_extensions

73b36c2

Init commit

8e95927

Merge remote-tracking branch 'origin/feature/synapse_data_collector' …

cabf93a

…into feature/synapse_data_collector

Merge branch 'feature/pipeline_extensions' into feature/synapse_data_…

ec33f67

…collector

Merge branch 'main' into feature/pipeline_extensions

440cc70

Merge remote-tracking branch 'origin/feature/synapse_data_collector' …

d6fcabb

…into feature/synapse_data_collector

fmt fixes

2988569

fmt fixes

f0c87d6

refactor

0764e01

Added tests for dependency

d188705

Added tests for dependency

68b0f83

Added Failure tests

5bd9408

installer as venv

5899a3c

fmt fixes

3cdbf7e

Merge branch 'feature/pipeline_extensions' into feature/synapse_data_…

822da6b

…collector

fmt fixes

299d293

Added Python Script

9186b89

Added Python Script

18f9b42

Fmt fixes

877a239

Merge branch 'main' into feature/pipeline_extensions

492a12f

Merge branch 'feature/pipeline_extensions' into feature/synapse_data_…

d0ca03f

…collector

intermediate commit

603aa0f

init commit

204c639

Merge branch 'main' into feature/synapse-cred-configuration

1af4de9

Merge branch 'main' into feature/pipeline_extensions

75fbfb5

fmt fixes

de87a2a

made source-tech optional param

451caf2

sundarshankar89 temporarily deployed to tool October 16, 2025 06:54 — with GitHub Actions Inactive

sundarshankar89 requested a review from Copilot October 16, 2025 06:56

Copilot AI reviewed Oct 16, 2025

View reviewed changes

src/databricks/labs/lakebridge/cli.py Outdated Show resolved Hide resolved

src/databricks/labs/lakebridge/cli.py Outdated Show resolved Hide resolved

src/databricks/labs/lakebridge/cli.py Outdated Show resolved Hide resolved

Fixed Copilot review comments

77cc4a7

sundarshankar89 temporarily deployed to tool October 16, 2025 07:01 — with GitHub Actions Inactive

sundarshankar89 requested a review from Copilot October 16, 2025 07:02

Copilot AI reviewed Oct 16, 2025

View reviewed changes

src/databricks/labs/lakebridge/cli.py Outdated Show resolved Hide resolved

src/databricks/labs/lakebridge/cli.py Outdated Show resolved Hide resolved

Fixed Copilot review comments

e2c0819

sundarshankar89 temporarily deployed to tool October 16, 2025 07:09 — with GitHub Actions Inactive

sundarshankar89 requested a review from Copilot October 16, 2025 07:10

Copilot AI reviewed Oct 16, 2025

View reviewed changes

src/databricks/labs/lakebridge/cli.py Show resolved Hide resolved

src/databricks/labs/lakebridge/cli.py Show resolved Hide resolved

gueniai reviewed Oct 19, 2025

View reviewed changes

sundarshankar89 commented Oct 22, 2025

View reviewed changes

sundarshankar89 requested a review from gueniai October 22, 2025 03:04

Merge branch 'main' into feature/profiler_entry_point

a5630cd

sundarshankar89 temporarily deployed to tool October 22, 2025 03:04 — with GitHub Actions Inactive

Add CLI command for publishing 'local' dashboards.

9847822

goodwillpunning temporarily deployed to tool October 27, 2025 12:25 — with GitHub Actions Inactive

goodwillpunning reviewed Oct 27, 2025

View reviewed changes

Merge branch 'main' into feature/profiler_entry_point

a3d03d6

sundarshankar89 temporarily deployed to tool October 27, 2025 16:03 — with GitHub Actions Inactive

Merge branch 'main' into feature/profiler_entry_point

0e659b0

sundarshankar89 temporarily deployed to tool October 28, 2025 11:32 — with GitHub Actions Inactive

removed local dashboard

37029bb

sundarshankar89 temporarily deployed to tool October 28, 2025 11:47 — with GitHub Actions Inactive

gueniai approved these changes Oct 28, 2025

View reviewed changes

Merge branch 'main' into feature/profiler_entry_point

fca09b9

sundarshankar89 temporarily deployed to tool October 29, 2025 12:51 — with GitHub Actions Inactive

Profiler CLI #1623

Are you sure you want to change the base?

Profiler CLI #1623

Uh oh!

Conversation

sundarshankar89 commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

What does this PR do?

Relevant implementation details

Caveats/things to watch out for when reviewing:

Linked issues

Functionality

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

gueniai left a comment

Choose a reason for hiding this comment

Uh oh!

gueniai Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

sundarshankar89 Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

sundarshankar89 Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sundarshankar89 Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

goodwillpunning Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

sundarshankar89 Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gueniai left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sundarshankar89 commented Jun 3, 2025 •

edited

Loading

codecov bot commented Oct 22, 2025 •

edited

Loading

sundarshankar89 Oct 28, 2025 •

edited

Loading