Skip to content

Commit 1abc9fe

Browse files
JCZuurmondJCZuurmond
authored
Add dashboard for tracking migration progress (#3016)
## Changes Add dashboard for tracking migration process, screenshot below as comments. Open changes: - [x] Update install to replace UCX catalog in queries, call the value to be replace `multiworkspaces` - [x] Add counter for tables and views migrated - [x] Use object serializer from #2743 - [x] Filter for latest scan only - [x] Add overview of the (unique) failure messages - [x] Reword "readiness" ### Linked issues Resolves #2596 References databrickslabs/lsql#306 Requires: #3083 ### Functionality - [x] added a new dashboard: migration-progress ### Tests - [x] manually tested - [x] added integration tests - [x] verified on staging environment (screenshot attached)
1 parent 35a0309 commit 1abc9fe

21 files changed

+529
-5
lines changed

‎src/databricks/labs/ucx/install.py

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -634,9 +634,16 @@ def _handle_existing_dashboard(self, dashboard_id: str, display_name: str, paren
634634
def _create_dashboard(self, folder: Path, *, parent_path: str) -> None:
635635
"""Create a lakeview dashboard from the SQL queries in the folder"""
636636
logger.info(f"Creating dashboard in {folder}...")
637-
metadata = DashboardMetadata.from_path(folder).replace_database(
638-
database=f"hive_metastore.{self._config.inventory_database}",
639-
database_to_replace="inventory",
637+
metadata = (
638+
DashboardMetadata.from_path(folder)
639+
.replace_database( # Assessment and migration dashboards
640+
database=f"hive_metastore.{self._config.inventory_database}",
641+
database_to_replace="inventory",
642+
)
643+
.replace_database( # Migration progress dashboard
644+
catalog=self._config.ucx_catalog,
645+
catalog_to_replace="ucx_catalog",
646+
)
640647
)
641648
metadata.display_name = f"{self._name('UCX ')} {folder.parent.stem.title()} ({folder.stem.title()})"
642649
reference = f"{folder.parent.stem}_{folder.stem}".lower()

‎src/databricks/labs/ucx/progress/install.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,13 +47,15 @@ class ProgressTrackingInstallation:
4747
_SCHEMA = "multiworkspace"
4848

4949
def __init__(self, sql_backend: SqlBackend, ucx_catalog: str) -> None:
50-
# `mod` is a required parameter, though, it's not used in this context without views.
51-
self._schema_deployer = SchemaDeployer(sql_backend, self._SCHEMA, mod=None, catalog=ucx_catalog)
50+
from databricks.labs import ucx # pylint: disable=import-outside-toplevel
51+
52+
self._schema_deployer = SchemaDeployer(sql_backend, self._SCHEMA, mod=ucx, catalog=ucx_catalog)
5253

5354
def run(self) -> None:
5455
self._schema_deployer.deploy_schema()
5556
self._schema_deployer.deploy_table("workflow_runs", WorkflowRun)
5657
self._schema_deployer.deploy_table("historical", Historical)
58+
self._schema_deployer.deploy_view("objects_snapshot", "queries/views/objects_snapshot.sql")
5759
logger.info("Installation completed successfully!")
5860

5961

‎src/databricks/labs/ucx/queries/progress/main/00_0_migration_progress.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
height: 4
3+
---
4+
5+
# Migration Progress
6+
7+
> If widgets show `Unable to render visualization.` verify if
8+
> the [UCX catalog exists](https://github.com/databrickslabs/ucx?tab=readme-ov-file#create-ucx-catalog-command).
9+
10+
This dashboard displays the migration progress, with data visualized from the `migration-progress-experimental`
11+
workflow. This workflow is designed to run regularly — either daily or weekly — to provide an up-to-date overview of the
12+
migration progress.
13+
14+
In addition to offering real-time insights into migration progress, the dashboard also facilitates planning and task
15+
division. For instance, you can choose to migrate one workspace or schema at a time. By assigning a migration owner to
16+
each workspace and/or schema, the dashboard shows how the resources allocated to that owner are progressing.

‎src/databricks/labs/ucx/queries/progress/main/01_0_percentage_migration_progress.sql

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
/* --title 'Overall progress (%)' --width 2 */
2+
SELECT
3+
ROUND(100 * try_divide(COUNT_IF(SIZE(failures) = 0), COUNT(*)), 2) AS percentage
4+
FROM ucx_catalog.multiworkspace.objects_snapshot
5+
WHERE object_type IN ('ClusterInfo', 'Grant', 'JobInfo', 'PipelineInfo', 'PolicyInfo', 'Table', 'Udf')

‎src/databricks/labs/ucx/queries/progress/main/01_1_percentage_udf_migration_progress.sql

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
/* --title 'UDF migration progress (%)' */
2+
SELECT
3+
ROUND(100 * TRY_DIVIDE(COUNT_IF(SIZE(failures) = 0), COUNT(*)), 2) AS percentage
4+
FROM ucx_catalog.multiworkspace.objects_snapshot
5+
WHERE object_type = "Udf"

‎src/databricks/labs/ucx/queries/progress/main/01_2_percentage_grant_migration_progress.sql

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
/* --title 'Grant migration progress (%)' */
2+
SELECT
3+
ROUND(100 * TRY_DIVIDE(COUNT_IF(SIZE(failures) = 0), COUNT(*)), 2) AS percentage
4+
FROM ucx_catalog.multiworkspace.objects_snapshot
5+
WHERE object_type = "Grant"

‎src/databricks/labs/ucx/queries/progress/main/01_3_percentage_job_migration_progress.sql

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
/* --title 'Job migration progress (%)' */
2+
SELECT
3+
ROUND(100 * TRY_DIVIDE(COUNT_IF(SIZE(failures) = 0), COUNT(*)), 2) AS percentage
4+
FROM ucx_catalog.multiworkspace.objects_snapshot
5+
WHERE object_type = "JobInfo"

‎src/databricks/labs/ucx/queries/progress/main/01_4_percentage_cluster_migration_progress.sql

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
/* --title 'Cluster migration progress (%)' */
2+
SELECT
3+
ROUND(100 * TRY_DIVIDE(COUNT_IF(SIZE(failures) = 0), COUNT(*)), 2) AS percentage
4+
FROM ucx_catalog.multiworkspace.objects_snapshot
5+
WHERE object_type = "ClusterInfo"

‎src/databricks/labs/ucx/queries/progress/main/01_5_percentage_table_migration_progress.sql

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
/* --title 'Table migration progress (%)' --width 2 */
2+
SELECT
3+
ROUND(100 * TRY_DIVIDE(COUNT_IF(SIZE(failures) = 0), COUNT(*)), 2) AS percentage
4+
FROM ucx_catalog.multiworkspace.objects_snapshot
5+
WHERE object_type = "Table"

‎src/databricks/labs/ucx/queries/progress/main/01_6_percentage_pipeline_migration_progress.sql

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
/* --title 'Pipeline migration progress (%)' */
2+
SELECT
3+
ROUND(100 * TRY_DIVIDE(COUNT_IF(SIZE(failures) = 0), COUNT(*)), 2) AS percentage
4+
FROM ucx_catalog.multiworkspace.objects_snapshot
5+
WHERE object_type = "PipelineInfo"

0 commit comments

Comments
 (0)