-
Notifications
You must be signed in to change notification settings - Fork 96
Open
Labels
CUJcritial user journeycritial user journeyEPICmigrate/codeAbstract Syntax Trees and other dark magicAbstract Syntax Trees and other dark magicmigrate/jobsStep 5 - Upgrading Jobs for External TablesStep 5 - Upgrading Jobs for External Tables
Description
Most of the Databricks Workflows are just scheduled Python notebooks. We need to migrate all relevant parts of the code in those notebooks including but not limited to:
- [EPIC] Lint all notebooks that belong to a job in scope of the assessment workflow #1085
- [FEATURE]: If a code computes a value dynamically, do value inference, at least at the state of linting #1205
- [FEATURE]: Rewrite incompatible SQL UDF declarations #1119
- [FEATURE]: Upgrade usage of s3fs library #1117
- [FEATURE]: Rewrite usages of DBFS Mount paths in Init Shell Scripts #1115
- [FEATURE]: Rewrite usages of DBFS Mount paths in Python code, including
/dbfs/mnt
local fs,dbfs:/
, and/mnt
#1109 - [FEATURE] Create mapping of directfs accesses into UC tables #392
- [FEATURE]: Regression testing harness for post-migration validation of configurations (Workload Assets) #1122
- [FEATURE]: Rewrite incompatible SQL UDF declarations #1119
- [FEATURE]: Rewrite usages of DBFS Mount paths in Init Shell Scripts #1115
- [FEATURE]: HMS Function Migration #885
- Migrate streaming checkpoints #492
- [FEATURE]: ensure Spark.sql-like code is not migrated accidentally #1213
- [FEATURE]: Mark notebooks as migrated #1327
Proposed Solution
- Create
migrate-notebooks-for-job
andrevert-migrate-notebooks-for-job
CLI commands - Build on top of https://github.com/databrickslabs/ucx/blob/main/src/databricks/labs/ucx/source_code/notebooks.py#L20-L32
- Backup notebooks before doing code migration
- Execute code migration
- tag relevant job as
uc-migrated
Additional Context
No response
Metadata
Metadata
Assignees
Labels
CUJcritial user journeycritial user journeyEPICmigrate/codeAbstract Syntax Trees and other dark magicAbstract Syntax Trees and other dark magicmigrate/jobsStep 5 - Upgrading Jobs for External TablesStep 5 - Upgrading Jobs for External Tables
Type
Projects
Status
No status