Add llm-transpile command with Switch runner and integration tests
#2078
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hiroyukinakazato-db
added
enhancement
hiroyukinakazato-db
temporarily deployed
to
tool
GitHub Actions
Inactive
|
✅ 46/46 passed, 6 flaky, 3m12s total Flaky tests:
Running from acceptance #2812 |
hiroyukinakazato-db
temporarily deployed
to
tool
GitHub Actions
Inactive
hiroyukinakazato-db
temporarily deployed
to
tool
GitHub Actions
Inactive
hiroyukinakazato-db
force-pushed
the
feature/llm-transpile
branch
from
804507e to
d884a20
Compare
hiroyukinakazato-db
had a problem deploying
to
tool
GitHub Actions
Failure
hiroyukinakazato-db
temporarily deployed
to
tool
GitHub Actions
Inactive
hiroyukinakazato-db
temporarily deployed
to
tool
GitHub Actions
Inactive
Implement llm-transpile command for LLM-based code transpilation: - Add SwitchInstaller for Switch transpiler package management - Install Switch package and deploy to workspace - Create and manage Databricks jobs with job-level parameters - Configure Switch resources (catalog, schema, volume) - Add SwitchRunner for executing Switch transpilation jobs - Upload source files to workspace volume - Execute transpilation via Databricks job - Download results and handle job lifecycle - Add llm-transpile CLI command with Switch transpiler support - Add comprehensive unit and integration tests
Move _get_switch_package_path() from WorkspaceInstallation to SwitchDeployment as a protected method, following Single Responsibility Principle. SwitchDeployment now resolves its own package path internally. Changes: - Add _get_switch_package_path() protected method to SwitchDeployment - Update SwitchDeployment.install() signature to remove path parameter - Remove duplicate _get_switch_package_path() from WorkspaceInstallation - Remove unused sys and TranspilerRepository imports from installation.py - Update tests to use new interface with mocked path resolution
Update test_installation.py to match the refactored SwitchDeployment.install() interface that now takes only resources parameter (path resolution is internal). Changes: - Remove switch_repository fixture parameter from test methods - Delete unused _StubTranspilerRepository stub class - Remove unused imports (Path, TranspilerRepository) - Update assertions to check only resources argument The tests verify that: 1. Switch installation uses configured resources correctly 2. Missing resources logs appropriate error message
hiroyukinakazato-db
force-pushed
the
feature/llm-transpile
branch
from
4f551ce to
bacd5f6
Compare
hiroyukinakazato-db
temporarily deployed
to
tool
GitHub Actions
Inactive
Sync with main branch to incorporate latest documentation updates # Conflicts: # labs.yml
hiroyukinakazato-db
temporarily deployed
to
tool
GitHub Actions
Inactive
The wait_for_completion option is intended for local CLI execution only and should not be included in Databricks job parameters. This change filters it out when building job parameter definitions. Changes: - Add excluded_options set to filter local-only options - Skip wait_for_completion when converting config.yml options - Add test using FriendOfSwitchDeployment pattern to verify exclusion
hiroyukinakazato-db
temporarily deployed
to
tool
GitHub Actions
Inactive
…ntrol Add test_switch_install_with_transpile for full workflow testing including job execution and output verification. Test automatically skips without LAKEBRIDGE_SWITCH_E2E=true environment variable. Refactor existing test to test_switch_install and extract helper functions for DRY implementation. All changes in single file with no CI impact.
…que suffix generation
Merged latest changes from main branch including: - Profiler skeleton and Synapse profiler scripts - Transpiler product_name to transpiler_id rename - Additional transpile command arguments support - Test improvements for MSSQL and transpiler repository
hiroyukinakazato-db
temporarily deployed
to
tool
GitHub Actions
Inactive
sundarshankar89
approved these changes
Oct 29, 2025
| ) -> None: | ||
| """Transpile source code to Databricks using LLM Transpiler (Switch)""" | ||
| ctx = ApplicationContext(w) | ||
| ctx.add_user_agent_extra("cmd", "llm-transpile") |
There was a problem hiding this comment.
Suggested change
| ctx.add_user_agent_extra("cmd", "llm-transpile") | |
| ctx.add_user_agent_extra("cmd", "switch") |
| from databricks.labs.lakebridge.transpiler.switch_runner import SwitchConfig, SwitchRunner | ||
|
|
||
|
|
||
| class FriendOfSwitchRunner(SwitchRunner): |
There was a problem hiding this comment.
Suggested change
| class FriendOfSwitchRunner(SwitchRunner): | |
| class MockSwitchRunner(SwitchRunner): |
| - name: interactive | ||
| description: "Whether installing in interactive mode, which may prompt for configuration settings." | ||
| default: auto | ||
| - name: include-llm-transpiler |
There was a problem hiding this comment.
confused why this is here in this PR?
gueniai
reviewed
Oct 30, 2025
| if ctx is None: | ||
| ctx = ApplicationContext(w) | ||
| del w | ||
| ctx.add_user_agent_extra("cmd", "transpile-switch") |
There was a problem hiding this comment.
the command is llm-transpile
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
This PR adds the
llm-transpilecommand for LLM-powered SQL conversion using the Switch transpiler.What does this PR do?
Adds
llm-transpileCLI command that runs Switch transpiler jobs with parameter passing support.Relevant implementation details
CLI Integration:
llm-transpilecommand to Lakebridge CLISwitch Runner Implementation:
SwitchConfig: manages Switch resources and job ID retrieval from InstallStateSwitchRunner: orchestrates Switch job execution with parametersTesting:
Development Environment:
.envto.gitignorefor local development credentialsCaveats/things to watch out for when reviewing:
transpileandreconcommand patternsreconpattern)--output-ws-folder(not--output-folder) to explicitly indicate workspace folder--include-llm-transpilerflag #2066 (Switch installation) to be merged firstLinked issues
Resolves #2047
Functionality
databricks labs lakebridge llm-transpileTests