Step Launchers supersession announcement

[danielgafni]

StepLauncher supersession announcement

TLDR

The StepLauncher is being superseded with Dagster Pipes. We recommend users start exploring Dagster Pipes and consider migrating (see the Spark example) existing StepLauncher usages to Pipes. While StepLauncher will remain available, this feature will no longer be receiving active development.

Users who prefer Step Launchers to Pipes because of some features like IOManager integration are welcome to do so, given they accept the increased development and deployment complexity.

Context

StepLauncher was an experimental Dagster resource which could be used to run Dagster steps in remote environments such as Spark. The following step launchers were implemented:

• dagster_databricks.databricks_pyspark_step_launcher
• dagster_aws.emr.emr_pyspark_step_launcher

The goal of step launchers was to seamlessly execute Dagster code in Spark or another remote environment. In particular, a StepLauncher is responsible for:

1. Setting up the remote environment - uploading Dagster code and installing dependencies
2. Executing the op/asset remotely
3. Fetching logs and Dagster events from the remote process and making them available to Dagster.

However, this power came at the cost of significant implementation complexity and feature coupling. If you've been using step launchers, especially in Spark, you might wonder why we're moving away from this approach. Historically, we tried to integrate business logic and orchestration in external runtimes like Spark using framework-level abstractions. While StepLauncher aimed to make remote execution easier, it faced several challenges that made adoption difficult:

• Rigid Code Structure: You had to structure your deployed code around Dagster code locations and definitions, which could be impractical if you had existing Spark jobs or dependency conflicts.
• Deployment Limitations: StepLauncher managed code deployment at runtime, which often conflicted with DevOps processes. Many users preferred managing deployments during push time instead.
• Complex Customization: Writing custom step launchers was complex, making them difficult to extend and use effectively.
• Runtime dependencies: The StepLauncher required to have dagster installed in the remote environment, which might introduce version conflicts or increase the complexity of the deployment process.
• Usage with other programming languages: because of the previous point, it was not possible to use StepLauncher with programming languages other than Python, which is a blocker for popular Scala/Java Spark workloads.

For those who do not want to structure their Spark business logic around Dagster definitions, we believe that Dagster Pipes -- a more composable and lightweight solution -- is the right path forward.

Dagster Pipes

Dagster Pipes is a wire protocol that handles parameter/context passing to the remote process, and log/metadata gathering from the remote process. This approach aligns better with Dagster's philosophy of modularity and extensibility, enabling users to create more flexible and powerful remote execution solutions, albeit with some additional setup responsibility.

Some of the improvements over StepLauncher are:

➕ Increased composability: Pipes components can be mixed and matched, allowing for use in a wide variety of environments.

➕ Decreased complexity: individual Pipes components can be implemented and tested in isolation, making it easier to develop and maintain custom solutions. This modular approach allows for greater flexibility and adaptability to specific use cases.

➕ Improved extensibility: Users can easily extend the existing existing family of Pipes components to meet their unique requirements, fostering a more diverse ecosystem of integrations.

➕ Lightweight: Pipes can execute unmodified scripts. In order to send additional Dagster events (such as Dagster metadata or asset check results) back to the orchestration process, a zero-dependency (and single-file) dagster-pipes Python package can be installed in the remote environment.

➕ Language-agnostic: Implementing Dagster Pipes in additional programming languages is very tractable. Dagster customers have already done this. Right now we only have an official implementation for Python, but we will be adding support to more languages in the near future (JVM languages being in progress).

As Pipes are more lightweight and give you greater control, they also come with some responsibilities:

➖ Pipes do not automatically set up the remote environment. This responsibility now falls to the user, typically handled through CI/CD processes.

➖ Pipes to not automatically execute op/asset body. Instead, the users are typically expected to provide an external script which will be launched by Pipes.

Pipes Clients

We have implemented a set of opinionated Pipes clients on top of the Pipes framework for some popular services.

• Subprocess
• Docker
• Kubernetes (via Pod)
• Databricks
• AWS Lambda
• AWS Glue
• AWS ECS
• AWS EMR Serverless
• AWS EMR

---

References

• Spark Step Launchers -> Pipes migration guide

[dinis-rodrigues]

This will imply major major breaking changes in the future. From the release notes the step launcher:

will not be removed from the codebase until Dagster 2.0 is released

Meaning that from v2 support it could be removed entirely.
Organizations currently using step launchers and io_managers (like us) will need to refactor the entire codebase to work with dagster pipes.

Yes the step launcher has its drawbacks, but it is easily customizable (like dagster pipes), by extending its class in my opinion.
If we compare the two, the boiler plate code which is needed to setup dagster pipes, rounds up to the same as a custom inherited step launcher.

And the biggest drawback of them all is the deprecation of io_manager with dagster pipes. For example, many spark jobs rely on intermediate outputs (instead of cache) to optimize the spark job. We for example, have intermediate outputs in all jobs, because we have complex queries which require this optimization.

Having to configure each op with a hardcoded "io_manager" in the end / beginning, just to make things work, is just friction to developers.

I think the idea behind dagster pipes is good, but it feels like we had a battletested steplauncher and now we are "obligated" to use a new less-featured almost blank canvas tool. For newcomers in dagster, this might be okay, but for existing users, its a barrier.

If the step launcher had its flaws, it should've been a matter of improving the existing solution, instead of building from the ground up a new one and making existing workflows deprecated.

Maybe this opinion is due to my lack of experience and experimentation with pipes, but just the thought of having to refactor everything, makes me demotivated just to try it

[jmccartin]

Hi, just come across this discussion after a ticket I was watching was closed, and this was referenced.

We are in the same boat as @dinis-rodrigues, our company makes use of remote execution on Azure Databricks quite heavily, with a somewhat complex chain of custom IO managers to handle data loading and writes. I haven't looked too deeply into dagster-pipes, but from the sounds of it, our codebase would be very difficult to port over. I would be fine continuing to use the current step launchers, but there is some long-term concern regarding the Databricks SDK (the existing step launcher is built on a deprecated SDK version). If there is no longer going to be active development from the core team, would it be up to the community to take over? It would be good to get an idea of how many people/teams are affected by this.

[danielgafni]

1. re: switching to Pipes in your case - you are right about it probably being too complicated right now. I suggest you stay on Step Launchers. Perhaps in the future Pipes could become more compatible with IOManagers which will make it easier for you to switch.
2. re: community taking over StepLauncher support - potentially, I don't see serious obstacles to that, but it's better if @yuhan or @schrockn could answer to this question

[jmccartin]

Any update on this? I notice that the documentation for the step launcher is essentially now removed, and the line "usage is discouraged" seems to conflict with what you guys are saying here.

Should I just create a fork of the library and see how far I get with regards to updating it for the new SDK version?

[danielgafni]

Hey @jmccartin!

We've updated the wording here. It should be aligned with this thread now.

As for your other question:

If there is no longer going to be active development from the core team, would it be up to the community to take over?

let's see if maybe @cmpadden @yuhan have an answer?

[schrockn]

Just to expand on this, @jmccartin thanks for the feedback here and as @danielgafni we've updated the docs to clarify the state of things.

To provide additional context and framing, Pipes is a "one step backwards two step forwards" type of situation. Pipes actually has less features than step launchers when it comes to executing on systems like Spark. But Pipes is easier to use, more composable, and more architecturally sound. For most people step launchers were just too hard to use and customize, and step launchers were too presumptuous about taking over responsibility for deploying code.

Our plan moving forward to build features that stack on top of Pipes to allow users to separate I/O and compute in the same way that I/O managers do today (and would likely be backwards compatible with I/O managers). We think this approach would be more flexible and composable: E.g. you could be able to deploy the transforms to a remote cluster without having to bring in all of dagster.

Until we have such a capability we will not eliminate or abandon Step Launchers. New users have much more success with Pipes so we are driving them towards that, hence the messaging. If you are one of the intrepid souls that have successfully bought into step launchers, it's quite a magical experience, and we don't want to take that away. We look forward to bringing similar capabilities to a broader audience in the future.

[jtv8]

Weighing in here as someone who evaluated both Pipes and Step Launchers and purposely chose Step Launchers: the Pipes experience is the exact opposite of what I want.

• It requires you to split your asset definitions into two Python files in two separate locations, one with your metadata and dependencies and the other with your actual asset code. The fact that you don't normally have to do this is one of the key selling points of Dagster.
• You can't locally test your code as a single artefact in the same way you could with no_step_launcher.
• It does not deploy your code for you and requires you to develop a whole additional CI/CD process to push your code to Databricks (I disagree with the argument that this is a deployment activity, any more than you "deploy" SQL to a database when you execute it).
• It requires you to wrap your code in boilerplate just to do simple things like read parameters and report asset materializations, rather than doing what Dagster is good at: leveraging simple Python functions.
• It does not appear to support I/O managers at all - these are perhaps the most useful feature in Dagster for platform teams who want to create abstractions to standardise how assets are read and written.

In summary: What I really want is a way to write Spark assets as simple Python functions, and I want to do it in a way that's completely agnostic of the execution environment - as an asset developer, most of the time, I don't want to handle low-level code for configuring and communicating with that execution environment. Those are concerns for platform developers. Until it's possible to use Pipes in such a way, I can't see myself recommending it.

If anything, it would be preferable to go in completely the opposite direction and implement a fully fledged run launcher for environments like Databricks and cut out the Kubernetes job entirely, if such a thing were possible. But absent that, I'll continue to use Step Launchers for as long as they're supported - and if we were forced to use Pipes in the future, we would probably end up building our own abstractions that make them work exactly how Step Launchers do now.

[schrockn]

Hey @jtv8 thanks for the thoughtful commentary. We understand completely that for people who are fully bought into that entire stack, Step Launchers provided a pretty compelling end-to-end product experience.

For Spark and Databricks specifically we are very excited about Spark/Databricks connect and are very actively looking at working on this. We have customers that already use it to great effect.

With Connect, we think we will be able to provide a true best-of-both-worlds experience, allowing you to keep your business logic in Dagster but do actual execution on remote clusters without having to manage deployment at all. Step Launchers are completely unnecessary in that world, but you get the lightweight functional API you desire.

My recommendation for you is the following: continue using step launchers and i/o managers since you are getting value out of them. As our Connect integration matures, you should be able to move that quite easily and eliminate step launchers entirely, which is a more robust approach the leverages more of the built-in capabilities in Spark/Databricks.

[jtv8]

Hi @schrockn - thanks for sharing that! Leveraging Databricks Connect would make perfect sense - it would separate concerns properly between Dagster as an orchestrator and Databricks as a compute environment, and would provide a consistent experience whether the code is running in Dagster or a local development environment. Also, I suspect it would improve cluster spin-up times quite a bit, which is always a big pain point with Databricks.

I'll recommend to my team that you we do what you suggested, and I'll keep an eye out for the Databricks Connect integration!