Discussion: Default working directories

[phil-opp]

There were some reports that our current working directory selection is too complicated and unintuitive. Let's try to find better defaults in this discussion.

Issue Report

Originally posted by @theol0403 in #1066 (comment):

I ran into this issue today when trying to set up distributed nodes.

Using just a local python node works fine:

  - id: keyboard_controller
    build: pip install -e keyboard_controller
    path: keyboard_controller/src/main.py
    inputs:
      tick: dora/timer/millis/100

  - id: rust_node
    build: cargo build -p rust_node
    path: target/debug/rust_node
    inputs:
      tick: dora/timer/millis/100

Then, dora build and dora start work as expected and start the python and rust nodes locally.

However, making just the rust node distributed:

  - id: rust_node
    _unstable_deploy:
      machine: other
    path: dynamic
    inputs:
      tick: dora/timer/millis/100

Causes keyboard_controller on the LOCAL daemon to fail.

keyboard_controller on default daemon : stdout    /opt/homebrew/Cellar/python@3.12/3.12.11/Frameworks/Python.framework/Versions/3.12/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/theol/Documents/github/rust-ethercat/_work/01981518-3271-7b98-9fcc-97fd5d61297c/keyboard_controller/src/main.py': [Errno 2] No such file or directory
keyboard_controller on default daemon : stdout    
keyboard_controller on default daemon : stdout    
keyboard_controller on default daemon : ERROR  daemon    exited with code 2 with stderr output:
---------------------------------------------------------------------------------
/opt/homebrew/Cellar/python@3.12/3.12.11/Frameworks/Python.framework/Versions/3.12/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/theol/Documents/github/rust-ethercat/_work/01981518-3271-7b98-9fcc-97fd5d61297c/keyboard_controller/src/main.py': [Errno 2] No such file or directory
---------------------------------------------------------------------------------

Then, changing the python node to:

  - id: keyboard_controller
    build: pip install -e ../../keyboard_controller
    path: ../../keyboard_controller/src/main.py
    inputs:
      tick: dora/timer/millis/100

solves the problem. But, I think this is very unintuitive and was very difficult to troubleshoot. Now, when commenting out the distributed rust node, the python node fails because the paths are wrong!

[phil-opp]

Let me summarize how we set the working directory currently:

• Use the path specified in _unstable_deploy.working_dir if set. Else:
• For dora run, use the parent directory of the dataflow file as working directory.
• For dora start, use a _work/<session_id> subdirectory of the directory where the dora daemon was started.
  • Where <session_id> is a unique identifier for the dataflow.yml that is started
  • We choose this subdirectory to avoid conflicts with other dataflows that are running on the same daemon
  • Exception: If no _unstable_deploy keys are specified in the dataflow and if the CLI and daemon are running on the same machine, we use the parent directory of the dataflow file as working directory (like for dora run)
• For nodes with git keys, we clone the repo into a git/<repo_domain>/<repo_path> subfolder of the working directory
  • Then we set the working directory of the node to this clone
• For dora build, we choose the same working directories as for dora start.
  • For dora build --local, we choose the same working directories as dora run

I agree that this is quite complex and that it would be nice to simplify this.

[phil-opp]

Proposal from @theol0403 :

It would be good if the nodes were spawned in the working directory of their daemons.

Could you clarify in which of the above cases this should apply in your opinion? Just for distributed dataflows? Or also for local dataflows, instead of using the parent directory of the dataflow yaml file?

What about nodes with git sources? We probably don't want to clone directly into the working directory and overwrite everything there, right? So I assume that we still want to clone into subfolder. Do we want to keep that subfolder as working dir for the build/run/start commands or would you prefer to use the working dir of the daemon instead (i.e. not use the clone dir as working dir)?

[haixuanTao]

yeah i know but i don't think we should use _unstable_deploy.working_dir and have if condition depending of it is set or not.

I would argue that dora build we should almost always build with the working dir where the build command is spawned.

And yes, people will have to manage where they run the command.

[phil-opp]

Ok, let me try to update the proposal accordingly:

• Remove the _unstable_deploy.working_dir key
• For dora run, use the directory the command was spawned in as working dir (since dora run will spawn a temporary daemon there).
  • This means that cd example; dora run dataflow.yml will set a different working dir than dora run example/dataflow.yml
• For dora start, use the working dir of the daemon directly (no matter if local or distributed)
  • If you start multiple dataflow instances (same or different dataflow file) on the same daemon, they will all use the same working directory (override each other logs, etc)
• For nodes with git keys, we still clone the repo into a git/<repo_domain>/<repo_path> subfolder of the working directory
  • But we set the working dir for build and run operations to the daemon's working dir instead (so the build command will not run inside the cloned repo)
• For dora build, we use the directory where the command is run
  • Remote machines need to use the same directory structure and the directory where dora build is run must also exist there, otherwise an error occurs

Like this?

[haixuanTao]

To me it sounds like a very natural behavior that is consistent.

I'm open for discussion, but at least, this I would believe this has the benefit of feeling natural

[phil-opp]

Ok, let me copy this to a top-level comment for consistency, then we can discuss.

[phil-opp]

I created a top-level comment at: https://github.com/orgs/dora-rs/discussions/1074#discussioncomment-13832322

[phil-opp]

Proposal distinct-deploy-command

Running a dataflow locally is quite different from deploying it to a distributed Dora network. To make this distinction clearer, this proposal introduces a separate CLI command dora deploy for running distributed dataflows. The dora build command will be changed to always do a local build.

Details:

• the working directory for all local operations will be the parent directory of the dataflow configuration file
  • explicit overrides still have priority
• dora run will now ignore any _unstable_deploy keys instead of throwing an error
• dora build will always perform a local build, ignoring any _unstable_deploy keys
• dora start assumes that the coordinator is running locally and sets the working dir to the parent directory of the dataflow configuration file too
  • we will remove the CLI arguments for specifying the coordinator IP address (the coordinator must run locally too)
• introduce a new dora deploy command
  • when run without arguments, it will first prompt for the coordinator IP address and port number and then open an interactive session to the coordinator
    • in the interactive session, you can issue build and start commands that behave like the existing distributed variants of these commands
    • we will also add the other commands that communicate with a coordinator there, e.g. stop or logs
  • You don't have to use the interactive session, you can also run them as non-interactive subcommands. For example, you can do dora deploy start dataflow.yml --coordinator-addr. This is useful for running scripts.
  • The working directory for dora deploy commands will be a _work/<session_id> subdirectory of the daemon's working directory
    • even if no _unstable_deploy keys are specified in the dataflow
    • to support running multiple dataflows in parallel in separate working dirs
    • for nodes that use git repositories we will keep cloning into a _work/<session_id>/git/<repo_domain>/<repo_path> directory and using that directory as working dir

This proposal removes any heuristic that we currently have for treating a dataflow as local or distributed. Instead, the user clearly specifies through the CLI how they want to run their nodes. For local builds and running, the working directory is always the parent dir of the dataflow file, independent of any _unstable_deploy keys. For deploy operations, the working dir a subdir of the special _work dir.

I think this distinction makes sense since deploying a dataflow typically also requires getting the node code from the development machine to the target machine. This could be done using a git key or through a build command that installs from some package registry. By having a separate dora deploy command, it's easy to test whether this setup works even when running it locally (because of the different working dir).

[phil-opp]

Proposal daemon-working-dir

(summary of https://github.com/orgs/dora-rs/discussions/1074#discussioncomment-13788652)

• Remove the _unstable_deploy.working_dir key
• For dora run, use the directory the command was spawned in as working dir (since dora run will spawn a temporary daemon there).
  • This means that cd example; dora run dataflow.yml will set a different working dir than dora run example/dataflow.yml
• For dora start, use the working dir of the daemon directly (no matter if local or distributed)
  • If you start multiple dataflow instances (same or different dataflow file) on the same daemon, they will all use the same working directory (override each other logs, etc)
• For nodes with git keys, we still clone the repo into a git/<repo_domain>/<repo_path> subfolder of the working directory
  • But we set the working dir for build and run operations to the daemon's working dir instead (so the build command will not run inside the cloned repo)
• For dora build, we use the directory where the command is run
  • Remote machines need to use the same directory structure and the directory where dora build is run must also exist there, otherwise an error occurs

[phil-opp]

There are a few things in this proposal that feel unintuitive and difficult to use to me :

• The fact that dora build and dora start use different working directories
• You don't know the working directory when writing the dataflow.yml file. It is decided only later when you run dora daemon and dora build. So you have to assume something.
  • The consequence of this is that you have to take a look at the dataflow.yml and see which working dir it expects before you start the daemons. Otherwise the dataflow might not work when it's started later.
  • The same applies to dora build: You have to read the dataflow.yml to figure out in which directory you need to run dora build.
• Sharing the same working directory for multiple dataflows can lead to conflicts if you're not careful.
  • Nodes can easily overwrite each others files by accident
• The requirement that all machines need to have the same directory layout seems quite limiting. There probably is no /home/philipp/ directory on cloud machines. So how do I build a distributed dataflow that targets both cloud machines and my local computer?
  • I could try to create a /home/philipp directory on the cloud machines, but this is additional effort and might not be allowed on some systems. So I probably need to set up the directory structure of the cloud machine locally.
  • What do I do if I have different working directories on different cloud services (e.g. because of different usernames)?
  • What happens if I want to run a distributed dataflow where some machines run Linux and others Windows? I probably need to create either a C:\Users\USERNAME folder on Linux or a /home/username folder on Windows (is this even possible?)...

[haixuanTao]

So I think that the current implementation of path used the dataflow as the working directory even when we provide a git repository.

I'm unable to run the following example:

  - id: hibiki
    git: https://github.com/haixuanTao/delayed-streams-modeling.git
    args: gen
    build: cargo build --manifest-path stt-rs/Cargo.toml -r --features cuda
    path: git/github.com/haixuanTao/delayed-streams-modeling.git/dd5e92a43b1e6709c8a72d9b58608f26ff62afa0/stt-rs/target/release/kyutai-stt-rs
    inputs:
      audio:
        source: NODE_ID/audio
        queue_size: 1000000
    outputs:
      - text
      - word

without specifying the full path including git.

Am I missing something?

[haixuanTao]

Ok somehow the problem vanished, if I use uv.

[haixuanTao]

okay now it's back with:

error: Failed to spawn: `stt-rs/target/release/kyutai-stt-rs`
  Caused by: No such file or directory (os error 2)

[phil-opp]

Thanks for reporting this, looking into it right now.

[phil-opp]

I cannot reproduce your error, but we do fall back to the base working directory when there is no known clone directory for the node. The most common cause of this is when no dora build was done before. Maybe this was the issue in your case?

I opened #1089 to throw an error in this case, instead of silently choosing a different working dir. This should make it more obvious what the error is. Also, the PR includes the full working directory in the log output, which should make debugging easier.