Support running ros2_bag binary from any directory

[kgreenek]

I ran into another potential snag. I created a bag target like so:

load("@com_github_mvukov_rules_ros2//ros2:bag.bzl", "ros2_bag")

ros2_bag(
    name = "bag",
    idl_deps = [
        "@ros2_common_interfaces//:nav_msgs",
        "@ros2_common_interfaces//:sensor_msgs",
        "@ros2_common_interfaces//:std_msgs",
        "@ros2_rcl_interfaces//:rcl_interfaces",
        "@ros2_rosbag2//:rosbag2_interfaces",
    ],
)

When I run it using bazel, it works as expected. However, when I run it directly from bazel-bin, I get the following error:

$ ./bazel-bin/romalcpp/examples/ros2/bag info [path-to-bagfile]
Traceback (most recent call last):
  File "/home/kgk/src/romalorg/romalcpp/./bazel-bin/romalcpp/examples/ros2/bag.runfiles/__main__/romalcpp/examples/ros2/bag_launch.py", line 25, in <module>
    sys.exit(ros2cli.cli.main(extension=extension))
  File "/home/kgk/src/romalorg/romalcpp/bazel-bin/romalcpp/examples/ros2/bag.runfiles/ros2cli/ros2cli/ros2cli/cli.py", line 46, in main
    extension.add_arguments(parser, script_name)
  File "/home/kgk/src/romalorg/romalcpp/bazel-bin/romalcpp/examples/ros2/bag.runfiles/com_github_mvukov_rules_ros2/ros2/ros2_cmd.py", line 104, in add_arguments
    add_subparsers(parser,
  File "/home/kgk/src/romalorg/romalcpp/bazel-bin/romalcpp/examples/ros2/bag.runfiles/com_github_mvukov_rules_ros2/ros2/ros2_cmd.py", line 86, in add_subparsers
    extension.add_arguments(command_parser, f'{cli_name} {name}')
  File "/home/kgk/src/romalorg/romalcpp/bazel-bin/romalcpp/examples/ros2/bag.runfiles/ros2_rosbag2/ros2bag/ros2bag/verb/info.py", line 24, in add_arguments
    add_standard_reader_args(parser)
  File "/home/kgk/src/romalorg/romalcpp/bazel-bin/romalcpp/examples/ros2/bag.runfiles/ros2_rosbag2/ros2bag/ros2bag/api/__init__.py", line 125, in add_standard_reader_args
    reader_choices = rosbag2_py.get_registered_readers()
RuntimeError: package 'rosbag2_storage' not found, searching: []

Here is the example repository I have been using if that it helpful:
https://github.com/romalorg/romalcpp

[mvukov]

Somewhat related to #57. What you get above is expected behavior at the moment. Please let me know what's your use-case where you would want to run e.g. rosbag utils w/o invoking bazel.

[kgreenek]

We install our software onto a robot and it is run there. There is no bazel on the robot / production system. We do need to run rosbag record (and other ros2 tools) on the robot. So this is really important for us.

[kgreenek]

The flow looks like this:

• CI uses bazel to build binaries (i.s. ROS2 nodes)
• CI deploys the compiled binaries onto the production robot

On the robot, we need to be able to record rosbags. We do not / will not install bazel anywhere on the robot, nor do we compile code there.

[mvukov]

Gotcha. This is not a problem with this repo, but related to packaging of bazel binaries. Long time ago I created a rule in https://github.com/mvukov/rules_ros/blob/main/third_party/packaging.bzl for this purpose (look at the main readme for more info). That solution is heavily based on rules_docker, just FYI. There is probably a simpler solution by now. At the time I created this rule rules_pkg didn't work well. The situation might have changed -- TBH, I would try a recent rules_pkg.

IIUC, you want to deploy on the robot several binaries, including a ros2_bag target. How do you group binaries to put them with Bazel into a single archive? Do you simply group them with a filegroup? If this is the case, the rule I put together isn't really going to cut it -- check rules_pkg first :)

You could also check https://github.com/lalten/rules_appimage.

[kgreenek]

Excellent questions. We are still trying out different options.

I have used rules_pkg for deployment at some pretty large scale in the past with success -- although it had lots of quirks. I have also used rules_docker to push docker images and deploy that way.

In this case, I do think it is an issue with this repository because trying to run the ros2_bag target from bazel-bin fails, even before doing any kind of packaging / deployment.

For example, in my example repo, the following does not work:

bazel build //romalcpp/examples/ros2:bag
./bazel-bin/romalcpp/examples/ros2/bag info [path-to-bagfile]

Note that the bag target hasn't been packaged at all. I'm just running the binary that was created by the build command. It gives the error from the description.

So even if it were deployed correctly using rules_pkg or something else, I believe it would still fail to run.

[mvukov]

OK, I see. Once you build the target, if you want to run it w/o bazel you actually have to do:

cd bazel-bin/package/binary.runfiles/workspace_name
./package/binary

And, indeed, rules_pkg alone wouldn't solve all this. For this purpose I generate a custom script within binary_pkg_tar rule, see https://github.com/mvukov/rules_ros/blob/main/third_party/packaging.bzl#L265-L267.

For a statically linked binary with no file-deps, you can just invoke the binary from wherever you want. Bazel built ROS2 binaries have to be invoked from a correct folder. Well, any binary that has e.g. bazel built shared lib deps or data files.

[kgreenek]

Got it. I can confirm that cd'ing into the runfiles directory worked. Here is what I did:

cd bazel-bin/romalcpp/examples/ros2/bag.runfiles/__main__
./romalcpp/examples/ros2/bag info [path to bagfile]

Unfortunately, this doesn't help me in my deployment use-case. My plan was to symlink the bag command somewhere that is in the $PATH when deployed to the robot. Creating a symlink is supported by the rules_pkg command when the tar file is uncompressed. This way, the bag command can be used from anywhere to record data.

Imagine ssh'ing into the robot and being able to do this:

cd ~/data/
bag record --all

This will work because the binary from my bag target is symlinked into the $PATH. When you run the command, it will create a bagfile named with the current timestamp in the current directory (e.g. ~/data/2023-04-17-12-12-00).

With the current implementation of the ros2_bag target, here is what our team would have to do on the robot in order to use it:

cd /opt/romalcpp/romalcpp/examples/ros2/bag.runfiles/__main__
./romalcpp/examples/ros2/bag record --all -o ~/data/my_bag_file

In this example scenario, /opt/romalcpp is the install path where we unpackaged the binaries onto the robot (or in a docker container, but you get the idea).

I would argue that this is pretty un-ergonomic. It requires $PWD to be set to something that is deeply nested and unintuitive. And I also have to pass the -o flag explicitly every time I run the bag command instead of allowing it to be generated with the current timestamp as in the first example.

Is there any way to support running ros2_bag targets without having a specific requirement on the current working directory?

[kgreenek]

Maybe there is an environment variable I could set? Or maybe the ros2_bag target could output a generated wrapper bash script that sets up the ros2 stuff to be relative to the path of the bash script itself?

[kgreenek]

I noticed that this appears to affect other ros2 targets and not just ros2_bag, e.g. ros2_cpp_binary, etc.

I played around a bit with the generated *_launch file for that target and I was able to make it run independently from $PWD by adding the following:

# Get the full path to the directory where this script is located.
dir="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
runfiles_dir="${dir}/target_name.runfiles/__main__"

Then for all the paths in that script, instead of making them relative to the current working directory, I made them relative to ${runfiles_dir}. For example:

ament_prefix_path="${runfiles_dir}/path/to/package/target_name_launch_ament_setup"

I think that should work for the ros2_cpp_binary target at least. The ros2_bag target does not appear to have a generated wrapper bash script, but maybe something similar could be done for it too.

[lalten]

@kgreenek maybe you'll find https://github.com/lalten/rules_appimage helpful. Its purpose is to package a executable Bazel target into a portable executable.

[kgreenek]

That is potentially useful, thank you.

I updated the description for more clarity. The feature request here is unrelated to deployment / packaging. What I really need is the ability to run the ros2_bag binary from any directory, rather than requiring $PWD to be in the runfiles directory.

[mvukov]

So, let's start with what happens with bazel run //path/to:a_bin: bazel changes cwd to the corresponding runfiles dir and then invokes the binary. That would be a runfiles dir in bazel cache. Even in this case invoking e.g. bazel run //path/to:my_rosbag_binary -- -o foo will create a dir somewhere in Bazel cache (foo is not absolute path).

That's for any binary that has some runtime references to files; nothing specific to this repo. This ensures that e.g. rpaths, referenced file/dir paths and symlinks are valid. I think you cannot escape this and need something like

(cd to/runfiles_dir && ./path/to/binary "$@")

... as I said above. You can try a custom Bazel rule e.g. based on my work mentioned above or create a shorthand/alias for this in e.g. bashrc.

It's not something very intuitive at first sight, but once you learn a bit about runfiles layout, it makes sense IMO.

[mvukov]

I noticed that this appears to affect other ros2 targets and not just ros2_bag, e.g. ros2_cpp_binary, etc.

I played around a bit with the generated *_launch file for that target and I was able to make it run independently from $PWD by adding the following:

# Get the full path to the directory where this script is located.
dir="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
runfiles_dir="${dir}/target_name.runfiles/__main__"

Then for all the paths in that script, instead of making them relative to the current working directory, I made them relative to ${runfiles_dir}. For example:

ament_prefix_path="${runfiles_dir}/path/to/package/target_name_launch_ament_setup"

I think that should work for the ros2_cpp_binary target at least. The ros2_bag target does not appear to have a generated wrapper bash script, but maybe something similar could be done for it too.

Something like this will invalidate symlinks that a binary may use, I am afraid. Also, ROS nodes have hard-coded path to rwm impl shared library -- for this reason one should not change cwd of nodes/deployments.