Roboscript

RoboScript is a platform designed to bridge the gap between high - level task planning and the practical deployment of code for robot manipulation: Arxiv. To minimize the sim-to-real gap in code generation for robotics control, the whole pipeline is built on ROS Noetic, Moveit! library and Gazebo Simulation. Thanks to the abstraction and uniformity in the aspect of perception and motion planning provided by the middle layer of ROS and Moveit, the generated code tested in Gazebo can be directly deployed to control real robots with close behavior. The project supports various arms integrated with Moveit stack. Project overview:

TODO List for release

Add ! emoji for the most important tasks.

• ❗ Mandatory:
  • Add support for Gamma
  • Add real robot setup instructions
  • Refactor scripts to reproduce experiments
• ❕ Favaorable:
  • Separate grounding model to a standalone ros node for better environment isolation
  • Add support for multimodal LMs (after refactoring the grounding node)
  • Better Environment & task generation
• ❓ Feature extensions:
  • Add support for Environment generation ---> task generation ---> code generation ---> execution and evaluation ---> data collection

Installation

Please read the notes below carefully before installing anything:

The whole package runs on ROS 1 Noetic. You are recommended to use Robostack to install & run ROS inside conda environment since you need multiple conda environments for different models. It is a well-known issue that native ROS 1 environment is not compatible with conda environment, so Robostack is here to help. You can anyway install original ROS packages in your system as written in the official ROS installation guide. Though DO NOT source the ROS environment (or comment out it in your .bashrc if you already setup one) in your terminal when you are using Robostack. Warnings from Robostack installation guide:

The components needed for this project:

• Franka ROS: Integration of libfranka and ROS control, which provides low-level connection/ low-level control to Franka robots.
• MoveIt: Full-stack library for robot mobile manipulation: planning, manipulation, perception, kinematics, control.
• Code generation dependencies.
• (Optional) any your custom model dependencies.

Download and Install mamba

You should download the mamba package manager to install the dependencies. Mamba is a faster version of conda, which is used to install the C++/python/ROS dependencies in this project.

There are two ways you can install mamba: 1) Fresh install with miniforge 2) Existing conda install (not recommended)

For the fresh install, see Fresh Insall. If you have previously installed conda environment, please disable it by commentting out the conda init scritps in ~/.bashrc or ~/.zshrc. See miniforge for download and initialization.

If you do want to keep using your installed conda environment, please refer to Existing conda install (not recommended) in the official document. You might need to resolve environment conflicts manually.

Install ROS environment with Robostack

Assuming you have installed conda and mamba in your computer. Create a new environment with conda-forge and robostack channels:

Special tip for CHINESE users: Make sure you have full access to Internet or enable proxy. Conda mirrors will NOT work. You need to access robostack channel to install ros packages.

mamba create -n ros_env
mamba activate ros_env

# this adds the conda-forge channel to the new created environment configuration 
conda config --env --add channels conda-forge
# and the robostack channel
conda config --env --add channels robostack-staging
# remove the defaults channel just in case, this might return an error if it is not in the list which is ok
conda config --env --remove channels defaults

Install ROS1 Noetic:

# Install ros-noetic into the environment (ROS1)
mamba install ros-noetic-desktop-full

Install common tools and dependency pacakges:

# Reactivate the environment to initialize the ros env
mamba deactivate
mamba activate ros_env

# Install ros-noetic common dev tools and pin gcc/g++ version
mamba install gcc=9.5.0 gxx=9.5.0 compilers cxx-compiler cmake pkg-config make ninja colcon-common-extensions catkin_tools boost-cpp ros-noetic-ros-numpy ros-noetic-rviz-tools

# Install ros gazebo packages
mamba install ros-noetic-gazebo-ros-pkgs

# Install realsense ros packages and image processing packages
mamba install ros-noetic-realsense2-camera ros-noetic-realsense2-description ros-noetic-librealsense2 ros-noetic-image-pipeline

# Install MoveIt ROS Packages
mamba install ros-noetic-moveit=1.1.11 ros-noetic-geometric-shapes=0.7.3

# Install Franka ROS Packages
mamba install ros-noetic-franka-ros

# Until Nov 2023, there is no pre-built packages in robostack for universal robots, so we need to build from source later

Prepare your catkin workspace

# make sure you have activated the ros environment
mamba activate ros_env
# create a catkin workspace
mkdir -p /path/to/catkin_ws/src
cd /path/to/catkin_ws
catkin init
cd /path/to/catkin_ws/src
git clone --recurse-submodules https://github.com/SgtVincent/RoboScript.git

Build the catkin packages

cd /path/to/catkin_ws
catkin build

Install code generation dependencies:

# code generation dependencies
pip install numpy==1.23 astunparse scipy shapely astunparse pygments openai open3d imageio==2.4.1 imageio-ffmpeg moviepy

Dev tools and tests builds

You can skip this section if you are not developing this project.

Please refer to dev_builds.md for more details.

Troubleshooting

Bug of ros-noetic-genpy 0.6.15 with gazebo:

[ERROR] [xxxxx]: Unpause physics service call failed: transport error completing service call: receive_once[/gazebo/unpause_physics]: DeserializationError cannot deserialize: unknown error handler name 'rosmsg'

Solution: Upgrade ros-noetic-genpy to 0.6.16 by building from source:

cd /path/to/catkin_ws/src
git clone --branch 0.6.16 https://github.com/ros/genpy.git
cd ..
catkin build genpy

Python 3.9 no longer supports thread.isAlive() method:

AttributeError: 'Thread' object has no attribute 'isAlive'

Solution: https://transang.me/add-apt-repository-does-not-work-after-updating-to-python-3-9/

Numpy >= 1.24 removed np.float property, which is used in ros_numpy:

AttributeError: module 'numpy' has no attribute 'float'.

Solution: numpy downgrade to <1.24

If you had some link failures during catkin make, please add the corresponding libraries to target_link_libraries() in the cmake file of moveit_tutorials (depending on the OS).

Test the environment installation

Before proceeding to the next step, please make sure you have the ROS+Gazebo+Moveit environment ready. You can test the environment with the following command:

# make sure you have activated the ros environment
mamba activate ros_env
cd /path/to/catkin_ws
source devel/setup.bash

# launch the gazebo environment
roslaunch instruct_to_policy run_panda_moveit_gazebo.launch verbose:=true

You will see the Gazebo and Moveit rviz window pop up. You can test the robot controller by moving the robot arm in rviz.

Build grasp detection package in a separate workspace

Please refer to grasp_detection package for more details.

Build grounding models pacakge in a separate workspace

TODO: move the grounding model to a separate ros package.

Please refer to grounding_model package for more details.

Data preparation

Please download the preprocessed data from:

Baidu Cloud

Google Drive

Please refer to data for more details.

Run

Please refer to instruct_to_policy package for more details.