Use your phone as a sensor in ROS2

While many projects exist to control your robot from your phone, this project is the other way around; your phone is the robot' sensors ! It will send the camera feed, IMU and GPS so that you may integrate the phone onto a mobile base.

The particularity of this project is that it relies on the mobile browser instead of a specific app. A webpage is served from the ROS2 node, opening the page from a mobile client will ask for permissions in the browser. The data is transmitted between the server and client using websockets for a modern and fast communication, as opposed to making HTTP requests or streaming UDP.

This repository is inspired by a project I did with students as a TA called phone-imu.

Build

source /opt/ros/humble/setup.bash
rosdep install -i --from-path src --rosdistro humble -y --ignore-src
colcon build --packages-up-to phone_sensors
# Or build for development, but remember to clean and rebuild every time the JS files are changed
# colcon build --symlink-install  --packages-up-to phone_sensors_examples --event-handlers console_direct+

phone_sensors usage

Quickstart

Run the following in a different terminal than the one used for building, otherwise the server might start from inside the build folder which will not contain the template and static folders for the webpage.

source install/setup.bash

# Ubuntu IP
EXTRA_IP=$(ip route get 8.8.8.8 | grep -oP 'src \K[^ ]+')
echo "My IP is $EXTRA_IP"
# Generate SSL certificates for local webserver 
ros2 run phone_sensors generate_dev_certificates.sh $EXTRA_IP

# Start server
ros2 run phone_sensors server --ros-args -p video_width:=1280 -p video_height:=720

Open the webpage from your mobile device. The URL contains the server host IP where the node is running. It depends on your network, but most likely is https://<EXTRA_IP>:2000. If you have multiple network interfaces, favour the fastest such as ethernet over wifi.

The page will prompt for permissions, then display the chosen camera

Parameters

Name	Type	Default	Unit	Description
host	string	"0.0.0.0"	IP	Use 0.0.0.0 to accept connections outside of localhost
port	int	2000		The port where the server listens on
debug	bool	True		Use Flask in debug mode
secret_key	string	"secret!"		Flask SECRET_KEY
ssl_certificate	string	"certs/certificate.crt"	path	Path to public SSL certificate
ssl_private_key	string	"certs/private.key"	path	Path to private SSL key
use_ros_time	bool	False		Use ROS time instead of device time for message timestamps
time_reference_source_device	string	"ros_to_device"		Source identifier for device TimeReference messages
time_reference_source_gnss	string	"device_to_gnss"		Source identifier for GNSS TimeReference messages
time_reference_frequency	float	-1.0	Hz	Rate to emit TimeReference data
imu_frequency	float	50.0	Hz	Rate to emit IMU data
gnss_frequency	float	10.0	Hz	Rate to emit GNSS data
frame_id_imu	string	package_name		Frame ID for IMU messages
frame_id_gnss	string	package_name		Frame ID for GNSS messages
frame_id_image	string	package_name		Frame ID for camera image messages
camera_device_label	string	"Facing front:1"		Label to identify which camera to use
video_fps	float	30.0	Hz	Video frame rate
video_width	int	1280	pixels	Video frame width
video_height	int	720	pixels	Video frame height
video_compression	float	0.3	0-1	JPEG compression quality (0=max compression, 1=best quality)
camera_calibration_file	string	""	path	Path to camera calibration YAML file (output from camera_calibration)

A negative value for the time reference, IMU, GNSS frequencies or video FPS will disable sending the corresponding data from the client device. This allows conserving bandwidth and processing power when certain sensors are not needed.

To find out the available camera_device_label, open the video test page

Coordinates

The device rotation rate and acceleration are defined in the device coordinate frame:

The device orientation is defined in ENU. The angles alpha, beta, gamma we get from the web API are the respective rotations around Z, X, Y. We convert these angles to quaternion by using beta as roll, gamma as pitch, alpha as yaw. The behaviour of a device with its Y-axis pointing up is unstable since the API is only defined between -90° to +90°. See the web API documentation for more details.

Browser compatibility

The current server is tested with Firefox and Chrome.

GeoLocation:

• Issue in Firefox: getting the position silently fails

Video:

• Browsers do not use the same device labels

phone_sensors_examples

Camera calibration & RVIZ

To calibrate you camera, print the checkerboard in maximum page size. While printing, do not adjust the size.

# Calibrate camera using GUI
source install/setup.bash
ros2 launch phone_sensors_examples calibrate.launch.py

# Generate SSL certificates for local webserver 
#ros2 run phone_sensors generate_dev_certificates.sh $(ip route get 8.8.8.8 | grep -oP 'src \K[^ ]+')

# Extract the calibration so that launch files will know where to look
tar -xvf /tmp/calibrationdata.tar.gz --exclude=*.png --directory src/phone_sensors_examples/config/
# Server, image rectification, RVIZ
ros2 launch phone_sensors_examples rviz.launch.py

# Start only the node
#ros2 run phone_sensors server --ros-args -p camera_calibration_file:=src/phone_sensors_examples/config/ost.yaml

TODO

• Explain usage with images and how the browser side behaves (in particular permissions)
• Test with different browsers (I am looking for testers though !)
• Document server files
• Publish TimeReference
• Publish IMU
• Publish GPS (NavSatFix)
• If GeoLocation provides speed and heading, publish an Odometry message
• Calibrate camera in examples package
• Publish CameraInfo
• Publish video stream (Image)
• Add Serial or Bluetooth, as a possible way for the phone to send data over USB to a microcontroller
• Launch file example to set parameters
• robot_localization example with control feedback in place of speed odometry
• robot_localization example with visual inertial odometry
• Fix issue where the video is not sent when show_video_preview is False
• Fix frequency issue
• Add tests for message_converters.py
• Use /camera/image/compressed/jpeg_quality output topic according to image_transport
• Use SocketIO namespaces to separate video and other data