Instructions for Setting Up On The Robot

On the Jetson Orin:

• Install ollama from jetson containers by following the tutorial: https://www.jetson-ai-lab.com/tutorial_ollama.html
• Pull gemma3 model weights by running "ollama pull gemma3"

On the NUC:

• Clone the fbot_agent_msgs repository in the ros2 workspace: https://github.com/fbotathome/fbot_agent_msgs
• Clone this repository (fbot_agent) in the ros2 workspace.
• Run script "./install_dependencies.sh"
• Update "./config/fbot_agent_config.yaml" with waypoints in the target environment.
• Include the ./launch/fbot_agent.launch.py from this package into some fbot_behavior launch with required navigation subsystems. The realsense must be publishing messages on the "/camera/camera/color/image_raw" topic.
• Query the agent by calling the "/fbot_agent/execute_command" service.
• The service type is "fbot_agent_msgs.srv.AgentCommand"
• Example:

$ ros2 service call /fbot_agent/execute_command fbot_agent_msgs/srv/AgentCommand "command: 'navigate to the dining room'"

• The robot will navigate to the waypoint named dining room, and the service will return a response with the string field 'response' set with a response generated by the agent.

TODO

Some things need further investigation:

• The "fbot_agent_node.py" is not being updated with the llm model and camera topic parameters in "config/fbot_agent_config.py". These must be manually set in "fbot_agent_node.py".

Instructions for Implementing Skills

To implement a new skill, follow these steps:

• Implement a new function in ./fbot_agent/agents/agent.py. The function must have a docstring explaining what it does to the agent, as well as it's input and output data types, so that it can reason about how to compose the different skills into a python program, in response to the given commands.
• Add the @tool decorator to the implemented function.
• Example:

  fbot_agent/fbot_agent/agents/agent.py

  Line 22 in 062d30b

  @tool

• Register the function by adding it to the list returned in AgentNode.get_tools() method in AgentNode class in the file ./fbot_agent/agents/agent.py.
• Example:

  fbot_agent/fbot_agent/agents/agent.py

  Line 184 in 062d30b

  navigate_to_pose,

Currently Implemented Skills

The following skills have already been implemented:

• navigate_to_pose
• move_forward
• rotate

Skills That Still Need to be Implemented

The following skills must be implemented:

• follow_person
• pick_object
• place_object
• speak
• listen