🍞 Future Breakfast: Teaching Robot Arm SO-ARM101 to Make Breakfast

Never Make Breakfast Again - Let AI Robot Arm Do It For You!

🎬 Demo Video

🎯 Overview

Tired of making breakfast every morning? This project demonstrates how to teach a robot arm to automate your entire breakfast routine using custom 3D-printed tools and machine learning. The SO-ARM101 robot arm learns through imitation to:

• 🍞 Toast Handling: Pick toast from basket and insert into toaster
• 🍽️ Plating: Retrieve toasted bread and place on plate
• 🥄 Yogurt Feeding: Scoop yogurt and feed directly to you

Key Features

• 🎮 Teleoperation: Leader-follower configuration for intuitive demonstration recording
• 🔧 Custom End-Effectors: 3D-printed grippers and spoon holders designed for food handling
• 🧠 Imitation Learning: Training with ACT (Action Chunking with Transformers) policy
• 📊 Open Source: All models, datasets, and CAD files publicly available

🛠️ System Requirements

Hardware

• Robot: SO-ARM101 (Leader-Follower configuration)
  • SO-ARM101 AI Arm Motor Kit Pro
  • SO-ARM101 3D Printed Parts
• Cameras:
  • USB camera (overhead view)
  • Intel RealSense D435i (hand view with depth sensing)
• 3D Printed Components:
  • Custom gripper for toast handling
  • Spoon holder for yogurt feeding
  • Food-safe materials recommended

Software Environment

• OS: Ubuntu 24.0
• Kernel: Linux-6.14.0-27-generic-x86_64-with-glibc2.39
• Python: 3.10
• GPU: NVIDIA GeForce RTX 4070
• Framework: Hugging Face LeRobot

🚀 Quick Start

1. Environment Setup

# Create a Conda environment for LeRobot
conda create -y -n lerobot python=3.10
conda activate lerobot

# Install LeRobot
pip install lerobot

2. Robot Arm Calibration

Follower Arm (for execution)

python -m lerobot.calibrate \
  --robot.type=so101_follower \
  --robot.port=/dev/tty.usbserial_lerobot_follower \
  --robot.id=lerobot_follower_arm

Leader Arm (for teleoperation)

python -m lerobot.calibrate \
  --teleop.type=so101_leader \
  --teleop.port=/dev/tty.usbserial_lerobot_leader \
  --teleop.id=lerobot_leader_arm

3. Data Collection

Record demonstrations for each breakfast task:

# Toast insertion task
python breakfast_toast_insertion_record.py

# Toast retrieval and plating
python breakfast_toast_retrieval_plating_record.py

# Yogurt feeding
python breakfast_yogurt_feeding_record.py

• Episodes per task: 10+
• Episode duration: 20-30 seconds
• Reset time: 10 seconds between episodes

4. Model Training

Train separate models for each breakfast task:

# Toast insertion
python -m lerobot.scripts.train \
  --dataset.repo_id=CreatorKanata/breakfast-toast-insertion \
  --policy.type=act \
  --output_dir=outputs/train/act-breakfast-toast-insertion \
  --policy.device=cuda \
  --wandb.enable=true \
  --policy.repo_id=CreatorKanata/act-breakfast-toast-insertion

# Toast retrieval & plating
python -m lerobot.scripts.train \
  --dataset.repo_id=CreatorKanata/breakfast-toast-retrieval-plating \
  --policy.type=act \
  --output_dir=outputs/train/act-breakfast-toast-retrieval-plating \
  --policy.device=cuda \
  --wandb.enable=true \
  --policy.repo_id=CreatorKanata/act-breakfast-toast-retrieval-plating

# Yogurt feeding
python -m lerobot.scripts.train \
  --dataset.repo_id=CreatorKanata/breakfast-yogurt-feeding \
  --policy.type=act \
  --output_dir=outputs/train/act-breakfast-yogurt-feeding \
  --policy.device=cuda \
  --wandb.enable=true \
  --policy.repo_id=CreatorKanata/act-breakfast-yogurt-feeding

5. Run Breakfast Automation

Run each task independently with the trained models:

# Toast insertion
python breakfast_toast_insertion_control.py

# Toast retrieval and plating
python breakfast_toast_retrieval_plating_control.py

# Yogurt feeding
python breakfast_yogurt_feeding_control.py

🤗 Hugging Face Resources

Trained Models

• 🍞 Toast Insertion Model
• 🍽️ Toast Retrieval & Plating Model
• 🥄 Yogurt Feeding Model

Training Datasets

• 📊 Toast Insertion Dataset
• 📊 Toast Retrieval & Plating Dataset
• 📊 Yogurt Feeding Dataset

Training Experiments (WandB)

• 📈 Toast Insertion Training
• 📈 Toast Retrieval & Plating Training
• 📈 Yogurt Feeding Training

🔧 3D Printing & Custom Tools

End-Effectors

All custom end-effectors are designed for FDM 3D printing:

stls/
├── jaw-toast.stl            # Toast gripper jaw component
├── roll-toast.stl           # Toast handling roller
├── toast-dish.stl           # Dish for toast placement
└── toast-holder.stl         # Toast holder attachment

Printing Recommendations

• Material: Food-safe PLA or PETG
• Layer Height: 0.2mm
• Infill: 20-30%
• Support: Required for gripper components

📁 Project Structure

future-breakfast-so-arm101/
├── breakfast_toast_insertion_record.py      # Record toast insertion demonstrations
├── breakfast_toast_insertion_control.py     # Run toast insertion with trained model
├── breakfast_toast_retrieval_plating_record.py   # Record toast retrieval demonstrations
├── breakfast_toast_retrieval_plating_control.py  # Run toast retrieval with trained model
├── breakfast_yogurt_feeding_record.py       # Record yogurt feeding demonstrations
├── breakfast_yogurt_feeding_control.py      # Run yogurt feeding with trained model
├── breakfast_test.py                        # Test script for debugging
├── stls/                                    # 3D printable components
│   ├── jaw-toast.stl                       # Toast gripper jaw
│   ├── roll-toast.stl                      # Toast handling roller
│   ├── toast-dish.stl                      # Toast placement dish
│   └── toast-holder.stl                    # Toast holder attachment
├── images/                                  # Documentation images
│   ├── future-breakfast.jpg                # Main project image
│   ├── toast-insertion.jpg                 # Toast insertion demo
│   ├── toast-retrieval-plating.jpg         # Toast retrieval demo
│   ├── tongs.jpg                           # Gripper detail
│   └── wandb.png                           # Training visualization
├── videos/                                  # Demo videos
├── docs/                                    # Additional documentation
│   ├── youtube.md                          # YouTube video description
│   └── gum-README.md                       # Reference project
└── README.md                                # This file

🔧 Customization

Camera Configuration

Adjust camera settings in the scripts:

camera_config = {
    "usb_camera": {
        "index_or_path": 6,  # Adjust based on your system
    },
    "realsense": {
        "serial_number_or_name": "841612072123",  # Your RealSense serial
    }
}

Task Parameters

• NUM_EPISODES: Number of episodes to record/evaluate
• EPISODE_TIME_SEC: Duration of each episode
• RESET_TIME_SEC: Reset time between episodes
• FPS: Frame rate (default: 30)

🏆 Maker Faire Bay Area 2025

This project was created for the Maker Faire Two-Minute Film Festival with the theme "Future Makers." It showcases:

• Practical AI applications for everyday problems
• Integration of 3D printing, robotics, and machine learning
• Open-source collaboration and reproducible research

📚 References

• Hugging Face LeRobot Documentation
• SO-ARM101 Documentation
• Seeed Studio SO-ARM101 Wiki
• ACT: Action Chunking with Transformers

🤝 Contributing

Contributions are welcome! Feel free to:

• Submit issues or feature requests
• Share your breakfast automation improvements
• Add new end-effector designs
• Contribute to the datasets

📄 License

This project is open source and available under the MIT License.

---

Made with ☕ by a maker who really doesn't like making breakfast

GitHub • Hugging Face • WandB