🤖 blocklyGPT

This repository contains the prototype implementation of the project described in the paper.

Gargioni, Luigi and Fogli, Daniela. "Integrating ChatGPT with Blockly for End-User Development of Robot Tasks"
Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, pages 478--482, 2024.
Publisher: ACM New York, NY.

Citation

If you use this project in your research, please cite the following paper:

@inproceedings{gargioni2024integrating,
  title={Integrating ChatGPT with Blockly for End-User Development of Robot Tasks},
  author={Gargioni, Luigi and Fogli, Daniela},
  booktitle={Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction},
  pages={478--482},
  year={2024}
}

Integrating ChatGPT with Blockly for End-User Development of Robot Tasks

This paper presents an End-User Development environment for collaborative robot programming, which integrates Open AI ChatGPT with Google Blockly. Within this environment, a user, who is neither expert in robotics nor in computer programming, can define the items characterizing the application domain (e.g., objects, actions, and locations) and define pick-and-place tasks involving these items. Task definition can be achieved with a combination of natural language and block-based interaction, which exploits the computational capabilities of ChatGPT and the graphical interaction features offered by Blockly, to check the correctness of generated robot programs and modify them through direct manipulation.

---

⚙️ BackEnd

• Default Port: 8000

---

🎯 Requirements

• Python 3.11.x
• Poetry (pip installation is not the official one, but the easiest)

---

🌟 Installing from scratch

poetry install

---

🔧 Start server

poetry run start

---

🔄 Update dependencies

Update with versions from pyproject.toml:

poetry update

Update pyproject.toml with latest versions retrieved from internet:

poetry run poetryup

---

💫 FrontEnd

• Folder: src

---

📚 Design libraries

• React 18.x.x
• Parcel
• Ant Design

---

📂 Install dependencies

npm install

---

🔀 Update dependencies in package.json

• Visual Studio Code Exstension: Versions Lens

---

🔎 Start debug server

npm start

🔑 Credentials

• Username: operator1
  Password: Operator_1!
  Type: Operator

• Username: manager1
  Password: passwordmanager1
  Type: Manager

For Django admin panel "127.0.0.1:8000/admin/":

• Username: admin
  Password: adminpassword
  Type: Administrator/Manager

Gazebo

ign gazebo -v 4 worldCobotta.sdf

🚀 Gazebo Workflow

:play_arrow: Startup Sequence

To start the complete Gazebo simulation environment, run the following batch files in order:

1. cobotta_node.bat - Initialize Cobotta robot node
2. flask_node.bat - Start Flask backend service
3. gazebo_bridge.bat - Launch Gazebo communication bridge
4. gazebo.bat - Start Gazebo simulation environment
5. npm start.bat - Launch frontend development server
6. poetry run start.bat - Start Python backend server