LLM-Controlled Robot Environment

This repository contains a project exploring the control of a simple simulated robot using Large Language Models (LLMs). The project is divided into several tasks, focusing on both unconstrained and constrained text generation for robot actions.

Project Overview

The core idea behind this project is to leverage LLMs to generate actions for a robot operating in a text-based interactive environment. The environment is designed with a single goal: to drop a knife in the living room. The robot interacts with the environment through textual commands like "go to living room" and "pick up knife".

Environment Details

The environment code is provided in run_agent.py and env.py. Here are the key rules and constraints governing the robot's actions:

Any pick up or drop action must be followed by a go to action. Any go to action must be followed by a pick up action. pick up x adds x to your inventory and removes it from the current room. drop x adds x to the current room and removes it from your inventory. You can only pick up items present in your current room and can only drop items you currently possess. You can manually play the game by running main.py.

Reading and Background

This project draws inspiration from recent research on controlling robots with LLMs, particularly the "SayCan" paper. Before diving into the code, it's recommended to read this paper. Understanding the language model component of SayCan (as opposed to the value function) is particularly relevant to this project.

https://say-can.github.io/assets/palm_saycan.pdf

Unconstrained Generation

The first programming task involves implementing the Agent class in agent.py. Specifically, implementing the generate_response() method. This method is responsible for generating a response (an action) from the LLM given a prompt from the environment, without any constraints on the generated output.

Constrained Generation

The environment provides a get_valid_actions() function in env.py, which returns a set of valid actions available to the agent at any given time. The second task is to implement a scoring-based approach similar to the one discussed in the SayCan paper.

Implemented the generate_constrained_response() function in agent.py. This function takes a prompt and a set of valid actions as input and must return one of the actions from the provided set.