Semantic-Parsing-of-picture-descriptions

This project is part of the Softwareproject "Language, Action and Perception".

General research question: Can we implement a model that learns a natural language from scratch through interaction?

Focused research question: Can we teach a computer a mapping from natural language picture descriptions to a logical representation?

SHAPELURN

An Interactive Language Learning Game

Demo of the Prototype

To play the Prototype of our language learning game you only have to clone the repository and run the gui_simple.py script. You will be asked to enter a name for your session and the programm will then create a new folder with your chosen name in the source code directory. In this folder the result data from your game will be stored. (If your are testing the game for us in order to collect data for our evaluation of the system, this folder contains all the data we need from you.)

Requirements
Python 3
PIL (Pillow)
PySimpleGUI
Numpy

To install the required packages run pip install -r requirements.txt

Basic Instructions

Hello! Welcome to SHAPELURN, where you can teach the computer any language of your choice!
You will be looking at different pictures and describing them to the computer in one sentence.
Please use rather short sentences and try not to use negation and conjunction."

In order to evaluate our model we would like to collect your data.
Please enter any name under which you would like to save your data.

For the detailed instructions and examples please refer to the Wiki

References

The project idea is based on Wang, S. I., Liang, P., & Manning, C. D. (2016). Learning language games through interaction. arXiv preprint arXiv:1606.02447.
Our parsing and learning framework is based on Liang, Percy and Christopher Potts. 2014. Bringing machine learning and compositional semantics together. Annual Review of Linguistics 1(1): 355–376. and uses code from the corresponding demonstration code.

Files

• folder marked_pictures: needed for running grammar.py, guesses for the test utterances in semdata.py will be saved here

• folder pictures: example pictures as created by PictureLevel.py

• BlockPictureGenerator.py: automatically creates and saves the pictures documented

• CalculCoordinates.py: used to calculate the coordinates for the picture generation in BlockPictureGenerator.py documented

• cosimforstem.py: cosine similarity based heuristic for stemming words from an unknown language documented

• PictureLevel.py: generates a picture for a specified level by calling BlockPictureGenerator.py with corresponding parameters documented

• Semantic_Learner.py documented

• eval_helper.py: functions needed in grammar.py to evaluate truth of description documented

• grammar.py: defines the grammar, the evaluation of logical forms and the parser documented

• gui_simple.py: working GUI **documented

• back_and_forth.py: An iterator class, that can go back and forth through a list

• learning.py: the Stochastic Gradient Descent learn algorithm documented

• semdata.py: training and test sentences documented

• word.py and world2.png: example picture used for demo of grammar.py with test sentences from semdata.py