Science Birds Basic Game Playing Software

Alpha v 0.6.2

Table of contents

1. Requirements
   1. Java Environment
   2. Python Environment
   3. The Science Birds Game
2. Run the Agent
3. Level Generation
4. Novel Levels Loading
5. Outline of the Provided Files
6. Outline of the Agent Code
7. Ground Truth Data Structure
8. Agent-Game Communication Protocols
9. Docker

Requirements

This system has been tested on

• Ubuntu 18.04
• MacOS 10.9 and 10.12
• Windows 10

Java Environment

The framework has been tested with the Java 12 and above.

Java Environment can be downloaded from this link.

Python Environment

• python version>=3.6
• numpy
• shapely
• open cv
• tensorflow
• install the dependencies

pip3 install -r requirements.txt

The Science Birds Game

Pre-compiled Science Birds for different platforms are in folder:

ScienceBirds

These files are tracked with Git Large File System.

Run the Agent

1. Run the game playing interface
   • Unzip the game that suits your OS in ./ScienceBirds folder
   • Navigate to the unzipped game folder.
   • use the code below to run:

java -jar game_playing_interface.jar Arguments:

• --headless: to run the software in headless mode (headless mode does not work on Windows now due to a potential bug in Unity)

• --trials-per-agent [number of trials per agent]: the number of trials per agent, default

• --trial-start-index [trial index]: the start trial index for the first agent, default 0

  • NOTE: this argument is only valid if --trials-per-agent is used and the its value is greater than zero
  • otherwise all agents will start at trial 0 and play through all trials

• --agent-name [agent name]: the agent to be run in this GPI

• --config-path [path to config.xml]: the path to the config.xml, default the same directory as game_playing_interface.jar

• --game-start-port [start port of SB]: the port of the first SB instance, it will increase by 1 afterwards. Default 9001

• --agent-port [port for the agent to connect]: the port that the game playing interface opens for agent connection. Default 2004

• --informed-only: only run agents on the trials trials that inform the novelty appearance

• --uninformed-only: only run agents on the trials trials that do not inform the novelty appearance

• --generate-config: generate a new config.xml for each agent in the current run. The configMeta.xml should be placed in

  • Linux

  ScienceBirds_Linux/ScienceBirds_Linux_Data/StreamingAssets/

  • MacOS

  ab.app/Contents/Resources/Data/StreamingAssets/

  • Windows

  ScienceBirds_Win/Science Birds_Data/StreamingAssets/

• --dev: run the game playing interface to access more information about the game objects

  • in dev mode, the type of the object in the ground truth is the true object type
  • in dev mode, agent command groundtruthwithscreenshot and groundtruthwithoutscreenshot will return non-noisy groundtruth
  • in dev mode, agent command ShootAndRecordGroundTruth will return non-noisy groundtruth
  • in dev mode, object current health value (currentLife) will be present. Unbreakable objects (slingshot and platform) will have max float value (3.402823E+38 in the ground truth json). In normal mode, currentLife will not be present. Ground and Trajectory do not have currentLife in any mode

• --noisy-batch-gt

  • if the argument is used, the batch ground truth command will always return noisy grount truths regardless of the dev mode

2. The Science Birds game does not need to be run separately, GPI will start it automatically

   • The Science Birds game used in this framework is a modified version of Lucas N. Ferreira's work that can be found in his Github repository

3. Run the Python naive agent python3 main.py

4. Run the Python deep Q-learning agent

   • The dq agent is placed in ./src/demo/ddqn_test_harness_agent.py
   • To run it: python3 ddqn_test_harness_agent.py

5. Eagle Wings

   • The eagle wings agent is located in the science birds repository
   • A readme file on running the agent is included in the agent's zip file

Generate Levels

1. Game levels are stored in the folder:

   • Linux

   ScienceBirds_Linux/ScienceBirds_Linux_Data/StreamingAssets/Levels/

   • MacOS

   ab.app/Contents/Resources/Data/StreamingAssets/Levels/

   • Windows

   ScienceBirds_Win/Science Birds_Data/StreamingAssets/Levels/

2. Game levels can be dynamically added or removed to the above folder during the running of the pipeline

3. New levels can be generated by running

python3 IratusAves.py

in folder levelgenerator

• the level generator is based on Matthew Stephenson's work.
• the latest changes on the level generator can be found in his repository

4. A brief description about the level generator is here
5. A detailed description of the level generator is here

Novel Levels Loading

1. In science birds game Levels folder as shwon in the previous section, there will be a set of folders named novelty_level_*
2. In each of these folders, there will be a set of folders named type*
3. Each set of novel levels are placed in one type* folder which contain two subfolders: [asset bundle] and [levels]
4. By default, there are 2600 levels pre-installed, where the first 200 levels are non-novelty levels used for generating the novelty levels
5. To change to levels installed: simply add/remove the type* folders. This operation can be done at the run time of the game

Outline of the Provided Files

This framework provided the following components:

1. The python naive agent

   • setup.py in root folder ./
     • the setup file for the python naive agent
   • requirements.txt in root folder ./
     • the dependencies for the python naive agent
   • all source code of the agent is in ./src and will be explained in detail in a separate section

2. The game playing interface

   • game_playing_interface.jar in root folder ./
     • the compiled game playing interface that handles commnication between the agent and the Science Birds game

3. The Science Birds game

   • three (Linux, MacOS and Windows) versions of the science birds game are provided in ./ScienceBirds

4. The level generator

   • located in ./levelgenerator
   • use the command:

   python3 IratusAves.py

   to generator levels

   • The levels will be generated into the same folder as the IratusAves.py
   • Currently the generated levels can be directly copied into the game level folders

Outline of the Agent Code

The ./src folder contains all the source code of the python naive agent. The agent is called naive agent because it ignores all other objects in the game but only aims and shoots at the pigs. Below is the outline of the code (this is a simple description. Detailed documentation in progress):

1. Files under ./src/demo/
   • naive_agent_groundtruth.py
     • A naive agent that interacts with the game by the pre-defined protocols
2. Files under ./src/client/
   • agent_client.py
     • it encodes the commands from the naive agent to send to the interface
     • it decodes the received interface messages into corresponding python object
   • server_client_config.json
     • the config file for the agent-interface connection
3. Files under ./src/trajectory_planner/
   • trajectory_planner.py
     • The implementation of the trajectory module
     • It calculates two possible trajectories given a directly reachable target point
     • It returns None if the target is non-reachable by the bird
4. Files under ./src/computer_vision
   • VisionMBR.py
     • image segmenting component that outputs a list of minimal bounding boxes (MBRs) to represent objects of a screenshot
   • VisionRealShape.py
     • image segmenting component that outputs a list of real shapes to represent objects a screenshot
   • ImageSegmentor.py
     • Implementation of region growing based image segmentation
   • GroundTruthReader.py
     • Parse the ground truth json object
     • Ground gruth data structure is detailed in this section
   • game_object.py
     • classes for representing game objects
   • cv_utils.py
     • utilities for vision
5. Files under ./src/utils
   • Point2D.py
     • A simple 2d point class used by the naive agent and the trajectory planner
6. ./main.py
   • main entry to run the agent

The Game Environment

1. The coordination system
   • in the python naive agent and the game playing interface, the game objects are represented in a Cartesian coordinate system with the origin point to be up-left corner. X coordinate increases along the right direction and Y coordinate increases along the downwards direction.
   • in the science birds game, the origin point (0,0) is the buttom-left corner and the Y coordinate increases along the upwards direction, otherwise the same as above.
   • this is to preserve compatibility with the existing game agents based on the Chrome game. We plan to change the coordinate system of ScienceBirds to match the default coordinates (origin at top left).

Ground Truth Data Structure

1. Ground truth data of game objects are stored in a Json object in GeoJSON inspired format. The differences between the used format and GeoJSON are:
   • The Polygon contours may not be closed, i.e., the last point in the contour may not be the same as the first one
   • The point coordinates are in integer format, but not long, lat format with decimals
   • The Ground object has an empty geometry element

The json object describs an array where each element describes a game object. Game object categories and their properties are described below: - Ground: the lowest unbreakable flat support surface - geometry: N/A - property: id = 'the unique id of the object' - property: type = 'Ground' - property: yindex = [the y coordinate of the ground line] - Platform: Unbreakable obstacles - geometry: polygon - property: id = 'the unique id of the object' - property: type = 'Object' or 'Platform' - property: currentLife = [remaining life point] (only appear in dev mode; max value for unbreakable object)
object] - property: colormap = [a list of compressed 8-bit (RRRGGGBB) colour and their percentage in the object] - Trajectory: the dots that represent the trajectories of the birds - geometry: MultiPoint - property: id = 'the unique id of the object' - property: type = 'Trajectory' - Slingshot: Unbreakable slingshot for shooting the bird - geometry: polygon - property: id = 'the unique id of the object' - property: type = 'Slingshot' - property: colormap = [a list of compressed 8-bit (RRRGGGBB) colour and their percentage in the object] - property: currentLife = [remaining life point] (only appear in dev mode; max value for unbreakable object) - Red Bird: - geometry: polygon - property: id = 'the unique id of the object' - property: type = 'Object'(in evaluation) or 'red_bird_[index]'(in dev mode) - property: colormap = [a list of compressed 8-bit (RRRGGGBB) colour and their percentage in the object] - property: currentLife = [remaining life point] (only appear in dev mode) - all objects below have the same representation as red bird - Blue Bird: - Yellow Bird: - White Bird: - Black Bird: - Small Pig: - Medium Pig: - Big Pig: - TNT: an explosive block - Wood Block: Breakable wooden blocks - Ice Block: Breakable ice blocks - Stone Block: Breakable stone blocks

2. ./src/computer_vision/GroundTruthReader.py is sample tool to read the parse ground truth json object.
3. Round objects are also represented as polygons with a list of vertices
4. Ground truth with noise
   • If noisy ground truth is requested, the noise will be applied to each point in vertices of the game objects except the ground, all birds and the slingshot
   • The noise for 'vertices' is applied to all vertices with the same amount within 2 pixels
   • The colour map has a noise of +/- 2%.
   • The colour is the colour map compresses 24 bit RGB colour into 8 bit
     • 3 bits for Red, 3 bits for Green and 2 bits for Blue
     • the percentage of the colour that accounts for the object is followed by colour
     • example: (127, 0.5) means 50% pixels in the objects are with colour 127
   • The noise is uniformly distributed
   • We will later offer more sophisticated and adjustable noise.
5. Example ground truth Json Object:

[
  {
    "type": "FeatureCollection",
    "features": [
      {
        "type": "Feature",
        "geometry": {},
        "properties": {
          "id": "-472",
          "label": "Ground",
          "yindex": 300,
          "colormap": []
        }
      },
	  {
		"type": "Feature",
		"geometry":{
			"type":"MultiPoint",
			"coordinates":[
				[195,188],[391,108],[591,143]
			]
		},
		"properties":{"id":"-18206","label":"Trajectory","colormap":[]}
	  },
      {
        "type": "Feature",
        "geometry": {
          "type": "Polygon",
          "coordinates": [
            [
              [207,216],[207,167],[224,167],[224,216]
            ]
          ]
        },
        "properties": {
          "id": "-608",
          "label": "Slingshot",
          "colormap": [],
          "currentLife": 3.402823e+38
        }
      },
	  {
        "type": "Feature",
        "geometry": {
          "type": "Polygon",
          "coordinates": [
            [
              [587,188],[587,187],[587,184],[583,184],[581,183]
            ]
          ]
        },
        "properties": {
          "id": "-716",
          "label": "pig_basic_medium_1",
          "colormap": [
            {"color": 0,"percent": 0.5},{"color": 4,"percent": 0.5}
          ],
          "currentLife": 2.5
        }
      }
    ]
  }
]

Communication Protocols

Message ID	Request	Format (byte[ ])	Return	Format (byte[ ])
1-10	Configuration Messages
1	Configure team ID Configure running mode	[1][ID][Mode] ID: 4 bytes Mode: 1 byte COMPETITION = 0 TRAINING = 1	Four bytes array. The first byte indicates the round; the second specifies the time limit in minutes; the third specifies the number of available levels	[round info][time limit][available levels] Note: in training mode, the return will be [0][0][0]. As the round info is not used in training, the time limit will be 600 hours, and the number of levels needs to be requested via message ID 15
2	Set simulation speed speed > 0 Note: this command can be sent at anytime during playing to change the simulation speed	[2][speed] speed: 4 bytes	OK/ERR	[1]/[0]
11-30	Query Messages
11	Do Screenshot	[11]	Width, height, image bytes Note: this command only returns screenshots without groundtruth	[width][height][image bytes] width, height: 4 bytes
12	Get game state	[12]	One byte indicates the ordinal of the state	[0]: UNKNOWN [1] : MAIN_MENU [2]: EPISODE_MENU [3]: LEVEL_SELECTION [4]: LOADING [5]: PLAYING [6]: WON [7]: LOST [8]: NEWTESTSET [9]: NEWTRAININGSET [10]: RESUMETRAINING [11]: NEWTRIAL [12]: REQUESTNOVELTYLIKELIHOOD [13]: EVALUATION_TERMINATED
14	Get the current level	[14]	four bytes array indicates the index of the current level	this command is deprecated and it will always return 1
15 (This request should not be used during the evaluation)	Get the number of levels	[15]	four bytes array indicates the number of available levels integer 0 will be returned if using this request during test harness	[number of level]
23 (This request should not be used during the evaluation)	Get my score	[23]	integer 0 will be returned if using this request during test harness A 4 bytes array indicating the number of levels followed by ([number_of_levels] * 4) bytes array with every four slots indicates a best score for the corresponding level	[number_of_levels][score_level_1]....[score_level_n] Note: This should be used carefully for the training mode, becaues there may be large amount of levels used in the training. Instead, when the agent is in winning state, use message ID 65 to get the score of a single level at winning state
31-50	In-Game Action Messages
31	Shoot using the Cartesian coordinates [Safe mode*]	[31][fx][fy][t1][t2] fx: the x coordinate of the release point coordinate of the bird on the sling shot fy: the y coordinate of the release point coordinate of the bird on the sling shot t1: the release time t2: the gap between the release time and the tap time. The tap time unit is in millisecond. The game world time elapse between two consective simulations is 20ms. If t1 is set to 0, the server will execute the shot immediately. The length of each parameter is 4 bytes	OK/ERR	[1]/[0]
32	Shoot using Polar coordinates [Safe mode*]	[32][theta][r][t1][t2] theta: release angle r: the radial coordinate The length of each parameter is 4 bytes	OK/ERR	[1]/[0]
33	Sequence of shots [Safe mode*]	[33][shots length][shot message ID][Params]...[shot message ID][Params] Maximum sequence length: 16 shots	An array with each slot indicates good/bad shot. The bad shots are those shots that are rejected by the server	For example, the server received 5 shots, and the third one was not executed due to some reason, then the server will return [1][1][0][1][1]
38	Shoot using the Cartesian coordinates [Safe mode*] with a batch of groundtruth as return value	[38][fx][fy][t1][t2][requested_gt_frequency][batch_gt_option] fx: the x coordinate of the release point coordinate of the bird on the sling shot fy: the y coordinate of the release point coordinate of the bird on the sling shot t1: the release time t2: the gap between the release time and the tap time. The tap time unit is in millisecond. The game world time elapse between two consective simulations is 20ms. requested_gt_frequency: indicates how frequent the ground truth is batched. This is independent to the game speed up factor. batch_gt_option: indicates the option of what portion of ground truths will be recorded. This parameter is optional with default value of 0 if not received from the agent [0]: Record all ground truths, maximum 200 ground truths will be recorded [1]: Only record the portion before the tap or the collision of the flying bird, whichever comes first If t1 is set to 0, the server will execute the shot immediately. The length of each parameter is 4 bytes	A batch of noisy groundtruth from the start of a shot to when the scene is stable again, 300 frames have been recorded or according to the specified condition The game will be paused when the scene is stable again or 200 frames have been recorded	[number_of_ground_truth_recorded][groundtruth 1 byte array length][groundtruth 1 bytes]...[groundtruth n byte array length][groundtruth n bytes] The number of groundtruth will be zero if the shot is not executed correctly
41	Shoot using the Cartesian coordinates [Fast mode**]	[41][fx][fy][t1][t2] The length of each parameter is 4 bytes	OK/ERR	[1]/[0]
42	Shoot using Polar coordinates [Fast mode**]	[42][theta][r][t1][t2] The length of each parameter is 4 bytes	OK/ERR	[1]/[0]
43	Sequence of shots [Fast mode**]	[43][shots length][shot message ID][Params]...[shot message ID][Params] Maximum sequence length: 16 shots	An array with each slot indicates good/bad shot. The bad shots are those shots that are rejected by the server	For example, the server received 5 shots, and the third one was not executed due to some reason, then the server will return [1][1][0][1][1]
34	Fully Zoom Out	[34]	OK/ERR	[1]/[0]
35	Fully Zoom In	[35]	OK/ERR	[1]/[0]
51-60	Level Selection Messages
51	Load a level	[51][Level] Level: 4 bytes	OK/ERR	[1]/[0]
52	Restart a level	[52]	OK/ERR	[1]/[0]
53	Load next available level	[53]	Loaded Level Index	[level index] level index: 4 bytes
61-70	Science Birds Specific Messages
61	Get Ground Truth With Screenshot	[61]	ground truth and corresponding screenshot	[groundtruth byte array length][ground truth bytes][image width][image height][image bytes] groundtruth byte array length: 4 bytes image width: 4 bytes image height: 4 bytes
62	Get Ground Truth Without Screenshot	[62]	ground truth	[groundtruth byte array length][ground truth bytes]
63	Get Noisy Ground Truth With Screenshot	[63]	noisy ground truth and corresponding screenshot	[groundtruth byte array length][ground truth bytes][image width][image height][image bytes]
64	Get Noisy Ground Truth Without Screenshot	[64]	noisy ground truth	[groundtruth byte array length][ground truth bytes]
65	Get Current Level Score	[65]	current score Note: this score can be requested at any time at Playing/Won/Lost state This is used for agents that take intermediate score seriously during training/reasoning To get the winning score, please make sure to execute this command when the game state is "WON"	[score] score: 4 bytes
66	Report Novelty Likelihood	[66][novelty_likelihood][non_novelty_likelihood][number_of_IDs][ID_1]...[ID_n][novelty_level][description_byte_array_length][novelty_description] [4 bytes][4 bytes][4 bytes][n*4 bytes][4 bytes][4 bytes][description_byte_array_length]	OK/ERR	[1][0]
68	Ready for New Set	[68]	a 19 bytes array indicating the time limit, interaction limit, number of levels, if in training or testing mode and if levels are sequenced or as a set, if the agent is allowed to query novelty information	[time_limit]:4 bbytes integer [interaction_limit] 4 byte integer [#levels] 4 bytes integer (integer 0 will be returned during evaluation) [attempts_per_level]4 bytes integer [mode] 1 byte, 0 -> training, 1 -> testing [if_as_a_set]:1 byte, 0 -> set, 1 -> sequence [allow_novelty_query]:1 byte, 0 -> not allowed, 1 -> allowed
69	Request Novelty Information	[69]	a 4 bytes interger indicating the novelty information	[novelty_info]:4 bbytes integer, -1 -> unknown, 0 -> novelty has not appeared, 1 -> novelty starts to appear
70	Batch Groundtruth	[70][requested_gt_frequency][number_of_frames_requested] number_of_frames_requested: indicates the number of ground truths to be recorded, with a maximal number of 300 frames. This parameter is optional with default value of 300 if not received from the agent	A batch of noisy groundtruth from the start of a shot to when 300 or the requested number of frames have been recorded The game will be paused again after the recording is finished	[number_of_ground_truth_recorded][groundtruth 1 byte array length][groundtruth 1 bytes]...[groundtruth n byte array length][groundtruth n bytes] The number of groundtruth will be zero if the ground truth is not available
71	Get Initial State ScreenShot	[71]	Width, height, image bytes Note: this command returns screenshot of the initial state of the game, but can be called at anytime during playing	[width][height][image bytes] width, height: 4 bytes
72	Get Novelty Hint	[72][hint_level] hint_level: integer that indicates the novel hint level 1-3. If hint_level<1 the ground truth dev mode will be turned off If hint_level>0 the ground truth dev mode will be turned on	novelty hint json object [{"hint_level":1,"novelty_type":$novelty_type}] novelty_type: {NOT_AVAILABLE, OBJECT, AGENT, ACTION, REACTION, INTERACTION, ENVIRONMENT, GOAL, EVENT} hint_level is always 1 in this version. More hint vocabularies will be added regarding hint level 2 and 3	[novelty hint byte array length][novelty hint bytes] novelty hint byte array length: 4 bytes
* Safe Mode: The server will wait until the state is static after making a shot.
** Fast mode: The server will send back a confirmation once a shot is made. The server will not do any check for the appearance of the won page.

Docker

A Docker image that uses the same setup in the evaluations is available.

1. To pull the docker image: docker pull aibirds/sbgame
2. To start the container: docker run -p 2004:2004 -p 2006:2006 aibirds/sbgame
3. The ENTRYPOINT is ["java", "-jar", "/sciencebirds/linux/game_playing_interface.jar", "--headless"] You may append any arguments as desired