Neuro API v2 Proposal

[Alexejhero]

These are my proposed changes for Neuro API v2.

Initial connection parameters message

Instead of having to send game in every single websocket message, it would be better idea to transmit it once when the websocket connection is established, and have the Neuro backend remember it. This would help if we want to add more parameters like this in the future.

This would take the place of the startup message and deprecate it.

There are two proposed ways to implement this:

Query parameters

Games would establish a connection to ws://localhost:port?game=My%20Game

Pros: Initial data is known as soon as the connection is established, so there is no period of time when the backend is waiting for it

Hello message

Websocket message sent upon connection to transmit all of the data

{
    "command": "hello", // or something else
    "data": {
        "game": "My Game"
    }
}

Pros: Easy to add new fields to this without overloading the query parameters

API versioning

Tracked by #6

In the initial message (specified above) we can include a new version field which will have a numerical integer value increasing. If no version is specified, assume api 1 (current version as of subathon 2). If a version is specified, handle all messages received by the backend with that backend version. This ensures backwards compatibility even after breaking changes.

Allow binary messages

It shouldn't be too difficult to allow binary ws messages as well as plaintext. (Remember Buckshot Roulette issue - this should help others not run into it as well)

Implement the Inscryption thing

Putting this here so no one forgets about it

Proper documentation for action/force and action/result

At the moment it's unclear what can and can't happen after an action/force is sent. It's written that Neuro doesn't support multiple action/force messages at once, but it's not documented what happens if that happens. (great english)

Additionally, while Neuro is waiting for action/result, it's known that you can send actions/unregister in the meantime, but nothing is said about other types of messages like context.

Having better documentation on this would be very valuable so we know how the backend works.

Stop retrying failed action forces

Tracked by #14

Let the client retry failed action forces - this allows them to send updated context, and also fixes a race condition deadlock. #14 (comment)

Mute/unmute API

Tracked by #48

Bring mute/unmute API to the generic game api.

{
    "command": "set_mute", // or something similar, or maybe have separate messages for mute and unmute (#44)
    "data": {
        "value": boolean
    }
}

Graceful shutdown API

Documentation is in here https://github.com/VedalAI/neuro-game-sdk/blob/main/API/PROPOSALS.md

#44 argues there should be separate messages for requesting a shutdown an cancelling it, instead of a bool param.

Filtering

Tracked by #49

We might want to have a way for the neuro backend to reject an actions/register or an actions/force and return that result to the client (for filtering). Or maybe we don't say when something was filtered and you give neuro an empty string instead? Doesn't seem like the best idea though.

Action overriding

When an action is received, if an action with the same name already exists, instead of discarding the new action, override the old one with the new one.

Websocket disconnection handling

When the websocket disconnects, the backend should automatically unregister all actions associated with it. Upon reconnection, the client will be responsible for reregistering all actions automatically. (So we don't have the actions/reregister_all thing anymore)

[CoolCat467]

I would like to suggest that there be a way to register temporary single-run actions with Neuro that the backend will only be able to run once and immediately be automatically unregistered. Although it is already possible to do this by unregistering the action before you send the action result back, this will eliminate race conditions if Neuro is attempting to run commands back to back.

[Alexejhero]

If you send actions/unregister before sending actions/result it achieves the same purpose.

[Vedal987]

Initial connection parameters message

Instead of having to send game in every single websocket message, it would be better idea to transmit it once when the websocket connection is established, and have the Neuro backend remember it. This would help if we want to add more parameters like this in the future.

I don't think the backend currently keeps track of individual connections. For some reason I think it's mostly stateless. It's easier to keep this currently, and keeps "support" for having multiple games connected at once - even through the same websocket connection I guess if you wanted that.

API versioning

In the initial message (specified above) we can include a new version field which will have a numerical integer value increasing. If no version is specified, assume api 1 (current version as of subathon 2). If a version is specified, handle all messages received by the backend with that backend version. This ensures backwards compatibility even after breaking changes.

API versioning seems like a good idea, though if we aren't doing a startup message it would have to go in every message like the game parameter currently - which I admit seems a bit unnecessary.

Allow binary messages

It shouldn't be too difficult to allow binary ws messages as well as plaintext. (Remember Buckshot Roulette issue - this should help others not run into it as well)

This was a limitation with the websocket server library I'm using but I can work around it.

Proper documentation for action/force and action/result

At the moment it's unclear what can and can't happen after an action/force is sent. It's written that Neuro doesn't support multiple action/force messages at once, but it's not documented what happens if that happens. (great english)

I really don't think you should do this but Neuro should try and handle both action/forces at once but this is untested. You generally wouldn't want to do this anyway since Neuro can't think about two things at once. Instead she would independently and simultaneously process both requests at once and the two states should be merged back afterwards.

Additionally, while Neuro is waiting for action/result, it's known that you can send actions/unregister in the meantime, but nothing is said about other types of messages like context.

You can send additional context messages while waiting for action/result but as with before she can only process one thing at once. This means it won't affect her action decision and instead the context will be merged after she is finished making her decision.

Stop retrying failed action forces

Tracked by #14

Let the client retry failed action forces - this allows them to send updated context, and also fixes a race condition deadlock. #14 (comment)

I think some of my implementations rely on this functionality for now but there should definitely be a better system here since game state can change in between attempts. Will take a look at this.

Mute/unmute API

Tracked by #48

Bring mute/unmute API to the generic game api.

Slightly hesitant on this since I feel like Neuro should be the one controlling this and not the game. This might be a good solution though until Neuro is smart enough to handle this herself.

Graceful shutdown API

Documentation is in here https://github.com/VedalAI/neuro-game-sdk/blob/main/API/PROPOSALS.md

This looks good to me. Most games should support Neuro starting/stopping the games by themselves. This is one of the main use cases for this SDK. Games like Slay the Spire that may want to quit once the next run is done can also take up to an hour for a run to finish which is interesting.

Filtering

Tracked by #49

We might want to have a way for the neuro backend to reject an actions/register or an actions/force and return that result to the client (for filtering). Or maybe we don't say when something was filtered and you give neuro an empty string instead? Doesn't seem like the best idea though.

It would be really hard for game clients to manage those rejections so I think the Neuro client should just handle it. Worst case I can attempt to rewrite/censor stuff I need to for Neuro automatically. Either way developers shouldn't worry about it.

Action overriding

When an action is received, if an action with the same name already exists, instead of discarding the new action, override the old one with the new one.

I agree. Sounds good to me.

Websocket disconnection handling

When the websocket disconnects, the backend should automatically unregister all actions associated with it. Upon reconnection, the client will be responsible for reregistering all actions automatically. (So we don't have the actions/reregister_all thing anymore)

I think this sounds like a good idea. I am slightly conscious of accidental disconnections due to network issues or sockets being connected for too long or something.

[Alexejhero]

Also it would be cool for the API to say who is playing the game, whether its neuro or evil

[KTrain5169]

Would it be possible to also add a way to check which packet is meant for which game? I can see a situation where Neuro receives an action force right after she sent an action meant for another game, this way the game sending an action force won't receive the invalid packet and accidentally send back an action result that should've been from another game.

I think if Neuro supports multi-connects this should be a thing so that she doesn't receive multiple action results per command, unless Neuro already ignores the action result from the unrelated game(s).

[Govorunb]

As far as I know, each game has its own websocket connection to the backend, so disambiguation is easy from the game's/SDK's perspective - the backend picks which websocket to send the action to and no other games hear the message.

[KTrain5169]

True, I just wanted to make sure because Randy sends to all connected WebSocket clients so I wasn't sure if Neuro had this ability as well.

[Govorunb]

Since this discussion got revived, I've been mulling over my old suggestion for routing and I think there's a better solution here that can wrap up versioning in a slightly nicer way as well.

TL;DR

• v1 remains on / (e.g. ws://localhost:8000) as legacy
• v2 and onwards connect to /v{version}/{game} (e.g. ws://localhost:8000/v2/Lorem%20Ipsum)
• A hello message is also nice for less important data

Yapsplanation

In my mind, the API version and game name are absolute requirements. After implementing a personal backend for testing, I can tell you that accepting connections without knowing those just makes me uncomfortable on the server's behalf.

This is a contrived example - I don't think anyone would do this on purpose - but right now a client can make a connection and then just sit there forever, sending keepalives to remain connected but never actually sending the init data. The backend has no choice but to accept the connection and then get left in this pre-init purgatory.
My argument isn't just this specific case - it's more about the 'gap in the state space' that's currently left unaccounted for. I feel like you'd have to work out/workaround quite a few weird edge cases before you could feel confident in the security/reliability of this part of an implementation. Knowing/validating this stuff upfront instead would automatically remove the need to answer those specific "what happens if this happens" questions because it can't happen anymore.

As for the specific URL shape - /vXYZ/ is standard for APIs, so there's not much to explain there. I like having the game be part of the path, but query (or, worst case, request headers) would work too I guess.

---

I also like the idea of a hello message - it doesn't have to be an exclusive "either/or" between it and URL params. It would be useful to negotiate other supplementary capabilities, like "this game benefits from vision" or "Evil is playing" or other data of similar significance. My only concern is that critical data should always go in the initial request.

[KTrain5169]

I pretty much agree with that, the one thing I think I would add is there could be a recommendations packet to specify recommendations throughout the game (e.g. "This part of the game, until otherwise stated, is enhanced with vision" and "You no longer have an enhanced experience with vision unless otherwise stated, please stop using it") could work as well. You can technically already do this with context packets but having a separate packet for it would likely save Neuro from confusion which can happen depending on the game's context.