Signify road to production

[iFergal]

Moving from #330.

--

The Signify libraries suffer from a number of problems at present which I think will harm our adoption. For me it's quite important to fix these issues given Signify is currently the lowest effort way to start using the KERI suite. Additionally, as new Signify implementations pop-up, the last thing we want to is to inherit more issues and have more re-work to do.

I'm just going to throw down some ideas so we can get a discussion going on an ideal path for Signify.

Main problems:

1. Poor typing across the board
2. Inconsistent ways to sign and submit data to APIs
3. Possible need for more flexible client creation
4. Lack of SSE integration

1. Poor typing across the board

This simply needs work to fix, and maybe some code generation based on the KERIA API doc could help. There are far too many instances of any used in the codebase - fixing these might cause breaking changes for clients in case the interface has changed from before. (e.g. checking for optional fields using the ? operator, but the field has been fully removed)

It would also be good to have nice consistency in how types are formed and used. Some APIs accept X parameters and some APIs accept 1 object with X fields. (this would be breaking though)

2. Inconsistent ways to sign and submit data to APIs

An API call generally:

• Creates some data structure based on the arguments
• Signs it
• Submits it to the KERIA API

In some cases, this is one Signify call (identifiers().create()) and in others it's multiple (ipex().grant() && ipex().submitGrant()).

In some cases, there are multiple events created - such as ACDC, TEL event and interaction event to anchor the TEL seal.

The need to split
Given this is an edge library where distributed reliability is important, I think it's necessary that the creation of the data and submission of the data is separated so the edge application can store it for re-try, queue it, etc. Otherwise, the data needs to be re-retrievable from KERIA.

I've already had to split the identifier creation into 2 on my fork so I could store a group inception event locally before submitting the /multisig/icp messages to the other group members, in case of a network failure.

Interface formats
Might be my Java background but I would prefer to chain results together like ipex().grant().submit() rather than having a root ipex().submitGrant() function. I think this leads to a cleaner interface for not too much overhead. (result of .grant() needs to be submittable instead of a plain object)

Remote signing
There may be use cases where the Signify client or wallet is external to the application forming the data. For example, remote wallet integration into a credential issuance tool. Delegation could make things easier here but I feel like it will still arise as a need eventually.

Perhaps you could have .ugrant() APIs etc for unsigned grants that require signing. Not sure.

Breaking changes
Splitting the remaining APIs like this is a complete breaking change. It could be done as a phased migration by keeping the original API which will internally call the split API, and finally removing those once ready in one big breaking change.

Otherwise I think it needs to be done on a giant feature branch which isn't a good idea as it will go stale or cause conflicts.

3. Possible need for more flexible client creation

@lenkan has some ideas here but ultimately it would be good to have a backwards compatible and intuitive way to:

• Specify transport medium of client (HTTP vs TCP vs XYZ in the future)
• Specify API protection (Signed vs ESSR)
• Other things?

On the API protection, main currently uses RFC-XXXX signed headers with a few issues:

• Unsigned query params
• Unsigned bodies

My fork uses tunneled ESSR:

• HTTP request is converted to a HTTP string
  • Contains sender AID as header
  • Encrypted with receiver AID (crypto box seal/HPKE)
• A wrapping HTTP request is sent
  • Sender AID, receiver AID, timestamp, signature as header
  • Ciphertext of HTTP string is the body
  • Signature is over the body's digest, timestamp and set of AIDs

The motivation for tunneling the HTTP request is because Falcon can unwrap the request in its middleware and just pass it along like before, so it was low effort.

ESSR makes debugging via the network difficult, and there may be certain cases where the extra encryption is simply need necessary, so it would be good to be able to disable it.

Some options:

1. Signed headers and ESSR option
2. Signed-only tunneled HTTP request and ESSR option

For option 2, I basically mean doing exactly as we do for ESSR at the moment, except don't bother encrypting the body. The advantage of that is it fixes the existing signed header issues and is consistent and easier to manage in both Signify and KERIA middlewares.

I know there are existing applications out there that rely on signed headers for communicating with backends, so signed_fetch could stay if needed.

4. Lack of SSE integration

This has been raised before and is still pending. It would be nice to have event driven operations and notifications to avoid this continuous polling which harms scalability.