Polishing up APIs for v2.0.0 stable

[kavitharaju]

Working towards getting out a stable release v2.0.0, we need to assess the current features and finalize them.
Based on internal discussions some areas of improvements have been identified and those changes at API endpoints level are listed in this document.

Some highlights of the proposed changes are:

• Change in terminologies
  • source and source_name to resource and resource_name
  • content to resource/type and resource_type
  • dictionaries to vocabularies
  • translation/projects to mt/token-based/projects
  • learn/gloss and learn/alignments to just /gloss and /alignments
• Major refactors
  • Introducing a new resource type called "parascripturals". This will provide a means to add all kinds of non-bible and related contents. Providing easy extensibility as new contents comes. This will remove infographics and bible video endpoints, db tables and all related components in code.
  • Removing the link between text bibles and audio bibles. This also will be reflected in endpoints, tables and code.
  • Introducing a new resource type for Sign Bible Videos

[kavitharaju]

Regarding the schema of parascripturals, this is what we have in mind right now. But certainly open to improvising it, as the idea is to be able to accommodate the contents like Bible project videos, inforgraphics and such new types that may come in the future. Basically a way to easily extend the scope to support more resource types.

{
   "type": "Bible project video"/"Bible Stories"/...,
   "title": ....,
   "description": ....,
   "content": ....,
   "link": ....,
   "reference": { book:..., chapter: ..., verseStart:..., verseEnd:...},
   "metadata": {...}
}

Only type and title fields are mandatory, rest can be used as per the needs of the usecase.

@RevantCI , sharing these details in response to the discussions we had.

[RevantCI]

Hi Kavitha if we could list the API's that are going to be removed it would be good.
And also the generic API if we could give a scope of the content that it will support,
So that I can list the use cases for the API's that are going to be removed so we can checked if the new API's cover them like videos, infographics etc.

[kavitharaju]

Have updated the document above with more details against each change indicating

• if it is a change in nomenclature
• if there are other APIs being replaced by it

The parascriptural resource type we are introducing is supposed to cover the existing infographics and bible project video usecase. Also we hope to keep it generic to be able to accommodate more usecases. One another we have thought about is Bible stories, but is not limited to it.

[kavitharaju]

For the parascriptural we are planning to let separate resources(sources) to be created for each new type and language and version. So

• hi_bible-stories_1.1.2023_parascriptual
• hi_infographic_3_parascriptual
• ml_bible-stories_1.1.2023_parascriptual
• hi_bible-project-videos_1.1.2023_parascriptual
• ml_bible-project-videos_1.1.2023_parascriptual
  could be some entries we an have in our resources each of these having rows in the above shared schema.

[RevantCI]

Would it be possible to add some other resources like ISL videos and bible reading plans in this API along with obs?
Sharing the links to the data used currently in case it help in the design of the db and request/response json

[kavitharaju]

For ISL videos, we are planning to introduce a new resource type.
Yes good to have Bible reading plans also supported by parascripturals. This is what I see in the link you shared

[ {date: <value>, 
   reading: [{ref: <value>, text: <value>}, ....]}
   ....
]

One way to represent it in parascriptural, would be as follows

{
   "type": "Bible Reading Plan",
   "title": "mcheyne",
   "description": "Day's theme or some sub title if available",
   "reference": { book:..., chapter: ..., verseStart:..., verseEnd:...},
   "link": ...., # link to the corresponding page in VO, if we can provide that
   "metadata": {date: "date-value"}
}

I this design, each reference will be one row, and all items of the day can be obtained by querying based on the date field in metadata. But this is just one design, that I am sharing to let know how this could be used. You can come up with better plans as per your usecase

[kavitharaju]

@joelthe1 @anjalyv
As we have decided to name the translation related API endpoints within vachan-api in the pattern

/v2/mt/token-based/projects
/v2/mt/token-based/project/tokens
/v2/mt/token-based/suggestions

How to go about naming the nllb APIs and stt APIs to keep this pattern consistant?

/v2/mt/nllb/translate
/v2/mt/nllb/models
/v2/stt/xlsr/transcribe
/v2/stt/mms/transcribe
/v2/stt/models ( /v2/stt/xlsr/models and /v2/stt/mms/models ?)

Please share your thoughts.

[anjalyv]

It looks fine.. But /v2/mt/nllb/models can be /v2/mt/models? Do we need different model apis for each tool?

[kavitharaju]

That depends on whether its going to be same app(micro service) or different.
In the case or /v2/mt there are endpoints starting like that in Vachan-api and also in nllb app. So routing the requests to correct service will be a problem.

Similarly for v2/stt/mms and v2/stt/xlsr, if all stt models can be hosted within one app(service), then using only /v2/stt as the common prefix will allow correct routing. But if they are going to be seperate apps, then we need longer prefix to identify them.

[anjalyv]

Decided to follow this pattern:

MT :

/v2/text/translate/nllb
/v2/text/translate/nllb/models

STT:

/v2/audio/transcribe/xlsr
/v2/audio/transcribe/mms

TTS:

/v2/audio/synthesis/mms
/v2/audio/synthesis/models

[shimilgithub]

Upgrading FASTAPI holds significant importance as VE currently running on an older version of FASTAPI(0.65.2). However, this upgradation presents a set of challenges that need to be addressed.

1. Latest version of FastAPI is not compatible with the graphql module, necessitating the library migration from graphene to the strawberry.
2. TestClient.delete() method no longer supports payloads in the latest FASTAPI release, thereby requiring the rewriting of all delete tests.
   For example.Now we have

      curl -X DELETE 'https://api.vachanengine.org/v2/dictionaries/{source_name}' \
      -H 'Authorization: Bearer <token>' \
      -H 'Content-Type: application/json' \
      -d '{
        "itemId": "your_word_id"
      }
   

And after upgradation,it will be changed to.

    curl -X DELETE 'https://api.vachanengine.org/v2/vocabularies/{source_name}?item_id={item_id}' \
    -H 'Authorization: Bearer <token>' \
    -H 'Content-Type: application/json'

[kavitharaju]

Even if we move the app to a stable version, as we are continuing to add more features and bring in major changes, I expect not all features might be in "stable" state. In that scenario, would it be worth adding beta versioning for specific API endpoints?

For instance, as per the plan we have now, the user registartion, login etc will have to moved to a separate micro-service soon. But for this release we are going ahead with what we have, by keeping them part of vachan-api app. But would be good to mark them as "beta" to indicate that changes should be expected. Similarly, for the basic gitlab-based streaming APIs we have included there, the USFM-Grammar based file processings that first needs USFM-Grammar to be stable, etc

Do you think its worth doing that? @joelthe1 or others?

[kavitharaju]

Invocation is same as it was before. This is just for adding a marker in the API documentation. It shows up like this in swagger, which is actually hard to even notice

[kavitharaju]

May be deprecating it is another option. It could be done for media streaming APIs. Here is how it looks like
https://git.door43.org/api/swagger#/repository/userTrackedTimes

[RevantCI]

Deprecated would normally refer to things that are old and are no longer going to be used in the next version, like the infographics api.
Tthe beta/experimental feature could have beta in the url itself maybe? /beta-2/feature. Saw a comment where someone tried it - OAI/OpenAPI-Specification#432 (comment)
Once the API is finalized a release can be made with the final API and once all apps are ported to new endpoint a second release can be done to remove /beta-2 endpoints
Just a suggestion else the deprecated option looks good if it seems cumbersome.

[joelthe1]

Thanks Revant for that link. I think the x-experimental: true could be a good enough way for us to do it for now.

[kavitharaju]

• Done Add x-experimental to a few endpoints #594

[kavitharaju]