Beta discussion

[iboB]

Items for beta. They are to be put into the roadmap and made into issues (if not already existing) and beta projects.

• Inference:
  • First and foremost: resolve Integration of llama.cpp and whisper.cpp #5 Loading inference libs as plugins #151 Integrating third party inference libs. Again #166
  • Schema generation: Redesign local inference API schemas and generation #149
  • API features: Potential API features #47
  • Llama:
    • leftover issues
    • Extend API llama ops ideas #50
    • Streaming
    • Improve chat
    • RAG
  • Whisper:
    • Improve API, Support of whisper params #58
    • Streaming
  • New inference engines:
    • Some TTS
    • Stable Diffusion
• Tools and demos
  • Acord:
    • WebSocket API
    • Assets and asset coherency (it is a daemon so #96 is not relevant)
    • Some app resembling copilot cli
    • Chat app
  • GUI API browser (ImGui)
  • Performance benchmarks
• Wrappers:
  • Swift: #153 #154
  • Java: some support for non-android gradle builds would be nice
  • NEW: Python
  • NEW: JS/TS (in order of priority)
    • React Native
    • Node modules
    • Browser
    • Low prio and may be abandoned: naked v8, JavaScriptCore... maybe only if using jsbind
• Other:
  • CUDA TEE integration
  • Mobile performance stats
  • Improve Android perf (Vulkan experiments)

Explicitly Out of scope

For other projects

server: server API, web app, #75

Not for Beta.

Will be discussed post beta:

• Deeper mobile support: #96 Reduce scope of support for mobile devices #139
• Training utilities:
  • Federated Learning
  • Reinforcement Learning from Human Feedback
  • Direct Preference Optimization

[pminev]

For Stable diffusion we might use this one.

[iboB]

Yep. This was my intention

[iboB]

For TTS, I'm eyeing tortoise.cpp but I haven't done thorough research for something better

[pminev]

I just found out that bark has .cpp version. We might add it in future.

[pminev]

I've found a model for 3D open-source model called TripoSR which seems interesting.