Skip to content

feat: added common embeddings providers components #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 37 commits into
base: main
Choose a base branch
from

Conversation

Rutik7066
Copy link
Contributor

@Rutik7066 Rutik7066 commented Jun 13, 2025

Currently, some providers do not support reranking, and some providers do not support image embedding generation

Provider Text Embedding Image Embedding Reranking
OpenAI
Cohere
Hugging Face
VoyageAI

/closes #16
/claim #16

@Rutik7066
Copy link
Contributor Author

OpenAI

OpenAI.Embed_compressed.mp4

@Rutik7066
Copy link
Contributor Author

HF

HF.Embed_compressed.mp4

@Rutik7066
Copy link
Contributor Author

Cohere

Cohere.Embed_compressed.mp4

@Rutik7066
Copy link
Contributor Author

VoyageAI

VoyageAI.Embed_compressed.mp4

@Rutik7066 Rutik7066 changed the title Rust WASM Components that Provide API to Common Embeddings Providers feat: added comman embeddings provider components Jun 16, 2025
@Rutik7066 Rutik7066 changed the title feat: added comman embeddings provider components feat: added common embeddings providers components Jun 16, 2025
@iambenkay
Copy link
Contributor

You might want to remove these videos and change them to a version that does not PUBLICLY expose your API Keys?
Just a suggestion anyways

@Rutik7066
Copy link
Contributor Author

Rutik7066 commented Jun 17, 2025

Thanks man!! I deleted keys after recording the videos.🙌

@iambenkay
Copy link
Contributor

Alright I was scared for you for a minute 😎

Copy link
Contributor

@iambenkay iambenkay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some helpful comments.

You also need to update this line

Makefile.toml#L156

and

Makefile.toml#L140

with entries like: embed_openai, embed_cohere, embed_voyageai

EDIT:
A better solution is to add a copy-release-artifacts and copy-debug-artifacts to the makefiles in the domains instead and call them from the parent Makefile.

I wrote it this way at a time when I was struggling with calling external makefiles in duckscript but that is no longer a challenge. You can edit it to reflect this behavior. That would help prevent the scope of a particular domain spilling into the general Makefile at the root

@Rutik7066
Copy link
Contributor Author

@vigoo Microsoft is discontinuing the Bing search api in August. Should I still add it? all others are implemented and tested

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create Rust WASM Components that Provide API to Common Embeddings Providers
2 participants