feat: Add support for SparseEncoder and sparse embedding models in Sentence Transformers

[Ryzhtus]

Related Issues

SentenceTransformers introduced support for sparse embedding models via the SparseEncoder class in v5.0.0. I thought it would be cool to support these in Haystack as well, since sparse models were previously only available through the FastEmbed integration (e.g. FastembedSparseTextEmbedder)

Proposed Changes:

Introduced two new embedder classes and also a class to manage these embedding classes:

• SentenceTransformersSparseTextEmbedder
• SentenceTransformersSparseDocumentEmbedder
• SentenceTransformersSparseEncoderEmbeddingBackend

How did you test it?

I added unit tests for both embedders

Notes for the reviewer

Some tests are currently failing — I’d appreciate your support in resolving them.
And we’ll likely need to add documentation as well.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[anakin87]

Hello and thanks for this idea!

I think it's a big topic and will probably require some work.

Some high-level notes:

1. I would create a completely separate _SentenceTransformersSparseEmbeddingBackendFactory as we do for FastEmbed.
2. Let's try to fit the returned sparse embedding into the existing Haystack SparseEmbedding dataclass.
3. Let's add tests for the backend and integration tests for the two embedders.
4. If you share a script or a raw Colab notebook with an end-to-end example, this would help validating and reviewing the implementation.