[SVS] SVS API and functionality update #676

rfsaliev · 2025-05-20T11:03:49Z

Describe the changes in the pull request

This PR integrates SVS updates including following:

Logging system integration
Batch Iterator API changes
Scalar Quantization functionality
non-Linux and non-x86 platforms support: MacOS and ARM

Which issues this PR fixes

SVS Index is not supported on MacOS and ARM
SVS Index does not support quantization on non-Intel HW
SVS Index Batch Iterator executes search routines at construction time

Main objects this PR modified

SVS Index

Mark if applicable

This PR introduces API changes
This PR introduces serialization changes

codecov · 2025-05-20T11:20:47Z

Codecov Report

Attention: Patch coverage is 93.25843% with 6 lines in your changes missing coverage. Please review.

Project coverage is 96.64%. Comparing base (ca1137a) to head (a1467b5).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/VecSim/algorithms/svs/svs.h	87.50%	5 Missing ⚠️
src/VecSim/algorithms/svs/svs_batch_iterator.h	93.75%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #676      +/-   ##
==========================================
- Coverage   96.65%   96.64%   -0.02%     
==========================================
  Files         121      121              
  Lines        6756     6792      +36     
==========================================
+ Hits         6530     6564      +34     
- Misses        226      228       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rfsaliev · 2025-05-23T10:21:27Z

src/VecSim/algorithms/svs/svs.h

+        auto sink = std::make_shared<spdlog::sinks::callback_sink_mt>(callback);
+        auto logger = std::make_shared<spdlog::logger>("SVSIndex", sink);
+        // Sink all messages to VecSim
+        logger->set_level(spdlog::level::trace);


@alonre24, this line causes printing al lot of tracing/debug information in flow tests (unit tests use custom log callback to filter-out "debug" logs).
Do you have an idea how can we properly avoid tracing/debug information to be dumped in tests keeping ability for SVS to sink needed logs to VecSim/Redisearch?
I can set logger level to spdlog::level::info here but it will prevent possibility to trace SVS index.

Can we redirect the logs output to a log file in flow tests somehow?

I think, there are different techniques could be used to manage VecSim logs in flow tests:

Extend VecSim python bindings to allow changing VecSim log call back and modify flow tests. See pybind11 docs

Manage pytest capture mode

Modify default callback to print "debug" logs to stdlog + p.2

@meiravgri can we address this issue in a separate PR? I believe that options 2 / 3 are better (so we still have full logs for failing tests if needed)

I'm not sure I fully follow — if these logs are too verbose for flow tests, wouldn't they also be noisy in production, which we probably want to avoid as well? Or is there a difference in how logging is handled between test and production environments?

In flow tests, all logs at every level are printed, while in production, the default level is "notice".
@dor-forer, can you please handle the logs redirection in tests?

Do you mean implementing option 2/3 in a separate pr?

Copilot

Pull Request Overview

This pull request updates the SVS API and functionality to integrate logging, update the batch iterator API, add scalar quantization support, and enable MacOS/ARM support. Key changes include:

Integration of a logging system and new log callback functions.
Refactoring and API changes in the SVS index and batch iterator.
Updates to Python bindings, CMake configuration, and dependency revisions.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/unit/test_svs.cpp	Updated tests to adapt to new vector addition/deletion logic and quantization skip conditions.
tests/flow/test_svs.py	Replaced window_size loop variable with epsilon_rt in range query tests.
src/python_bindings/bindings.cpp	Added bindings for the new epsilon parameter in SVSRuntimeParams.
src/VecSim/vec_sim_common.h	Introduced the scalar quantization enum value.
src/VecSim/index_factories/svs_factory.cpp	Added support for scalar quantization in index and vector size calculations.
src/VecSim/algorithms/svs/svs_utils.h	Updated distance conversion functions from float to double.
src/VecSim/algorithms/svs/svs_extensions.h	Adjusted SVSStorageTraits specialization enable_if condition.
src/VecSim/algorithms/svs/svs_batch_iterator.h	Refactored batch iterator implementation with added done flag.
src/VecSim/algorithms/svs/svs.h	Integrated logger creation and adjusted query result construction.
cmake/svs.cmake	Modified SVS LVQ support check and updated the pre-compiled SVS URL.
deps/ScalableVectorSearch	Updated subproject commit hash.

dor-forer · 2025-06-09T13:59:57Z

@rfsaliev Can we merge this?

rfsaliev · 2025-06-09T14:23:05Z

@rfsaliev Can we merge this?

Yes, I think so.

dor-forer · 2025-06-10T06:21:37Z

@rfsaliev I think there are some conflicts after merging #684

rfsaliev · 2025-06-10T15:36:59Z

@rfsaliev I think there are some conflicts after merging #684

@dor-forer, I've rebased this PR on the last main

rfsaliev · 2025-06-11T09:13:17Z

@dor-forer, your approval is dismissed by rebase.
Can you please review again?
Thank you.

rfsaliev requested review from meiravgri and Copilot May 20, 2025 11:03

This comment was marked as outdated.

Sign in to view

rfsaliev commented May 23, 2025

View reviewed changes

rfsaliev requested a review from alonre24 June 2, 2025 15:38

rfsaliev force-pushed the rfsaliev/svs-new-api-v008-dev branch from 917dfcf to 19629e1 Compare June 3, 2025 17:36

rfsaliev mentioned this pull request Jun 4, 2025

[SVS] Add multi-vector support to SVS Non-Tiered Index [MOD-10186] #690

Open

2 tasks

alonre24 requested review from Copilot and dor-forer and removed request for meiravgri June 5, 2025 09:29

Copilot AI reviewed Jun 5, 2025

View reviewed changes

dor-forer previously approved these changes Jun 5, 2025

View reviewed changes

rfsaliev mentioned this pull request Jun 6, 2025

[SVS] Add SVS LeanVec compression support #684

Merged

2 tasks

rfsaliev added 13 commits June 10, 2025 11:29

[SVS] Integrate VecSim and SVS logging systems

d54ef7a

[SVS] Changed SVS batch iterator API

09462d7

Refactor SVS_BatchIterator and remove max_batch_size

515b6df

Update SVS submodule

d77d12c

Enable SVS for non-x86 platforms

6e7fa2d

Disable SVS for cmake >= v.4 due to robin-map incompatibility

5213198

Update SVS submodule and enable SVS for cmake v4

6bd584b

cleanup and fix allocation and construct_at issues

2cfca25

Fix aggregare initilialization issue (C++ P0960R3)

14fe420

Workaround C++ LWG 3436 issue

4b85bce

First draft of SVS scalar quantization support

42de8a9

Allow Scalar quantization for non-LVQ case

8d122d5

Add scalar quantization query test

8d4f66d

rfsaliev added 9 commits June 10, 2025 11:50

Filter debug log in svs tests and format

5b1244c

Update SVS shared library package

d7395f3

Fix MacOS build

228b96a

Update SVS with eve compilation fix

c49b714

Workaround SVS batch interator completion issue

ad1a500

Fix/update SVS flow test

7c9b6b8

fixup! Workaround SVS batch interator completion issue

01f3613

Improve code coverage

88b4f9e

Fix isSVSQuantBitsSupported() should return SQ as fallback quantization

5997492

rfsaliev dismissed dor-forer’s stale review via 5997492 June 10, 2025 14:02

rfsaliev force-pushed the rfsaliev/svs-new-api-v008-dev branch from 19629e1 to 5997492 Compare June 10, 2025 14:02

Post-rebase fix

2d65413

rfsaliev requested a review from dor-forer June 11, 2025 08:25

Improve SQ tests coverage

a1467b5

dor-forer approved these changes Jun 11, 2025

View reviewed changes

alonre24 enabled auto-merge June 11, 2025 12:26

alonre24 added this pull request to the merge queue Jun 11, 2025

Merged via the queue into main with commit 6963d58 Jun 11, 2025
16 checks passed

alonre24 deleted the rfsaliev/svs-new-api-v008-dev branch June 11, 2025 14:08

[SVS] SVS API and functionality update #676

[SVS] SVS API and functionality update #676

Uh oh!

Conversation

rfsaliev commented May 20, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

codecov bot commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

dor-forer commented Jun 9, 2025

Uh oh!

rfsaliev commented Jun 9, 2025

Uh oh!

dor-forer commented Jun 10, 2025

Uh oh!

rfsaliev commented Jun 10, 2025

Uh oh!

rfsaliev commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

codecov bot commented May 20, 2025 •

edited

Loading