Feature: Merging Small Indexes

[ashvardanian]

Describe what you are looking for

As a performance optimization, we can implement merge for existing indexes instead of moving (inserting) elements one by one.

Can you contribute to the implementation?

• I can contribute

Is your feature request specific to a certain interface?

It applies to everything

Contact Details

No response

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[rationalga]

Hi,
Has this feature been developed or not yet? While loading indexes that are bigger than RAM is great but its not useful if we can't create them in the first place i.e. during insertion index size can't outgrow the RAM (as far as I know). Thank you very much.

[mbautin]

It would also be great to allow filtering entries during this merge process based on a predicate (e.g. filtering out entries that were later marked as deleted).

[abetomo]

I would like this feature too.
Is it correct that a large index for merging is created in a writable mmap because it needs to be written when merging?

[kou]

How about #572?