fix fit memory test #461

meiravgri · 2024-05-07T10:04:22Z

Describe the changes in the pull request

Fix memoryfit test
Which issues this PR fixes

#...
MOD...

Main objects this PR modified

...
...

Mark if applicable

This PR introduces API changes
This PR introduces serialization changes

codecov · 2024-05-07T10:31:09Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.10%. Comparing base (c521d82) to head (c38cfcc).

Additional details and impacted files

@@                    Coverage Diff                    @@
##           meiravg_feature_bfloat16     #461   +/-   ##
=========================================================
  Coverage                     97.10%   97.10%           
=========================================================
  Files                            86       86           
  Lines                          4633     4633           
=========================================================
  Hits                           4499     4499           
  Misses                          134      134

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

alonre24 · 2024-05-07T10:44:08Z

tests/unit/test_bruteforce.cpp

+    // Fit memory to initial capacity shouldn't have any affect since the ctor initializes label2id
+    // size to the initial capacity.
+    size_t initial_memory = index->getAllocationSize();


So for the current state comment is not accurate right? "the ctor initializes label2id size to the initial capacity"

* fix fit memory test * remove xenial checks * remove irrelevant comment * free index in seralization test

* remove bw_vl arch (not needed, avx512 is enough) * remove xenial from merge queue flow * fix fit memory test (#461) * fix fit memory test * remove xenial checks * remove irrelevant comment * free index in seralization test * install gcc10 bionic * install newer vg ubuntu18 * exit valgrind directory

* upgrade codcov instance to c7i * [MOD-6763] bfloat16 - distance calculation functions (#445) * update arch_opt macros in bf16 BM * fix macro check * Remove arch_opt enum and use cpu features flags instead (#459) * FP16 distance functions (#453) * [MOD-6764] Introduce bfloat16 datatype (#454) * Introduce bfloat16 datatype add Datatype to `VecSimIndexAbstract` template args. add template to SetDistFunc and have specialized implemntation for each <dataType, distType> combintation now the dist function pointer argument is not neseccesary, we can simple return the function. move using dist_func_t to spaces_aux.h (should be in a file included by both L2_spaces.h/IP.spaces.h and spaces.h) * move getdistance imp to spaces.cpp * move normalize function from vec_utils to spaces: spaces.h: get normalize function. spaces.cpp: specialized templates according to the datatype for the normalization functions spaces/normalize/normalize_naive.h: templated naive implementations of normalization functions VecSimIndexAbstract must have both DataType and DistType add VecSimType_BFLOAT16 update vec_utile accordingly bfloat16.h: add bfloat16_to_float32 for little and big TieredFactory::NewIndex calls TieredHNSWFactory::TieredFactory::NewIndex instead of implmentation duplication add bfloat to index factory for NewIndex and EstimateInitialSize * add no_opt functions tests for bfloat16 also add normalization function test for bf16 * fixed nohrmalized value * add bf16 benchmarks: add serialized hnsw bf16 files hnswfactory: add ctor from file for bf16 add fitMemory and run it at the begining of each benchmark to tighten memory. * update bm_files script * add flow tests to bf16 bruteforce: L2 batch iterator range query multivalue hnsw: serialization L2 batch iterator range multi tiered: create search and insert * add bfloat16 module to toml * upgrade scipy nupy * also use python 3.10 * upgrade PR python to 3.10 * update serializer * bm: fix casts unit tests: add test_bf16 * more unit tests * tiered tests * add index_type assertion ti hnsw serialization test add desgnated classes for bf and flat in bf16 unit test * unit test final * repalce cast to dynamic casts * add get neighors test to tiered add normalize test for bfloat16 in testcommon * use bit_cast * upgrade codecov machine * Revert "upgrade codecov machine" This reverts commit 7619b05. * Revert "use bit_cast" This reverts commit 59b8fab. * fix alignment intialization by changing the order * make bfloat16_to_float32 templated accroding to system's endianess * eliminate default in GetDistFunc add test for invalid metric * add test fot fit memory * split fit memory tests * run merge queue flow * add override to fit memory * upload valgrind artifacts on failure * _mm256_loadu_epi16 is supported as of gcc1, use _mm256_lddqu_si256 instead * fix fit memory test (#461) * fix fit memory test * remove xenial checks * remove irrelevant comment * free index in seralization test * Remove avx512_BW_VL (#460) * remove bw_vl arch (not needed, avx512 is enough) * remove xenial from merge queue flow * fix fit memory test (#461) * fix fit memory test * remove xenial checks * remove irrelevant comment * free index in seralization test * install gcc10 bionic * install newer vg ubuntu18 * exit valgrind directory * [MOD-6775] introduce fp16 type! (#462) * remove bw_vl arch (not needed, avx512 is enough) * remove xenial from merge queue flow * introduce fp16 type! change bfloat and fp16 to struct typedef to avoid ambiguty between them * added flow tests * fp16 benchmarks * fix hnsw flow test * add float616 to serializing script add fp16 files to bm files decrease testSizeEstimation dim size to make it run fasetr in debug * set log call back to null in benchmarks * remove struct typedef add default ctor * add comma seriaizer file * disable temp flow * disabe for real * remove constexpr from bfloat default ctor declaring a default ctor as constexpr is allowed as of c++20, and this feature in particular is supported as of gcc10 I don't see any benefit using this feature... * test focal * disable temp flow --------- Co-authored-by: alonre24 <alonreshef24@gmail.com>

* fix fit memory test * remove xenial checks * remove irrelevant comment * free index in seralization test (cherry picked from commit a0b70c8)

* remove bw_vl arch (not needed, avx512 is enough) * remove xenial from merge queue flow * fix fit memory test (#461) * fix fit memory test * remove xenial checks * remove irrelevant comment * free index in seralization test * install gcc10 bionic * install newer vg ubuntu18 * exit valgrind directory (cherry picked from commit d2e1d4f)

* upgrade codcov instance to c7i (cherry picked from commit b5cc93b) * [MOD-6763] bfloat16 - distance calculation functions (#445) (cherry picked from commit 850d6a8) * update arch_opt macros in bf16 BM (cherry picked from commit 253dcd3) * fix macro check (cherry picked from commit 26b047d) * Remove arch_opt enum and use cpu features flags instead (#459) (cherry picked from commit f98793a) * FP16 distance functions (#453) (cherry picked from commit 9420e77) * [MOD-6764] Introduce bfloat16 datatype (#454) * Introduce bfloat16 datatype add Datatype to `VecSimIndexAbstract` template args. add template to SetDistFunc and have specialized implemntation for each <dataType, distType> combintation now the dist function pointer argument is not neseccesary, we can simple return the function. move using dist_func_t to spaces_aux.h (should be in a file included by both L2_spaces.h/IP.spaces.h and spaces.h) * move getdistance imp to spaces.cpp * move normalize function from vec_utils to spaces: spaces.h: get normalize function. spaces.cpp: specialized templates according to the datatype for the normalization functions spaces/normalize/normalize_naive.h: templated naive implementations of normalization functions VecSimIndexAbstract must have both DataType and DistType add VecSimType_BFLOAT16 update vec_utile accordingly bfloat16.h: add bfloat16_to_float32 for little and big TieredFactory::NewIndex calls TieredHNSWFactory::TieredFactory::NewIndex instead of implmentation duplication add bfloat to index factory for NewIndex and EstimateInitialSize * add no_opt functions tests for bfloat16 also add normalization function test for bf16 * fixed nohrmalized value * add bf16 benchmarks: add serialized hnsw bf16 files hnswfactory: add ctor from file for bf16 add fitMemory and run it at the begining of each benchmark to tighten memory. * update bm_files script * add flow tests to bf16 bruteforce: L2 batch iterator range query multivalue hnsw: serialization L2 batch iterator range multi tiered: create search and insert * add bfloat16 module to toml * upgrade scipy nupy * also use python 3.10 * upgrade PR python to 3.10 * update serializer * bm: fix casts unit tests: add test_bf16 * more unit tests * tiered tests * add index_type assertion ti hnsw serialization test add desgnated classes for bf and flat in bf16 unit test * unit test final * repalce cast to dynamic casts * add get neighors test to tiered add normalize test for bfloat16 in testcommon * use bit_cast * upgrade codecov machine * Revert "upgrade codecov machine" This reverts commit 7619b05. * Revert "use bit_cast" This reverts commit 59b8fab. * fix alignment intialization by changing the order * make bfloat16_to_float32 templated accroding to system's endianess * eliminate default in GetDistFunc add test for invalid metric * add test fot fit memory * split fit memory tests (cherry picked from commit e018d91) * run merge queue flow (cherry picked from commit 4670c59) * add override to fit memory (cherry picked from commit f994c79) * upload valgrind artifacts on failure (cherry picked from commit 7ece68d) * _mm256_loadu_epi16 is supported as of gcc1, use _mm256_lddqu_si256 instead (cherry picked from commit c521d82) * fix fit memory test (#461) * fix fit memory test * remove xenial checks * remove irrelevant comment * free index in seralization test (cherry picked from commit a0b70c8) * Remove avx512_BW_VL (#460) * remove bw_vl arch (not needed, avx512 is enough) * remove xenial from merge queue flow * fix fit memory test (#461) * fix fit memory test * remove xenial checks * remove irrelevant comment * free index in seralization test * install gcc10 bionic * install newer vg ubuntu18 * exit valgrind directory (cherry picked from commit d2e1d4f) * [MOD-6775] introduce fp16 type! (#462) * remove bw_vl arch (not needed, avx512 is enough) * remove xenial from merge queue flow * introduce fp16 type! change bfloat and fp16 to struct typedef to avoid ambiguty between them * added flow tests * fp16 benchmarks * fix hnsw flow test * add float616 to serializing script add fp16 files to bm files decrease testSizeEstimation dim size to make it run fasetr in debug * set log call back to null in benchmarks * remove struct typedef add default ctor * add comma seriaizer file * disable temp flow * disabe for real * remove constexpr from bfloat default ctor declaring a default ctor as constexpr is allowed as of c++20, and this feature in particular is supported as of gcc10 I don't see any benefit using this feature... * test focal * disable temp flow (cherry picked from commit 704282b) --------- Co-authored-by: meiravgri <meirav.grimberg@redis.com> Co-authored-by: meiravgri <109056284+meiravgri@users.noreply.github.com> Co-authored-by: alonre24 <alonreshef24@gmail.com>

fix fit memory test

d97f550

meiravgri force-pushed the meiravg_fix_fitMemoryTest branch from 6910e93 to d97f550 Compare May 7, 2024 10:07

meiravgri requested a review from alonre24 May 7, 2024 10:08

remove xenial checks

b6770e0

alonre24 reviewed May 7, 2024

View reviewed changes

remove irrelevant comment

c13d34b

meiravgri requested a review from alonre24 May 7, 2024 11:10

alonre24 previously approved these changes May 7, 2024

View reviewed changes

meiravgri enabled auto-merge May 7, 2024 11:16

meiravgri added this pull request to the merge queue May 7, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 7, 2024

free index in seralization test

c38cfcc

meiravgri dismissed alonre24’s stale review via c38cfcc May 7, 2024 12:51

meiravgri requested a review from alonre24 May 7, 2024 12:52

alonre24 approved these changes May 7, 2024

View reviewed changes

meiravgri enabled auto-merge May 7, 2024 12:58

meiravgri added this pull request to the merge queue May 7, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 7, 2024

meiravgri merged commit a0b70c8 into meiravg_feature_bfloat16 May 7, 2024
16 checks passed

meiravgri deleted the meiravg_fix_fitMemoryTest branch May 7, 2024 16:10

meiravgri added a commit that referenced this pull request May 7, 2024

fix fit memory test (#461)

ef91aff

* fix fit memory test * remove xenial checks * remove irrelevant comment * free index in seralization test

github-actions bot pushed a commit that referenced this pull request May 12, 2024

fix fit memory test (#461)

98beeba

* fix fit memory test * remove xenial checks * remove irrelevant comment * free index in seralization test (cherry picked from commit a0b70c8)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix fit memory test #461

fix fit memory test #461

Uh oh!

meiravgri commented May 7, 2024

Uh oh!

codecov bot commented May 7, 2024 •

edited

Loading

Uh oh!

alonre24 May 7, 2024

Uh oh!

meiravgri May 7, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fix fit memory test #461

fix fit memory test #461

Uh oh!

Conversation

meiravgri commented May 7, 2024

Uh oh!

codecov bot commented May 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alonre24 May 7, 2024

Choose a reason for hiding this comment

Uh oh!

meiravgri May 7, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented May 7, 2024 •

edited

Loading