java.lang.NegativeArraySizeException

[david-liu]

when i found the algo on 1.5 billion items, I got the following exception

Caused by: java.lang.NegativeArraySizeExceptionat 
com.github.jelmerk.knn.hnsw.HnswIndex.add(HnswIndex.java:212)at 
com.github.jelmerk.knn.Index.lambda$addAll$0(Index.java:125)at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)

My configuration is

    val hnsw = new HnswSimilarity()
      .setIdentifierCol("item_id")
      .setQueryIdentifierCol("item_id")
      .setFeaturesCol("embeddings")
      .setNumPartitions(150)
      .setNumReplicas(5)
      .setK(5)
      .setEf(20)
      .setSimilarityThreshold(0.5)
      .setDistanceFunction("inner-product")
      .setPredictionCol("approximate")
      .setExcludeSelf(true)
      .setM(16)
      .setEfConstruction(200)

[jelmerk]

That looks like a bug, I'll see if I can fix it.

Tbh you'll likely have trouble getting it to work with billions of items right now. I've been working on some variation of this idea https://arxiv.org/pdf/1906.10602.pdf to make this possible but haven't gotten around to finishing it