Memory Issues with Sparse Vectors in XGBoost4j-Spark: Disabling Sparse-to-Dense Conversion

[stepanov1997]

Hi everyone,

I’m using sparse vectors with about 10 features out of a possible 50 million. However, the conversion to dense vectors is causing heap exhaustion. Is there a way to disable the sparse-to-dense conversion?

Right now, I can’t even train on a small batch of vectors without running into memory issues, but I ultimately need to train on 200 million rows.

Any help would be greatly appreciated. I’m using XGBoost4j-Spark version 3.0.0 with the Java.

Thanks!

[trivialfis]

cc @wbo4958

[stepanov1997]

Hey @trivialfis and @wbo4958, the lack of proper handling for SparseVector (as noted in the TODO) causes memory issues. If I have 50 million features, converting it to a dense vector can lead to serious problems. With proper sparse vector support, this wouldn’t be an issue

xgboost/jvm-packages/xgboost4j-spark/src/main/scala/ml/dmlc/xgboost4j/scala/spark/Utils.scala

Line 52 in 94bb1da

// TODO support sparsevector

Can we expect this issue to be fixed in the near future, or should I consider giving up on distributed training for now?

[trivialfis]

It's in the backlog. But may I ask how did you get the 50 million features? Are you using some kind of encoding?

[stepanov1997]

Thx, I'm using Murmurhash to encode features.