Does Not Run on Large Datasets when using python 3.7 #6
Description
Hi,
I am not able to use MS PHATE with python 3.7.
Reproducible example:
virtualenv --no-download phate_test_env # change path to env as needed
source phate_test_env/bin/activate # load env
pip install git+https://github.com/KrishnaswamyLab/Multiscale_PHATE
python
import numpy as np
from multiscale_phate import multiscale_phate
data = np.random.uniform(size=(5000000,200)) # make sure its big
msphate_obj = multiscale_phate.Multiscale_PHATE(n_pca=None, n_jobs=48, knn=200)
msphate_obj.fit(data)
sterr is:
File "[...]/lib/python3.7/site-packages/multiscale_phate/multiscale_phate.py", line 158, in fit
self.hash = utils.hash_object(X)
File "[...]/lib/python3.7/site-packages/multiscale_phate/utils.py", line 18, in hash_object
return hash(pickle.dumps(X))
OverflowError: cannot serialize a bytes object larger than 4 GiB
edited to make data shape more realistic