Precomputing large feature vectors

[ingmarschuster]

Hi, I want to precompute a feature vector of 1900 dimensions for over 500 data points and store it in an HF dataset. RAM is becoming a problem. How can I make sure to write directly to disk?

[mariosasko]

Hi! You can reduce writer_batch_size (default 10_000) (also batch_size if batched=True to work with smaller batches) in map to lower the memory consumption. This is how the embedding computation part could look like then:

dset_with_embeddings = dset.map(lambda ex: {"embedding": compute_embedding(ex)}, writer_batch_size=10)