Skip to content

Commit 5ca11aa

Browse files
authored
Don't do multipart uploads for chunks (#845)
Some leftover code was doing unconditional multipart uploads of chunks, in the object store that use `object_store` (gcs, local, etc). This was producing for usual chunk sizes ~ 3x performance penalty. We should add multipart back, but only for large objects.
1 parent 6a9c8f5 commit 5ca11aa

File tree

1 file changed

+1
-5
lines changed

1 file changed

+1
-5
lines changed

icechunk/src/storage/object_store.rs

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -459,11 +459,7 @@ impl Storage for ObjectStorage {
459459
bytes: bytes::Bytes,
460460
) -> Result<(), StorageError> {
461461
let path = self.get_chunk_path(&id);
462-
let upload = self.get_client().await.put_multipart(&path).await?;
463-
// TODO: new_with_chunk_size?
464-
let mut write = object_store::WriteMultipart::new(upload);
465-
write.write(&bytes);
466-
write.finish().await?;
462+
self.get_client().await.put(&path, bytes.into()).await?;
467463
Ok(())
468464
}
469465

0 commit comments

Comments
 (0)