Skip to content

Commit 8edd708

Browse files
authored
fix: fix batch transform modify schema (#1483)
## Which issue does this PR close? - Previously, we used `record_batch.with_schema(target_schema.clone())` to change the schema name of a record batch, but unfortunately, `with_schema` would check the schema name and throw an error. So I think we should convert the RecordBatch like how `BatchTransform::Modify` does, but keep the same columns. ```rust pub fn with_schema(self, schema: SchemaRef) -> Result<Self, ArrowError> { if !schema.contains(self.schema.as_ref()) { return Err(ArrowError::SchemaError(format!( "target schema is not superset of current schema target={schema} current={}", self.schema ))); } Ok(Self { schema, columns: self.columns, row_count: self.row_count, }) } ``` - Closes #. ## What changes are included in this PR? ## Are these changes tested?
1 parent 69686ba commit 8edd708

File tree

1 file changed

+8
-1
lines changed

1 file changed

+8
-1
lines changed

crates/iceberg/src/arrow/record_batch_transformer.rs

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,14 @@ impl RecordBatchTransformer {
153153
)?
154154
}
155155
Some(BatchTransform::ModifySchema { target_schema }) => {
156-
record_batch.with_schema(target_schema.clone())?
156+
let options = RecordBatchOptions::default()
157+
.with_match_field_names(false)
158+
.with_row_count(Some(record_batch.num_rows()));
159+
RecordBatch::try_new_with_options(
160+
target_schema.clone(),
161+
record_batch.columns().to_vec(),
162+
&options,
163+
)?
157164
}
158165
None => {
159166
self.batch_transform = Some(Self::generate_batch_transform(

0 commit comments

Comments
 (0)