feat: introduce SchemaCollector #391
Conversation
SchemaCollector can be used to generate a CalciteSchema from a Substrait Rel
| SubstraitSchema newSubSchema = new SubstraitSchema(); | ||
| schema = schema.add(schemaName, newSubSchema); | ||
| } | ||
| } |
There was a problem hiding this comment.
The existing LookupCalciteSchema code didn't handle nested schemas. I've added support for it here.
| if (!existingSchema.equals(namedScan.getInitialSchema())) { | ||
| throw new IllegalArgumentException( | ||
| String.format( | ||
| "NamedScan for %s is present multiple times with different schemas", tableName)); |
There was a problem hiding this comment.
It's helpful to guard against this because it can result in some very weird errors. The same table can appear multiple times in a plan (i.e. through joins), and if an external producer applies field trimming and those tables are trimmed to have different fields, unpleasantness can ensue.
|
The This works by introducing 2 classes that extend from Calcite:
And then using these to build out a schema from a Substrait relation using the new public SchemaCollector visitor. The new collector also handles nested schemas and performs basic validation on schemas. There are tests to go along with is as well. This new implementation can be upgraded to Calcite 1.39 without issues. |
SchemaCollector can be used to produce a CalciteSchema from a Substrait Rel
BREAKING CHANGE: LookupCalciteSchema has been removed