[GCS Destination] JSON output omits null fields, add option to include nulls #57538
marcoaanogueira
started this conversation in
Connector Ideas and Features
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the problem
When using the GCS destination connector, fields that have
null
values are being omitted from the output JSON.This causes downstream systems that rely on a consistent schema (e.g. expecting all fields, even if null) to break or require workarounds.
Steps to Reproduce
null
values.null
values in optional fields.Expected Behavior
The GCS destination should have an option to include fields with
null
values in the output JSON.Example of expected record:
Instead, the current behavior outputs:
Context
This behavior breaks compatibility with systems expecting a fixed schema.
Suggested Solution
Add a configuration option to the GCS destination connector (e.g., include_null_fields: true) that controls whether null fields are kept in the output JSON.
Additional context
Connector: destination-gcs
GCS destination connector version: v0.4.8
Format: JSON Lines (jsonl)
Let me know if you need help testing or validating this feature, happy to contribute!
Beta Was this translation helpful? Give feedback.
All reactions