Skip to content
This repository was archived by the owner on Sep 23, 2024. It is now read-only.
This repository was archived by the owner on Sep 23, 2024. It is now read-only.

Field-Level Metadata Removed #101

@ers81239

Description

@ers81239

Describe the bug
Field level metadata is removed from the catalog when discovery is performed

To Reproduce
Steps to reproduce the behavior:

  1. Provide field-level metadata in the catalog, for example:
{
          'breadcrumb': ['properties', 'char_name'],
          'metadata': {
	          'arbitrary_field_metadata': 'should be preserved'
          }
}
  1. Run the tap

This affects at least Meltano users who use the meltano select feature to populate the selected metadatum to indicate whether fields should be copied or not. In that case, the selected field is being stripped and therefore ignored and all fields get copied regardless.

Expected behavior
Field-level metadata should be preserved through the stream_utils.refresh_streams_schema() with only discoverable metadata being updated/added.

Your environment

  • Version of tap: master
  • Version of python: 3.8

Additional context
I'm using the tap with meltano and setting a field metadatum: selected: True, however this is stripped from the metadata and in turn not honored by the tap.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions