Skip to content

[pg_replication] the initial replication for the whole database without specifying all table names is not possible #602

@VioletM

Description

@VioletM

dlt version

1.7.0

Source name

pg_replication

Describe the problem

The empty table_names argument for the init_replication function throws an error:

Traceback (most recent call last):
  File "/Users/vmishechk/PycharmProjects/customers_questions/pg_replication_lendbuzz/pg_replication_pipeline.py", line 46, in <module>
    replicate_with_initial_load()
  File "/Users/vmishechk/PycharmProjects/customers_questions/pg_replication_lendbuzz/pg_replication_pipeline.py", line 28, in replicate_with_initial_load
    snapshot = init_replication(  # requires the Postgres user to have the REPLICATION attribute assigned
  File "/Users/vmishechk/Library/Caches/pypoetry/virtualenvs/dlt-verified-sources-5afoIPXt-py3.9/lib/python3.9/site-packages/dlt/common/configuration/inject.py", line 259, in _wrap
    return f(*bound_args.args, **bound_args.kwargs)
  File "/Users/vmishechk/PycharmProjects/customers_questions/pg_replication_lendbuzz/pg_replication/helpers.py", line 151, in init_replication
    snapshot_table_names = [
TypeError: 'NoneType' object is not iterable

Expected behavior

All tables in the database are added to the initial snapshot

Steps to reproduce

dest_pl = dlt.pipeline(
        pipeline_name="pg_replication_pipeline2",
        destination='duckdb',
        dataset_name="replication_postgres",
    )

# initialize replication for the source table
slot_name = "example_slot"
pub_name = "example_pub"
schema_name = "public"

snapshot = init_replication(  # requires the Postgres user to have the REPLICATION attribute assigned
    slot_name=slot_name,
    pub_name=pub_name,
    schema_name=schema_name,
    # table_names=["customers", "orders"],
    persist_snapshots=True,  # persist snapshot table(s) and let function return resource(s) for initial load
    reset=True,
)

How you are using the source?

I run this source in production.

Operating system

macOS

Runtime environment

Local

Python version

3.9

dlt destination

No response

Additional information

No response

Metadata

Metadata

Assignees

Labels

supportan issue from SE

Type

No type

Projects

Status

Planned

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions