Skip to content

FATAL error #47

@jayadevanm

Description

@jayadevanm

I am testing this on Alpine Linux, with PostgreSQL 12.5, TimescaleDB 2.0.1. With a batch size of 1000000, it worked fine. When I increased the batch size to 10000000, I started getting and error.
FATAL: terminating connection because protocol synchronization was lost
Google led me to this lib/pq#473 (comment)
Not sure if that is the same. The file has 133 million records and PostgreSQL's COPY works without any issues. I am trying to optimize the batch-size/number-of-workers combinations and see if I can get a better throughput with parallel-copy. So far, I have been able to get roughly 90,000-1,00,000 per second with default COPY while parallel-copy gave me 50,000-60,000 per second.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions