Proposal: Binary Data Support for libpq's Extended Protocol with Consumption Tracking

[avamingli]

Description

We( @avamingli and @wangliang03) propose enhancing libpq's Extended Protocol to support binary data transmission while adding message tracking capabilities, addressing request identified in previous PR:

#990 (comment) by @yjhjstz

The current struct-based implementation(bitmap) suffers from extened-subtags and lacks proper validation of unconsumed messages.

Our solution introduces a structured binary format with type/length headers while maintaining backward compatibility, making it more general and easier subtags-extend for for developers. They don't need to consider what messages are of other subtags and only care about the ones created by themselves.

The key innovation is a producer-consumer tracking system that will log warnings when messages remain unconsumed at transaction commit, preventing silent data leaks.

The implementation is adding opt-in binary support with instrumentation to detect unconsumed messages. Memory management will leverage TopTransactionContext for automatic cleanup while adding reference counting for shared payloads.

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[avamingli]

#1098

[avamingli]

Merged.