Replies: 3 comments 5 replies
-
|
Alright, after debugging and updating this post I seem to realise that this might be an Istio problem, as istio-proxy stills holds the old endpoints. Not going to delete this post as this might help somebody else |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @seilerre for sharing this! |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, this was only party the answer, see next comment. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, this seems to be due to the fact that Hyper in Clickhouse Sink is doing connection pooling and envoy in Istio is as well. Is there any way to deactivate connection pooling in the Clickhouse Sink and let this be handled by Istio? See discussion here istio/istio#54539 |
Beta Was this translation helpful? Give feedback.
-
|
I don't see any relevant option exposed in the public docs: https://vector.dev/docs/reference/configuration/sinks/clickhouse/ Note that internally the ClickHouse sink relies on HTTP. |
Beta Was this translation helpful? Give feedback.
-
|
HTTP also doesn't expose this option. Wouldn't this be a nice addition? I can think of more people using vector in a Kubernetes context using a Service Mesh, which handles stuff like connection pooling. |
Beta Was this translation helpful? Give feedback.
-
|
You are welcome to open a feature request (if you do so, please make sure to provide sufficient context for potential contributors). |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if this should be a discussion. Something seems to be broken. Apart from istio/istio#54539 there are also other issues mentioning this: VictoriaMetrics/helm-charts#1938 |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
A note for the community
Problem
We are running Clickhouse and Vector in Kubernetes. Vector is receiving events from Kafka and is sinking those to Clickhouse. When updating Clickhouse, thereby rolling out a new Pod, data is incoming again without dropped data, however Vector is complaining with WARN messages. This doesn't go away on its own. A 503 seems to me, like Vector is trying to reach the old Clickhouse instances? Could this be possible? Failed requests are retried and written successfully. Requests seem to time out after 10 or 30s, as can be seen in the istio-proxy logs below. A restart of Vector helps, but this can't be the right solution.
2025-01-03T07:48:00.264670Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15212892}: vector::sinks::util::retries: Internal log [Retrying after response.] has been suppressed 22 times. 2025-01-03T07:48:00.264683Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15212892}: vector::sinks::util::retries: Retrying after response. reason=503 Service Unavailable: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection timeout internal_log_rate_limit=true 2025-01-03T07:48:00.264715Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15212891}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding. 2025-01-03T07:48:11.196311Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15213093}: vector::sinks::util::retries: Internal log [Retrying after response.] has been suppressed 24 times. 2025-01-03T07:48:11.196327Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15213093}: vector::sinks::util::retries: Retrying after response. reason=503 Service Unavailable: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection timeout internal_log_rate_limit=true 2025-01-03T07:48:11.196372Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15213091}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding. 2025-01-03T07:48:22.164840Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15213341}: vector::sinks::util::retries: Internal log [Retrying after response.] has been suppressed 29 times. ### Configuration ```text clickhouse_sink: type: clickhouse compression: zstd healthcheck: enabled: true acknowledgements: enabled: true inputs: - clickhouse_tables endpoint: http://service-clickhouse-prod.clickhouse-prod.svc.cluster.local:8123 format: json_each_row database: ep_sdc_edge table: '{{ "{{kafkaSrcTopicToDbName}}" }}' skip_unknown_fields: true auth: strategy: basic user: ${DEST_CLICKHOUSE_USERNAME} password: ${DEST_CLICKHOUSE_PASSWORD} date_time_best_effort: true encoding: timestamp_format: rfc3339 batch: max_events: 2500 timeout_secs: 2Version
0.42.0-distroless-libc
Debug Output
Debug Logs of one request, that fails after rolling the Clickhouse Pods
istio-proxy logs outgoing from vector pod:
This doesn't seem to be a DNS issue, IPs, are correct:
Example Data
No response
Additional Context
No response
References
No response
Beta Was this translation helpful? Give feedback.
All reactions