Batching of records

[davidkingg]

Hey Guys,
I need help or clarification on how I can properly batch data.
As of now, I cannot accurately predict my data traffic so I would like data to be inserted into clickhouse once in every 30mins.

is there a connector config to handle this effectively?

I tried using the consumer.override.fetch.max.wait.ms but it seems the consumer can only wait for up to 9mins but fails for longer wait time with this log message:

2025-05-26 06:36:24,125 INFO   ||  [Consumer clientId=connector-consumer-clickhouse-connect-0, groupId=connect-clickhouse-connect] Node 1001 disconnected.   [org.apache.kafka.clients.NetworkClient]
2025-05-26 06:36:24,126 INFO   ||  [Consumer clientId=connector-consumer-clickhouse-connect-0, groupId=connect-clickhouse-connect] Cancelled in-flight FETCH request with correlation id 11 due to node 1001 being disconnected (elapsed time since creation: 600012ms, elapsed time since send: 600010ms, throttle time: 0ms, request timeout: 1000000ms)   [org.apache.kafka.clients.NetworkClient]
2025-05-26 06:36:24,126 INFO   ||  [Consumer clientId=connector-consumer-clickhouse-connect-0, groupId=connect-clickhouse-connect] Cancelled in-flight METADATA request with correlation id 112 due to node 1001 being disconnected (elapsed time since creation: 303029ms, elapsed time since send: 303029ms, throttle time: 0ms, request timeout: 1000000ms)   [org.apache.kafka.clients.NetworkClient]
2025-05-26 06:36:24,126 INFO   ||  [Consumer clientId=connector-consumer-clickhouse-connect-0, groupId=connect-clickhouse-connect] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1001:   [org.apache.kafka.clients.FetchSessionHandler]
org.apache.kafka.common.errors.DisconnectException

is there a way to control the connect worker or connector to pull for records every 30mins?

My second question:
is there a way to implement automatic schema evolution?

my last question.
currently, if a column is added to the source table, the connector just skips this and continues writing the existing columns to clickhouse.
I would prefer if an error is thrown so I can make the schema changes on clickhouse,restart the connector and not lose any data.
is there a way to change this behavior ?

Thank you!