Push queries abort after 10 minutes of no activitiy

[mrt181]

I am using push queries on (source) tables that have infrequent updates.
The connections are aborted after 4 minutes.

I am using confluent cloud which does not allow to configure: ksql.idle.connection.timeout.seconds

A workaround is mentioned here: confluentinc/ksql#6970 (comment)

That involves calling some endpoint like listStreams periodically.

When creating a push query I have to use IKSqlDBContext

     IQbservable<MyType> qbservable = context.CreatePushQuery<MyType>();

This use a client under the hood to which I do not have access.
So I can not issue a periodic request using that client to keep the connection open.

Is there a way to keep push query connections alive similar to the mentioned workaround?

[tomasfabian]

You might find a configuration for HttpClient that can be set as follows:

public static void Configure(this IServiceCollection services, string ksqlDbUrl)
{
  services.AddDbContext<IKSqlDBContext, KSqlDBContext>(c =>
  {
    c.UseKSqlDb(ksqlDbUrl);

    c.ReplaceHttpClient<IHttpClientFactory, ksqlDB.RestApi.Client.KSql.RestApi.Http.HttpClientFactory>(httpClient => { 
        httpClient.DefaultRequestHeaders.Add("Connection", "Keep-Alive"); //Find a proper configuration option
      })
      .AddHttpMessageHandler(_ => new CustomMessageHandler());
  });
}

Check also https://learn.microsoft.com/en-us/dotnet/api/system.net.http.socketshttphandler.connectcallback?view=net-7.0#examples

[mrt181]

So what we did is to configure the sockets handler to send keep alive pings.

  builder
    .Services.AddHttpClient<
      ksqlDB.RestApi.Client.KSql.RestApi.Http.IHttpClientFactory,
      ksqlDB.RestApi.Client.KSql.RestApi.Http.HttpClientFactory
    >()
    .UseSocketsHttpHandler(
      (handler, _) =>
      {
        handler.PooledConnectionLifetime = TimeSpan.FromHours(1);
        handler.KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always;
        handler.KeepAlivePingDelay = TimeSpan.FromMinutes(1);
        handler.KeepAlivePingTimeout = TimeSpan.FromMinutes(1);
      }
    )
    .SetHandlerLifetime(Timeout.InfiniteTimeSpan);

This is a good solution but did not work for me because the cloud infrastructure I use has a proxy managing all connections. That proxy timeouts the connection to the server and does not forward the keep alives. That proxy is not configurable.

[tomasfabian]

Thanks for sharing the solution.