How to handle 100's of CRDs? Can we process them concurrently

[ewassef]

I would have swore that at some point, you would get multiple calls to ReconcileAsync at the same time. However, it now seems that this is not the case. Is there a way to run say X entities in parallel? our reconcile loops are taking an hour now because of the sheer number of CRDs we are creating

[ewassef]

@buehler what do you think?

[buehler]

Hey @ewassef
The current mechanism is serial. There is (sadly) no way around it. I'm currently on vacation, I'll have a look at v8 when I am home :-)
If I recall it correctly, v8 could theoretically run in parallel.

[ewassef]

I was curious about your plans for that. It looks like the reactive extensions are being replaced with a concurrent dictionary. However, I think we may be able to process multiple entities concurrently (safely also) by modifying this line to this:

.Merge(maxConcurrent: _operatorSettings.ConcurrentExecutionLimit ?? 2)

how do you mean theoretically run in parallel in v8?

[buehler]

Hey @ewassef
I just checked, in v8 the logic is:

private async Task ReconcileModification(TEntity entity)
{
    var latestGeneration = _entityCache.GetOrAdd(entity.Uid(), 0);
    if (entity.Generation() <= latestGeneration)
    {
        _logger.LogDebug(
            """Entity "{kind}/{name}" modification did not modify generation. Skip event.""",
            entity.Kind,
            entity.Name());
        return;
    }

    _entityCache.TryUpdate(entity.Uid(), entity.Generation() ?? 1, latestGeneration);
    await using var scope = _provider.CreateAsyncScope();
    var controller = scope.ServiceProvider.GetRequiredService<IEntityController<TEntity>>();
    await controller.ReconcileAsync(entity);
}

And this method is called on an event base of the watcher. This means, that 100's of CRDs could theoretically run in parallel. There is no semaphore lock anymore.

[jaccus]

Hi @buehler! Would it be possible to enable running reconciliation for multiple entities of the same type in parallel without modifying the source code of operator sdk library? Or with "could theoretically" were you just referring to a potential feature improvement?

I am struggling with a similar problem as author. Multiple independent instances of the same entity running their reconciliations serially result in drastically increased deployment times. This unfortunately makes it hard to justify using the library in our productions scenarios.

[buehler]

Hmm, fair enough. I thought that using async await here can help to reconcile the entities in parallel. However, if that's not the case we need to investigate a threading mechanism. This would introduce a lot of complexity in the system.

[slacki123]

Hey, I'm working with something similar now and I'm noticing some strange behaviour. I'm wondering if this is related or there's something wrong with the requeue mechanism

When using the same custom resource definition, and then creating something like 10 custom resources based on it with varying requeue time varying from 5 seconds to 60 seconds per resource.

I'm noticing that on the startup of the operator, all 10 resources get updated and processed, but then after a while I only see 2 or 3 resources getting processed while the remaining resources are ignored despite having been added to the requeue.

The way I set up my requeue is this:

  public class ClusterController : IEntityController<MyEntity>
{
    private readonly EntityRequeue<MyEntity> _requeue;

    public ClusterController(
        EntityRequeue<MyEntity> requeue)
    {
        _requeue = requeue;
    }

    public async Task ReconcileAsync(MyEntity myEntity)
    {
        if (!myEntity.Spec.Enabled)
        {
            return; // will not trigger a requeue
        }
        
        // Update status here
  
        var pollingRate = TimeSpan.FromSeconds(myEntity.Spec.ReconciliationLoopPollSeconds);
        _requeue(myEntity, pollingRate);
    }

Has anyone encountered this?

EDIT:
now related to #707

[slacki123]

Building on the question of processing 100s of CRDS further. What is the expected behaviour when dealing with long running Reconciliation processes?

For example say that:

1. we add a custom resource - it gets reconciled
2. the process runs for some time
3. In the meantime, we modify the custom resource, which should lead to triggering another reconciliation loop

now the question is, will the process wait for the existing process to finish, or will it start running another one in parallel? (which could be problematic if conflicts are an issue)

I have some ideas how to deal with this issue on the implementation side, as well as the KubeOps resource watcher side, but would like to understand how deeply has this already been thought through first

[buehler]

Yeah, it's a complicated topic.

Making it configurable seems to be a great idea. When I built a similar logic in v7, a lot of problems surfaced. For example: if the reconciliation loop takes more time than the requested requeue, you create an infinite backlog.

Then I started to use time durations to "memorize" the reconcile loop and if something is happening right now, ignore the current reconcile and requeue after.

Then another implementation took place which I don't really remember anymore.

The queue always caused problems in some cases. That is one of the reasons I rewrote the whole thing for v8. Right now, it is plain and simple. We can do such a queue mechanism, but it should be well thought through.

Is there someone who knows (more) about best practices in terms of Kubernetes Operators? Should it be possible to have long running reconcile loops? Or should you make multiple operators with multiple resources?

As a sidenote: maybe we can make this caching / queueing mechanism with an Interface, then, one can add their own strategy without having to add it directly to the package.

[slacki123]

I see the infinite loop backlog - although could be solved if you don't build the backlog and only ever process the last event grouped by event type? Although I agree that a highly experienced veteran's opinion would be ideal on this. The interface is a nice idea, but depending on how useful that would be. As for right now I can also write my own task manager that tracks the ongoing reconciliations, and if the current one is still processing then skip, and if not then reconcile for example. Either way, I'd end up managing my own code for this

[neil-119]

Has this been solved for? Running into the same issue where we can only reconcile one resource at a time, and if that resource takes time, everyone else is SOL. We have a CRD that is particularly long running, and in that time no one else has their CR reconciled.

[buehler]

Hey @neil-119

I think - in the current state of the SDK - it would make sense to split up those long running CRDs in their own operator. Since everything is .net, you can easily share the business logic. Would that be possible?

[neil-119]

We need to parallelize the same CRD, as some of these custom resources will take 40+ minutes to reconcile and there are currently 20 of them sitting in the queue for the same CRD. I'd like all of them (for the same CRD) to run in parallel. I thought about just spinning off another thread and just returning from the controller, and updating the entity after the fact but I'm not sure how feasible that is.