Components with multiple roles (component that is both a sink and a source)

[esensar]

Recently, my PR has been merged (#21348), which added a new type of enrichment table, which can also act as a sink. That has caused some changes in the topology, because enrichment tables were not meant to act as sinks and were supposed to be accessible only from VRL remap transform.

We have had a brief discussion on how to handle this in other ways in the PR (#21348 (comment)) - more granular approach to building the topology, giving each component parts that might make it a sink, source, transform, or other things that support each of these roles, turning all of them into generic nodes, which can then be combined into a topology in more ways, depending on what each of the components support.

Right now, I am maintaining a branch which also enables this memory enrichment table to act as a source. You can see the changes here: master...esensar:vector:feat/vrl-cache-as-source (it is not very clean, since I assumed that something like this would not be accepted as a contribution, at least not in the near future, but I might be wrong). This enables some interesting new ways to use this table and I assume it might make sense for many more components.

My question is, does this sound like a direction it makes sense for Vector to move in, or does it introduce too much complexity? Does the current state of that memory enrichment table as a source branch make any sense? Would it be acceptable to merge with some changes? If not, do you have a suggestion on how to provide similar functionality in an acceptable way (dumping data from the cache)?

[pront]

Hi @esensar, thank you for sharing this. I will definitely take a look at your branch and comment. I agree that having generic nodes in the topology opens up new capabilities and use cases. For example, we can inspect sink output. Now, how to do this refactoring elegantly is another big challenge :)

In the meantime, sharing this prior art in case it helps:

• Support dead letter queue on sinks #1772 (dead letter queue, chaining sinks)
• chore: RFC for handling discarded events #14708
• Various linked issues contain more context

[esensar]

I have opened a draft PR with a modified version of this memory enrichment table feature: #22466
Implementing node based approach where each node can take on multiple roles may take much longer to implement, so I have made some changes to provide the feature we require in that PR, with the least effect on the rest of the topology.

[pront]

I will take a look, however I need some heads down time to think about the overall approach. Treating components (source, transform, sink, maybe enrichment table) as a node with inputs/outputs is the desired end result.