Memory enrichment table: Difference between global enrichment_tables config vs usage as sink

[ChristianCiach]

I am trying to wrap my head around the the memory enrichment tables as highlighted in https://vector.dev/highlights/2025-02-24-memory_enrichment_table/

I pretty much understand everything written in that article, until the very last section called "Use as a sink":

What is the difference between defining the enrichment table using the global configuration option enrichment_tables vs configuring it as a sink with type: memory_enrichment_table?

• Both configurations use an inputs of sources or transformers to populate the table and also share the exact same configuration options like tty.
• What is the name of the enrichment table when defining it as a sink, to reference it using the get_enrichment_table_record VRL function? Is it the name of the sink?
• When would I prefer the sink over the global configuration or vice versa?
• Why is the sink type memory_enrichment_table not listed at https://vector.dev/docs/reference/configuration/sinks/?

I am so confused.

Kindly pinging @esensar, because I strongly assume he knows it best :)

[ChristianCiach]

I think I might get it, after looking at the sources. The last section of https://vector.dev/highlights/2025-02-24-memory_enrichment_table/ is highly misleading, maybe even wrong. This is my understanding now:

• You never explicitly define a memory enrichment table as a sink. The sink will be implicitly created from the globally configured enrichment tables having type: memory.
• The name of the sink is identical to the name of the enrichment table. This is not documented anywhere, but it also doesn't really matter, since we never explicitly reference a sink as outputs. Instead, it's the other way around: Sinks define sources and transformers as inputs, which we do in enrichtment_tables.[id].inputs, which is btw an attribute not documented at all at https://vector.dev/docs/reference/configuration/global-options/. In fact, when having an enrichment_table of type memory, this enrichment_table can then be configured with all common attributes of a sink. Weird!

Quote from the linked article:

For example, here is how to introduce this new component as a sink if you have another source that can populate the cache:

 memory_table_sink:
   inputs: [ "another_source_or_transform" ]
   type: memory_enrichment_table
   ttl: 60
   flush_interval: 5

This sentence probably tries to convey the fact that the configured enrichment table automatically generates a sink that looks roughly like that YAML snippet. (Although that is not entirely correct, since the name of the sink will be memory_table, not memory_table_sink, I think)

[ChristianCiach]

I think this is all very confusing. I wonder why the sinks are even created automatically, instead of properly exposing a new sink type memory_enrichment_table to the user. Something like this:

enrichment_tables:
  some_cache:
    type: memory
    ttl: 60
    flush_interval: 5

sinks:
  some_cache_sink:
    type: memory_enrichment_table
    inputs: [a_source_or_transformer]
    table_key: some_cache

This would mean that there could be multiple sinks that contribute to the same table. This may be the reason it was not done this way.

I just see that an enrichment table can now be used as a source by configuring enrichment_tables.[id].source_config. Wouldn't it make a lot of sense to introduce an analogous enrichment_tables.[id].sink_config?:

enrichment_tables:
  some_cache:
    type: memory
    ttl: 60
    flush_interval: 5
    sink_config:
      inputs: [another_source_or_transform]
      sink_key: some_cache_sink  # could default to the name of the table

@esensar Did i get any of that wrong?

[esensar]

Hey @ChristianCiach
Sorry about the confusing feature 😄
I agree, it is generally pretty confusing in its current state and I didn't do a good job of explaining it in the docs.

You are right, the sink is generated automatically (and the source too, if you use it). This whole component is a special case in vector codebase, which contributes to this unusual way of defining it.

The reasoning for not having a separate sink, but having it be automatically generated is that it is useless if not used as a sink. The only way to populate that table is to use it as a sink. It also requires data formatted in a specific way, which means in most cases you will also need a special remap transform made just to feed the data into the table.

Regarding the last item, I agree, sink_config would make it more similar to source_config and make it a bit more clear what is going on, but again, the reason for putting source_config in its own field is that it is optional.

So to sum it up, this is not listed in sinks because it is not really a sink when it comes to vector components, but it is also not useful as a sink. The sink part of it is just required for it to function, but you would never build your configuration around feeding data to a memory table.

Let me know if you have more questions.

[ChristianCiach]

Great answer! Thank you for your quick response and for your awesome work on Vector!

[ChristianCiach]

The reasoning for not having a separate sink, but having it be automatically generated is that it is useless if not used as a sink.

I can follow that logic, but I don't think it's a particularly strong argument. I can define all sorts of sources that are useless if not referenced by a transformer or sink. It should be possible to implement a validator that ensures that every table has a referencing sink (if this is even something you want to actually ensure).

I am not saying any of this needs to change. I just believe it would be easier to grasp as a user. Also, from a model standpoint, this would also avoid this weird merger of the MemoryTable and Sink models.

[esensar]

Yeah, I realized that as I was typing up the response 😄
But I couldn't figure out a better reply. I guess the main reason is that most of that documentation is automatically generated, and this components is a special case, so it is missed in that, but I also think it could be misleading, because users would then think that they can place it in the sinks part of the configuration and they can't.

[pront]

I agree this part https://github.com/vectordotdev/vector/blob/master/website/content/en/highlights/2025-02-24-memory_enrichment_table.md?plain=1#L105-L117 is a bit confusing. @esensar do you have a few minutes to provide a minimal but complete config there?

We can also add it here: https://vector.dev/docs/reference/configuration/#using-memory-enrichment-table-as-a-source (I realized this doesn't show up as a result in the search bar and is a bit hard to find)