Events dropped and Error logs when using unix socket as a source

[bmallya]

Hello,
The vector agent that I am using is doing a couple of things. It has HTTP, TCP, Unix and file as sources. The events that get in through these sources are routed through a transform (basic validation and appending additional metadata) before being sent to another instance of vector on an upstream device which will act as a collector. The version of vector that I am currently using is 0.30 and here is the config file

type = "http"
address = "0.0.0.0:50002"
encoding = "text"
path = "/ot/log"
strict_path = true

[sources.tcp_source]
type = "socket"
mode = "tcp"
address = "0.0.0.0:50003"

[sources.udp_source]
type = "socket"
mode = "unix_stream"
path = "/tmp/unix_sock"

[sources.file_source]
type = "file"
read_from = "beginning"
include = ["/var/log/**"]

[transforms.format]
type = "remap"
inputs = ["http_source","tcp_source","udp_source"]
file = "/logEventValidator.vrl"
drop_on_error = true
drop_on_abort = true


[sinks.forward]
type = "vector"
inputs = ["format","nonFormat"]
address = "upstream:50335"
version = "2"

With this running configuration, I have so far been able to acquire and forward all the events. But recently ran into an issue with the unix socket.
The events that are being sent to the agent are quite big ~8KB. I was told by the person utilising this configuration that they are seeing these error memory allocation of 4194304 bytes failed error ( I did not get a chance to look at the exact message and where it was popping up).
To recreate this, I started pumping in 8KB sized event every 10 ms to the vector agent. I was not able to see any issues with it. Then when I brought this time gap to 1 ms. ie. send 8KB sized event every 1 ms. With these after some time, I am seeing these errors being logged.

2023-09-15T08:40:47.662096Z ERROR sink{component_kind="sink" component_id=forward component_type=vector component_name=forward}:request{request_id=56469}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] has been suppressed 6 times.
2023-09-15T08:40:47.662156Z ERROR sink{component_kind="sink" component_id=forward component_type=vector component_name=forward}:request{request_id=56469}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=Request failed: status: OutOfRange, message: "Error, message length too large: found 8108587 bytes, the limit is: 4194304 bytes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Fri, 15 Sep 2023 08:40:47 GMT", "content-length": "0"} } internal_log_rate_limit=true

Once we get into this situation, no more events are sent through to the collector unless we restart the vector agent.
I wanted to understand a bit more about this error and see why this is happening? would anyone know if memory allocation error has ever been seen and if the memory allocation error and what I have put in above are the same?

[jszwedko]

Hi @bmallya !

The error you are seeing seems to be related to #17926 which was fixed in v0.32.1.

memory allocation of 4194304 bytes failed error sounds a lot like it ran out of memory.

[bmallya]

Thanks @jszwedko . I understand that there was a memory allocation error, but I am trying to understand how it could have happened. The message size being sent is around 8KB where as #17926 talks about payload greater than 4MB.
In the case of unix socket used as a source (or in general) is there a buffering that happens before the events are sent out? Does the buffer parameter get initialised to some default values and get used?

[jszwedko]

The error was reported by the vector sink when it attempted to forward data downstream to another Vector instance. In this case the Vector instance running the vector source responded to the incoming request with the "payload too big error" which is what caused the vector sink to log that error.

The vector sink batches together events in requests. By default it'll batch 1000 events, which for 8k events would create payloads of 8 MB.

[bmallya]

Thanks @jszwedko, Let me upgrade to v32.0 and see if the issue that I am seeing goes away