Question on buffer.max_size when using "vector" as sink

[bmallya]

Hello,

I needed to understand how the buffering works. Correct me if my understanding is wrong, but I assumed that when buffering is turned on using buffer.max_size, the events will be written to the location pointed out by data_dir and once the batch age meets batch.timeout_secs, it will be flushed out. When this is done, the events will no longer be available in the local buffer. Is this correct?

This is the configuration that I have for the sink

[sinks.forward]
type = "vector"
inputs = ["validator"]
address = "is1:50335"
buffer.type = "disk"
buffer.max_size = 50000
#batch.max_bytes = 50000
buffer.when_full = "block"
batch.timeout_secs = 10
version = "2"

When an event is generated, I notice that the file size in the forward_id directory increases but once the timeout is met, the size does not decrease

[root@is1 forward_id]# ls -lrth
total 20K
-rw-r--r-- 1 root root    0 May 26 13:02 LOCK
-rw-r--r-- 1 root root   50 May 26 13:02 MANIFEST-000004
-rw-r--r-- 1 root root   60 May 26 13:02 LOG.old
-rw-r--r-- 1 root root   16 May 26 13:02 CURRENT
-rw-r--r-- 1 root root  181 May 26 13:02 LOG
-rw-r--r-- 1 root root 1.2K May 26 13:03 000005.log
[root@is1 forward_id]# date
Thu May 26 13:14:55 UTC 2022
(reverse-i-search)`dat': ^Cte
[root@is1 forward_id]# curl -XPOST -d '{"severity_text": "Error","severity_number": "5","name": "NetError","body": "ABC","flags":"A","trace_id": "xxxnd-548","span_id": "","resource": { "attributes":{"os.type":"linux","os.name":"fedora","device.type":"372","device.sub_type":"001"}}}' '127.0.0.1:50332/ot/log'
[root@is1 forward_id]# ls -lrth
total 20K
-rw-r--r-- 1 root root    0 May 26 13:02 LOCK
-rw-r--r-- 1 root root   50 May 26 13:02 MANIFEST-000004
-rw-r--r-- 1 root root   60 May 26 13:02 LOG.old
-rw-r--r-- 1 root root   16 May 26 13:02 CURRENT
-rw-r--r-- 1 root root  181 May 26 13:02 LOG
-rw-r--r-- 1 root root 1.5K May 26 13:15 000005.log
[root@is1 forward_id]# date
Thu May 26 13:15:16 UTC 2022 <<==At this point the event should have been moved out? ==
[root@is1 forward_id]# ls -lrth
total 20K
-rw-r--r-- 1 root root    0 May 26 13:02 LOCK
-rw-r--r-- 1 root root   50 May 26 13:02 MANIFEST-000004
-rw-r--r-- 1 root root   60 May 26 13:02 LOG.old
-rw-r--r-- 1 root root   16 May 26 13:02 CURRENT
-rw-r--r-- 1 root root  181 May 26 13:02 LOG
-rw-r--r-- 1 root root 1.5K May 26 13:15 000005.log
[root@is1 forward_id]#

I would like to ideally want to be using batch.max_size (with batch.max_size = buffer.max_size), so that we can send all the events together and then clear out the buffer
I understand that the event size in buffer might not be same as the one used in batch due to serialization. Is there a known overhead that we could possibly account for? say buffer.max_size = batch.max_size + X

[spencergilbert]

Hey @bmallya,

I needed to understand how the buffering works. Correct me if my understanding is wrong, but I assumed that when buffering is turned on using buffer.max_size, the events will be written to the location pointed out by data_dir and once the batch age meets batch.timeout_secs, it will be flushed out. When this is done, the events will no longer be available in the local buffer. Is this correct?

This is not correct, batching and buffering are not connected today. That is to say, when a disk buffer is enabled all events sent to the sink will be written to that buffer and persisted to disk.

Batching happens in memory and once the events are batched and successfully sent the buffer will ack that the events sent in the batch can be deleted but the deletion is not immediately done at this time. The delay between acknowledgment and actual deletion of the events in buffer is arbitrary.

disk_v2 (which will be the default starting in 0.22.0) will keep it's data files open until the entire file has been processed and acknowledged before closing the file (max size 128MB).

The original disk buffer uses leveldb and releases disk usage via compaction based on a ratio of un-compacted events to unprocessed events to buffer max size limit.

[bmallya]

Thanks for the clarification @spencergilbert. I have a couple of questions around using disk_v2

1. I know its currently in beta, but would you know when the release version will be available?
2. I am looking at a use case where I want to be writing X amount of data (events) into the disk buffer and once its filled, its flushed out and then the disk buffer is cleared up. With disc_v2, would I be able to achieve this? Or would it also depend on the batching behaviour that you have mentioned above?

[spencergilbert]

We're targeting Monday, May 30, for the 0.22.0 release which will have disk_v2 as the default as well as transparently migrate any v1 buffers to the new v2 buffer.

The batching behavior isn't changing with the buffer update - I don't believe we can support that particular use case today. We have a couple of issues open around creating batches on disk, which could be something that solves that for you: #10392 and #3829

[bmallya]

Thanks again @spencergilbert .

• For some reason, I have not seen any of the files under the forward_id directory being reduced (Even after quite some time of the events being sent successfully).
• Is there a way to check how we are honouring the buffer.max_size? Say I have set buffer.max_size as 10KB, and batch size say 50KB (or even batch.timeout_secs to 20 mins), would I be able to check the disk buffer is not going beyond the max 10KB?
• With v22.0, would I be able to check the same as above?

[spencergilbert]

We expose a number of metrics for buffers that you can generate with an internal_metrics source and expose via your preferred metrics sink. 0.22.0 will have the same set of metrics exposed, as far as I'm aware (also the release got bumped a day as we failed to remember Monday being a holiday for us US folks).

For some reason, I have not seen any of the files under the forward_id directory being reduced (Even after quite some time of the events being sent successfully).

Is this using disk or disk_v2?