Buffering to disk based on health status

[hemanthofficial3009]

Hello Team, We have a case where we cannot continuously write to disk for buffering the logs(being streamed out from http/tcp/udp) because of certain hardware limitations. To address this case we will need to persist logs in the disk space of the host system only when streaming is not possible i.e., when the health status of the down stream vector instance is bad. Can you suggest any solution that would come handy to address this case. Thanks

[tobz]

👋🏻

Just to understand the use case better: what's the limitation around not being able to continuously write to disk? Space, I/O, wear on the storage device, etc?

[hemanthofficial3009]

As we wanted to use this approach on a wide range of connected systems, the limitations on hardware would include Space (there are few systems with a very minimal space), wear of SD cards in few cases, limited read/write cycles

[tobz]

In the case where minimizing disk write activity overall is preferred, we do have an experimental feature you can try out: overflow buffers.

Normally, you would choose between either in-memory or disk buffers, and deal with the trade-offs of whichever one you picked, but overflow buffers allow you to compose a buffer topology consisting of multiple buffer stages. An additional when_full option called overflow instructs the buffer to overflow events to the next buffer in the topology if it cannot currently accept any more events.

This feature is still experimental, and we have some issues around actually documenting it (since it's a rather advanced feature), but here's an example of what it might look like to configure:

[sinks.http]
type = "http"
...

# This first stage of the buffer is an in-memory buffer that can hold 500 events. We set it to
# "overflow" into the disk buffer in the second stage rather than block or drop.
[[sinks.http.buffer]]
type = "memory"
max_events = 500
when_full = "overflow"

# This second stage of the buffer is a disk buffer that can hold up to 1GiB. Once this stage fills
# up, though, we have to block.
[[sinks.http.buffer]]
type = "disk"
max_size = 1073741824
when_full = "block"

This configures the http sink to primarily use an in-memory buffer that should hold a maximum 500 events, and when that stage is full, events will temporarily flow to the disk stage, which is configured to hold a maximum of 1GiB. If both stages fill up, we still end up blocking and propagating backpressure... but crucially, we shouldn't write to the disk buffer under normal, steady-state workloads.

All of the same caveats apply to each stage of the buffer, such as the disk buffer still having files on disk even if all events have been drained and processed, and so on.

[zamazan4ik]

@tobz Thank you for the explanation. We have almost the same case: we want to store events on a disk, only if a downstream sink is not available (in our case - Vector sink). Since I did not find an information in the documentation, could you please update the documentation?

I can explain, why this case is not so rare in real life. In our case we have the following log topology: Haproxy -> Vector instance 1 -> Vector instance 2 -> File. We want to have an ability to update software on Vector instance 2 without a complete downtime for the whole system. In this case we decided simply configure large enough on-disk buffer on Vector instance 1. This buffer holds events when Vector instance 2 is on maintenance. When it will be ready to receive events again, Vector instance 1 will send all events from the buffer and clear it. And according to the current documentation about buffers, this scenario cannot be efficiently implemented.

By the way, one interesting moment about the buffers behaviour. Here about disk buffer I see the following:

Stores the sink’s buffer on disk. This is less performant, but durable. Data will not be lost between restarts. Will also hold data in memory to enhance performance.

What is the memory buffer size, when disk type is selected? Can I configure it somehow? E.g. what is going on, if my disk buffer size is 16Gib and free RAM is only 4Gib?

[zamazan4ik]

And one more note. Since disk buffer could be quite large - is there any optional built-in mechanism for compressing events in the buffer? It could help with saving memory/disk space during buffering at cost of some overhead on compression/decompression. Right now this scenario should be implemented only via OS-specific tooling (like zram or built-in FS file compression (like zstd-based compression in BTRFS)). But compression via 3rd party tooling could be less efficient since it has less context informtation which could be useful for more efficient compression.