Putting a upper limit on data FluenD ships to defined output

[sahil-sawhney]

Is there a way to limit the amount of logs sent by FluentD per day to the specified output (http for example).

So that FluentD stops sending, let's say, more than 10GB logs for a given day.

[daipom]

Normally, we need to adjust the amount of input data by setting Input pluings.

For example, in_tail has an option group to adjust it.

• https://docs.fluentd.org/input/tail#less-than-group-greater-than-section

If we limit the sending amount in Output pugins, the data will accumulate in the buffer of Fluentd.
It is not desirable.
So, it is unlikely to adjust it in Output plugin.

[sahil-sawhney]

Thanks @daipom for the direction, it's very helpful.

So does FluentD input stops forwarding data to output till the defined rate_period is reached? Or does it keep forwarding data till the limit is reached for a particular rate_period?

What I mean is, If I want to send 100K log lines per day. Would that be achievable by this? Or since rate_period is 1 day here, the data from the input will flow to the output only after the day, which would not be desirable.

[daipom]

So does FluentD input stops forwarding data to output till the defined rate_period is reached? Or does it keep forwarding data till the limit is reached for a particular rate_period?

It keeps reading data till the limit is reached.
So this would be close to what you need.
Please note that the periods are not aligned at the start of the day (00:00), and there seems to be a small bug that the limit is judged by the average number of lines collected for a group, not the total number. (#3535 (comment))

Finally, please note that group setting of in_tail is just an example.
If you use other Input plugins, you need to adjust their settings as well, but some plugins may not have such a setting.

[sahil-sawhney]

Thanks, @daipom for the details.

My input would be from fluentbit (forward). I was checking the docs but couldn't find the same concept as the "group" there. So with fluentbit as a source, is there a way to achieve the same use case?

[daipom]

in_forward?
Certainly, it doesn't have such a setting.
There doesn't seem to be an easy way in this case.

It is common to suppress the peak within a limited time, but suppressing the daily sending amount is not a common story for Fluentd, which is responsible for data transfer.

For example, what do you do with data that exceeds that limit?
If the average data flow exceeds that limit, the data must be discarded or it will overflow.

If you can discard the data exceeding that limit, you can use some Filter plugins.
For example, you can use filter-flowcounter to count up the data size in one day.