-
Notifications
You must be signed in to change notification settings - Fork 11
Filter Buffer
This filter receives logs, stores them in redis, and sends them at the right moment to the right filter connected in output. Its main uses are, waiting a certain amount of time and send all logs received during this time (set interval to what you need and required_log_lines to 1), or waiting to have enough logs before sending them (set a low interval and required_log_lines to what you need), or you can make a mix of both (set interval and required_log_lines to what you need). Interval and required_log_lines are set in Buffer's config file, see Config section of this wiki page.
0x62756672
No specific dependencies
Example of darwin configuration for this filter :
Note that, for this filter, next_filter, nb_thread and cache_size are mandatory (for legacy) but have no impact on filter's behavior.
{
"buffer_1": {
"exec_path": "/path/to/darwin/build/darwin_buffer",
"config_file":"/path/to/filter.conf",
"output": "LOG",
"next_filter": "",
"nb_thread": 1,
"log_level": "DEBUG",
"cache_size": 0
}
}The config file is made of 3 parts: The redis config, input format and outputs.
Redis :
redis_socket_path: The redis server on which to store logs.
{
"redis_socket_path": "path/to/redis/server/socket.sock"
}Input Format :
input_format: An array of json objects, this array must contain all values needed by the output filters.
Each object contains:
- name: the name of the value
- type: the type of the value (string or int only for now)
For example, an input_format for a sofa connection would be: (see Sofa documentation)
{
"input_format": [
{"name": "ip", "type": "string"},
{"name": "hostname", "type": "string"},
{"name": "os", "type": "string"},
{"name": "proto", "type": "string"},
{"name": "port", "type": "string"}
]
}For each connection between Buffer Filter and another Filter, you need to provide to Buffer the same data than you would have provide to the according Filter. Except for Anomaly Filter.
For an Anomaly connection you need to provide the same data as for Tanomaly.
For example, an input_format for an anomaly connection would be: (see Tanomaly documentation)
{
"input_format": [
{"name": "net_src_ip", "type": "string"},
{"name": "net_dst_ip", "type": "string"},
{"name": "net_dst_port", "type": "string"},
{"name": "ip_proto", "type": "string"}
]
}Outputs :
outputs: An array of json objects, containing configs to connect to output filters.
Each object contains:
- filter_type: f + filter_name (for now: fanomaly or fsofa)
- filter_socket_path: the socket of the filter receiving the resulting output
- interval: the interval (in seconds) between two sending tries (sent only if there is more logs than required_log_lines)
- required_log_lines: the number of required logs before sending to output filter
-
redis_lists: an array of json objects containing a source and a list name (see Body section of this doc)
- source: a source name which will always be the first element of a log line (empty source in config will match all sources in body)
- name: the name of the redis storage associated to the source.
{
"outputs": [
{
"filter_type": "fanomaly",
"filter_socket_path": "path/to/output/anomaly/socket.sock",
"interval": 300,
"required_log_lines": 11,
"redis_lists": [{
"source": "source_1",
"name": "anomaly_list_1"
},
{
"source": "source_2",
"name": "anomaly_list_2"
}]
},
{
"filter_type": "fsofa",
"filter_socket_path": "path/to/output/sofa/socket.sock",
"interval": 200,
"required_log_lines": 15,
"redis_lists": [{
"source": "source_1",
"name": "sofa_list_1"
},
{
"source": "source_3",
"name": "sofa_list_3"
},
{
"source": "",
"name": "sofa_list_all"
}]
}
]
}In the above example, a log line from:
- source_1 will be stored in anomaly_list_1 AND sofa_list_1
- source_2 will be stored in anomaly_list_2
-
source_3 will be stored in sofa_list_3
Additionally, all log lines will end up in sofa_list_all.
Any filter linked to a buffer will get from a log line only the data that it needs to perform, not paying attention to other data in log line. The only error handling is made on the type of the data, the content inspection is left to the output filter.
You can have several filters in output, all of them having one or more source/name redis storage pair(s).
Here is a complete config file example:
{
"redis_socket_path": "/var/sockets/redis/redis.sock",
"input_format": [
{"name": "net_src_ip", "type": "string"},
{"name": "net_dst_ip", "type": "string"},
{"name": "net_dst_port", "type": "string"},
{"name": "ip_proto", "type": "string"},
{"name": "ip", "type": "string"},
{"name": "hostname", "type": "string"},
{"name": "os", "type": "string"},
{"name": "proto", "type": "string"},
{"name": "port", "type": "string"},
],
"outputs": [
{
"filter_type": "fanomaly",
"filter_socket_path": "/var/sockets/darwin/anomaly.sock",
"interval": 300,
"required_log_lines": 11,
"redis_lists": [{
"source": "source_1",
"name": "darwin_buffer_anomaly_source_1"
},
{
"source": "source_2",
"name": "darwin_buffer_anomaly_source_2"
}]
},
{
"filter_type": "fanomaly",
"filter_socket_path": "/var/sockets/darwin/anomaly_2.sock",
"interval": 300,
"required_log_lines": 11,
"redis_lists": [{
"source": "",
"name": "darwin_buffer_anomaly_all_sources"
}]
},
{
"filter_type": "fsofa",
"filter_socket_path": "/var/sockets/darwin/sofa.sock",
"interval": 100,
"required_log_lines": 15,
"redis_lists": [{
"source": "source_1",
"name": "darwin_buffer_sofa_1"
},
{
"source": "source_2",
"name": "darwin_buffer_sofa_2"
}]
}
]
}Unlike the other filters, the content of the body will change depending on the buffer's config.
If we take this input_format as an example:
{
"input_format": [
{"name": "net_src_ip", "type": "string"},
{"name": "net_dst_ip", "type": "string"},
{"name": "net_dst_port", "type": "string"},
{"name": "ip_proto", "type": "string"}
]
}The body would look like this:
[
["<source>", "<net_src_ip>", "<net_dst_ip">, <"net_dst_port">, "<ip_proto>"],
...
]
The order in the input_format has no importance as long as you keep the same order in the body.
Every line from body must start by the source and contain as many elements as in the config input_format part (+1 for source).
An empty source in the body will only trigger empty sources in output config.
Here is an example of a body:
[
["source_1", "127.0.0.1", "127.0.0.4", "22", "6"],
["source_1", "127.0.0.2", "127.0.0.5", "24", "6"],
["source_2", "127.0.0.1", "127.0.0.3", "22", "17"],
["", "127.0.0.2", "127.0.0.1", "21", "17"],
...
]
In a case where you have two different filters in output, you need to provide a complete log line that will be split between the different filters, according to what they need. The order in a body line is therefore really important as it is the only way to ensure that a data will end up in the correct filter.
In case of multiple sources for a same filter, each redis storage will be treated independently and data will be sent separately from buffer to filter. Multiple sources can be relevant if you want to treat with a single filter different data sets from different origins.
This filter returns 0 if the line was correctly parsed, and 101 if not.
It does not raise alerts.