axosyslog-4.13.0

4.13.0

AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.

We provide cloud-ready container images and Helm charts.

Packages are available in our APT and RPM repositories (Ubuntu, Debian, AlmaLinux, Fedora).

Check out the AxoSyslog documentation for all the details.

License

The licensing of AxoSyslog has been simplified and upgraded from a combination of LGPL-2.1-or-later and
GPL-2.0-or-later to GPL-3.0-or-later.

As before, Contributory Licensing Agreements (CLAs) are NOT required to contribute to AxoSyslog: contributors retain
their own copyright, making AxoSyslog a combination of code from hundreds of individuals and companies.
This, and the use of GPL v3 ensures that AxoSyslog or AxoSyslog derived code cannot become proprietary software.

While this has basically no impact on users of AxoSyslog, it reflects a step towards a more open and more
community-friendly project. Read more here

FilterX features

• format_xml() and format_windows_eventlog_xml(): new functions added

  Example usage:

  $MSG = format_xml({"a":{"b":"foo"}});
  

  (#684)

• protobuf_message(): Added a new function to create arbitrary protobuf data

  Usage:

  protobuf_data = protobuf_message(my_dict, schema_file="my_schema_file.proto");
  

  (#678)

• clickhouse(), bigquery() destination: Added proto-var() option

  This option can be used to send a FilterX prepared protobuf payload.
  (#678)

• format_cef(), format_leef(): Added new functions for CEF and LEEF formatting
  (#690)

• parse_cef(), parse_leef(): Extensions are no longer put under the extensions inner dict

  By default now they get placed on the same level as the headers.
  The new separate_extensions=true argument can be used for the
  old behavior.
  (#690)

FilterX bugfixes

• Fixed some FilterX evaluation error messages being printed to stderr.
  (#654)

• parse_cef(), parse_leef(): Fixed some failed parsing around escaped delimiters
  (#699)

[1] syslog-ng is a trademark of One Identity.

Discord

For a bit more interactive discussion, join our Discord server:

Credits

AxoSyslog is developed as a community project, and as such it relies
on volunteers, to do the work necessary to produce AxoSyslog.

Reporting bugs, testing changes, writing code or simply providing
feedback is an important contribution, so please if you are a user
of AxoSyslog, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Attila Szakacs, Balazs Scheidler, László Várady, Szilard Parrag,
Tamás Kosztyu, shifter

axosyslog-charts-0.15.0

AxoSyslog for Kubernetes

axosyslog-4.12.0

4.12.0

AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.

We provide cloud-ready container images and Helm charts.

Packages are available in our APT and RPM repositories (Ubuntu, Debian, AlmaLinux, Fedora).

Check out the AxoSyslog documentation for all the details.

License

The licensing of AxoSyslog has been simplified and upgraded from a combination of LGPL-2.1-or-later and GPL-2.0-or-later to GPL-3.0-or-later.

As before, Contributory Licensing Agreements (CLAs) are NOT required to contribute to AxoSyslog: contributors retain their own copyright, making AxoSyslog a combination of code from hundreds of individuals and companies.
This, and the use of GPL v3 ensures that AxoSyslog or AxoSyslog derived code cannot become proprietary software.

While this has basically no impact on users of AxoSyslog, it reflects a step towards a more open and more community-friendly project. Read more here

Features

• clickhouse() destination: Added server-side-schema() option.

  Sending data to ClickHouse in Protobuf format has some limitations.
  The Protobuf formatted data does not carry type information,
  so there are two ways of determining the type mapping in ClickHouse:

  1. Using autogenerated schema: https://clickhouse.com/docs/interfaces/formats/Protobuf#using-autogenerated-protobuf-schema

     This method looks at the table that the data is getting inserted
     into and generates the default protobuf mapping for it.

     The problem with this solution is that some more complex column
     types can receive data from different input types, for example
     DateTime64 can recieve from uint64, string, float, but with the
     autogenerated schema you can utilize only one of them.

  2. Using format schema: https://clickhouse.com/docs/interfaces/formats#formatschema

     This method solves the issue of the autogenerated schema, but
     needs a more complex setup. First you need to place a .proto
     file along your server in a specific directory, then you can
     reference that schema during insert, which is done by the
     server-side-schema() option.

  Example: server-side-schema("my_proto_file_on_server:my_message_schema_name")
  (#666)

• log-flow-control(yes/no) global option

  This option allows enabling flow control for all log paths. When set to yes,
  flow control is globally enabled, but it can still be selectively disabled
  within individual log paths using the no-flow-control flag.

  WARNING: Enabling global flow control can cause the system() source to block.
  As a result, if messages accumulate at the destination, applications that log
  through the system may become completely stalled, potentially halting their
  operation. We don't recommend enabling flow control in log paths that
  include the system() source.

  For example,

  options {
    log-flow-control(yes);
  };
  
  log {
    source { system(); };
    destination { network("server" port(5555)); };
    flags(no-flow-control);
  };
  
  log { ... };
  

  (#606)

FilterX features

• Failure information tracking

  The following functions have been added to allow tracking failures in FilterX code:

  • failure_info_enable(), optional parameter: collect_falsy=true/false, defaults to false:
    Enable failure information collection from this point downwards through all branches of the pipeline.

  • failure_info_clear():
    Clear accumulated failure information

  • failure_info_meta({}):
    Attach metadata to the given section of FilterX code. The metadata remains in effect until the next call
    or until the end of the enclosing FilterX block, whichever comes first.

  • failure_info():
    Return failure information as a FilterX dictionary. This should ideally be called as late as possible, for example, in the last log path of your configuration or within a fallback path.

  Example output:

  [
    {
      "meta": {
        "step": "Setting common fields"
      },
      "location": "/etc/syslog-ng/syslog-ng.conf:33:7",
      "line": "nonexisting.key = 13;",
      "error": "No such variable: nonexisting"
    }
  ]
  

  (#629)

• - * / %: Added new filterx arithmetic operators.

  Example usage:

  if (3 - 3 == 0)
  if (3.0 * 3 == 9.0)
  if (3.0 / 3.0 == 1.0)
  if (4 % 3 == 1)
  

  Note: A general + operator already exists. % only accepts integer values.
  (#625)

• in: Added new filterx operator for membership check.

  Example usage:

  s_arr = ['foo', 'bar', 'asd'];
  if ("foo" in s_arr)
  

  or

  s_arr = ['foo', 'bar', 'asd'];
  if ("bar" not in s_arr)
  

  (#617)

• get_timestamp(): query the timestamps of log messages
  (#586)

• strcasecmp(): case insensitive string comparison
  (#580)

• update_metric(): Labels containing null and "" (empry string) values are now skipped.
  (#671)

Bugfixes

• Fixed some time parsing and time formatting issues.
  (#626)

• network(), syslog() destinations: handle async TLS messages (KeyUpdate, etc.)
  (#609)

• rate-limit(): fix precision issue that could occur at a very low message rate
  (#599)

• rewrite: fix not creating empty values for non-matching regexp optional match groups
  (#669)

• regexp-parser(): fix double-free with invalid configuration
  (#670)

• metrics: fix syslogng_last_config_file_modification_timestamp_seconds
  (#612)

• pubsub-grpc() destination: Fixed message size counting.
  (#652)

• collectd(): fix not reading server responses
  (#609)

• network()/syslog() TLS destinations: fix a possible infinite looping issue
  (#615)

FilterX bugfixes

• metrics_labels: Fixed a crash that occurred when trying to get a label from an empty object.
  (#601)

• regexp_*(): fix not creating empty values for non-matching optional match groups
  (#669)

• cache_json_file(): fix updating json content on file changes
  (#612)

Other changes

• loggen: statistics output has slightly changed

  The new --perf option can be used to measure the log throughput of AxoSyslog.
  (#598)

• stats(): freq() now defaults to 0

  Internal statistic messages were produced every 10 minutes by default.
  Metrics are available through syslog-ng-ctl, we believe modern monitoring and
  observability render this periodic message obsolete.
  (#600)

[1] syslog-ng is a trademark of One Identity.

Discord

For a bit more interactive discussion, join our Discord server:

Credits

AxoSyslog is developed as a community project, and as such it relies
on volunteers, to do the work necessary to produce AxoSyslog.

Reporting bugs, testing changes, writing code or simply providing
feedback is an important contribution, so please if you are a user
of AxoSyslog, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Attila Szakacs, Balazs Scheidler, Bálint Horváth,
Christian Heusel, Eli Schwartz, Franco Fichtner, Hofi,
Kovacs, Gergo Ferenc, László Várady, Mate Ory, Tamás Kosztyu, shifter

axosyslog-charts-0.14.0

AxoSyslog for Kubernetes

axosyslog-4.11.0

4.11.0

AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.

We provide cloud-ready container images and Helm charts.

Packages are available in our APT and RPM repositories (Ubuntu, Debian, AlmaLinux, Fedora).

Check out the AxoSyslog documentation for all the details.

Features

• webhook(): headers support

  include-request-headers(yes) stores request headers under the ${webhook.headers} key,
  allowing further processing, for example, in FilterX:

  filterx {
    headers = json(${webhook.headers});
    $type = headers["Content-Type"][-1];
  };
  

  proxy-header("x-forwarded-for") helps retain the sender's original IP and the proxy's IP address
  ($SOURCEIP, $PEERIP).
  (#524)

• network(), syslog() sources: add $PEERIP and $PEERPORT macros

  The $PEERIP and $PEERPORT macros always display the address and port of the direct sender.
  In most cases, these values are identical to $SOURCEIP and $SOURCEPORT.
  However, when dealing with proxied protocols, $PEERIP and $PEERPORT reflect the proxy's address and port,
  while $SOURCEIP and $SOURCEPORT indicate the original source of the message.
  (#523)

• gRPC based destinations: Added response-action() option

  With this option, it is possible to fine tune how AxoSyslog
  behaves in case of different gRPC results.

  Supported by the following destination drivers:

  • opentelemetry()
  • loki()
  • bigquery()
  • clickhouse()
  • google-pubsub-grpc()

  Supported gRPC results:

  • ok
  • unavailable
  • cancelled
  • deadline-exceeded
  • aborted
  • out-of-range
  • data-loss
  • unknown
  • invalid-argument
  • not-found
  • already-exists
  • permission-denied
  • unauthenticated
  • failed-precondition
  • unimplemented
  • internal
  • resource-exhausted

  Supported actions:

  • disconnect
  • drop
  • retry
  • success

  Usage:

  google-pubsub-grpc(
    project("my-project")
    topic("my-topic")
    response-action(
      not-found => disconnect
      unavailable => drop
    )
  );
  

  (#561)

FilterX features

• set_pri(): Added new filterx function to set the message priority value.

  Example usage:

  set_pri(pri=100);
  

  Note: Second argument must be between 0 and 191 inclusive.
  (#521)

• set_timestamp(): Added new filterx function to set the message timestamps.

  Example usage:

  set_timestamp(datetime, stamp="stamp");
  

  Note: Second argument can be "stamp" or "recvd", based on the timestamp to be set.
  Default is "stamp".
  (#510)

• cache_json_file(): inotify-based reloading of JSON file
  (#517)

FilterX bugfixes

• switch: Fixed a crash that occurred when the selector or case failed to evaluate.
  (#527)

Notes to developers

• editorconfig: configure supported editors for the project's style
  (#550)

• We have clarified the meaning of the required "Signed-off-by" line in commit messages
  (CONTRIBUTING.md)

• Light, AxoSyslog's lightweight end-to-end testing framework is available as a PyPi package:
  https://pypi.org/project/axosyslog-light/

  It allows you to extend the framework and the test suite out-of-tree (licensed under GPL-2.0-or-later).

Other changes

• azure-monitor(): unified destination

  The azure-monitor-builtin() and azure-monitor-custom() destinations are deprecated in favor of azure-monitor().
  azure-monitor() requires the stream-name() option instead of the table name.
  (#531)

Discord

For a bit more interactive discussion, join our Discord server:

Credits

AxoSyslog is developed as a community project, and as such it relies
on volunteers, to do the work necessary to produce AxoSyslog.

Reporting bugs, testing changes, writing code or simply providing
feedback is an important contribution, so please if you are a user
of AxoSyslog, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Attila Szakacs, Balazs Scheidler, David Mandelberg,
Hofi, Janos Szigetvari, László Várady, Szilard Parrag, Tamás Kosztyu,
shifter

axosyslog-4.10.1

4.10.1

These are news entries of AxoSyslog 4.10.0.
4.10.1 fixed two crashes related to FilterX strings and JSON objects.

AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.

Explore and learn more about the new features in our release announcement blog post.

We provide cloud-ready container images and Helm charts.

Packages are available in our APT and RPM
repositories (Ubuntu, Debian, AlmaLinux, Fedora).

Check out the AxoSyslog documentation for all the details.

Highlights

Google Pub/Sub gRPC destination

Sending logs to Google Pub/Sub via the gRPC interface.

Example config:

google-pubsub-grpc(
  project("my_project")
  topic($topic)

  data($MESSAGE)
  attributes(
    timestamp => $S_ISODATE,
    host => $HOST,
  )

  workers(4)
  batch-timeout(1000) # ms
  batch-lines(1000)
);

The project() and topic() options are templatable.
The default service endpoint can be changed with the service_endpoint() option.

(#373)

Azure Monitor destination

Sending logs to Azure Monitor using OAuth 2 authentication.

Example config:

azure-monitor-custom(
  table-name("table")
  dcr-id("dcr id")
  dce-uri("https://dce-uri.ingest.monitor.azure.com")

  auth(tenant-id("tenant id") app-id("app id") app-secret("app secret"))

  workers(4)
  batch_timeout(1000) # ms
  batch_lines(5000)
  batch_bytes(4096KiB)
);

Note: Table name should not contain the trailing "_CL" string for custom tables.

(#457)

Features

• syslog() source driver: add support for RFC6587 style auto-detection of
  octet-count based framing to avoid confusion that stems from the sender
  using a different protocol to the server. This behaviour can be enabled
  by using transport(auto) option for the syslog() source.
  (#4814)

• syslog(transport(proxied-*)) and network(transport(proxied-*)): changed
  where HAProxy transport saved the original source and destination addresses.
  Instead of using dedicated PROXIED_* name-value pairs, use the usual
  $SOURCEIP, $SOURCEPORT, $DESTIP and $DESTPORT macros, making haproxy
  based connections just like native ones.

  $SOURCEPORT: added new macro which expands to the source port of the peer.
  (#361)

• check-program: Introduced as a flag for global or source options.

  By default, this flag is set to false. Enabling the check-program flag triggers program name validation for RFC3164 messages. Valid program names must adhere to the following criteria:

  Contain only these characters: [a-zA-Z0-9-_/().]
  Include at least one alphabetical character.
  If a program name fails validation, it will be considered part of the log message.

  Example:

  source { network(flags(check-hostname, check-program)); };
  

  (#380)

• s3 destination: Added content-type() option.
  (#408)

• bigquery(), google-pubsub-grpc(): Added service-account() authentication option

  Example usage:

  destination {
      google-pubsub-grpc(
          project("test")
          topic("test")
          auth(service-account(key ("path_to_service_account_key.json")))
      );
  };
  

  Note: In contrary to the http() destination's similar option,
  we do not need to manually set the audience here as it is
  automatically recognized by the underlying gRPC API.
  (#412)

• metrics: add syslogng_stats_level metric to monitor the current metric verbosity level
  (#493)

• webhook(),opentelemetry() sources: support input_event_bytes metrics
  (#494)

Bugfixes

• network(), syslog() sources and destinations: fix TCP/TLS shutdown
  (#420)

• network(), syslog(): Fixed a potential crash for TLS destinations during reload

  In case of a TLS connection, if the handshake didn't happen before reloading AxoSyslog,
  it crashed on the first message sent to that destination.
  (#418)

• axosyslog-otlp() destination: Fixed a crash.
  (#384)

• http: Fixed a batching related bug that happened with templated URLs and a single worker.
  (#464)

Other changes

• Crash report (backtrace) on x86-64 and ARM-based Linux systems
  (#350)

• FilterX and log path information for perf stackdumps
  (#433)

FilterX features

• FilterX performance improvements
  (#253, #257, #258, #330, #365, #385, #390, #395, #396, #397, #400, #421, #426, #428, #429, #430, #432, #436, #437, #446, #448, #452, #453, #467, #468, #469, #470, #471, #472, #473, #474, #476, #491)

• strftime(): Added new filterx function to format datetimes.

  Example usage:

  $MSG = strftime("%Y-%m-%dT%H:%M:%S %z", datetime);
  

  Note: %Z currently does not respect the datetime's timezone,
  usage of %z works as expected, and advised.
  (#402)

• keys(): Add keys Function to Retrieve Top-Level Dictionary Keys

  This feature introduces the keys function, which returns the top-level keys of a dictionary. It provides a simple way to inspect or iterate over the immediate keys without manually traversing the structure.

  • Returns an Array of Keys: Provides a list of dictionary keys as an array.
  • Current Level Only: Includes only the top-level keys, ignoring nested structures.
  • Direct Index Access: The resulting array supports immediate indexing for quick key retrieval.

  Example:

      dict = {"foo":{"bar":{"baz":"foobarbaz"}},"tik":{"tak":{"toe":"tiktaktoe"}}};
      # empty dictionary returns []
      empty = keys(json());
  
      # accessing the top level results ["foo", "tik"]
      a = keys(dict);
  
      # acccessing nested levels directly results ["bar"]
      b = keys(dict["foo"]);
  
      # directly index the result of keys() to access specific keys is possible (returns ["foo"])
      c = keys(dict)[0];

  (#435)

• Added support for switch cases.

  This syntax helps to organize the code for multiple
  if, elif, else blocks and also improves
  the branch finding performance.

  Cases with literal string targets are stored in a map,
  and the lookup is started with them.

  Other case targets can contain any expressions,
  and they are evaluated in order.

  Please note that although literal string and default
  target duplications are checked and will cause init failure,
  non-literal expression targets are not checked, and only
  the first maching case will be executed.

  Example config:

  switch ($MESSAGE) {
    case "foobar":
      $MESSAGE = "literal-case";
      break;
    case any_expression:
      $MESSAGE = "variable-case";
      break;
    default:
      $MESSAGE = "default";
      break;
  };
  

  (#473)

• vars(): add exclude_msg_values parameter
  (#505)

• vars(): $ is now prepended for the names of message variables.
  (...

axosyslog-charts-0.13.0

AxoSyslog for Kubernetes

axosyslog-4.10.0

4.10.0

AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.

Explore and learn more about the new features in our release announcement blog post.

We provide cloud-ready container images and Helm charts.

Packages are available in our APT and RPM
repositories (Ubuntu, Debian, AlmaLinux, Fedora).

Check out the AxoSyslog documentation for all the details.

Highlights

Google Pub/Sub gRPC destination

Sending logs to Google Pub/Sub via the gRPC interface.

Example config:

google-pubsub-grpc(
  project("my_project")
  topic($topic)

  data($MESSAGE)
  attributes(
    timestamp => $S_ISODATE,
    host => $HOST,
  )

  workers(4)
  batch-timeout(1000) # ms
  batch-lines(1000)
);

The project() and topic() options are templatable.
The default service endpoint can be changed with the service_endpoint() option.

(#373)

Azure Monitor destination

Sending logs to Azure Monitor using OAuth 2 authentication.

Example config:

azure-monitor-custom(
  table-name("table")
  dcr-id("dcr id")
  dce-uri("https://dce-uri.ingest.monitor.azure.com")

  auth(tenant-id("tenant id") app-id("app id") app-secret("app secret"))

  workers(4)
  batch_timeout(1000) # ms
  batch_lines(5000)
  batch_bytes(4096KiB)
);

Note: Table name should not contain the trailing "_CL" string for custom tables.

(#457)

Features

• syslog() source driver: add support for RFC6587 style auto-detection of
  octet-count based framing to avoid confusion that stems from the sender
  using a different protocol to the server. This behaviour can be enabled
  by using transport(auto) option for the syslog() source.
  (#4814)

• syslog(transport(proxied-*)) and network(transport(proxied-*)): changed
  where HAProxy transport saved the original source and destination addresses.
  Instead of using dedicated PROXIED_* name-value pairs, use the usual
  $SOURCEIP, $SOURCEPORT, $DESTIP and $DESTPORT macros, making haproxy
  based connections just like native ones.

  $SOURCEPORT: added new macro which expands to the source port of the peer.
  (#361)

• check-program: Introduced as a flag for global or source options.

  By default, this flag is set to false. Enabling the check-program flag triggers program name validation for RFC3164 messages. Valid program names must adhere to the following criteria:

  Contain only these characters: [a-zA-Z0-9-_/().]
  Include at least one alphabetical character.
  If a program name fails validation, it will be considered part of the log message.

  Example:

  source { network(flags(check-hostname, check-program)); };
  

  (#380)

• s3 destination: Added content-type() option.
  (#408)

• bigquery(), google-pubsub-grpc(): Added service-account() authentication option

  Example usage:

  destination {
      google-pubsub-grpc(
          project("test")
          topic("test")
          auth(service-account(key ("path_to_service_account_key.json")))
      );
  };
  

  Note: In contrary to the http() destination's similar option,
  we do not need to manually set the audience here as it is
  automatically recognized by the underlying gRPC API.
  (#412)

• metrics: add syslogng_stats_level metric to monitor the current metric verbosity level
  (#493)

• webhook(),opentelemetry() sources: support input_event_bytes metrics
  (#494)

Bugfixes

• network(), syslog() sources and destinations: fix TCP/TLS shutdown
  (#420)

• network(), syslog(): Fixed a potential crash for TLS destinations during reload

  In case of a TLS connection, if the handshake didn't happen before reloading AxoSyslog,
  it crashed on the first message sent to that destination.
  (#418)

• axosyslog-otlp() destination: Fixed a crash.
  (#384)

• http: Fixed a batching related bug that happened with templated URLs and a single worker.
  (#464)

Other changes

• Crash report (backtrace) on x86-64 and ARM-based Linux systems
  (#350)

• FilterX and log path information for perf stackdumps
  (#433)

FilterX features

• FilterX performance improvements
  (#253, #257, #258, #330, #365, #385, #390, #395, #396, #397, #400, #421, #426, #428, #429, #430, #432, #436, #437, #446, #448, #452, #453, #467, #468, #469, #470, #471, #472, #473, #474, #476, #491)

• strftime(): Added new filterx function to format datetimes.

  Example usage:

  $MSG = strftime("%Y-%m-%dT%H:%M:%S %z", datetime);
  

  Note: %Z currently does not respect the datetime's timezone,
  usage of %z works as expected, and advised.
  (#402)

• keys(): Add keys Function to Retrieve Top-Level Dictionary Keys

  This feature introduces the keys function, which returns the top-level keys of a dictionary. It provides a simple way to inspect or iterate over the immediate keys without manually traversing the structure.

  • Returns an Array of Keys: Provides a list of dictionary keys as an array.
  • Current Level Only: Includes only the top-level keys, ignoring nested structures.
  • Direct Index Access: The resulting array supports immediate indexing for quick key retrieval.

  Example:

      dict = {"foo":{"bar":{"baz":"foobarbaz"}},"tik":{"tak":{"toe":"tiktaktoe"}}};
      # empty dictionary returns []
      empty = keys(json());
  
      # accessing the top level results ["foo", "tik"]
      a = keys(dict);
  
      # acccessing nested levels directly results ["bar"]
      b = keys(dict["foo"]);
  
      # directly index the result of keys() to access specific keys is possible (returns ["foo"])
      c = keys(dict)[0];

  (#435)

• Added support for switch cases.

  This syntax helps to organize the code for multiple
  if, elif, else blocks and also improves
  the branch finding performance.

  Cases with literal string targets are stored in a map,
  and the lookup is started with them.

  Other case targets can contain any expressions,
  and they are evaluated in order.

  Please note that although literal string and default
  target duplications are checked and will cause init failure,
  non-literal expression targets are not checked, and only
  the first maching case will be executed.

  Example config:

  switch ($MESSAGE) {
    case "foobar":
      $MESSAGE = "literal-case";
      break;
    case any_expression:
      $MESSAGE = "variable-case";
      break;
    default:
      $MESSAGE = "default";
      break;
  };
  

  (#473)

• vars(): add exclude_msg_values parameter
  (#505)

• vars(): $ is now prepended for the names of message variables.
  (#393)

• regex_search(): Function Reworked

  The `regex_sear...

axosyslog-charts-0.12.0

AxoSyslog for Kubernetes

axosyslog-4.9.0

4.9.0

AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.

Explore and learn more about the new features in our release announcement blog post.

We provide cloud-ready container images and Helm charts.

Packages are available for Debian and Ubuntu from our APT repository.
RPM packages are available in the Assets section (we’re working on an RPM repository as well, and hope to have it up and running for the next release).

FilterX (AxoSyslog's advanced parsing and filtering language) became a publicly available feature in AxoSyslog after the 4.8 release.
As it is currently under heavy development, FilterX related news entries can be found in separate sections.
Please note that although its syntax approaches its final form, it may break in subsequent releases.

Check out the AxoSyslog documentation for all the details.

Highlights

Sending data to ClickHouse

The new clickhouse() destination uses ClickHouse's gRPC
interface to insert logs.

Please note, that as of today, ClickHouse Cloud does not support
the gRPC interface. The clickhouse() destination is currently
only useful for self hosted ClickHouse servers.

If you would like to send logs to ClickHouse Cloud, gRPC support
can be requested from the ClickHouse Cloud team or a HTTP based
driver can be implemented in AxoSyslog.

Example config:

clickhouse(
  database("default")
  table("my_first_table")
  user("default")
  password("pw")
  schema(
    "user_id" UInt32 => $R_MSEC,
    "message" String => "$MSG",
    "timestamp" DateTime => "$R_UNIXTIME",
    "metric" Float32 => 3.14
  )
  workers(4)
  batch-lines(1000)
  batch-timeout(1000)
);

(#354)

Features

• opentelemetry(), loki() destination: Added support for templated header() values.
  (#334)

• opentelemetry(), axosyslog-otlp(): Added keep-alive() options.

  Keepalive can be configured with the time(), timeout()
  and max-pings-without-data() options of the keep-alive() block.

  opentelemetry(
      ...
      keep-alive(time(20000) timeout(10000) max-pings-without-data(0))
  );
  

  (#276)

• bigquery(): Added auth() options.

  Similarly to other gRPC based destination drivers, the bigquery()
  destination now accepts different authentication methods, like
  adc(), alts(), insecure() and tls().

  bigquery (
      ...
      auth(
          tls(
              ca-file("/path/to/ca.pem")
              key-file("/path/to/key.pem")
              cert-file("/path/to/cert.pem")
          )
      )
  );
  

  (#276)

• loki(): Added batch-bytes() and compression() options.
  (#276)

• socket based sources: Added a new option called idle-timeout().

  Setting this option makes AxoSyslog close the client connection
  if no data is received for the set amount of seconds.
  (#355)

• socket based sources: Added new flag, called exit-on-eof.

  Setting this flag to a source makes AxoSyslog stop,
  when EOF is received.
  (#351)

• syslog-ng-ctl: Added attach subcommand.

  With attach, it is possible to attach to the
  standard IO of the syslog-ng proccess.

  Example usage:

  # takes the stdio fds for 10 seconds and displays syslog-ng output in that time period
  $ syslog-ng-ctl attach stdio --seconds 10
  

  # steal trace level log messages for 10 seconds
  $ syslog-ng-ctl attach logs --seconds 10 --log-level trace
  

  (#326)

Bugfixes

• Config @version: Fixed compat-mode inconsistencies when @version
  was not specified at the top of the configuration file or was not specified at all.
  (#312)

• s3(): Eliminated indefinite memory usage increase for each reload.

  The increased memory usage is caused by the botocore library, which
  caches the session information. We only need the Session object, if
  role() is set. The increased memory usage still happens with that set,
  currently we only fixed the unset case.
  (#318)

• opentelemetry(), axosyslog-otlp() sources: Fixed source hang-up on flow-controlled paths.
  (#314)

• opentelemetry(), axosyslog-otlp() sources: Fixed a crash when workers() is set to > 1.
  (#310)

• file(), wildcard-file(): Fixed a crash and persist name collision issues.

  If multiple wildcard-file() sources or a wildcard-file() and a file() source were
  reading the same input file, it could result in log loss, log duplication, and various crashes.
  (#291)

• wildcard-file(): Fixed a crash that occurs after config reload when the source is flow-controlled.
  (#293)

• file(), stdout(): Fixed log sources getting stuck.

  Due to an acknowledgment bug in the file() and stdout() destinations,
  sources routed to those destinations may have gotten stuck as they were
  flow-controlled incorrectly.

  This issue occured only in extremely rare cases with regular files, but it
  occured frequently with /dev/stderr and other slow pseudo-devices.
  (#303)

• metrics: syslog-ng-ctl --reset will no longer reset Prometheus metrics
  (#370)

• stats: Fixed free_window counters.
  (#296)

FilterX features

• Added new filterx code flow controls.

  • drop: Drops the currently processed message and returns success.
  • done: Stops the processing and returns success.
    (#269)

• update_metric(): Added a new function similar to metrics-probe parser.

  Example usage:

  update_metric("filterx_metric", labels={"msg": $MSG, "foo": "foovalue"}, level=1, increment=$INCREMENT);
  

  (#220)

• startswith(), endswith(), includes(): Added string matching functions.

  • First argument is the string that is being matched.
  • Second argument is either a single substring or a list of substrings.
  • Optionally the ignorecase argument can be set to configure case sensitivity
    • default: false

  Example usage:

  startswith(string, prefix, ignorecase=false);
  startswith(string, [prefix_1, prefix_2], ignorecase=true);
  
  endswith(string, suffix, ignorecase=false);
  endswith(string, [suffix_1, suffix_2], ignorecase=true);
  
  includes(string, substring, ignorecase=false);
  includes(string, [substring_1, substring_2], ignorecase=true);
  

  (#297)

• parse_xml(): Added new function to parse XMLs.

  Example usage:

  my_structured_data = parse_xml(raw_xml);
  

  Converting XML to a dict is not standardized.

  Our intention is to create the most compact dict as possible,
  which means certain nodes will have different types and
  structures based on a number of different qualities of the
  input XML element.

  The following points will demonstrate the choices we made in our parser.
  In the examples we will use the JSON dict implementation.

  1. Empty XML elements become empty strings.

    XML:  <foo></foo>
    JSON: {"foo": ""}
  

  2. Attributions are stored in @attr key-value pairs,
     similarly to some other converters (e.g.: python xmltodict).

    XML:  <foo bar="123" baz="bad"/>
    JSON: {"foo": {"@bar": "123", "@baz": "bad"}}
  

  3. If an XML element has both attributes and a value,
     we need to store them in a dict, and the value needs a key.
     We store the text value under the #text key.

    XML:  <foo bar="123">baz</foo>
    JSON: {"foo": {"@bar": "123", "#text": "baz"}}
  

  4. An XML element can have both a value and inner elements.
     We use the #text key here, too.

    XML:  <foo>bar<baz>123</baz></foo>
    JSON: {"foo": {"#text": "bar", "baz": "123"}}
  

  5. An XML element can have multiple values separated by inner elements.
     In that case we concatenate the values.

    XML:  <foo>bar<a></a>baz</foo>
    JSON: {"foo": {"#text": "barbaz", "a": ""}}
  

  (#251)

• parse_windows_eventlog_xml(): Added a new function to parse Windows EventLog XMLs.

  This parser is really similar to parse_xml() with
  a couple of small differences:

  1. There is a quick schema validation.
  2. The Event->EventData field automatically handles named Data elements.
     (#282)

• parse_cef(), `parse_le...