Skip to content

config-log-forwarding-3 #94412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: standalone-logging-docs-main
Choose a base branch
from
Open

config-log-forwarding-3 #94412

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions configuring/configuring-log-forwarding.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,11 @@ Administrators can configure the following types of filters:

include::modules/enabling-multi-line-exception-detection.adoc[leveloffset=+2]
include::modules/logging-http-forward.adoc[leveloffset=+2]
include::modules/cluster-logging-collector-log-forward-loki.adoc[leveloffset=+1]
include::modules/cluster-logging-collector-log-forward-kafka.adoc[leveloffset=+1]
include::modules/cluster-logging-collector-log-forward-syslog.adoc[leveloffset=+2]
// Cloudwatch docs
include::modules/cluster-logging-collector-log-forward-cloudwatch.adoc[leveloffset=+1]
include::modules/content-filter-drop-records.adoc[leveloffset=+2]
include::modules/audit-filtering.adoc[leveloffset=+2]
include::modules/input-spec-filter-labels-expressions.adoc[leveloffset=+2]
Expand Down
74 changes: 36 additions & 38 deletions modules/cluster-logging-collector-log-forward-cloudwatch.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
[id="cluster-logging-collector-log-forward-cloudwatch_{context}"]
= Forwarding logs to Amazon CloudWatch

You can forward logs to Amazon CloudWatch, a monitoring and log storage service hosted by Amazon Web Services (AWS). You can forward logs to CloudWatch in addition to, or instead of, the default log store.

To configure log forwarding to CloudWatch, you must create a `ClusterLogForwarder` custom resource (CR) with an output for CloudWatch, and a pipeline that uses the output.

.Procedure
Expand Down Expand Up @@ -33,47 +31,43 @@ $ oc apply -f cw-secret.yaml
+
[source,yaml]
----
apiVersion: logging.openshift.io/v1
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: <log_forwarder_name> <1>
namespace: <log_forwarder_namespace> <2>
name: <log_forwarder_name>
namespace: <log_forwarder_namespace>
spec:
serviceAccountName: <service_account_name> <3>
serviceAccountName:
name: <service_account_name> <1>
outputs:
- name: cw <4>
type: cloudwatch <5>
- name: cw <2>
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in v6 we require 3 or more chars for a name. Probably best to find a new example name.

type: cloudwatch <3>
cloudwatch:
groupBy: logType <6>
groupPrefix: <group prefix> <7>
region: us-east-2 <8>
secret:
name: cw-secret <9>
authentication:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indentation: authentication is a child of cloudwatch

awsAccessKey:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing authentication.type awsAccessKey

keyId:
key: <key>
secretName: cw-secret <4>
groupName: logType <5>
Copy link

@cahartma cahartma Jun 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This api has changed from groupBy to groupName, and is no longer an enumeration. It now requires a static value, or a template requiring a fallback field.

$ oc explain clf.spec.outputs.cloudwatch.groupName
...
DESCRIPTION:
    GroupName defines the strategy for grouping logstreams
    
    The GroupName can be a combination of static and dynamic values consisting
    of field paths followed by "\|\|" followed by another field path or a static
    value.
    
    A dynamic value is encased in single curly brackets "{}" and MUST end with a
    static fallback value separated with "\|\|".
    
    Static values can only contain alphanumeric characters along with dashes,
    underscores, dots and forward slashes.
    
    Example:
     1. foo-{.bar\|\|"none"}
     2. {.foo\|\|.bar\|\|"missing"}
     3.
    foo.{.bar.baz\|\|.qux.quux.corge\|\|.grault\|\|"nil"}-waldo.fred{.plugh\|\|"none"}

region: us-east-2 <6>
pipelines:
- name: infra-logs <10>
inputRefs: <11>
- name: infra-logs <7>
inputRefs: <8>
- infrastructure
- audit
- application
outputRefs:
- cw <12>
----
<1> In legacy implementations, the CR name must be `instance`. In multi log forwarder implementations, you can use any name.
<2> In legacy implementations, the CR namespace must be `openshift-logging`. In multi log forwarder implementations, you can use any namespace.
<3> The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the `openshift-logging` namespace.
<4> Specify a name for the output.
<5> Specify the `cloudwatch` type.
<6> Optional: Specify how to group the logs:
+
* `logType` creates log groups for each log type.
* `namespaceName` creates a log group for each application name space. It also creates separate log groups for infrastructure and audit logs.
* `namespaceUUID` creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.
<7> Optional: Specify a string to replace the default `infrastructureName` prefix in the names of the log groups.
<8> Specify the AWS region.
<9> Specify the name of the secret that contains your AWS credentials.
<10> Optional: Specify a name for the pipeline.
<11> Specify which log types to forward by using the pipeline: `application,` `infrastructure`, or `audit`.
<12> Specify the name of the output to use when forwarding logs with this pipeline.
- cw <9>
----
<1> The name of your service account.
<2> Specify a name for the output.
<3> Specify the `cloudwatch` type.
<4> Specify the name of the secret that contains your AWS credentials.
<5> Specify the strategy for grouping logstreams. The `GroupName` can be a combination of static and dynamic values consisting of field paths followed by "\|\|" followed by another field path or a static value. A dynamic value is encased in single curly brackets "{}" and must end with a static fallback value separated with `\|\|`.
<6> Specify the AWS region.
<7> Specify a name for the pipeline.
<8> Specify which log types to forward by using the pipeline: `application,` `infrastructure`, or `audit`.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same inputRef comment

<9> Specify the name of the output to use when forwarding logs with this pipeline.

. Create the CR object:
+
Expand Down Expand Up @@ -121,11 +115,12 @@ $ oc get ns/app -ojson | jq .metadata.uid
"794e1e1a-b9f5-4958-a190-e76a9b53d7bf"
----


In your `ClusterLogForwarder` custom resource (CR), you configure the `infrastructure`, `audit`, and `application` log types as inputs to the `all-logs` pipeline. You also connect this pipeline to `cw` output, which forwards the logs to a CloudWatch instance in the `us-east-2` region:

[source,yaml]
----
apiVersion: "logging.openshift.io/v1"
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: instance
Expand All @@ -135,10 +130,12 @@ spec:
- name: cw
type: cloudwatch
cloudwatch:
groupBy: logType
authentication:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indent all this, and it's missing authentication.type

awsAccessKey:
keyId:
key: <key>
secretName: cw-secret
region: us-east-2
secret:
name: cw-secret
pipelines:
- name: all-logs
inputRefs:
Expand All @@ -155,7 +152,7 @@ Each region in CloudWatch contains three levels of objects:
** log stream
*** log event


////
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jcantrill, i did not finf groupBy in the kafka output api:

FIELDS:
  authentication	<Object>
    sasl	<Object>
      mechanism	<string>
      password	<Object>
        key	<string> -required-
        secretName	<string> -required-
      username	<Object>
        key	<string> -required-
        secretName	<string> -required-
  brokers	<[]string>
  topic	<string>
  tuning	<Object>
    compression	<string>
    enum: none, snappy, zstd, lz4
    deliveryMode	<string>
    enum: AtLeastOnce, AtMostOnce
    maxWrite	<Object>
  url	<string>

Is this content still correct?

With `groupBy: logType` in the `ClusterLogForwarding` CR, the three log types in the `inputRefs` produce three log groups in Amazon Cloudwatch:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With groupName: {.log_type||"unknown"} in the...


[source,terminal]
Expand Down Expand Up @@ -290,3 +287,4 @@ $ aws --output json logs describe-log-groups | jq .logGroups[].logGroupName
----

The `groupBy` field affects the application log group only. It does not affect the `audit` and `infrastructure` log groups.
////
Copy link

@cahartma cahartma Jun 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any info in the lines above referencing groupBy is outdated and incorrect in v6. This example can probably be done away with completely.
For example, if you want UUID to show after the app namespaces, you would use groupName: my-cluster-7977k.{.namespace_id||"unknown} and place this in a pipeline for app logs only.

106 changes: 51 additions & 55 deletions modules/cluster-logging-collector-log-forward-kafka.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@
[id="cluster-logging-collector-log-forward-kafka_{context}"]
= Forwarding logs to a Kafka broker

You can forward logs to an external Kafka broker in addition to, or instead of, the default log store.

To configure log forwarding to an external Kafka instance, you must create a `ClusterLogForwarder` custom resource (CR) with an output to that instance, and a pipeline that uses the output. You can include a specific Kafka topic in the output or use the default. The Kafka output can use a TCP (insecure) or TLS (secure TCP) connection.

.Procedure
Expand All @@ -17,67 +15,66 @@ To configure log forwarding to an external Kafka instance, you must create a `Cl
+
[source,yaml]
----
apiVersion: logging.openshift.io/v1
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: <log_forwarder_name> <1>
namespace: <log_forwarder_namespace> <2>
name: <log_forwarder_name>
namespace: <log_forwarder_namespace>
spec:
serviceAccountName: <service_account_name> <3>
serviceAccountName:
name: <service_account_name> <1>
outputs:
- name: app-logs <4>
type: kafka <5>
url: tls://kafka.example.devlab.com:9093/app-topic <6>
secret:
name: kafka-secret <7>
- name: app-logs <2>
type: kafka <3>
kafka:
authentication:
sasl:
username:
key: <key>
secretName: kafka-secret <4>
url: tls://kafka.example.devlab.com:9093/app-topic <5>
- name: infra-logs
type: kafka
url: tcp://kafka.devlab2.example.com:9093/infra-topic <8>
kafka:
url: tcp://kafka.devlab2.example.com:9093/infra-topic <6>
- name: audit-logs
type: kafka
url: tls://kafka.qelab.example.com:9093/audit-topic
secret:
name: kafka-secret-qe
kafka:
authentication:
sasl:
username:
key: <key>
secretName: kafka-secret <6>
url: tls://kafka.qelab.example.com:9093/audit-topic
pipelines:
- name: app-topic <9>
inputRefs: <10>
- application
outputRefs: <11>
- app-logs
labels:
logType: "application" <12>
- name: infra-topic <13>
inputRefs:
- infrastructure
outputRefs:
- infra-logs
labels:
logType: "infra"
- name: audit-topic
inputRefs:
- audit
outputRefs:
- audit-logs
labels:
logType: "audit"
- name: app-topic
inputRefs: <7>
- application
outputRefs: <8>
- app-logs
- name: infra-topic <9>
inputRefs:
- infrastructure
outputRefs:
- infra-logs
- name: audit-topic
inputRefs:
- audit
outputRefs:
- audit-logs
----
<1> In legacy implementations, the CR name must be `instance`. In multi log forwarder implementations, you can use any name.
<2> In legacy implementations, the CR namespace must be `openshift-logging`. In multi log forwarder implementations, you can use any namespace.
<3> The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the `openshift-logging` namespace.
<4> Specify a name for the output.
<5> Specify the `kafka` type.
<6> Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the `tcp` (insecure) or `tls` (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.
<7> If you are using a `tls` prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain a `ca-bundle.crt` key that points to the certificate it represents. In legacy implementations, the secret must exist in the `openshift-logging` project.
<8> Optional: To send an insecure output, use a `tcp` prefix in front of the URL. Also omit the `secret` key and its `name` from this output.
<9> Optional: Specify a name for the pipeline.
<10> Specify which log types to forward by using the pipeline: `application,` `infrastructure`, or `audit`.
<11> Specify the name of the output to use when forwarding logs with this pipeline.
<12> Optional: String. One or more labels to add to the logs.
<13> Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:
<1> The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the `openshift-logging` namespace.
<2> Specify a name for the output.
<3> Specify the `kafka` type.
<4> If you are using a `tls` prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain a `ca-bundle.crt` key that points to the certificate it represents. In legacy implementations, the secret must exist in the `openshift-logging` project.
<5> Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the `tcp` (insecure) or `tls` (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.
<6> Optional: To send an insecure output, use a `tcp` prefix in front of the URL. Also omit the `secret` key and its `name` from this output.
<7> Specify which log types to forward by using the pipeline: `application,` `infrastructure`, or `audit`.
<8> Specify the name of the output to use when forwarding logs with this pipeline.
<9> Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:
** A name to describe the pipeline.
** The `inputRefs` is the log type to forward by using the pipeline: `application,` `infrastructure`, or `audit`.
** The `outputRefs` is the name of the output to use.
** Optional: String. One or more labels to add to the logs.

. Optional: To forward a single output to multiple Kafka brokers, specify an array of Kafka brokers as shown in the following example:
+
Expand All @@ -88,13 +85,12 @@ spec:
outputs:
- name: app-logs
type: kafka
secret:
name: kafka-secret-dev
kafka: <1>
brokers: <2>
kafka: #<1>
# ...
brokers: #<2>
- tls://kafka-broker1.example.com:9093/
- tls://kafka-broker2.example.com:9093/
topic: app-topic <3>
topic: app-topic #<3>
# ...
----
<1> Specify a `kafka` key that has a `brokers` and `topic` key.
Expand Down
71 changes: 36 additions & 35 deletions modules/cluster-logging-collector-log-forward-loki.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@
[id="cluster-logging-collector-log-forward-loki_{context}"]
= Forwarding logs to an external Loki logging system

You can forward logs to an external Loki logging system in addition to, or instead of, the default log store.

To configure log forwarding to Loki, you must create a `ClusterLogForwarder` custom resource (CR) with an output to Loki, and a pipeline that uses the output. The output to Loki can use the HTTP (insecure) or HTTPS (secure HTTP) connection.

.Prerequisites
Expand All @@ -20,51 +18,54 @@ To configure log forwarding to Loki, you must create a `ClusterLogForwarder` cus
+
[source,yaml]
----
apiVersion: logging.openshift.io/v1
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: <log_forwarder_name> <1>
namespace: <log_forwarder_namespace> <2>
name: <log_forwarder_name>
namespace: <log_forwarder_namespace>
spec:
serviceAccountName: <service_account_name> <3>
serviceAccountName:
name: <service_account_name> #<1>
outputs:
- name: loki-insecure <4>
type: "loki" <5>
url: http://loki.insecure.com:3100 <6>
loki:
tenantKey: kubernetes.namespace_name
- name: loki-insecure #<2>
type: loki #<3>
loki:
url: http://loki.insecure.com:3100 #<4>
labelKeys:
- kubernetes.labels.foo
- name: loki-secure <7>
type: "loki"
url: https://loki.secure.com:3100
secret:
name: loki-secret <8>
- kubernetes.labels.foo
tenantKey: kubernetes.namespace_name
- name: loki-secure #<5>
type: loki
loki:
tenantKey: kubernetes.namespace_name <9>
authentication:
token:
from: <secret_or_serviceAccount>
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indent under token

secret: #<6>
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this spec has changed in v6. The example should likely be authentication.token.from and be set to serviceAccount. Then the old tls info is now in its own spec, with key and configMap/secretName

key: <secret_key>
name: loki-secret
labelKeys:
- kubernetes.labels.foo <10>
- kubernetes.labels.foo #<7>
tenantKey: kubernetes.namespace_name #<8>
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is now a template and combination of static and dynamic values inside curly brackets (with a static fallback)

url: https://loki.secure.com:3100
pipelines:
- name: application-logs <11>
inputRefs: <12>
- name: my-pipeline
inputRefs: #<9>
- application
- audit
outputRefs: <13>
outputRefs: #<10>
- loki-secure
----
<1> In legacy implementations, the CR name must be `instance`. In multi log forwarder implementations, you can use any name.
<2> In legacy implementations, the CR namespace must be `openshift-logging`. In multi log forwarder implementations, you can use any namespace.
<3> The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the `openshift-logging` namespace.
<4> Specify a name for the output.
<5> Specify the type as `"loki"`.
<6> Specify the URL and port of the Loki system as a valid absolute URL. You can use the `http` (insecure) or `https` (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. Loki's default port for HTTP(S) communication is 3100.
<7> For a secure connection, you can specify an `https` or `http` URL that you authenticate by specifying a `secret`.
<8> For an `https` prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a `ca-bundle.crt` key that points to the certificates it represents. Otherwise, for `http` and `https` prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the `openshift-logging` project. For more information, see the following "Example: Setting a secret that contains a username and password."
<9> Optional: Specify a metadata key field to generate values for the `TenantID` field in Loki. For example, setting `tenantKey: kubernetes.namespace_name` uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section.
<10> Optional: Specify a list of metadata field keys to replace the default Loki labels. Loki label names must match the regular expression `[a-zA-Z_:][a-zA-Z0-9_:]*`. Illegal characters in metadata keys are replaced with `_` to form the label name. For example, the `kubernetes.labels.foo` metadata key becomes Loki label `kubernetes_labels_foo`. If you do not set `labelKeys`, the default value is: `[log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]`. Keep the set of labels small because Loki limits the size and number of labels allowed. See link:https://grafana.com/docs/loki/latest/configuration/#limits_config[Configuring Loki, limits_config]. You can still query based on any log record field using query filters.
<11> Optional: Specify a name for the pipeline.
<12> Specify which log types to forward by using the pipeline: `application,` `infrastructure`, or `audit`.
<13> Specify the name of the output to use when forwarding logs with this pipeline.
<1> The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the `openshift-logging` namespace.
<2> Specify a name for the output.
<3> Specify the type as `loki`.
<4> Specify the URL and port of the Loki system as a valid absolute URL. You can use the `http` (insecure) or `https` (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. Loki's default port for HTTP(S) communication is 3100.
<5> For a secure connection, you can specify an `https` or `http` URL that you authenticate by specifying a `secret`.
<6> For an `https` prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a `ca-bundle.crt` key that points to the certificates it represents. Otherwise, for `http` and `https` prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the `openshift-logging` project. For more information, see the following "Example: Setting a secret that contains a username and password."
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tls is now its own spec and no longer looks for specific keys

<7> Optional: Specify a metadata key field to generate values for the `TenantID` field in Loki. For example, setting `tenantKey: kubernetes.namespace_name` uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section.
<8> Optional: Specify a list of metadata field keys to replace the default Loki labels. Loki label names must match the regular expression `[a-zA-Z_:][a-zA-Z0-9_:]*`. Illegal characters in metadata keys are replaced with `_` to form the label name. For example, the `kubernetes.labels.foo` metadata key becomes Loki label `kubernetes_labels_foo`. If you do not set `labelKeys`, the default value is: `[log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]`. Keep the set of labels small because Loki limits the size and number of labels allowed. See link:https://grafana.com/docs/loki/latest/configuration/#limits_config[Configuring Loki, limits_config]. You can still query based on any log record field using query filters.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is now a template, so this needs to be rewritten.

<9> Specify which log types to forward by using the pipeline: `application,` `infrastructure`, or `audit`.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same inputRef comment

<10> Specify the name of the output to use when forwarding logs with this pipeline.

+
[NOTE]
====
Expand Down