config-log-forwarding-3 #94412

theashiot · 2025-06-07T16:39:34Z

Version(s):

Issue:

Link to docs preview:

QE review:

QE has approved this change.

Additional information:

openshift-ci · 2025-06-07T16:39:39Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

theashiot · 2025-06-07T16:39:50Z

/test all

ocpdocs-previewbot · 2025-06-07T16:55:35Z

🤖 Fri Jun 13 15:44:10 - Prow CI generated the docs preview:

https://94412--ocpdocs-pr.netlify.app/
https://94412--ocpdocs-pr.netlify.app/openshift-logging/latest/configuring/configuring-log-forwarding.html

jcantrill

/hold

theashiot · 2025-06-13T15:33:42Z

modules/cluster-logging-collector-log-forward-cloudwatch.adoc

@@ -155,7 +152,7 @@ Each region in CloudWatch contains three levels of objects:
 ** log stream
 *** log event

-
+////


@jcantrill, i did not finf groupBy in the kafka output api:

FIELDS: authentication <Object> sasl <Object> mechanism <string> password <Object> key <string> -required- secretName <string> -required- username <Object> key <string> -required- secretName <string> -required- brokers <[]string> topic <string> tuning <Object> compression <string> enum: none, snappy, zstd, lz4 deliveryMode <string> enum: AtLeastOnce, AtMostOnce maxWrite <Object> url <string>

Is this content still correct?

openshift-ci · 2025-06-13T15:44:53Z

@theashiot: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

cahartma · 2025-06-20T21:49:39Z

modules/cluster-logging-collector-log-forward-cloudwatch.adoc

-       region: us-east-2 <8>
-     secret:
-        name: cw-secret <9>
+     authentication:


indentation: authentication is a child of cloudwatch

cahartma · 2025-06-20T21:50:14Z

modules/cluster-logging-collector-log-forward-cloudwatch.adoc

  outputs:
-   - name: cw <4>
-     type: cloudwatch <5>
+   - name: cw <2>


in v6 we require 3 or more chars for a name. Probably best to find a new example name.

cahartma · 2025-06-20T21:52:04Z

modules/cluster-logging-collector-log-forward-cloudwatch.adoc

-     secret:
-        name: cw-secret <9>
+     authentication:
+       awsAccessKey:


missing authentication.type awsAccessKey

cahartma · 2025-06-20T21:54:04Z

modules/cluster-logging-collector-log-forward-cloudwatch.adoc

+         keyId:
+           key: <key>
+           secretName: cw-secret <4>
+       groupName: logType <5>


This api has changed from groupBy to groupName, and is no longer an enumeration. It now requires a static value, or a template requiring a fallback field.

$ oc explain clf.spec.outputs.cloudwatch.groupName ... DESCRIPTION: GroupName defines the strategy for grouping logstreams The GroupName can be a combination of static and dynamic values consisting of field paths followed by "\|\|" followed by another field path or a static value. A dynamic value is encased in single curly brackets "{}" and MUST end with a static fallback value separated with "\|\|". Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes. Example: 1. foo-{.bar\|\|"none"} 2. {.foo\|\|.bar\|\|"missing"} 3. foo.{.bar.baz\|\|.qux.quux.corge\|\|.grault\|\|"nil"}-waldo.fred{.plugh\|\|"none"}

cahartma · 2025-06-20T21:56:25Z

modules/cluster-logging-collector-log-forward-cloudwatch.adoc

+<5> Specify the strategy for grouping logstreams.  The `GroupName` can be a combination of static and dynamic values consisting of field paths followed by "\|\|" followed by another field path or a static value.  A dynamic value is encased in single curly brackets "{}" and must end with a static fallback value separated with `\|\|`. 
+<6> Specify the AWS region.
+<7> Specify a name for the pipeline.
+<8> Specify which log types to forward by using the pipeline: `application,` `infrastructure`, or `audit`.


same inputRef comment

cahartma · 2025-06-20T22:01:26Z

modules/cluster-logging-collector-log-forward-cloudwatch.adoc

@@ -135,10 +130,12 @@ spec:
   - name: cw
     type: cloudwatch
     cloudwatch:
-       groupBy: logType
+     authentication:


indent all this, and it's missing authentication.type

cahartma · 2025-06-20T22:04:11Z

modules/cluster-logging-collector-log-forward-cloudwatch.adoc

@@ -155,7 +152,7 @@ Each region in CloudWatch contains three levels of objects:
 ** log stream
 *** log event

-
+////
 With `groupBy: logType` in the `ClusterLogForwarding` CR, the three log types in the `inputRefs` produce three log groups in Amazon Cloudwatch:


With groupName: {.log_type||"unknown"} in the...

cahartma · 2025-06-20T22:10:38Z

modules/cluster-logging-collector-log-forward-cloudwatch.adoc

@@ -290,3 +287,4 @@ $ aws --output json logs describe-log-groups | jq .logGroups[].logGroupName
 ----

 The `groupBy` field affects the application log group only. It does not affect the `audit` and `infrastructure` log groups.
+////


Any info in the lines above referencing groupBy is outdated and incorrect in v6. This example can probably be done away with completely.
For example, if you want UUID to show after the app namespaces, you would use groupName: my-cluster-7977k.{.namespace_id||"unknown} and place this in a pipeline for app logs only.

cahartma · 2025-06-20T22:25:27Z

modules/cluster-logging-collector-log-forward-loki.adoc

-      tenantKey: kubernetes.namespace_name <9>
+      authentication:
+        token:
+        from: <secret_or_serviceAccount>


indent under token

cahartma · 2025-06-20T22:27:22Z

modules/cluster-logging-collector-log-forward-loki.adoc

+<3> Specify the type as `loki`.
+<4> Specify the URL and port of the Loki system as a valid absolute URL. You can use the `http` (insecure) or `https` (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. Loki's default port for HTTP(S) communication is 3100.
+<5> For a secure connection, you can specify an `https` or `http` URL that you authenticate by specifying a `secret`.
+<6> For an `https` prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a `ca-bundle.crt` key that points to the certificates it represents. Otherwise, for `http` and `https` prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the `openshift-logging` project. For more information, see the following "Example: Setting a secret that contains a username and password."


tls is now its own spec and no longer looks for specific keys

cahartma · 2025-06-20T22:29:39Z

modules/cluster-logging-collector-log-forward-loki.adoc

+      authentication:
+        token:
+        from: <secret_or_serviceAccount>
+        secret: #<6>


this spec has changed in v6. The example should likely be authentication.token.from and be set to serviceAccount. Then the old tls info is now in its own spec, with key and configMap/secretName

cahartma · 2025-06-20T22:31:11Z

modules/cluster-logging-collector-log-forward-loki.adoc

      labelKeys:
-      - kubernetes.labels.foo <10>
+        - kubernetes.labels.foo #<7>
+      tenantKey: kubernetes.namespace_name #<8>


Is now a template and combination of static and dynamic values inside curly brackets (with a static fallback)

cahartma · 2025-06-20T22:31:58Z

modules/cluster-logging-collector-log-forward-loki.adoc

+<5> For a secure connection, you can specify an `https` or `http` URL that you authenticate by specifying a `secret`.
+<6> For an `https` prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a `ca-bundle.crt` key that points to the certificates it represents. Otherwise, for `http` and `https` prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the `openshift-logging` project. For more information, see the following "Example: Setting a secret that contains a username and password."
+<7> Optional: Specify a metadata key field to generate values for the `TenantID` field in Loki. For example, setting `tenantKey: kubernetes.namespace_name` uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section.
+<8> Optional: Specify a list of metadata field keys to replace the default Loki labels. Loki label names must match the regular expression `[a-zA-Z_:][a-zA-Z0-9_:]*`. Illegal characters in metadata keys are replaced with `_` to form the label name. For example, the `kubernetes.labels.foo` metadata key becomes Loki label `kubernetes_labels_foo`. If you do not set `labelKeys`, the default value is: `[log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]`. Keep the set of labels small because Loki limits the size and number of labels allowed. See link:https://grafana.com/docs/loki/latest/configuration/#limits_config[Configuring Loki, limits_config]. You can still query based on any log record field using query filters.


Is now a template, so this needs to be rewritten.

cahartma · 2025-06-20T22:32:26Z

modules/cluster-logging-collector-log-forward-loki.adoc

+<6> For an `https` prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a `ca-bundle.crt` key that points to the certificates it represents. Otherwise, for `http` and `https` prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the `openshift-logging` project. For more information, see the following "Example: Setting a secret that contains a username and password."
+<7> Optional: Specify a metadata key field to generate values for the `TenantID` field in Loki. For example, setting `tenantKey: kubernetes.namespace_name` uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section.
+<8> Optional: Specify a list of metadata field keys to replace the default Loki labels. Loki label names must match the regular expression `[a-zA-Z_:][a-zA-Z0-9_:]*`. Illegal characters in metadata keys are replaced with `_` to form the label name. For example, the `kubernetes.labels.foo` metadata key becomes Loki label `kubernetes_labels_foo`. If you do not set `labelKeys`, the default value is: `[log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]`. Keep the set of labels small because Loki limits the size and number of labels allowed. See link:https://grafana.com/docs/loki/latest/configuration/#limits_config[Configuring Loki, limits_config]. You can still query based on any log record field using query filters.
+<9> Specify which log types to forward by using the pipeline: `application,` `infrastructure`, or `audit`.


same inputRef comment

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 7, 2025

openshift-ci bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 7, 2025

jcantrill reviewed Jun 9, 2025

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 9, 2025

config-log-forwarding-3

482846d

theashiot force-pushed the config-log-forwarding-3 branch from ea9bc99 to 482846d Compare June 13, 2025 15:28

theashiot marked this pull request as ready for review June 13, 2025 15:30

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 13, 2025

theashiot commented Jun 13, 2025

View reviewed changes

cahartma reviewed Jun 20, 2025

View reviewed changes

config-log-forwarding-3 #94412

Are you sure you want to change the base?

config-log-forwarding-3 #94412

Conversation

theashiot commented Jun 7, 2025

Uh oh!

openshift-ci bot commented Jun 7, 2025

Uh oh!

theashiot commented Jun 7, 2025

Uh oh!

ocpdocs-previewbot commented Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jcantrill left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Jun 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cahartma Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cahartma Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ocpdocs-previewbot commented Jun 7, 2025 •

edited

Loading

cahartma Jun 20, 2025 •

edited

Loading

cahartma Jun 20, 2025 •

edited

Loading