Otel doc fixes (#10150) (#10171)

Author: opensearch-trigger-bot[bot]

_data-prepper/pipelines/configuration/sinks/opensearch.md

@@ -66,7 +66,9 @@ Option | Required | Type | Description
 `insecure` | No | Boolean  | Whether or not to verify SSL certificates. If set to `true`, then certificate authority (CA) certificate verification is disabled and insecure HTTP requests are sent instead. Default is `false`.
 `proxy` | No | String | The address of the [forward HTTP proxy server](https://en.wikipedia.org/wiki/Proxy_server). The format is `"<hostname or IP>:<port>"` (for example, `"example.com:8100"`, `"http://example.com:8100"`, `"112.112.112.112:8100"`). The port number cannot be omitted.
 `index` | Conditionally | String | The name of the export index. Only required when the `index_type` is `custom`. The index can be a plain string, such as `my-index-name`, contain [Java date-time patterns](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html), such as `my-index-%{yyyy.MM.dd}` or `my-%{yyyy-MM-dd-HH}-index`, be formatted using field values, such as `my-index-${/my_field}`, or use [Data Prepper expressions]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `my-index-${getMetadata(\"my_metadata_field\"}`. All formatting options can be combined to provide flexibility when creating static, dynamic, and rolling indexes. 
-`index_type` | No | String | Tells the sink plugin what type of data it is handling. Valid values are `custom`, `trace-analytics-raw`, `trace-analytics-service-map`, or `management-disabled`. Default is `custom`.
+`index_type` | No | String | Specifies the type of data the sink plugin handles. Valid values include `custom`, `trace-analytics-raw`, `trace-analytics-plain-raw`, `trace-analytics-service-map`, `log-analytics`, `log-analytics-plain`, `metric-analytics`, `metric-analytics-plain`, and `management-disabled`. <br><br>To produce Amazon Security Lake–compliant data from the `otel_logs_source` with `output_format: otel`, set `index_type` to `log-analytics-plain`. <br>For `otel_metrics_source` with `output_format: otel`, set `index_type` to `metric-analytics-plain`. <br>For `otel_trace_source` with `output_format: otel`, set `index_type` to `trace-analytics-plain-raw`. <br><br>Default is `custom`.
+
+
 `template_type` | No | String | Defines what type of OpenSearch template to use. Available options are `v1` and `index-template`. The default value is `v1`, which uses the original OpenSearch templates available at the `_template` API endpoints. The `index-template` option uses composable [index templates]({{site.url}}{{site.baseurl}}/opensearch/index-templates/), which are available through the OpenSearch `_index_template` API. Composable index types offer more flexibility than the default and are necessary when an OpenSearch cluster contains existing index templates. Composable templates are available for all versions of OpenSearch and some later versions of Elasticsearch. When `distribution_version` is set to `es6`, Data Prepper enforces the `template_type` as `v1`.
 `template_file` | No | String | The path to a JSON [index template]({{site.url}}{{site.baseurl}}/opensearch/index-templates/) file, such as `/your/local/template-file.json`, when `index_type` is set to `custom`. For an example template file, see [otel-v1-apm-span-index-template.json](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/resources/otel-v1-apm-span-index-template.json). If you supply a template file, then it must match the template format specified by the `template_type` parameter.
 `template_content` | No | JSON | Contains all the inline JSON found inside of the index  [index template]({{site.url}}{{site.baseurl}}/opensearch/index-templates/). For an example of template content, see [the example template content](#example_template_content).

_data-prepper/pipelines/configuration/sources/otel-logs-source.md

@@ -29,6 +29,8 @@ You can configure the `otel_logs_source` source with the following options.
 | unframed_requests | Boolean | Enables requests that are not framed using the gRPC wire protocol. Default value is `false`. |
 | thread_count  | int | The number of threads to keep in the `ScheduledThreadPool`. Default value is `500`. |
 | max_connection_count | int | The maximum number of open connections allowed. Default value is `500`. |
+| `output_format` | String | Specifies the output format of the generated events. Valid values are `otel` or `opensearch`. Default is `opensearch`. |
+
 
 ### SSL
 
@@ -52,6 +54,15 @@ source:
     - otel_logs_source:
 ```
 
+To generate data in the OpenTelemetry format, set the `output_format` setting to `otel`, as shown in the following example:
+
+```yaml
+source:
+    - otel_logs_source:
+        output_format: otel
+```
+{% include copy.html %}
+
 ## Metrics
 
 You can use the following metrics with the `otel_logs_source` source.
@@ -65,4 +76,4 @@ You can use the following metrics with the `otel_logs_source` source.
 | `internalServerError` | Counter | Measures the total number of requests that are erroneous due to errors other than `requestTimeouts` or `requestsTooLarge`. |
 | `successRequests` | Counter | Measures the total number of requests successfully written to the buffer. |
 | `payloadSize` | Distribution summary | Measures the distribution of all incoming payload sizes. |
-| `requestProcessDuration` | Timer | Measures the duration of request processing. |
+| `requestProcessDuration` | Timer | Measures the duration of request processing. |

_data-prepper/pipelines/configuration/sources/otel-metrics-source.md

@@ -19,6 +19,7 @@ proto_reflection_service | No | Boolean | Enables a reflection service for Proto
 unframed_requests | No | Boolean | Enables requests not framed using the gRPC wire protocol.
 thread_count | No | Integer | The number of threads to keep in the `ScheduledThreadPool`. Default value is `200`.
 max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is `500`.
+| `output_format` | String | Specifies the output format of the generated events. Valid values are `otel` or `opensearch`. Default is `opensearch`. |
 max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`.
 ssl | No | Boolean | Enables connections to the OpenTelemetry source port over TLS/SSL. Default value is `true`.
 sslKeyCertChainFile | Conditionally | String | File-system path or Amazon Simple Storage Service (Amazon S3) path to the security certificate (for example, `"config/demo-data-prepper.crt"` or `"s3://my-secrets-bucket/demo-data-prepper.crt"`). Required if `ssl` is set to `true`.
@@ -28,9 +29,25 @@ acmCertificateArn | Conditionally | String | Represents the ACM certificate ARN.
 awsRegion | Conditionally | String | Represents the AWS Region used by ACM or Amazon S3. Required if `useAcmCertForSSL` is set to `true` or `sslKeyCertChainFile` and `sslKeyFile` is the Amazon S3 path.
 authentication | No | Object | An authentication configuration. By default, an unauthenticated server is created for the pipeline. This uses pluggable authentication for HTTPS. To use basic authentication, define the `http_basic` plugin with a `username` and `password`. To provide customer authentication, use or create a plugin that implements [GrpcAuthenticationProvider](https://github.com/opensearch-project/data-prepper/blob/1.2.0/data-prepper-plugins/armeria-common/src/main/java/com/amazon/dataprepper/armeria/authentication/GrpcAuthenticationProvider.java).
 
-<!--- ## Configuration
+## Usage
+
+To use the `otel-metrics` source, create the following `pipeline.yaml` file with `otel_metrics_source` as the source:
+
+```yaml
+source:
+    - otel_metrics_source:
+```
+{% include copy.html %}
+
+To use the OpenTelemetry format for your output, set the `output_format` to `otel`, as shown in the following example:
+
+```yaml
+source:
+    - otel_metrics_source:
+        output_format: otel
+```
+{% include copy.html %}
 
-Content will be added to this section.--->
 
 ## Metrics
 

_data-prepper/pipelines/configuration/sources/otel-trace-source.md

@@ -24,6 +24,7 @@ proto_reflection_service | No | Boolean | Enables a reflection service for Proto
 unframed_requests | No | Boolean | Enable requests not framed using the gRPC wire protocol.
 thread_count | No | Integer | The number of threads to keep in the ScheduledThreadPool. Default value is `200`.
 max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is `500`.
+| `output_format` | String | Specifies the output format of the generated events. Valid values are `otel` or `opensearch`. Default is `opensearch`. |
 max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`.
 ssl | No | Boolean | Enables connections to the OTel source port over TLS/SSL. Defaults to `true`.
 sslKeyCertChainFile | Conditionally | String | File system path or Amazon Simple Storage Service (Amazon S3) path to the security certificate (for example, `"config/demo-data-prepper.crt"` or `"s3://my-secrets-bucket/demo-data-prepper.crt"`). Required if `ssl` is set to `true`.
@@ -33,6 +34,24 @@ acmCertificateArn | Conditionally | String | Represents the ACM certificate ARN.
 awsRegion | Conditionally | String | Represents the AWS region used by ACM or Amazon S3. Required if `useAcmCertForSSL` is set to `true` or `sslKeyCertChainFile` and `sslKeyFile` are Amazon S3 paths.
 authentication | No | Object | An authentication configuration. By default, an unauthenticated server is created for the pipeline. This parameter uses pluggable authentication for HTTPS. To use basic authentication, define the `http_basic` plugin with a `username` and `password`. To provide customer authentication, use or create a plugin that implements [GrpcAuthenticationProvider](https://github.com/opensearch-project/data-prepper/blob/1.2.0/data-prepper-plugins/armeria-common/src/main/java/com/amazon/dataprepper/armeria/authentication/GrpcAuthenticationProvider.java).
 
+## Usage
+
+To use the `otel-metrics` source, create the following `pipeline.yaml` file with `otel_metrics_source` as the source:
+
+```yaml
+source:
+    - otel_trace_source:
+```
+{% include copy.html %}
+
+If you want to use the OpenTelemetry format for your output, set the `output_format` to `otel`, as shown in the following example:
+
+```yaml
+source:
+    - otel_trace_source:
+        output_format: otel
+```
+{% include copy.html %}
 
 ## Metrics
 

_data-prepper/pipelines/functions.md

@@ -13,6 +13,7 @@ OpenSearch Data Prepper offers a range of built-in functions that can be used wi
 - [`cidrContains()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/cidrcontains/)
 - [`contains()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/contains/)
 - [`getMetadata()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/get-metadata/)
+- [`getEventType()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/get-eventtype/)
 - [`hasTags()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/has-tags/)
 - [`join()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/join/)
 - [`length()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/length/)

_data-prepper/pipelines/get-eventtype.md

@@ -0,0 +1,20 @@
+---
+layout: default
+title: getEventType()
+parent: Functions
+grand_parent: Pipelines
+nav_order: 45
+---
+
+# getEventType()
+
+The `getEventType()` function returns the internal event type of the current event.
+
+The return value is one of the event types defined in the `EventType.java` file. For example, if the event is an OpenTelemetry trace event, the returned event type is `TRACE`.
+
+Use this function to check the event type before performing conditional processing, as shown in the following example:
+
+```json
+getEventType() == "TRACE"
+```
+{% include copy.html %}