[DOCS-12084] Add OP search syntax and upgrade guide

content/en/observability_pipelines/guide/upgrade_your_filter_queries_to_the_new_search_syntax.md

@@ -0,0 +1,131 @@
+---
+title: Upgrade Your Filter Queries to the New Search Syntax
+description: Learn how to update your Observability Pipelines filter queries to use the new search syntax.
+aliases:
+    - /observability_pipelines/guide/upgrade_to_the_new_search_syntax/
+private: true
+disable_toc: false
+further_reading:
+- link: "/observability_pipelines/search_syntax/"
+  tag: "Documentation"
+  text: "Learn more about Observability Pipelines search syntax"
+---
+
+## Overview
+
+Worker versions 2.11 and newer use an updated search syntax. This document goes over:
+
+- [How to upgrade to the new syntax](#how-to-upgrade-to-the-new-search-syntax)
+- [What's new in the updated search syntax](#whats-new-in-the-updated-search-syntax)
+
+## How to upgrade to the new search syntax
+
+See the steps based on whether you:
+
+- [Created the pipeline in the Pipeline UI](#created-the-pipeline-in-pipeline-ui)
+- [Created the pipeline using the API or Terraform](#created-the-pipeline-using-the-api-or-terraform)
+
+### Created the pipeline in Pipeline UI
+
+If you created your pipeline using the Pipeline UI:
+
+1. [Upgrade to Observability Pipelines Worker][1] version 2.11.
+1. Navigate to the [Pipeline UI][2] for that pipeline and update your filter queries to the new syntax. See the [What's new in the updated search syntax](#whats-new-in-the-updated-search-syntax) section for more information.
+1. On the pipeline editor page, by default the `Use legacy search syntax` box is checked because your pipeline is running Worker 2.10 or older, which uses the old search syntax.
+{{< img src="observability_pipelines/guide/legacy_search_checkbox.png" alt="The pipelines editor showing the legacy search checkbox selected" style="width:100%;" >}}
+1. After you've updated all your queries in that pipeline, uncheck the `Use legacy search syntax` box and deploy your pipeline
+
+### Created the pipeline using the API or Terraform
+
+If your pipeline was created using the public API or Terraform:
+- Within the same request that you make to update your pipeline queries to the new search syntax, set `use_legacy_search_syntax` to `false`.
+- **Note**: You **must** set `use_legacy_search_syntax` to `false` when you update your queries because if `use_legacy_search_syntax` is left unpopulated, it defaults to `true` in the Worker.
+
+
+## What's new in the updated search syntax
+
+The following table lists the differences between the legacy and new search syntax:
+
+| Legacy syntax | New syntax                        |
+| ------------- | ------------------------------- |
+| Must use the `@` symbol for attribute search, except when referencing [reserved fields](#legacy-syntax-reserved-fields). | Do not need to use the `@` symbol for attribute search. **Note**: The `@` symbol in queries gets stripped to preserve backwards compatibility. |
+| Since `@` indicates an attribute search, tag searches do not include an `@` so are matched under the attributes `tags` and `ddtags`.<br><br>Attribute search queries without an `@` symbol are matched against the `tags` or `ddtags` array.<br><br>Example attribute search syntax: `env:prod` | Tags syntax must be explicitly entered.<br><br>Inspect your data with Live Capture to determine which fields to match against.<br><br>Example attribute search syntax: `tags:"env:prod" OR ddtags:"env:prod"`  |
+| [Reserved fields](#legacy-syntax-reserved-fields) do not need the `@` symbol. | Reserved fields do not need the `@` symbol. |
+
+**Note**: The upgraded search syntax does not need the `@` symbol for attribute searches. While you do not need to remove the `@` symbol from filter queries that were previously using them, Datadog recommends that you remove the `@` symbol.
+
+The following examples show matched logs, along with the legacy syntax and new syntax that matche the logs.
+
+`{"user": "firstname.lastname"}`
+: **Legacy syntax**: `@user:firstname.lastname`
+: **New syntax**: `user:firstname.lastname`
+: **Difference**: The new syntax does not require the `@` symbol for attribute search.
+
+`{"message": {"log_level": "ERROR"}}`
+: **Legacy syntax**: `@message.log_level:ERROR`
+: **New syntax**: `message.log_level:ERROR`
+: **Difference**: The new syntax does not require the `@` symbol for attribute search.
+
+`{"status": "INFO"}`
+: **Legacy syntax**: `status:INFO`
+: **New syntax**: `status:INFO`
+: **Difference**: No changes because `status` was previously a [reserved field](#legacy-syntax-reserved-fields) that could be filtered without using the `@` symbol. The new syntax does not use the `@` symbol for attribute searches.
+
+`{"message": "Hello, world" }`<br>`{"message: "hello world"}`<br>`{"message": "Hello-world"}`
+: **Legacy syntax**: `message:"hello world"`
+: **New syntax**: `message:"hello world"`
+: **Difference**: There are no changes between the legacy and new syntax because `message` was a reserved field in the legacy search syntax and didn't require the `@` symbol. The new syntax does not use the `@` symbol for attribute searches.
+
+`{"message": "hEllo world"}`
+: **Legacy syntax**: `HELLO OR hello OR Hello`
+: **New syntax**: `hello`
+: **Difference**: With the new syntax, [free text search][4] is case insensitive.
+
+`{"user": "name"}`
+: **Legacy syntax**: `@user:(name OR Name OR nAme)`
+: **New syntax**: `user:(name OR Name or nAme)`
+: **Difference**: With the new syntax, [attribute search][4] is case sensitive and the `@` symbol is not required for attribute search.
+
+`{"tags": ["env:prod"] }`<br>`{"ddtags": ["env:prod"] }`
+: **Legacy syntax**: `env:prod`
+: **New syntax**: `tags:"env:prod" OR ddtags:"env:prod"`
+: **Difference**: With the legacy syntax, when the syntax does not contain the `@` symbol and is not searching for a reserved field, all terms are matched with the `tags` or `ddtags` field. With the new search syntax, there are no reserved fields so all searches must be entered explicitly.
+
+`{"tags": ["message.log_level:INFO"] }`<br>`{"ddtags": ["message.log_level:INFO"]}}`
+: **Legacy syntax**: `message.log_level:INFO`
+: **New syntax**: `tags:"message.log_level:INFO" OR ddtags:"message.log_level:INFO"`
+: **Difference**: Same reason as the previous query for `env:prod` query.
+
+`{"source": "postgres" }`<br>`{"ddsource":"postgres" }`
+: **Legacy syntax**: `source:postgres`
+: **New syntax**: `source:postgres OR ddsource:postgres`
+: **Difference**: With the legacy syntax, attribute search with `source` field matches both `source` and `ddsource` fields. The new syntax no longer does this so you must enter `source` or `ddsource` explicitly.
+
+**Note**: Using wildcards for field names in attribute search is not supported for either the legacy or new syntax. For example, the following usage of wildcard does not work:
+
+- Legacy syntax: `*:something`
+- New syntax: `*:something`
+
+### Legacy syntax reserved fields
+
+For the legacy syntax, these are the reserved fields:
+
+* host
+* source
+* status
+* service
+* trace_id
+* message
+* timestamp
+* tags
+
+See [Reserved attributes][3] for more information.
+
+## Further reading
+
+{{< partial name="whats-next/whats-next.html" >}}
+
+[1]: /observability_pipelines/install_the_worker/?tab=docker#upgrade-the-worker
+[2]: https://app.datadoghq.com/observability-pipelines
+[3]: /logs/log_configuration/attributes_naming_convention/#reserved-attributes
+[4]: /observability_pipelines/search_syntax/#attribute-search

content/en/observability_pipelines/search_syntax.md

@@ -0,0 +1,241 @@
+---
+title: Search Syntax
+description: Learn the search syntax to create filter queries for your Observability Pipelines processors.
+disable_toc: false
+private: true
+---
+## Overview
+
+When you add a processor to a pipeline, you can filter your logs so only a subset of them go through a processor. This document goes over the following information:
+
+- [Free text search](#free-text-search): when you want to search the `message` field.
+- [Attribute search](#attribute-search): when you want to search attribute keys and values.
+- [Arrays](#arrays) when you want to search within an array of nested values.
+- [Boolean operators](#boolean-operators) that you can use in your search query.
+- [Special characters and spaces that need to be escaped](#escape-special-characters-and-spaces) in search queries.
+- Using [wildcards](#wildcards) in your search queries.
+
+**Note**: Worker version 2.11 and newer uses an upgraded search syntax. After you upgrade the Worker to version 2.11, you might need to update your filter queries to match the new syntax. See the [Upgrade to the New Search Syntax](/observability_pipelines/guide/upgrade_to_the_next_search_syntax/) for more information.
+
+## Search syntax
+
+There are two types of filter queries you can use:
+
+- [Free text](#free-text-search)
+- [Attribute](#attribute-search)
+
+### Free text search
+
+Free text search only searches the `message` field and is case insensitive. It is composed of terms and operators. There are two types of terms:
+
+- A single term is a single word such as `test` or `hello`.
+- A sequence is a group of words surrounded by double quotes, such as `"hello dolly"`.
+
+The following are free text search examples:
+
+Search syntax: `hello`
+: Searches for the exact string `hello`. For example, `{"message": "hello world"}` is a matching log.
+
+Search syntax: `Hello world`
+: Searches for `hello` and `world`. For example, `"hello beautiful world"` is a match.
+: This query can also be written as: `Hello AND world`.
+: **Note**: The message must contain both `hello` and `world` to match.
+
+Search syntax: `"hello world"`
+: Searches for a sequence of words. For example `"hello world"`, `"hello-world"`, and `"Hello, world"` are all matches.
+
+### Attribute search
+
+You can search attribute keys and values. For example, if your attribute key is `url` and you want to filter on the `url` value `www.datadoghq.com`, enter: `url:www.datadoghq.com`.
+
+To filter for events that have a specific attribute key, use the `_exists_` syntax. For example if you use the query `_exists_:service`, the event `{"service": "postgres"}` matches the query, but the event `{"env": "prod"}` does not match.
+
+**Note**: Attribute searches are case sensitive.
+
+The following are attribute search syntax examples and logs that match the syntax:
+
+Search syntax: `status:ok service:flask-web-app`
+: Matches logs with the status `ok` from your `flask-web-app` service.
+: This query can also be written as: `status:ok AND service:flask-web-app`.
+
+Search syntax: `user.status:inactive`
+: Matches logs with the status `inactive` nested under the `user` attribute.
+
+Search syntax: `http.url:/api-v1/*`
+: Matches logs containing a value in the `http.url` attribute that starts with `/api-v1/`.
+
+Search syntax: `http.status_code:[200 TO 299] http.url_details.path:/api-v1/*`
+: Matches logs containing an `http.status_code` value that is greater than or equal to `200` and less than or equal to `299`, and containing a value in the `http.url_details.path` attribute that start with `/api-v1/`.
+
+Search syntax: `http.status:[200 TO 299]`
+: Matches logs containing an `http.status` value that is greater than or equal to `200` and less than or equal to `299`.
+: **Notes**:
+: - `[..]` Square brackets mean the ranges are inclusive.
+: - Ranges can be used across any attribute.
+
+Search syntax: `http.status:{200 TO 299}`
+: Matches logs containing an `http.status` value that is greater than `200` or less than `299`. **Notes**: - `{..}` Curly brackets mean the ranges are exclusive. - Ranges can be used across any attribute.
+
+Search syntax: `"service.status":disabled`
+: Matches logs with `"service.status": "disabled"`. This filter syntax searches for a literal `.` in the attribute key.
+: See [Path notation](#path-notation) for more information.
+
+Search syntax: `_exists_:service`
+: Matches logs with the attribute key `service`. For example, the query matches `{"service": "postgres"}`, but does not match `{"env": "prod"}`.
+
+#### Path notation
+
+For the following log structure:
+
+```json
+{
+    "outer_key": {
+        "inner_key": "inner_value",
+        "a": {
+            "double_inner_key": "double_inner_value",
+            "b": "b value"
+        },
+        "c": "c value"
+    },
+    "d": "d value"
+}
+```
+
+- Use `outer_key.inner_key` to reference the key with the value `inner_value`.
+- Use `outer_key.inner_key.double_inner_key` to reference the key with the value `double_inner_value`.
+
+If you want to search for a literal `.` in the attribute key, wrap the key in escaped quotes in the search query. For example, the search query `"service.status":disabled` matches the event `{"service.status": "disabled"}`.
+
+### Boolean operators
+
+You can use the following case sensitive Boolean operators to combine multiple terms in a search query.
+
+| Operator     | Description                                            |
+|--------------|--------------------------------------------------------|
+| `AND`        | Intersection: both terms are in the event.             |
+| `OR`         | Union: either term is contained in the event.          |
+| `-` or `NOT` | Exclusion: the following term is **not** in the event. |
+
+The follow are example queries that use Boolean operators:
+
+Search syntax: `NOT (status:debug)`
+: Matches logs that do not have the status `DEBUG`.
+
+Search syntax: `host:COMP-A9JNGYK OR host:COMP-J58KAS`
+: Only matches logs from those specific hosts.
+
+Search syntax: `Hello AND World`
+: Searches for `hello` and `world`. For example, "hello beautiful world" is a match.
+: This query can also be written as: `Hello world`.
+: **Note**: The message must contain both `hello` and `world` to match.
+
+Search syntax: `hello` AND `status:info`
+: Matches logs with a message field that contains `hello` and with `status:info`.
+
+Search syntax: `-http.status_code:200`
+: Matches logs where http.status_code is not equal to 200
+
+Search syntax: `service:(postgres OR datadog_agent)`
+: Matches logs with the values `postgres` or `datadog_agent` for the `service` attribute. This query can also be written as: `service:postgres OR service:datadog_agent`
+
+## Escape special characters and spaces
+
+The following characters are considered special and require escaping with the following characters: `-` `!` `&&` `||` `>` `>=` `<` `<=` `(` `)` `{` `}` `[` `]` `"` `*` `?` `:` `#`, and spaces.
+
+**Notes**:
+
+- `/` is not considered a special character and doesn't need to be escaped.
+- You can search for special characters inside of an attribute. See [Search an attribute that contains special characters](#search-an-attribute-that-contains-special-characters).
+- If you want to match logs that contain the special character `!` in the `message` field, use the attribute search syntax: `message:*!*`. **Note**: You cannot use free text search queries to filter for log messages with special characters.
+
+### Search an attribute that contains special characters
+
+Searching for an attribute value that contains special characters requires escaping or double quotes. For example, to search for an attribute `my_app` with the value `hello:world`, use the syntax: `my_app:hello:world` or `my_app:"hello:world"`.
+
+### Match a single special character or space
+
+To match a single special character or space, use the `?` wildcard. For example, to search for an attribute `my_app` with the value `hello world again`, use the syntax: `my_app:hello?world?again`.
+
+### Examples
+
+For the following example log:
+
+```
+{
+    "service": "postgres",
+    "status": "INFO",
+    "tags": [
+        "env:prod",
+        "namespace:something",
+        "reader:logs",
+        "my_app:hello world again"
+    ]
+}
+```
+
+The following are search syntax examples that escape special characters and spaces:
+
+Search syntax: `tags:env*`
+: Matches logs with a `tag` attribute value of `env`.
+
+Search syntax: `tags:(env\:prod OR env\:test)`
+: Matches logs with the tag `env:prod` or `env:test` in the `tags` array.
+: This query can also be written as `tags:("env:prod" OR "env:test")`.
+
+Search syntax: `tags:env\:prod AND -tags:version\:beta`
+: Matches logs that have `env:prod` and does not have `version:beta` in the `tag` array.
+: This query can also be written as `tags:"env:prod" AND -tags:"version:beta"`.
+
+Search syntax: `my_app:hello\:world`
+: Matches logs that contain `my_app:hello:world`.
+: This query can also be written as `my_app:"hello:world"`.
+
+Search syntax: `my_app:hello?world?again`
+: Matches logs that contain `"my_app":"hello world again"`.
+
+### Arrays
+
+In the following example, CloudWatch logs for Windows contain an array of JSON objects under `Event.EventData.Data`.
+
+```
+Event
+{
+EventData {
+    Data [
+        {"Name":"SubjectUserID1", "value":"12345"},
+        {"Name":"SubjectUserID2", "value":"Admin"},
+        {"Name":"ObjectServer", "value":"Security"}
+	]
+    }
+}
+```
+
+If you use the filter query: `Event.EventData.Data.Name:ObjectServer`, the above log event is matched because it contains a nested object with the attribute key `Name` and the value `ObjectServer`.
+
+## Wildcards
+
+​​You can use `*` for wildcard searches. The following are wildcard search examples:
+
+Search syntax: `*network*`
+: Matches logs with a `message` field value that contains `network`.
+
+Search syntax: `web*`
+: Matches logs with a `message` field value that starts with `web`.
+
+Search syntax: `*web`
+: Matches logs with a `message` field value that ends with `web`.
+
+Search syntax: `service:*mongo`
+: Matches logs with `service` attribute values that ends with `mongo`.
+
+Search syntax: `service:web*`
+: Matches logs that have a `service` attribute value that starts with `web`.
+
+**Notes**:
+- You cannot use wildcards to search attribute keys, such as `*:app` or `service*:app`.
+- Wildcards only work as wildcards outside of double quotes.
+    - For example, `"*test*"` matches a log which has the string `*test*` in its `message` field, while `*test*` matches a log which has the string `test` anywhere in the `message` field.
+
+#### Search for special characters or escaped characters
+
+When searching for an attribute that contains special characters or requires escaping or double quotes, use the `?` wildcard to match a single special character or space. For example, to search for an attribute `my_attribute` with the value `hello world`, use the syntax: `my_attribute:hello?world`.