From 64b6bc172dfa7dffb16f92fd68c7a23bc169bae8 Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Tue, 27 May 2025 11:17:29 +0200 Subject: [PATCH 1/7] First passs --- apl/scalar-functions/array-functions.mdx | 4 + .../array-functions/bag-has-key.mdx | 159 +++++++++++++++++ .../array-functions/bag-keys.mdx | 164 +++++++++++++++++ .../array-functions/bag-merge.mdx | 166 ++++++++++++++++++ .../array-functions/bag-pack.mdx | 153 ++++++++++++++++ docs.json | 4 + 6 files changed, 650 insertions(+) create mode 100644 apl/scalar-functions/array-functions/bag-has-key.mdx create mode 100644 apl/scalar-functions/array-functions/bag-keys.mdx create mode 100644 apl/scalar-functions/array-functions/bag-merge.mdx create mode 100644 apl/scalar-functions/array-functions/bag-pack.mdx diff --git a/apl/scalar-functions/array-functions.mdx b/apl/scalar-functions/array-functions.mdx index 506d9abd..8dbf6dc1 100644 --- a/apl/scalar-functions/array-functions.mdx +++ b/apl/scalar-functions/array-functions.mdx @@ -23,6 +23,10 @@ The table summarizes the array functions available in APL. | [array_slice](/apl/scalar-functions/array-functions/array-slice) | Extracts a slice of a dynamic array. | | [array_split](/apl/scalar-functions/array-functions/array-split) | Splits an array to multiple arrays according to the split indices and packs the generated array in a dynamic array. | | [array_sum](/apl/scalar-functions/array-functions/array-sum) | Calculates the sum of elements in a dynamic array. | +| [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key) | Calculates the sum of elements in a dynamic array. | +| [bag_keys](/apl/scalar-functions/array-functions/bag-keys) | Calculates the sum of elements in a dynamic array. | +| [bag_merge](/apl/scalar-functions/array-functions/bag-merge) | Calculates the sum of elements in a dynamic array. | +| [bag_pack](/apl/scalar-functions/array-functions/bag-pack) | Calculates the sum of elements in a dynamic array. | | [isarray](/apl/scalar-functions/array-functions/isarray) | Checks whether a value is an array. | | [pack_array](/apl/scalar-functions/array-functions/pack-array) | Packs all input values into a dynamic array. | | [strcat_array](/apl/scalar-functions/array-functions/strcat-array) | Takes an array and returns a single concatenated string with the array’s elements separated by the specified delimiter. | diff --git a/apl/scalar-functions/array-functions/bag-has-key.mdx b/apl/scalar-functions/array-functions/bag-has-key.mdx new file mode 100644 index 00000000..c9fda2e5 --- /dev/null +++ b/apl/scalar-functions/array-functions/bag-has-key.mdx @@ -0,0 +1,159 @@ +--- +title: bag_has_key +description: 'This page explains how to use the bag_has_key function in APL.' +--- + +Use the `bag_has_key` function in APL to check whether a dynamic property bag contains a specific key. This is helpful when your data includes semi-structured or nested fields encoded as dynamic objects, such as JSON-formatted logs or telemetry metadata. + +You often encounter property bags in observability data where log entries, spans, or alerts carry key–value metadata. Use `bag_has_key` to filter, conditionally process, or join such records based on the existence of specific keys, without needing to extract the values themselves. + +## For users of other query languages + +If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. + + + + +In Splunk SPL, you often check whether a key exists in a JSON object using `spath` and conditional logic. APL simplifies this with `bag_has_key`, which returns a boolean directly and avoids explicit parsing. + + +```sql Splunk example +| eval hasKey=if(isnull(spath(data, "keyName")), false, true) +```` + +```kusto APL equivalent +['sample-http-logs'] +| where bag_has_key(dynamic_field, 'keyName') +``` + + + + + + +ANSI SQL doesn’t include native support for property bags or dynamic fields. You typically use JSON functions to access keys in JSON-formatted strings. In APL, dynamic fields are first-class, and `bag_has_key` provides direct support for key existence checks. + + +```sql SQL example +SELECT * +FROM logs +WHERE JSON_EXTRACT(json_column, '$.keyName') IS NOT NULL +``` + +```kusto APL equivalent +['sample-http-logs'] +| where bag_has_key(dynamic_field, 'keyName') +``` + + + + + + +## Usage + +### Syntax + +```kusto +bag_has_key(bag: dynamic, key: string) +``` + +### Parameters + +| Name | Type | Description | +| ----- | --------- | ---------------------------------------------------------------- | +| `bag` | `dynamic` | A dynamic value representing a property bag (e.g., JSON object). | +| `key` | `string` | The key to check for within the property bag. | + +### Returns + +Returns a `bool` value: + +- `true` if the specified key exists in the property bag +- `false` otherwise + +## Use case examples + + + + +Use `bag_has_key` to filter log entries that include a specific metadata key embedded in a dynamic object. + +**Query** + +```kusto +['sample-http-logs'] +| extend metadata = bag_pack('source', 'cdn', 'env', 'prod') +| where bag_has_key(metadata, 'env') +| project _time, id, method, uri, status, metadata +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B%27sample-http-logs%27%5D%20%7C%20extend%20metadata%20%3D%20bag_pack%28%27source%27%2C%20%27cdn%27%2C%20%27env%27%2C%20%27prod%27%29%20%7C%20where%20bag_has_key%28metadata%2C%20%27env%27%29%20%7C%20project%20_time%2C%20id%2C%20method%2C%20uri%2C%20status%2C%20metadata%22%7D) + +**Output** + +| _time | id | method | uri | status | metadata | +| ----------------- | ---- | ------ | -------------- | ------ | ----------------------------- | +| 2025-05-27T12:30Z | u123 | GET | /login | 200 | {'source':'cdn','env':'prod'} | +| 2025-05-27T12:31Z | u124 | POST | /cart/checkout | 500 | {'source':'cdn','env':'prod'} | + +The query filters logs where the synthetic `metadata` bag includes the key `'env'`. + + + + +Use `bag_has_key` to filter spans that include specific dynamic span attributes. + +**Query** + +```kusto +['otel-demo-traces'] +| extend attributes = bag_pack('user', 'alice', 'feature_flag', 'beta') +| where bag_has_key(attributes, 'feature_flag') +| project _time, trace_id, span_id, ['service.name'], kind, attributes +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B%27otel-demo-traces%27%5D%20%7C%20extend%20attributes%20%3D%20bag_pack%28%27user%27%2C%20%27alice%27%2C%20%27feature_flag%27%2C%20%27beta%27%29%20%7C%20where%20bag_has_key%28attributes%2C%20%27feature_flag%27%29%20%7C%20project%20_time%2C%20trace_id%2C%20span_id%2C%20%5B%27service.name%27%5D%2C%20kind%2C%20attributes%22%7D) + +**Output** + +| _time | trace_id | span_id | ['service.name'] | kind | attributes | +| ----------------- | --------- | -------- | ----------------- | ------ | --------------------------------------- | +| 2025-05-27T10:02Z | abc123 | span567 | frontend | client | {'user':'alice','feature_flag':'beta'} | + +The query selects spans with dynamic `attributes` bags containing the `'feature_flag'` key. + + + + +Use `bag_has_key` to identify HTTP logs where the request metadata contains sensitive audit-related keys. + +**Query** + +```kusto +['sample-http-logs'] +| extend audit_info = bag_pack('action', 'delete', 'reason', 'admin_override') +| where bag_has_key(audit_info, 'reason') +| project _time, id, uri, status, audit_info +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B%27sample-http-logs%27%5D%20%7C%20extend%20audit_info%20%3D%20bag_pack%28%27action%27%2C%20%27delete%27%2C%20%27reason%27%2C%20%27admin_override%27%29%20%7C%20where%20bag_has_key%28audit_info%2C%20%27reason%27%29%20%7C%20project%20_time%2C%20id%2C%20uri%2C%20status%2C%20audit_info%22%7D) + +**Output** + +| _time | id | uri | status | audit_info | +| ----------------- | ---- | ------------- | ------ | ---------------------------------------------- | +| 2025-05-27T13:45Z | u999 | /admin/delete | 403 | {'action':'delete','reason':'admin_override'} | + +The query returns only logs where the `audit_info` bag includes the `'reason'` key, indicating administrative override events. + + + + +## List of related functions + +- [parse_json](/apl/scalar-functions/parse-json): Converts a JSON string into a dynamic object. Use it before `bag_has_key` when working with string-encoded JSON. +- [bag_keys](/apl/bag-functions/bag-keys): Returns all keys in a dynamic property bag. Use it when you need to enumerate available keys. +- [bag_unpack](/apl/bag-functions/bag-unpack): Converts a property bag into individual columns. Use it when you want to extract all key–value pairs. +- [has](/apl/scalar-functions/has): Checks whether a string contains a substring. Use it for string matching, not dynamic objects. +- [getpath](/apl/scalar-functions/getpath): Retrieves a value from a dynamic object using a key path. Use it when you want the value, not just check for presence. diff --git a/apl/scalar-functions/array-functions/bag-keys.mdx b/apl/scalar-functions/array-functions/bag-keys.mdx new file mode 100644 index 00000000..adc45dd7 --- /dev/null +++ b/apl/scalar-functions/array-functions/bag-keys.mdx @@ -0,0 +1,164 @@ +--- +title: bag_keys +description: 'This page explains how to use the bag_keys function in APL.' +--- + +Use the `bag_keys` function in APL to extract the keys of a dynamic (bag) object as an array of strings. This is useful when you want to inspect or manipulate the structure of a dynamic field—such as JSON-like nested objects—without needing to know its exact schema in advance. + +Use `bag_keys` when you’re working with semi-structured data and want to: + +- Discover what properties are present in a dynamic object. +- Iterate over the keys programmatically using other array functions. +- Perform validation or debugging tasks to ensure all expected keys exist. + +This function is especially helpful in log analytics, observability pipelines, and security auditing, where dynamic properties are often collected from various services or devices. + +## For users of other query languages + +If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. + + + + +In Splunk SPL, you typically interact with JSON-like fields using the `spath` command or use `keys(_raw)` to retrieve field names. In APL, `bag_keys` serves a similar purpose by returning an array of keys from a dynamic object. + + +```sql Splunk example +| eval key_list=keys(data_field) +```` + +```kusto APL equivalent +datatable(data: dynamic) +[ + dynamic({ "ip": "127.0.0.1", "status": "200", "method": "GET" }) +] +| extend keys = bag_keys(data) +``` + + + + + + +ANSI SQL doesn’t have native support for dynamic objects or JSON key introspection in the same way. However, some SQL dialects (like PostgreSQL or BigQuery) provide JSON-specific functions for extracting keys. `bag_keys` is the APL equivalent for dynamically introspecting JSON objects. + + +```sql SQL example +SELECT JSON_OBJECT_KEYS(data) FROM logs; +``` + +```kusto APL equivalent +datatable(data: dynamic) +[ + dynamic({ "ip": "127.0.0.1", "status": "200", "method": "GET" }) +] +| extend keys = bag_keys(data) +``` + + + + + + +## Usage + +### Syntax + +```kusto +bag_keys(bag) +``` + +### Parameters + +| Name | Type | Description | +| ----- | --------- | -------------------------------------------------- | +| `bag` | `dynamic` | The dynamic object whose keys you want to extract. | + +### Returns + +An array of type `string[]` containing the names of the keys in the dynamic object. If the input is not a dynamic object, the function returns `null`. + +## Use case examples + + + + +Use `bag_keys` to audit dynamic metadata fields in HTTP logs where each record contains a nested object representing additional request attributes. + +**Query** + +```kusto +['sample-http-logs'] +| extend metadata = dynamic({ 'os': 'Windows', 'browser': 'Firefox', 'device': 'Desktop' }) +| extend key_list = bag_keys(metadata) +| project _time, uri, metadata, key_list +``` + +[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D) %7C extend metadata%20%3D%20dynamic(%7B%20'os'%3A%20'Windows'%2C%20'browser'%3A%20'Firefox'%2C%20'device'%3A%20'Desktop'%20%7D) %7C extend key_list%20%3D%20bag_keys(metadata) %7C project%20_time%2C%20uri%2C%20metadata%2C%20key_list%22%7D) + +**Output** + +| _time | uri | metadata | key_list | +| ------------------- | ------ | ------------------------------------------------ | ---------------------------- | +| 2025-05-26 12:01:23 | /login | {os: Windows, browser: Firefox, device: Desktop} | [‘os’, ‘browser’, ‘device’] | + +This query inspects a simulated metadata object and returns the list of its keys, helping you debug inconsistencies or missing fields. + + + + +Use `bag_keys` to examine custom span attributes encoded as dynamic fields within OpenTelemetry trace events. + +**Query** + +```kusto +['otel-demo-traces'] +| extend attributes = dynamic({ 'user_id': 'abc123', 'feature_flag': 'enabled' }) +| extend attribute_keys = bag_keys(attributes) +| project _time, ['service.name'], kind, attributes, attribute_keys +``` + +[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D) %7C extend attributes%20%3D%20dynamic(%7B%20'user_id'%3A%20'abc123'%2C%20'feature_flag'%3A%20'enabled'%20%7D) %7C extend attribute_keys%20%3D%20bag_keys(attributes) %7C project%20_time%2C%20%5B'service.name'%5D%2C%20kind%2C%20attributes%2C%20attribute_keys%22%7D) + +**Output** + +| _time | ['service.name'] | kind | attributes | attribute_keys | +| ------------------- | ----------------- | ------ | ------------------------------------------ | ------------------------------ | +| 2025-05-26 13:14:01 | frontend | client | {user_id: abc123, feature_flag: enabled} | [‘user_id’, ‘feature_flag’] | + +This query inspects the custom span-level attributes and extracts their keys to verify attribute coverage or completeness. + + + + +Use `bag_keys` to list all security-related fields captured dynamically during request monitoring for auditing or compliance. + +**Query** + +```kusto +['sample-http-logs'] +| extend security_context = dynamic({ 'auth_status': 'success', 'role': 'admin', 'ip': '192.168.1.5' }) +| extend fields = bag_keys(security_context) +| project _time, status, ['geo.country'], security_context, fields +``` + +[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D) %7C extend security_context%20%3D%20dynamic(%7B%20'auth_status'%3A%20'success'%2C%20'role'%3A%20'admin'%2C%20'ip'%3A%20'192.168.1.5'%20%7D) %7C extend fields%20%3D%20bag_keys(security_context) %7C project%20_time%2C%20status%2C%20%5B'geo.country'%5D%2C%20security_context%2C%20fields%22%7D) + +**Output** + +| _time | status | ['geo.country'] | security_context | fields | +| ------------------- | ------ | ---------------- | ----------------------------------------------------- | ------------------------------- | +| 2025-05-26 15:32:10 | 200 | US | {auth_status: success, role: admin, ip: 192.168.1.5} | [‘auth_status’, ‘role’, ‘ip’] | + +This helps you audit security metadata in requests and ensure key fields are present across records. + + + + +## List of related functions + +- [bag_unpack](/apl/dynamic-functions/bag_unpack): Converts a dynamic object into a set of columns. Use it when you want to flatten dynamic objects into tabular columns. +- [bag_keys_nested](/apl/dynamic-functions/bag_keys_nested): Returns keys at all levels of a dynamic object, not just the top level. Use this if you need a recursive view. +- [parse_json](/apl/scalar-functions/parse_json): Converts a JSON string into a dynamic object. Use this before `bag_keys` if your data is stored as raw JSON. +- [todynamic](/apl/scalar-functions/todynamic): Converts a string to a dynamic type. Use it to prepare strings for inspection with `bag_keys`. +- [array_length](/apl/array-functions/array_length): Returns the number of keys returned by `bag_keys` if you want to count them. diff --git a/apl/scalar-functions/array-functions/bag-merge.mdx b/apl/scalar-functions/array-functions/bag-merge.mdx new file mode 100644 index 00000000..7451b60d --- /dev/null +++ b/apl/scalar-functions/array-functions/bag-merge.mdx @@ -0,0 +1,166 @@ +--- +title: bag_merge +description: 'This page explains how to use the bag_merge function in APL.' +--- + +Use the `bag_merge` function in APL (Axiom Processing Language) to merge two or more dynamic (JSON-like) bags into a single dynamic value. This function is useful when you want to aggregate multiple properties from different sources or rows into one unified structure. The function is especially valuable in scenarios involving flexible schemas, such as semi-structured log or trace data, where each record might contain partial metadata that you want to consolidate. + +Use `bag_merge` in aggregations with `summarize` to combine dynamic values across groups or time ranges. When multiple input bags share the same property name, `bag_merge` returns the last one by default, replacing earlier ones. + +## For users of other query languages + +If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. + + + + +In Splunk, you use `spath`, `mvcombine`, or `eval` with `json_object` and `json_merge`-style logic to work with JSON objects. APL’s `bag_merge` provides a simpler and more direct way to merge dynamic values (bags) across rows or groups. + + +```sql Splunk example +| stats values(field1) as field1_values values(field2) as field2_values +| eval merged=json_object("field1", field1_values, "field2", field2_values) +```` + +```kusto APL equivalent +datatable(x: dynamic) +[ + dynamic({'a': 1}), + dynamic({'b': 2}) +] +| summarize merged = bag_merge(x) +``` + + + + + + +ANSI SQL doesn’t have a built-in equivalent to APL’s `bag_merge` because it lacks native support for dynamic (JSON-like) objects and merging them. You often need to write complex expressions using `JSON_OBJECT`, `JSON_MERGE`, or user-defined functions. APL handles this use case natively with `bag_merge`. + + +```sql SQL example +SELECT JSON_OBJECT_AGG(key, value) +FROM ( + SELECT 'a' AS key, '1' AS value + UNION + SELECT 'b', '2' +) t +``` + +```kusto APL equivalent +datatable(x: dynamic) +[ + dynamic({'a': 1}), + dynamic({'b': 2}) +] +| summarize merged = bag_merge(x) +``` + + + + + + +## Usage + +### Syntax + +```kusto +bag_merge(expr) +``` + +### Parameters + +| Name | Type | Description | +| ---- | ------- | ------------------------------------------ | +| expr | dynamic | A column or expression that returns a bag. | + +### Returns + +A single dynamic value that merges all bags in the group. If the same key appears in multiple bags, the value from the last bag takes precedence. + +## Use case examples + + + + +Combine metadata from multiple logs for each user into a single object to simplify downstream analysis. + +**Query** + +```kusto +['sample-http-logs'] +| project id, metadata = pack('uri', uri, 'status', status, 'method', method) +| summarize merged_metadata = bag_merge(metadata) by id +``` + +[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D) %7C project id%2C metadata %3D pack('uri'%2C uri%2C 'status'%2C status%2C 'method'%2C method) %7C summarize merged_metadata %3D bag_merge(metadata) by id%22%7D) + +**Output** + +| id | merged_metadata | +| ----- | ----------------------------------------------------- | +| user1 | `{'uri': '/home', 'status': '200', 'method': 'GET'}` | +| user2 | `{'uri': '/cart', 'status': '404', 'method': 'POST'}` | + +Each row shows merged HTTP metadata for each unique user. + + + + +Aggregate span metadata across service kinds in a trace to create a unified summary. + +**Query** + +```kusto +['otel-demo-traces'] +| project trace_id, metadata = pack('service', ['service.name'], 'kind', kind, 'status', status_code) +| summarize merged_metadata = bag_merge(metadata) by trace_id +``` + +[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D) %7C project trace_id%2C metadata %3D pack('service'%2C %5B'service.name'%5D%2C 'kind'%2C kind%2C 'status'%2C status_code) %7C summarize merged_metadata %3D bag_merge(metadata) by trace_id%22%7D) + +**Output** + +| trace_id | merged_metadata | +| --------- | ----------------------------------------------------------------- | +| abc123 | `{'service': 'frontend', 'kind': 'client', 'status': 'OK'}` | +| def456 | `{'service': 'cartservice', 'kind': 'server', 'status': 'ERROR'}` | + +The query combines span-level metadata across services into trace-level summaries. + + + + +Merge location metadata from multiple access attempts into a single object per user ID. + +**Query** + +```kusto +['sample-http-logs'] +| project id, location = pack('city', ['geo.city'], 'country', ['geo.country']) +| summarize merged_location = bag_merge(location) by id +``` + +[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D) %7C project id%2C location %3D pack('city'%2C %5B'geo.city'%5D%2C 'country'%2C %5B'geo.country'%5D) %7C summarize merged_location %3D bag_merge(location) by id%22%7D) + +**Output** + +| id | merged_location | +| ----- | --------------------------------------- | +| user1 | `{'city': 'New York', 'country': 'US'}` | +| user2 | `{'city': 'Berlin', 'country': 'DE'}` | + +This query builds a consolidated location profile for each user. + + + + +## List of related functions + +- [bag_pack](/apl/dynamic-functions/bag_pack): Creates a dynamic value from key-value pairs. Use when you need to build a bag, not merge existing ones. +- [pack](/apl/dynamic-functions/pack): Similar to `bag_pack`, but optimized for constructing dynamic values from scalar values. +- [parse_json](/apl/dynamic-functions/parse_json): Converts a JSON string into a dynamic object. Use it when your source data is in text format. +- [todynamic](/apl/scalar-functions/todynamic): Converts any scalar value into a dynamic value. Useful when preparing inputs for `bag_merge`. +- [bag_keys](/apl/dynamic-functions/bag_keys): Returns the list of keys from a bag. Use it to inspect contents before or after merging. diff --git a/apl/scalar-functions/array-functions/bag-pack.mdx b/apl/scalar-functions/array-functions/bag-pack.mdx new file mode 100644 index 00000000..2084feb9 --- /dev/null +++ b/apl/scalar-functions/array-functions/bag-pack.mdx @@ -0,0 +1,153 @@ +--- +title: bag_pack +description: 'This page explains how to use the bag_pack function in APL.' +--- + +Use the `bag_pack` function in APL to construct a dynamic property bag from a list of key-value pairs. A property bag is a flexible data structure where keys are strings and values are dynamic types. This function is useful when you want to combine multiple values into a single dynamic object, often to simplify downstream processing or export. + +You typically use `bag_pack` in aggregation or projection scenarios to consolidate structured data—for example, packing related request metadata into one field, or grouping trace data by contextual attributes. This makes it easier to output, filter, or transform nested information. + +## For users of other query languages + +If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. + + + + +In Splunk, you can use `mvzip` and `eval` to create key-value mappings, or use `spath` to interpret JSON data. However, packing data into a true key-value structure for export or downstream use requires JSON manipulation. APL’s `bag_pack` provides a native and type-safe way to do this. + + +```sql Splunk example +| eval metadata=tojson({"status": status, "duration": req_duration_ms}) +```` + +```kusto APL equivalent +project metadata = bag_pack('status', status, 'duration', req_duration_ms) +``` + + + + + + +SQL doesn’t have a direct built-in function like `bag_pack`. To achieve similar behavior, you typically construct JSON objects using functions like `JSON_OBJECT` or use user-defined types. In APL, `bag_pack` is the idiomatic way to construct dynamic objects with labeled fields. + + +```sql SQL example +SELECT JSON_OBJECT('status' VALUE status, 'duration' VALUE req_duration_ms) AS metadata FROM logs; +``` + +```kusto APL equivalent +project metadata = bag_pack('status', status, 'duration', req_duration_ms) +``` + + + + + + +## Usage + +### Syntax + +```kusto +bag_pack(key1, value1, key2, value2, ...) +``` + +### Parameters + +| Name | Type | Description | +| --------------------- | -------- | ------------------------------------------------------------------------ | +| `key1, key2, ...` | `string` | The names of the fields to include in the property bag. | +| `value1, value2, ...` | `scalar` | The corresponding values for the keys. Values can be of any scalar type. | + +The number of keys must equal the number of values. Keys must be string literals or string expressions. + +### Returns + +A `dynamic` value representing a property bag (dictionary) where keys are strings and values are the corresponding values. + +## Use case examples + + + + +Use `bag_pack` to create a structured object that captures key request attributes for easier inspection or export. + +**Query** + +```kusto +['sample-http-logs'] +| where status == '500' +| project _time, error_context = bag_pack('uri', uri, 'method', method, 'duration_ms', req_duration_ms) +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20where%20status%20%3D%3D%20'500'%20%7C%20project%20_time%2C%20error_context%20%3D%20bag_pack%28'uri'%2C%20uri%2C%20'method'%2C%20method%2C%20'duration_ms'%2C%20req_duration_ms%29%22%7D) + +**Output** + +| _time | error_context | +| -------------------- | -------------------------------------------------------------- | +| 2025-05-27T10:00:00Z | `{ "uri": "/api/data", "method": "GET", "duration_ms": 342 }` | +| 2025-05-27T10:05:00Z | `{ "uri": "/api/auth", "method": "POST", "duration_ms": 879 }` | + +The query filters HTTP logs to 500 errors and consolidates key request fields into a single dynamic column named `error_context`. + + + + +Use `bag_pack` to enrich trace summaries with service metadata for each span. + +**Query** + +```kusto +['otel-demo-traces'] +| where ['service.name'] == 'checkoutservice' +| project trace_id, span_id, span_info = bag_pack('kind', kind, 'duration', duration, 'status_code', status_code) +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D%20%7C%20where%20%5B'service.name'%5D%20%3D%3D%20'checkoutservice'%20%7C%20project%20trace_id%2C%20span_id%2C%20span_info%20%3D%20bag_pack%28'kind'%2C%20kind%2C%20'duration'%2C%20duration%2C%20'status_code'%2C%20status_code%29%22%7D) + +**Output** + +| trace_id | span_id | span_info | +| --------- | -------- | ------------------------------------------------------------------------------ | +| a1b2... | f9c3... | `{ "kind": "server", "duration": "00:00:00.1240000", "status_code": "OK" }` | +| c3d4... | h7e2... | `{ "kind": "client", "duration": "00:00:00.0470000", "status_code": "ERROR" }` | + +The query targets spans from the `checkoutservice` and combines attributes into a single object per span. + + + + +Use `bag_pack` to create a compact event summary combining user ID and geographic info for anomaly detection. + +**Query** + +```kusto +['sample-http-logs'] +| where status == '403' +| project _time, id, geo_summary = bag_pack('city', ['geo.city'], 'country', ['geo.country']) +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20where%20status%20%3D%3D%20'403'%20%7C%20project%20_time%2C%20id%2C%20geo_summary%20%3D%20bag_pack%28'city'%2C%20%5B'geo.city'%5D%2C%20'country'%2C%20%5B'geo.country'%5D%29%22%7D) + +**Output** + +| _time | id | geo_summary | +| -------------------- | -------- | --------------------------------------- | +| 2025-05-27T12:00:00Z | user_01 | `{ "city": "Berlin", "country": "DE" }` | +| 2025-05-27T12:01:00Z | user_02 | `{ "city": "Paris", "country": "FR" }` | + +The query helps identify patterns in failed access attempts by summarizing location data per event. + + + + +## List of related functions + +- [pack](/apl/array-functions/pack): Similar to `bag_pack`, but uses values as keys and values alternately without requiring string keys. +- [parse_json](/apl/parse-functions/parse_json): Converts a JSON string into a dynamic object; use it when the source data is raw JSON. +- [pack_all](/apl/array-functions/pack_all): Automatically packs all columns into a dynamic object; use when you want to capture the entire row. +- [todynamic](/apl/convert-functions/todynamic): Converts a value into a dynamic type; use it to prepare data for `bag_pack`. +- [project](/apl/operators/project): Use with `bag_pack` to construct structured columns inline in your result set. diff --git a/docs.json b/docs.json index ab941da5..d999f842 100644 --- a/docs.json +++ b/docs.json @@ -326,6 +326,10 @@ "apl/scalar-functions/array-functions/array-slice", "apl/scalar-functions/array-functions/array-split", "apl/scalar-functions/array-functions/array-sum", + "apl/scalar-functions/array-functions/bag-has-key", + "apl/scalar-functions/array-functions/bag-keys", + "apl/scalar-functions/array-functions/bag-merge", + "apl/scalar-functions/array-functions/bag-pack", "apl/scalar-functions/array-functions/isarray", "apl/scalar-functions/array-functions/pack-array", "apl/scalar-functions/array-functions/strcat-array" From 618ad83b4abfd0b709d586a874502851c5348922 Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Wed, 28 May 2025 11:48:45 +0200 Subject: [PATCH 2/7] Fixes --- .../array-functions/bag-has-key.mdx | 16 +++++++-------- .../array-functions/bag-keys.mdx | 20 +++++++++---------- .../array-functions/bag-merge.mdx | 18 ++++++++--------- .../array-functions/bag-pack.mdx | 17 +++++++--------- 4 files changed, 31 insertions(+), 40 deletions(-) diff --git a/apl/scalar-functions/array-functions/bag-has-key.mdx b/apl/scalar-functions/array-functions/bag-has-key.mdx index c9fda2e5..950dc3c4 100644 --- a/apl/scalar-functions/array-functions/bag-has-key.mdx +++ b/apl/scalar-functions/array-functions/bag-has-key.mdx @@ -94,8 +94,8 @@ Use `bag_has_key` to filter log entries that include a specific metadata key emb | _time | id | method | uri | status | metadata | | ----------------- | ---- | ------ | -------------- | ------ | ----------------------------- | -| 2025-05-27T12:30Z | u123 | GET | /login | 200 | {'source':'cdn','env':'prod'} | -| 2025-05-27T12:31Z | u124 | POST | /cart/checkout | 500 | {'source':'cdn','env':'prod'} | +| 2025-05-27T12:30Z | u123 | GET | /login | 200 | \{'source':'cdn','env':'prod'\} | +| 2025-05-27T12:31Z | u124 | POST | /cart/checkout | 500 | \{'source':'cdn','env':'prod'\} | The query filters logs where the synthetic `metadata` bag includes the key `'env'`. @@ -119,7 +119,7 @@ Use `bag_has_key` to filter spans that include specific dynamic span attributes. | _time | trace_id | span_id | ['service.name'] | kind | attributes | | ----------------- | --------- | -------- | ----------------- | ------ | --------------------------------------- | -| 2025-05-27T10:02Z | abc123 | span567 | frontend | client | {'user':'alice','feature_flag':'beta'} | +| 2025-05-27T10:02Z | abc123 | span567 | frontend | client | \{'user':'alice','feature_flag':'beta'\} | The query selects spans with dynamic `attributes` bags containing the `'feature_flag'` key. @@ -143,7 +143,7 @@ Use `bag_has_key` to identify HTTP logs where the request metadata contains sens | _time | id | uri | status | audit_info | | ----------------- | ---- | ------------- | ------ | ---------------------------------------------- | -| 2025-05-27T13:45Z | u999 | /admin/delete | 403 | {'action':'delete','reason':'admin_override'} | +| 2025-05-27T13:45Z | u999 | /admin/delete | 403 | \{'action':'delete','reason':'admin_override'\} | The query returns only logs where the `audit_info` bag includes the `'reason'` key, indicating administrative override events. @@ -152,8 +152,6 @@ The query returns only logs where the `audit_info` bag includes the `'reason'` k ## List of related functions -- [parse_json](/apl/scalar-functions/parse-json): Converts a JSON string into a dynamic object. Use it before `bag_has_key` when working with string-encoded JSON. -- [bag_keys](/apl/bag-functions/bag-keys): Returns all keys in a dynamic property bag. Use it when you need to enumerate available keys. -- [bag_unpack](/apl/bag-functions/bag-unpack): Converts a property bag into individual columns. Use it when you want to extract all key–value pairs. -- [has](/apl/scalar-functions/has): Checks whether a string contains a substring. Use it for string matching, not dynamic objects. -- [getpath](/apl/scalar-functions/getpath): Retrieves a value from a dynamic object using a key path. Use it when you want the value, not just check for presence. +- [bag_keys](/apl/scalar-functions/array-functions/bag-keys): Returns all keys in a dynamic property bag. Use it when you need to enumerate available keys. +- [bag_pack](/apl/scalar-functions/array-functions/bag-pack): Converts a list of key-value pairs to a dynamic property bag. Use when you need to build a bag. +- [bag_merge](/apl/scalar-functions/array-functions/bag-merge): Merge several dynamic bags into a single dynamic value. diff --git a/apl/scalar-functions/array-functions/bag-keys.mdx b/apl/scalar-functions/array-functions/bag-keys.mdx index adc45dd7..f9b8cc79 100644 --- a/apl/scalar-functions/array-functions/bag-keys.mdx +++ b/apl/scalar-functions/array-functions/bag-keys.mdx @@ -94,13 +94,13 @@ Use `bag_keys` to audit dynamic metadata fields in HTTP logs where each record c | project _time, uri, metadata, key_list ``` -[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D) %7C extend metadata%20%3D%20dynamic(%7B%20'os'%3A%20'Windows'%2C%20'browser'%3A%20'Firefox'%2C%20'device'%3A%20'Desktop'%20%7D) %7C extend key_list%20%3D%20bag_keys(metadata) %7C project%20_time%2C%20uri%2C%20metadata%2C%20key_list%22%7D) +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20extend%20metadata%20%3D%20dynamic(%7B%20'os'%3A%20'Windows'%2C%20'browser'%3A%20'Firefox'%2C%20'device'%3A%20'Desktop'%20%7D)%20%7C%20extend%20key_list%20%3D%20bag_keys(metadata)%20%7C%20project%20_time%2C%20uri%2C%20metadata%2C%20key_list%22%7D) **Output** | _time | uri | metadata | key_list | | ------------------- | ------ | ------------------------------------------------ | ---------------------------- | -| 2025-05-26 12:01:23 | /login | {os: Windows, browser: Firefox, device: Desktop} | [‘os’, ‘browser’, ‘device’] | +| 2025-05-26 12:01:23 | /login | \{os: Windows, browser: Firefox, device: Desktop\} | [‘os’, ‘browser’, ‘device’] | This query inspects a simulated metadata object and returns the list of its keys, helping you debug inconsistencies or missing fields. @@ -118,13 +118,13 @@ Use `bag_keys` to examine custom span attributes encoded as dynamic fields withi | project _time, ['service.name'], kind, attributes, attribute_keys ``` -[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D) %7C extend attributes%20%3D%20dynamic(%7B%20'user_id'%3A%20'abc123'%2C%20'feature_flag'%3A%20'enabled'%20%7D) %7C extend attribute_keys%20%3D%20bag_keys(attributes) %7C project%20_time%2C%20%5B'service.name'%5D%2C%20kind%2C%20attributes%2C%20attribute_keys%22%7D) +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D%20%7C%20extend%20attributes%20%3D%20dynamic(%7B%20'user_id'%3A%20'abc123'%2C%20'feature_flag'%3A%20'enabled'%20%7D)%20%7C%20extend%20attribute_keys%20%3D%20bag_keys(attributes)%20%7C%20project%20_time%2C%20%5B'service.name'%5D%2C%20kind%2C%20attributes%2C%20attribute_keys%22%7D) **Output** | _time | ['service.name'] | kind | attributes | attribute_keys | | ------------------- | ----------------- | ------ | ------------------------------------------ | ------------------------------ | -| 2025-05-26 13:14:01 | frontend | client | {user_id: abc123, feature_flag: enabled} | [‘user_id’, ‘feature_flag’] | +| 2025-05-26 13:14:01 | frontend | client | \{user_id: abc123, feature_flag: enabled\} | [‘user_id’, ‘feature_flag’] | This query inspects the custom span-level attributes and extracts their keys to verify attribute coverage or completeness. @@ -142,13 +142,13 @@ Use `bag_keys` to list all security-related fields captured dynamically during r | project _time, status, ['geo.country'], security_context, fields ``` -[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D) %7C extend security_context%20%3D%20dynamic(%7B%20'auth_status'%3A%20'success'%2C%20'role'%3A%20'admin'%2C%20'ip'%3A%20'192.168.1.5'%20%7D) %7C extend fields%20%3D%20bag_keys(security_context) %7C project%20_time%2C%20status%2C%20%5B'geo.country'%5D%2C%20security_context%2C%20fields%22%7D) +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20extend%20security_context%20%3D%20dynamic(%7B%20'auth_status'%3A%20'success'%2C%20'role'%3A%20'admin'%2C%20'ip'%3A%20'192.168.1.5'%20%7D)%20%7C%20extend%20fields%20%3D%20bag_keys(security_context)%20%7C%20project%20_time%2C%20status%2C%20%5B'geo.country'%5D%2C%20security_context%2C%20fields%22%7D) **Output** | _time | status | ['geo.country'] | security_context | fields | | ------------------- | ------ | ---------------- | ----------------------------------------------------- | ------------------------------- | -| 2025-05-26 15:32:10 | 200 | US | {auth_status: success, role: admin, ip: 192.168.1.5} | [‘auth_status’, ‘role’, ‘ip’] | +| 2025-05-26 15:32:10 | 200 | US | \{auth_status: success, role: admin, ip: 192.168.1.5\} | [‘auth_status’, ‘role’, ‘ip’] | This helps you audit security metadata in requests and ensure key fields are present across records. @@ -157,8 +157,6 @@ This helps you audit security metadata in requests and ensure key fields are pre ## List of related functions -- [bag_unpack](/apl/dynamic-functions/bag_unpack): Converts a dynamic object into a set of columns. Use it when you want to flatten dynamic objects into tabular columns. -- [bag_keys_nested](/apl/dynamic-functions/bag_keys_nested): Returns keys at all levels of a dynamic object, not just the top level. Use this if you need a recursive view. -- [parse_json](/apl/scalar-functions/parse_json): Converts a JSON string into a dynamic object. Use this before `bag_keys` if your data is stored as raw JSON. -- [todynamic](/apl/scalar-functions/todynamic): Converts a string to a dynamic type. Use it to prepare strings for inspection with `bag_keys`. -- [array_length](/apl/array-functions/array_length): Returns the number of keys returned by `bag_keys` if you want to count them. +- [bag_pack](/apl/scalar-functions/array-functions/bag-pack): Converts a list of key-value pairs to a dynamic property bag. Use when you need to build a bag. +- [bag_merge](/apl/scalar-functions/array-functions/bag-merge): Merge several dynamic bags into a single dynamic value. +- [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key): Check whether a dynamic property bag contains a specific key. diff --git a/apl/scalar-functions/array-functions/bag-merge.mdx b/apl/scalar-functions/array-functions/bag-merge.mdx index 7451b60d..bfc77285 100644 --- a/apl/scalar-functions/array-functions/bag-merge.mdx +++ b/apl/scalar-functions/array-functions/bag-merge.mdx @@ -67,14 +67,14 @@ datatable(x: dynamic) ### Syntax ```kusto -bag_merge(expr) +bag_merge(Bag1, Bag2, Bag3, ...) ``` ### Parameters | Name | Type | Description | | ---- | ------- | ------------------------------------------ | -| expr | dynamic | A column or expression that returns a bag. | +| `Bag1, Bag2, ...` | dynamic | The bags you want to merge. | ### Returns @@ -95,7 +95,7 @@ Combine metadata from multiple logs for each user into a single object to simpli | summarize merged_metadata = bag_merge(metadata) by id ``` -[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D) %7C project id%2C metadata %3D pack('uri'%2C uri%2C 'status'%2C status%2C 'method'%2C method) %7C summarize merged_metadata %3D bag_merge(metadata) by id%22%7D) + **Output** @@ -119,7 +119,7 @@ Aggregate span metadata across service kinds in a trace to create a unified summ | summarize merged_metadata = bag_merge(metadata) by trace_id ``` -[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D) %7C project trace_id%2C metadata %3D pack('service'%2C %5B'service.name'%5D%2C 'kind'%2C kind%2C 'status'%2C status_code) %7C summarize merged_metadata %3D bag_merge(metadata) by trace_id%22%7D) + **Output** @@ -143,7 +143,7 @@ Merge location metadata from multiple access attempts into a single object per u | summarize merged_location = bag_merge(location) by id ``` -[Run in Playground]([https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D) %7C project id%2C location %3D pack('city'%2C %5B'geo.city'%5D%2C 'country'%2C %5B'geo.country'%5D) %7C summarize merged_location %3D bag_merge(location) by id%22%7D) + **Output** @@ -159,8 +159,6 @@ This query builds a consolidated location profile for each user. ## List of related functions -- [bag_pack](/apl/dynamic-functions/bag_pack): Creates a dynamic value from key-value pairs. Use when you need to build a bag, not merge existing ones. -- [pack](/apl/dynamic-functions/pack): Similar to `bag_pack`, but optimized for constructing dynamic values from scalar values. -- [parse_json](/apl/dynamic-functions/parse_json): Converts a JSON string into a dynamic object. Use it when your source data is in text format. -- [todynamic](/apl/scalar-functions/todynamic): Converts any scalar value into a dynamic value. Useful when preparing inputs for `bag_merge`. -- [bag_keys](/apl/dynamic-functions/bag_keys): Returns the list of keys from a bag. Use it to inspect contents before or after merging. +- [bag_keys](/apl/scalar-functions/array-functions/bag-keys): Returns all keys in a dynamic property bag. Use it when you need to enumerate available keys. +- [bag_pack](/apl/scalar-functions/array-functions/bag-pack): Converts a list of key-value pairs to a dynamic property bag. Use when you need to build a bag. +- [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key): Check whether a dynamic property bag contains a specific key. diff --git a/apl/scalar-functions/array-functions/bag-pack.mdx b/apl/scalar-functions/array-functions/bag-pack.mdx index 2084feb9..3f3cec5a 100644 --- a/apl/scalar-functions/array-functions/bag-pack.mdx +++ b/apl/scalar-functions/array-functions/bag-pack.mdx @@ -102,11 +102,11 @@ Use `bag_pack` to enrich trace summaries with service metadata for each span. ```kusto ['otel-demo-traces'] -| where ['service.name'] == 'checkoutservice' +| where ['service.name'] == 'checkout' | project trace_id, span_id, span_info = bag_pack('kind', kind, 'duration', duration, 'status_code', status_code) ``` -[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D%20%7C%20where%20%5B'service.name'%5D%20%3D%3D%20'checkoutservice'%20%7C%20project%20trace_id%2C%20span_id%2C%20span_info%20%3D%20bag_pack%28'kind'%2C%20kind%2C%20'duration'%2C%20duration%2C%20'status_code'%2C%20status_code%29%22%7D) +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D%20%7C%20where%20%5B'service.name'%5D%20%3D%3D%20'checkout'%20%7C%20project%20trace_id%2C%20span_id%2C%20span_info%20%3D%20bag_pack%28'kind'%2C%20kind%2C%20'duration'%2C%20duration%2C%20'status_code'%2C%20status_code%29%22%7D) **Output** @@ -115,7 +115,7 @@ Use `bag_pack` to enrich trace summaries with service metadata for each span. | a1b2... | f9c3... | `{ "kind": "server", "duration": "00:00:00.1240000", "status_code": "OK" }` | | c3d4... | h7e2... | `{ "kind": "client", "duration": "00:00:00.0470000", "status_code": "ERROR" }` | -The query targets spans from the `checkoutservice` and combines attributes into a single object per span. +The query targets spans from the `checkout` and combines attributes into a single object per span. @@ -126,11 +126,10 @@ Use `bag_pack` to create a compact event summary combining user ID and geographi ```kusto ['sample-http-logs'] -| where status == '403' | project _time, id, geo_summary = bag_pack('city', ['geo.city'], 'country', ['geo.country']) ``` -[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20where%20status%20%3D%3D%20'403'%20%7C%20project%20_time%2C%20id%2C%20geo_summary%20%3D%20bag_pack%28'city'%2C%20%5B'geo.city'%5D%2C%20'country'%2C%20%5B'geo.country'%5D%29%22%7D) +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20project%20_time%2C%20id%2C%20geo_summary%20%3D%20bag_pack('city'%2C%20%5B'geo.city'%5D%2C%20'country'%2C%20%5B'geo.country'%5D)%22%7D) **Output** @@ -146,8 +145,6 @@ The query helps identify patterns in failed access attempts by summarizing locat ## List of related functions -- [pack](/apl/array-functions/pack): Similar to `bag_pack`, but uses values as keys and values alternately without requiring string keys. -- [parse_json](/apl/parse-functions/parse_json): Converts a JSON string into a dynamic object; use it when the source data is raw JSON. -- [pack_all](/apl/array-functions/pack_all): Automatically packs all columns into a dynamic object; use when you want to capture the entire row. -- [todynamic](/apl/convert-functions/todynamic): Converts a value into a dynamic type; use it to prepare data for `bag_pack`. -- [project](/apl/operators/project): Use with `bag_pack` to construct structured columns inline in your result set. +- [bag_keys](/apl/scalar-functions/array-functions/bag-keys): Returns all keys in a dynamic property bag. Use it when you need to enumerate available keys. +- [bag_merge](/apl/scalar-functions/array-functions/bag-merge): Merge several dynamic bags into a single dynamic value. +- [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key): Check whether a dynamic property bag contains a specific key. From 1ebc89cd4a89863d557117786c2e471fcd0d6254 Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Wed, 28 May 2025 11:50:35 +0200 Subject: [PATCH 3/7] Fixes --- apl/scalar-functions/array-functions.mdx | 10 +++++----- apl/scalar-functions/array-functions/bag-has-key.mdx | 2 +- apl/scalar-functions/array-functions/bag-keys.mdx | 4 ++-- apl/scalar-functions/array-functions/bag-merge.mdx | 2 +- apl/scalar-functions/array-functions/bag-pack.mdx | 4 ++-- 5 files changed, 11 insertions(+), 11 deletions(-) diff --git a/apl/scalar-functions/array-functions.mdx b/apl/scalar-functions/array-functions.mdx index 8dbf6dc1..8a302580 100644 --- a/apl/scalar-functions/array-functions.mdx +++ b/apl/scalar-functions/array-functions.mdx @@ -23,10 +23,10 @@ The table summarizes the array functions available in APL. | [array_slice](/apl/scalar-functions/array-functions/array-slice) | Extracts a slice of a dynamic array. | | [array_split](/apl/scalar-functions/array-functions/array-split) | Splits an array to multiple arrays according to the split indices and packs the generated array in a dynamic array. | | [array_sum](/apl/scalar-functions/array-functions/array-sum) | Calculates the sum of elements in a dynamic array. | -| [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key) | Calculates the sum of elements in a dynamic array. | -| [bag_keys](/apl/scalar-functions/array-functions/bag-keys) | Calculates the sum of elements in a dynamic array. | -| [bag_merge](/apl/scalar-functions/array-functions/bag-merge) | Calculates the sum of elements in a dynamic array. | -| [bag_pack](/apl/scalar-functions/array-functions/bag-pack) | Calculates the sum of elements in a dynamic array. | +| [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key) | Checks whether a dynamic property bag contains a specific key. | +| [bag_keys](/apl/scalar-functions/array-functions/bag-keys) | Returns all keys in a dynamic property bag. | +| [bag_merge](/apl/scalar-functions/array-functions/bag-merge) | Merges several dynamic bags into a single dynamic value. | +| [bag_pack](/apl/scalar-functions/array-functions/bag-pack) | Converts a list of key-value pairs to a dynamic property bag. | | [isarray](/apl/scalar-functions/array-functions/isarray) | Checks whether a value is an array. | | [pack_array](/apl/scalar-functions/array-functions/pack-array) | Packs all input values into a dynamic array. | | [strcat_array](/apl/scalar-functions/array-functions/strcat-array) | Takes an array and returns a single concatenated string with the array’s elements separated by the specified delimiter. | @@ -35,4 +35,4 @@ The table summarizes the array functions available in APL. Most array functions accept a dynamic array as their parameter. Dynamic arrays allow you to add or remove elements. You can change a dynamic array with an array function. -A dynamic array expands as you add more elements. This means that you don’t need to determine the size in advance. \ No newline at end of file +A dynamic array expands as you add more elements. This means that you don’t need to determine the size in advance. diff --git a/apl/scalar-functions/array-functions/bag-has-key.mdx b/apl/scalar-functions/array-functions/bag-has-key.mdx index 950dc3c4..ce34d7cc 100644 --- a/apl/scalar-functions/array-functions/bag-has-key.mdx +++ b/apl/scalar-functions/array-functions/bag-has-key.mdx @@ -154,4 +154,4 @@ The query returns only logs where the `audit_info` bag includes the `'reason'` k - [bag_keys](/apl/scalar-functions/array-functions/bag-keys): Returns all keys in a dynamic property bag. Use it when you need to enumerate available keys. - [bag_pack](/apl/scalar-functions/array-functions/bag-pack): Converts a list of key-value pairs to a dynamic property bag. Use when you need to build a bag. -- [bag_merge](/apl/scalar-functions/array-functions/bag-merge): Merge several dynamic bags into a single dynamic value. +- [bag_merge](/apl/scalar-functions/array-functions/bag-merge): Merges several dynamic bags into a single dynamic value. diff --git a/apl/scalar-functions/array-functions/bag-keys.mdx b/apl/scalar-functions/array-functions/bag-keys.mdx index f9b8cc79..ca07120b 100644 --- a/apl/scalar-functions/array-functions/bag-keys.mdx +++ b/apl/scalar-functions/array-functions/bag-keys.mdx @@ -158,5 +158,5 @@ This helps you audit security metadata in requests and ensure key fields are pre ## List of related functions - [bag_pack](/apl/scalar-functions/array-functions/bag-pack): Converts a list of key-value pairs to a dynamic property bag. Use when you need to build a bag. -- [bag_merge](/apl/scalar-functions/array-functions/bag-merge): Merge several dynamic bags into a single dynamic value. -- [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key): Check whether a dynamic property bag contains a specific key. +- [bag_merge](/apl/scalar-functions/array-functions/bag-merge): Merges several dynamic bags into a single dynamic value. +- [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key): Checks whether a dynamic property bag contains a specific key. diff --git a/apl/scalar-functions/array-functions/bag-merge.mdx b/apl/scalar-functions/array-functions/bag-merge.mdx index bfc77285..510eebcd 100644 --- a/apl/scalar-functions/array-functions/bag-merge.mdx +++ b/apl/scalar-functions/array-functions/bag-merge.mdx @@ -161,4 +161,4 @@ This query builds a consolidated location profile for each user. - [bag_keys](/apl/scalar-functions/array-functions/bag-keys): Returns all keys in a dynamic property bag. Use it when you need to enumerate available keys. - [bag_pack](/apl/scalar-functions/array-functions/bag-pack): Converts a list of key-value pairs to a dynamic property bag. Use when you need to build a bag. -- [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key): Check whether a dynamic property bag contains a specific key. +- [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key): Checks whether a dynamic property bag contains a specific key. diff --git a/apl/scalar-functions/array-functions/bag-pack.mdx b/apl/scalar-functions/array-functions/bag-pack.mdx index 3f3cec5a..7929830c 100644 --- a/apl/scalar-functions/array-functions/bag-pack.mdx +++ b/apl/scalar-functions/array-functions/bag-pack.mdx @@ -146,5 +146,5 @@ The query helps identify patterns in failed access attempts by summarizing locat ## List of related functions - [bag_keys](/apl/scalar-functions/array-functions/bag-keys): Returns all keys in a dynamic property bag. Use it when you need to enumerate available keys. -- [bag_merge](/apl/scalar-functions/array-functions/bag-merge): Merge several dynamic bags into a single dynamic value. -- [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key): Check whether a dynamic property bag contains a specific key. +- [bag_merge](/apl/scalar-functions/array-functions/bag-merge): Merges several dynamic bags into a single dynamic value. +- [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key): Checks whether a dynamic property bag contains a specific key. From 9d36f8008df74b755192d33580699c2c17096a6e Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Thu, 12 Jun 2025 10:54:24 +0200 Subject: [PATCH 4/7] Implement review --- apl/scalar-functions/array-functions/bag-merge.mdx | 2 +- apl/scalar-functions/array-functions/bag-pack.mdx | 10 ++++++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/apl/scalar-functions/array-functions/bag-merge.mdx b/apl/scalar-functions/array-functions/bag-merge.mdx index 510eebcd..4e42aabe 100644 --- a/apl/scalar-functions/array-functions/bag-merge.mdx +++ b/apl/scalar-functions/array-functions/bag-merge.mdx @@ -5,7 +5,7 @@ description: 'This page explains how to use the bag_merge function in APL.' Use the `bag_merge` function in APL (Axiom Processing Language) to merge two or more dynamic (JSON-like) bags into a single dynamic value. This function is useful when you want to aggregate multiple properties from different sources or rows into one unified structure. The function is especially valuable in scenarios involving flexible schemas, such as semi-structured log or trace data, where each record might contain partial metadata that you want to consolidate. -Use `bag_merge` in aggregations with `summarize` to combine dynamic values across groups or time ranges. When multiple input bags share the same property name, `bag_merge` returns the last one by default, replacing earlier ones. +Use `bag_merge` to combine dynamic values across groups or time ranges. When multiple input bags share the same property name, `bag_merge` returns the last one by default, replacing earlier ones. ## For users of other query languages diff --git a/apl/scalar-functions/array-functions/bag-pack.mdx b/apl/scalar-functions/array-functions/bag-pack.mdx index 7929830c..5700b70f 100644 --- a/apl/scalar-functions/array-functions/bag-pack.mdx +++ b/apl/scalar-functions/array-functions/bag-pack.mdx @@ -1,11 +1,17 @@ --- title: bag_pack -description: 'This page explains how to use the bag_pack function in APL.' +description: 'This page explains how to use the bag_pack and pack functions in APL.' --- Use the `bag_pack` function in APL to construct a dynamic property bag from a list of key-value pairs. A property bag is a flexible data structure where keys are strings and values are dynamic types. This function is useful when you want to combine multiple values into a single dynamic object, often to simplify downstream processing or export. -You typically use `bag_pack` in aggregation or projection scenarios to consolidate structured data—for example, packing related request metadata into one field, or grouping trace data by contextual attributes. This makes it easier to output, filter, or transform nested information. +You typically use `bag_pack` in projection scenarios to consolidate structured data—for example, packing related request metadata into one field, or grouping trace data by contextual attributes. This makes it easier to output, filter, or transform nested information. + + +The `pack` and `bag-pack` functions are equivalent in APL. + +A common use is `bag_pack(*)` that gets all fields of your dataset as a bag. This can be useful when you want to get sets of values. + ## For users of other query languages From ca1dbd34815eda13326823f0b4593cbcd35d2170 Mon Sep 17 00:00:00 2001 From: Mano Toth <71388581+tothmano@users.noreply.github.com> Date: Fri, 13 Jun 2025 08:29:24 +0200 Subject: [PATCH 5/7] Update apl/scalar-functions/array-functions/bag-pack.mdx --- apl/scalar-functions/array-functions/bag-pack.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apl/scalar-functions/array-functions/bag-pack.mdx b/apl/scalar-functions/array-functions/bag-pack.mdx index 5700b70f..63694b36 100644 --- a/apl/scalar-functions/array-functions/bag-pack.mdx +++ b/apl/scalar-functions/array-functions/bag-pack.mdx @@ -8,7 +8,7 @@ Use the `bag_pack` function in APL to construct a dynamic property bag from a li You typically use `bag_pack` in projection scenarios to consolidate structured data—for example, packing related request metadata into one field, or grouping trace data by contextual attributes. This makes it easier to output, filter, or transform nested information. -The `pack` and `bag-pack` functions are equivalent in APL. +The `pack` and `bag_pack` functions are equivalent in APL. A common use is `bag_pack(*)` that gets all fields of your dataset as a bag. This can be useful when you want to get sets of values. From e07b2469c9637363dcae2c91d091d5b9957c306f Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Fri, 13 Jun 2025 08:44:10 +0200 Subject: [PATCH 6/7] Update bag-merge.mdx --- apl/scalar-functions/array-functions/bag-merge.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apl/scalar-functions/array-functions/bag-merge.mdx b/apl/scalar-functions/array-functions/bag-merge.mdx index 4e42aabe..670240e7 100644 --- a/apl/scalar-functions/array-functions/bag-merge.mdx +++ b/apl/scalar-functions/array-functions/bag-merge.mdx @@ -159,6 +159,6 @@ This query builds a consolidated location profile for each user. ## List of related functions +- [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key): Checks whether a dynamic property bag contains a specific key. - [bag_keys](/apl/scalar-functions/array-functions/bag-keys): Returns all keys in a dynamic property bag. Use it when you need to enumerate available keys. - [bag_pack](/apl/scalar-functions/array-functions/bag-pack): Converts a list of key-value pairs to a dynamic property bag. Use when you need to build a bag. -- [bag_has_key](/apl/scalar-functions/array-functions/bag-has-key): Checks whether a dynamic property bag contains a specific key. From 3bfff5286f1ce0bd89de53aa808ef612ac43855f Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Mon, 16 Jun 2025 09:24:58 +0200 Subject: [PATCH 7/7] Add note about private preview --- apl/scalar-functions/array-functions/bag-merge.mdx | 10 ++++------ getting-started-guide/feature-states.mdx | 1 + 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/apl/scalar-functions/array-functions/bag-merge.mdx b/apl/scalar-functions/array-functions/bag-merge.mdx index 670240e7..47aad341 100644 --- a/apl/scalar-functions/array-functions/bag-merge.mdx +++ b/apl/scalar-functions/array-functions/bag-merge.mdx @@ -7,6 +7,10 @@ Use the `bag_merge` function in APL (Axiom Processing Language) to merge two or Use `bag_merge` to combine dynamic values across groups or time ranges. When multiple input bags share the same property name, `bag_merge` returns the last one by default, replacing earlier ones. + +`bag_merge` is currently in private preview. To try it out, [contact Axiom](https://axiom.co/contact). + + ## For users of other query languages If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. @@ -95,8 +99,6 @@ Combine metadata from multiple logs for each user into a single object to simpli | summarize merged_metadata = bag_merge(metadata) by id ``` - - **Output** | id | merged_metadata | @@ -119,8 +121,6 @@ Aggregate span metadata across service kinds in a trace to create a unified summ | summarize merged_metadata = bag_merge(metadata) by trace_id ``` - - **Output** | trace_id | merged_metadata | @@ -143,8 +143,6 @@ Merge location metadata from multiple access attempts into a single object per u | summarize merged_location = bag_merge(location) by id ``` - - **Output** | id | merged_location | diff --git a/getting-started-guide/feature-states.mdx b/getting-started-guide/feature-states.mdx index fbf24bab..23bf86a3 100644 --- a/getting-started-guide/feature-states.mdx +++ b/getting-started-guide/feature-states.mdx @@ -22,6 +22,7 @@ Private and public preview features are experimental, are not guaranteed to work Current private preview features: - [Flow](/process-data/introduction) +- [bag_merge function](/apl/scalar-functions/array-functions/bag-merge) Current public preview features: - [Cursor-based pagination](/restapi/pagination)