[AISP-312] Add Signals + ML end-to-end tutorial

.github/styles/Snowplow/Acronyms.yml

@@ -10,6 +10,7 @@ second: '(?:\b[A-Z][a-z]+ )+\(([A-Z]{3,5})\)'
 exceptions:
   - API
   - ASP
+  - CDN
   - CLI
   - CPU
   - CSS

.github/styles/Snowplow/Headings.yml

@@ -9,6 +9,7 @@ exceptions:
   - CLI
   - Cosmos
   - Docker
+  - DOM
   - Emmet
   - gRPC
   - I

.github/styles/config/vocabularies/snowplow/accept.txt

@@ -139,3 +139,5 @@ READMEs
 [uU]tils
 [vV]iewport
 [wW]alkthrough
+
+agentic

.gitignore

@@ -28,3 +28,6 @@ yarn-error.log*
 # Python
 __pycache__
 manifests
+
+# Local Netlify folder
+.netlify

README.md

@@ -67,7 +67,7 @@ Vale only checks normal prose. Text that's marked as code—code blocks or in-li
 
 To install the extension, find "[Vale VSCode](https://marketplace.visualstudio.com/items?itemName=ChrisChinchilla.vale-vscode)" in the Extensions Marketplace within VS Code, then click **Install**.
 
-The Vale extension will automatically check files when they're opened or saved. It underlines the flagged sections in different colours, based on the severity of the alert - red for errors, orange for warnings, and blue for suggestions. Mouse-over the underlined section to see the alert message, or check the VS Code **Problems** tab.
+The Vale extension will automatically check files when they're opened or saved. It underlines the flagged sections in different colors, based on the severity of the alert - red for errors, orange for warnings, and blue for suggestions. Mouse-over the underlined section to see the alert message, or check the VS Code **Problems** tab.
 
 ### Vale command-line interface
 

docs/api-reference/failed-events/index.md

@@ -116,7 +116,7 @@ Adapter failure schema can be found [here](https://github.com/snowplow/iglu-cent
 
 ## Tracker protocol violation
 
-This failure type is produced by the [Enrich](/docs/pipeline/enrichments/index.md) application, when an HTTP request does not conform to our [Snowplow Tracker Protocol](/docs/sources/trackers/snowplow-tracker-protocol/index.md).
+This failure type is produced by the [Enrich](/docs/pipeline/enrichments/index.md) application, when an HTTP request does not conform to our [Snowplow Tracker Protocol](/docs/events/index.md).
 
 <details>
 

docs/api-reference/loaders-storage-targets/bigquery-loader/_deploy-overview.md

@@ -19,4 +19,4 @@ To run the loader, mount your config file into the docker image, and then provid
   --iglu-config /myconfig/iglu.hocon
 `}</CodeBlock>
 
-Where `loader.hocon` is loader's [configuration file](/docs/api-reference/loaders-storage-targets/bigquery-loader/#configuring-the-loader) and `iglu.hocon` is [iglu resolver](/docs/api-reference/iglu/iglu-resolver/index.md) configuration.
+Where `loader.hocon` is loader's [configuration file](/docs/api-reference/loaders-storage-targets/bigquery-loader/index.md#configuring-the-loader) and `iglu.hocon` is [iglu resolver](/docs/api-reference/iglu/iglu-resolver/index.md) configuration.

docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/index.md

@@ -9,7 +9,6 @@ import {versions} from '@site/src/componentVersions';
 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
 import DeltaConfig from '@site/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/_delta_config.md';
-import HudiConfig from '@site/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/_hudi_config.md';
 import IcebergBigLakeConfig from '@site/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/_iceberg_biglake_config.md';
 import IcebergGlueConfig from '@site/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/_iceberg_glue_config.md';
 import PubsubConfig from '@site/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/_pubsub_config.md';
@@ -52,23 +51,6 @@ import Admonition from '@theme/Admonition';
     </table>
   </TabItem>
 
-  <TabItem value="hudi" label="Hudi">
-    <Admonition type="note" title="Alternative Docker image">
-    To use the Lake Loader with Hudi support, pull the appropriate alternative image from Docker Hub, e.g. <code>snowplow/lake-loader-aws:{`${versions.lakeLoader}`}-hudi</code>.
-    </Admonition>
-    <table>
-        <thead>
-            <tr>
-                <th>Parameter</th>
-                <th>Description</th>
-            </tr>
-        </thead>
-        <tbody>
-          <HudiConfig/>
-        </tbody>
-    </table>
-  </TabItem>
-
 </Tabs>
 
 ### Streams configuration

docs/api-reference/loaders-storage-targets/lake-loader/index.md

@@ -17,7 +17,7 @@ The Lake Loader is an application that loads Snowplow events to a cloud storage
 
 :::info Open Table Formats
 
-The Lake Loader supports the three major Open Table Formats: [Delta](https://delta.io/), [Iceberg](https://iceberg.apache.org/) and [Hudi](https://hudi.apache.org/).
+The Lake Loader supports the two major Open Table Formats: [Delta](https://delta.io/) and [Iceberg](https://iceberg.apache.org/).
 
 For Iceberg tables, the loader supports [AWS Glue](https://docs.aws.amazon.com/glue/) as catalog.
 

docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/postgres-loader-configuration-reference/index.md

@@ -6,10 +6,10 @@ sidebar_position: 0
 
 This is a complete list of the options that can be configured in the postgres loader's HOCON config file. The [example configs in github](https://github.com/snowplow-incubator/snowplow-postgres-loader/tree/master/config) show how to prepare an input file.
 
-<table class="has-fixed-layout"><tbody><tr><td><code>input.type</code></td><td>Required. Can be "Kinesis", "PubSub" or "Local". Configures where input events will be read from.</td></tr><tr><td><code>input.streamName</code></td><td>Required when <code>input.type</code> is Kinesis. Name of the Kinesis stream to read from.</td></tr><tr><td><code>input.region</code></td><td>Required when <code>input.type</code> is Kinesis. AWS region in which the Kinesis stream resides.</td></tr><tr><td><code>input.initialPosition</code></td><td>Optional. Used when <code>input.type</code> is Kinesis. Use "TRIM_HORIZON" (the default) to start streaming at the last untrimmed record in the shard, which is the oldest data record in the shard. Or use "LATEST" to start streaming just after the most recent record in the shard.</td></tr><tr><td><code>input.retrievalMode.type</code></td><td>Optional. When <code>input.type</code> is Kinesis, this sets the polling mode for retrieving records. Can be "FanOut" (the default) or "Polling".</td></tr><tr><td><code>input.retrievalMode.maxRecords</code></td><td>Optional. Used when <code>input.retrievalMode.type</code> is "Polling". Configures how many records are fetched in each poll of the kinesis stream. Default 10000.</td></tr><tr><td><code>input.projectId</code></td><td>Required when <code>input.type</code> is PubSub. The name of your GCP project.</td></tr><tr><td><code>input.subscriptionId</code></td><td>Required when <code>input.type</code> is PubSub. Id of the PubSub subscription to read events from</td></tr><tr><td><code>input.path</code></td><td>Required when <code>input.type</code> is Local. Path for event source. It can be directory or file. If it is directory, all the files under given directory will be read recursively. Also, given path can be both absolute path or relative path w.r.t. executable.</td></tr><tr><td><code>output.good.host</code></td><td>Required. Hostname of the postgres database.</td></tr><tr><td><code>output.good.port</code></td><td>Optional. Port number of the postgres database. Default 5432.</td></tr><tr><td><code>output.good.database</code></td><td>Required. Name of the postgres database.</td></tr><tr><td><code>output.good.username</code></td><td>Required. Postgres role name to use when connecting to the database</td></tr><tr><td><code>output.good.password</code></td><td>Required. Password for the postgres user.</td></tr><tr><td><code>output.good.schema</code></td><td>Required. The Postgres schema in which to create tables and write events.</td></tr><tr><td><code>output.good.sslMode</code></td><td>Optional. Configures how the client and server agree on ssl protection. Default "REQUIRE"</td></tr><tr><td><code>output.bad.type</code></td><td>Optional. Can be "Kinesis", "PubSub", "Local" or "Noop". Configures where failed events will be sent. Default is "Noop" which means failed events will be discarded</td></tr><tr><td><code>output.bad.streamName</code></td><td>Required when <code>bad.type</code> is Kinesis. Name of the Kinesis stream to write to.</td></tr><tr><td><code>output.bad.region</code></td><td>Required when <code>bad.type</code> is Kinesis. AWS region in which the Kinesis stream resides.</td></tr><tr><td><code>output.bad.projectId</code></td><td>Required when <code>bad.type</code> is PubSub. The name of your GCP project.</td></tr><tr><td><code>output.bad.topicId</code></td><td>Required when <code>bad.type</code> is PubSub. Id of the PubSub topic to write failed events to</td></tr><tr><td><code>output.bad.path</code></td><td>Required when <code>bad.type</code> is Local. Path of the file to write failed events</td></tr><tr><td><code>purpose</code></td><td>Optional. Set this to "ENRICHED_EVENTS" (the default) when reading the stream of enriched events in tsv format. Set this to "JSON" when reading a stream of self-describing json, e.g. snowplow [bad rows](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows).</td></tr><tr><td><code>monitoring.metrics.cloudWatch</code></td><td>Optional boolean, with default true. For kinesis input, this is used to disable sending metrics to cloudwatch.</td></tr></tbody></table>
+<table className="has-fixed-layout"><tbody><tr><td><code>input.type</code></td><td>Required. Can be "Kinesis", "PubSub" or "Local". Configures where input events will be read from.</td></tr><tr><td><code>input.streamName</code></td><td>Required when <code>input.type</code> is Kinesis. Name of the Kinesis stream to read from.</td></tr><tr><td><code>input.region</code></td><td>Required when <code>input.type</code> is Kinesis. AWS region in which the Kinesis stream resides.</td></tr><tr><td><code>input.initialPosition</code></td><td>Optional. Used when <code>input.type</code> is Kinesis. Use "TRIM_HORIZON" (the default) to start streaming at the last untrimmed record in the shard, which is the oldest data record in the shard. Or use "LATEST" to start streaming just after the most recent record in the shard.</td></tr><tr><td><code>input.retrievalMode.type</code></td><td>Optional. When <code>input.type</code> is Kinesis, this sets the polling mode for retrieving records. Can be "FanOut" (the default) or "Polling".</td></tr><tr><td><code>input.retrievalMode.maxRecords</code></td><td>Optional. Used when <code>input.retrievalMode.type</code> is "Polling". Configures how many records are fetched in each poll of the kinesis stream. Default 10000.</td></tr><tr><td><code>input.projectId</code></td><td>Required when <code>input.type</code> is PubSub. The name of your GCP project.</td></tr><tr><td><code>input.subscriptionId</code></td><td>Required when <code>input.type</code> is PubSub. Id of the PubSub subscription to read events from</td></tr><tr><td><code>input.path</code></td><td>Required when <code>input.type</code> is Local. Path for event source. It can be directory or file. If it is directory, all the files under given directory will be read recursively. Also, given path can be both absolute path or relative path w.r.t. executable.</td></tr><tr><td><code>output.good.host</code></td><td>Required. Hostname of the postgres database.</td></tr><tr><td><code>output.good.port</code></td><td>Optional. Port number of the postgres database. Default 5432.</td></tr><tr><td><code>output.good.database</code></td><td>Required. Name of the postgres database.</td></tr><tr><td><code>output.good.username</code></td><td>Required. Postgres role name to use when connecting to the database</td></tr><tr><td><code>output.good.password</code></td><td>Required. Password for the postgres user.</td></tr><tr><td><code>output.good.schema</code></td><td>Required. The Postgres schema in which to create tables and write events.</td></tr><tr><td><code>output.good.sslMode</code></td><td>Optional. Configures how the client and server agree on ssl protection. Default "REQUIRE"</td></tr><tr><td><code>output.bad.type</code></td><td>Optional. Can be "Kinesis", "PubSub", "Local" or "Noop". Configures where failed events will be sent. Default is "Noop" which means failed events will be discarded</td></tr><tr><td><code>output.bad.streamName</code></td><td>Required when <code>bad.type</code> is Kinesis. Name of the Kinesis stream to write to.</td></tr><tr><td><code>output.bad.region</code></td><td>Required when <code>bad.type</code> is Kinesis. AWS region in which the Kinesis stream resides.</td></tr><tr><td><code>output.bad.projectId</code></td><td>Required when <code>bad.type</code> is PubSub. The name of your GCP project.</td></tr><tr><td><code>output.bad.topicId</code></td><td>Required when <code>bad.type</code> is PubSub. Id of the PubSub topic to write failed events to</td></tr><tr><td><code>output.bad.path</code></td><td>Required when <code>bad.type</code> is Local. Path of the file to write failed events</td></tr><tr><td><code>purpose</code></td><td>Optional. Set this to "ENRICHED_EVENTS" (the default) when reading the stream of enriched events in tsv format. Set this to "JSON" when reading a stream of self-describing json, e.g. snowplow [bad rows](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows).</td></tr><tr><td><code>monitoring.metrics.cloudWatch</code></td><td>Optional boolean, with default true. For kinesis input, this is used to disable sending metrics to cloudwatch.</td></tr></tbody></table>
 
 #### Advanced options
 
 We believe these advanced options are set to sensible defaults, and hopefully you won't need to ever change them.
 
-<table class="has-fixed-layout"><tbody><tr><td><code>backoffPolicy.minBackoff</code></td><td>If producer (PubSub or Kinesis) fails to send item, it will retry to send it again. This field configures backoff time for first retry. Every retry will double the backoff time of previous one.</td></tr><tr><td><code>backoffPolicy.maxBackoff</code></td><td>Maximum backoff time for retry. After this value is reached, backoff time will no more increase.</td></tr><tr><td><code>input.checkpointSettings.maxBatchSize</code></td><td>Used when <code>input.type</code> is Kinesis. Determines the max number of records to aggregate before checkpointing the records. Default is 1000.</td></tr><tr><td><code>input.checkpointSettings.maxBatchWait</code></td><td>Used when <code>input.type</code> is Kinesis. Determines the max amount of time to wait before checkpointing the records. Default is 10 seconds.</td></tr><tr><td><code>input.checkpointSettings.maxConcurrent</code></td><td>Used when <code>input.type</code> is PubSub. The max number of concurrent evaluation for checkpointer.</td></tr><tr><td><code>output.good.maxConnections</code></td><td>Maximum number of connections database pool is allowed to reach. Default 10</td></tr><tr><td><code>output.good.threadPoolSize</code></td><td>Size of the thread pool for blocking database operations. Default is value of "maxConnections"</td></tr><tr><td><code>output.bad.delayThreshold</code></td><td>Set the delay threshold to use for batching. After this amount of time has elapsed (counting from the first element added), the elements will be wrapped up in a batch and sent. Default 200 milliseconds</td></tr><tr><td><code>output.bad.maxBatchSize</code></td><td>A batch of messages will be emitted when the number of events in batch reaches the given size. Default 500</td></tr><tr><td><code>output.bad.maxBatchBytes</code></td><td>A batch of messages will be emitted when the size of the batch reaches the given size. Default 5 MB</td></tr></tbody></table>
+<table className="has-fixed-layout"><tbody><tr><td><code>backoffPolicy.minBackoff</code></td><td>If producer (PubSub or Kinesis) fails to send item, it will retry to send it again. This field configures backoff time for first retry. Every retry will double the backoff time of previous one.</td></tr><tr><td><code>backoffPolicy.maxBackoff</code></td><td>Maximum backoff time for retry. After this value is reached, backoff time will no more increase.</td></tr><tr><td><code>input.checkpointSettings.maxBatchSize</code></td><td>Used when <code>input.type</code> is Kinesis. Determines the max number of records to aggregate before checkpointing the records. Default is 1000.</td></tr><tr><td><code>input.checkpointSettings.maxBatchWait</code></td><td>Used when <code>input.type</code> is Kinesis. Determines the max amount of time to wait before checkpointing the records. Default is 10 seconds.</td></tr><tr><td><code>input.checkpointSettings.maxConcurrent</code></td><td>Used when <code>input.type</code> is PubSub. The max number of concurrent evaluation for checkpointer.</td></tr><tr><td><code>output.good.maxConnections</code></td><td>Maximum number of connections database pool is allowed to reach. Default 10</td></tr><tr><td><code>output.good.threadPoolSize</code></td><td>Size of the thread pool for blocking database operations. Default is value of "maxConnections"</td></tr><tr><td><code>output.bad.delayThreshold</code></td><td>Set the delay threshold to use for batching. After this amount of time has elapsed (counting from the first element added), the elements will be wrapped up in a batch and sent. Default 200 milliseconds</td></tr><tr><td><code>output.bad.maxBatchSize</code></td><td>A batch of messages will be emitted when the number of events in batch reaches the given size. Default 500</td></tr><tr><td><code>output.bad.maxBatchBytes</code></td><td>A batch of messages will be emitted when the size of the batch reaches the given size. Default 5 MB</td></tr></tbody></table>