Declarative Pipelines

Author: jaceklaskowski

docs/declarative-pipelines/.pages

@@ -0,0 +1,4 @@
+title: Declarative Pipelines
+nav:
+    - index.md
+    - ...

docs/declarative-pipelines/index.md

@@ -0,0 +1,14 @@
+---
+subtitle: 4.1.0-SNAPSHOT
+---
+
+# Declarative Pipelines
+
+**Spark Declarative Pipelines (SDP)** is a declarative framework for building ETL pipelines on Apache Spark.
+
+!!! danger
+    Declarative Pipelines framework is only available in the development branch of Apache Spark 4.1.0-SNAPSHOT.
+
+## Learning Resources
+
+1. [Spark Declarative Pipelines Programming Guide](https://github.com/apache/spark/blob/master/docs/declarative-pipelines-programming-guide.md)

docs/features/index.md

@@ -12,6 +12,7 @@ The following are the features of Spark SQL that help place it in the top of the
 * [Catalog Plugin API](../connector/catalog/index.md)
 * [Columnar Execution](../columnar-execution/index.md)
 * [Connector API](../connector/index.md)
+* [Declarative Pipelines](../declarative-pipelines/index.md) (4.1.0-SNAPSHOT)
 * [Default Columns](../default-columns/index.md)
 * [Direct Queries on Files](../direct-queries-on-files/index.md)
 * [Dynamic Partition Pruning](../dynamic-partition-pruning/index.md)

mkdocs.yml

@@ -206,6 +206,7 @@ nav:
       - SizeInBytesOnlyStatsPlanVisitor: cost-based-optimization/SizeInBytesOnlyStatsPlanVisitor.md
       - Statistics: cost-based-optimization/Statistics.md
     - ... | default-columns/**.md
+    - ... | declarative-pipelines/**.md
     - ... | direct-queries-on-files/**.md
     - Dynamic Partition Pruning:
       - dynamic-partition-pruning/index.md