Skip to content

Commit 9e17cba

Browse files
jackywang-dbsryza
authored andcommitted
[SPARK-52714][SDP] Remove unused comment arg in append_flow decorator
### What changes were proposed in this pull request? In Spark Declarative Pipelines (SDP), users can define append flows in Python using the [append_flow](https://github.com/apache/spark/blob/e3321aa44ea255365222c491657b709ef41dc460/python/pyspark/pipelines/api.py#L34-L41) decorator. The append_flow decorator currently accepts a `comment` arg. However, there is no way for user to see flow comments as of now. Therefore, this argument is unused and not referenced in function body. ```py def append_flow( *, target: str, name: Optional[str] = None, comment: Optional[str] = None, # <--- Removing spark_conf: Optional[Dict[str, str]] = None, once: bool = False, ) -> Callable[[QueryFunction], None]: ``` This PR removes the field. ### Why are the changes needed? The `comment` arg is not being used anywhere and having it in the API will confuse the user thinking they can see flow comments somewhere. ### Does this PR introduce _any_ user-facing change? Yes, the previously optional `comment` arg is removed from the `append_flow` API. However, SDP has not been released yet (pending release in v4.1), so no user should be impacted by this change. ### How was this patch tested? Examined all testcases to make sure none of the current append_flow usage is supplying this argument ### Was this patch authored or co-authored using generative AI tooling? No Closes #51404 from JiaqiWang18/SPARK-52714-remove-append_flow-comment. Authored-by: Jacky Wang <jacky.wang@databricks.com> Signed-off-by: Sandy Ryza <sandy.ryza@databricks.com>
1 parent 18e995f commit 9e17cba

File tree

1 file changed

+0
-2
lines changed
  • python/pyspark/pipelines

1 file changed

+0
-2
lines changed

python/pyspark/pipelines/api.py

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,6 @@ def append_flow(
3535
*,
3636
target: str,
3737
name: Optional[str] = None,
38-
comment: Optional[str] = None,
3938
spark_conf: Optional[Dict[str, str]] = None,
4039
once: bool = False,
4140
) -> Callable[[QueryFunction], None]:
@@ -44,7 +43,6 @@ def append_flow(
4443
4544
:param name: The name of the flow. If unspecified, the query function's name will be used.
4645
:param target: The name of the dataset this flow writes to. Must be specified.
47-
:param comment: Description of the flow. If unspecified, the dataset's comment will be used.
4846
:param spark_conf: A dict whose keys are the conf names and values are the conf values. \
4947
These confs will be set when the flow is executed; they can override confs set for the \
5048
destination, for the pipeline, or on the cluster.

0 commit comments

Comments
 (0)