You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-52714][SDP] Remove unused comment arg in append_flow decorator
### What changes were proposed in this pull request?
In Spark Declarative Pipelines (SDP), users can define append flows in Python using the [append_flow](https://github.com/apache/spark/blob/e3321aa44ea255365222c491657b709ef41dc460/python/pyspark/pipelines/api.py#L34-L41) decorator. The append_flow decorator currently accepts a `comment` arg. However, there is no way for user to see flow comments as of now. Therefore, this argument is unused and not referenced in function body.
```py
def append_flow(
*,
target: str,
name: Optional[str] = None,
comment: Optional[str] = None, # <--- Removing
spark_conf: Optional[Dict[str, str]] = None,
once: bool = False,
) -> Callable[[QueryFunction], None]:
```
This PR removes the field.
### Why are the changes needed?
The `comment` arg is not being used anywhere and having it in the API will confuse the user thinking they can see flow comments somewhere.
### Does this PR introduce _any_ user-facing change?
Yes, the previously optional `comment` arg is removed from the `append_flow` API. However, SDP has not been released yet (pending release in v4.1), so no user should be impacted by this change.
### How was this patch tested?
Examined all testcases to make sure none of the current append_flow usage is supplying this argument
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#51404 from JiaqiWang18/SPARK-52714-remove-append_flow-comment.
Authored-by: Jacky Wang <jacky.wang@databricks.com>
Signed-off-by: Sandy Ryza <sandy.ryza@databricks.com>
0 commit comments