Proposal to Enhance WasbPrefixSensor with XCom Push for Blob Name Detection

[samuelkhtu]

Hello Airflow Community,

I would like to propose an enhancement to the WasbPrefixSensor to include an XCom push feature that captures the names of the blobs detected. This enhancement would allow downstream tasks to easily access the list of detected blob names using XCom pull.

Current Functionality:
The WasbPrefixSensor currently detects the presence of blobs with a specified prefix in an Azure Blob Storage container. However, it does not provide a straightforward way to pass the names of these detected blobs to downstream tasks.

Proposed Enhancement:
XCom Push: Modify the WasbPrefixSensor to push the list of detected blob names to XCom.
XCom Key: Use a specific XCom key to store the list of blob names, making it easy for downstream tasks to retrieve this information.

Benefits:
Improved Data Flow: Downstream tasks can directly access the list of detected blob names, simplifying the workflow and reducing the need for additional custom operators or tasks to fetch this information.
Enhanced Flexibility: Users can leverage the detected blob names in various downstream tasks, such as processing the blobs, logging, or triggering further actions based on the blob names.

Example Usage:

# Define the WasbPrefixSensor
detect_blobs = WasbPrefixSensor(
    task_id='detect_blobs',
    container_name='my-container',
    prefix='my-prefix',
    azure_conn_id='my_azure_conn',
    xcom_push=True  # New parameter to enable XCom push
)

# Downstream task using XCom pull to get the list of detected blob names
process_blobs = PythonOperator(
    task_id='process_blobs',
    python_callable=my_processing_function,
    op_args=[{{ ti.xcom_pull(task_ids='detect_blobs', key='blob_names') }}]
)

Questions for the Community:
Do you find this enhancement useful for your workflows?

• Are there any additional features or considerations that should be included in this proposal?
• Any potential challenges or drawbacks you foresee with this implementation?
• I look forward to hearing your thoughts and feedback on this proposal.

Thanks!!