Reevaluating Pipeline Separation: Addressing Complexity from Separated Lambda defined in Step Functions

[tzuwei93]

Note: This comparative analysis was developed with GitHub Copilot assistance to ensure comprehensive coverage of data modeling patterns.

Problem Statement

Splitting the pipeline into multiple Lambda functions and managing them with Step Functions has added significant complexity to this project.

Many of the functions in our pipeline are highly reusable. Spreading them across different Lambda containers can result in unnecessary duplication and increased maintenance overhead. For instance, functions like extract_s3_path and GooglePlacesClient are used in multiple places. And it is also difficult to trace relationships of table dependencies when there are bugs. Rather than managing this shared logic in a scattered manner, you could package these functions into a custom Python package or centralize all data processing logic within a single Python module. However, even these approaches fall short compared to adopting an open-source tool that is purpose-built for data pipeline orchestration and data transformation management.

Currently, the Lambda stages are tightly coupled: data and configurations are passed directly between them, so modifying or maintaining any single stage can affect the entire pipeline. While Step Functions are being used to orchestrate multiple Lambda workflows, they are not specifically designed for data pipelines. The UI and YAML configurations only offer a basic view of dependencies between stages, making the overall management of data transformations, dependencies, and pipeline maintenance less intuitive. For example, details such as data schemas or table definitions must still be defined and maintained separately within the code, rather than being natively managed by the orchestration tool.

To address these challenges more efficiently and intuitively, it is highly recommended to adopt an open-source tool that is specifically designed for development of data pipeline, data transformation.

[tzuwei93]

To address the pain described above, I'd like to adopt the following solution leveraging modern open-source tools:

1. Centralized Orchestration with Dagster
   I will use Dagster as the main orchestration framework for managing the entire data pipeline. Dagster is a orchestrator platform, which designed specifically for data pipeline development. By running Dagster locally, I can define, schedule, and monitor all pipeline stages in a unified and intuitive interface.

2. Data Transformation with dbt
   For all SQL-based data transformation tasks, I will integrate dbt (data build tool). dbt allows me to define data models, manage dependencies, and implement data quality tests in a maintainable and version-controlled manner. Its native support for modularity and documentation further reduces code duplication and improves pipeline transparency.

3. Compute Engine: Local Spark Thrift Server
   To handle data processing and transformations, I will deploy a local Spark Thrift Server. This setup allows dbt to execute SQL queries and transformations on Spark and Output result to Cloud Data Lake like S3.

How This Approach Addresses the Problems:

• Minimizes Duplication: Shared transformation logic is centralized in dbt models and Python modules, eliminating the need to duplicate functions across Lambda containers.
• Native Support for Data Pipeline Needs: The combination of Dagster and dbt provides first-class support for managing data schemas, dependencies, and pipeline metadata—something that Step Functions and Lambda lack natively.

[tzuwei93]

Addressing above issue in this new repository nbe-user-code .