Skip to content

Parallelize fork #15

@stoney95

Description

@stoney95

Context

The package is providing the fork function. This function takes multiple functions and forwards the intermediate result to every of these functions. By this multiple independet branches are created that all consume the result of the previous pipeline step.

Description

Currently the fork function is executing the branches sequentially. It would be nice to have a version that is parallelizing these steps.

Hints

  • A new module should provide a parallel version of fork. The new function should be imported by from pypely.parallel import parallel_fork
  • It needs to guaranteed that the order of the result is kept. The order the functions are given in needs to reflect the order of their outputs.
  • Take care about variable references. It might be reasonable to create a copy of the intermediate result.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions