[RFD-0001] Multi-Project Workflow Hierarchy

[AleksandarCole]

[RFD - 0001] - Multi-Project Workflow Hierarchy

Authors	State
@darkofabijan @AleksandarCole @DamjanBecirovic	Draft

What

Semaphore allows users to trigger workflows via API. In certain cases, a workflow in Project A may trigger a workflow in Project B, creating a chain of dependent workflows. The multi-project workflow hierarchy feature will enable users to:

• Visualize dependencies between these workflows.
• Easily navigate between related workflows.
• Understand the parent-child relationships among workflows.

Why

Currently, projects in Semaphore are repository-based and operate in isolation, without sharing information. However, there are many scenarios where an action in one repository needs to trigger a build in another. For users with such interconnected workflows, there is presently no way to see the relationships between these builds. This feature will provide visibility into these inter-project dependencies, improving the management and understanding of complex workflows.

---

Details

Use-cases

In an organization with several projects and independent teams, API calls are employed within each project to initiate workflows across various projects based on pipeline conditions.

For instance, a workflow within project A can trigger workflows across project B, project C, and project D. Moreover, project C can further trigger workflows within project E and project F.

The team working on project C wants to:

• Easily find the parent workflow on project A that triggered their workflow.
• Easily find child workflows on project E and project F that were triggered by their workflow.
• See the state and status of these workflows.
• Navigate between them seamlessly.

The lead or DevOps person wants to:

• Understand the entire dependency tree.
• See the status of each build and identify which are not passing.
• Easily navigate between them.

Problems

Problem 1 - Traceability > No parent traceability in API

Users can run the workflow through the API in the following way:

curl -i -H "Authorization: Token {api_token}" \
-d "project_id={project_id}&reference={reference}" \
-X POST  "https://{org_name}.semaphoreci.com/api/v1alpha/plumber-workflows"

However, there is no way to forward additional metadata about the parent workflow that initiated the API call.

Problem 2 - Storage > No way to store dependency tree

There is no place in-app that allows you to store this dependency tree. On idea was to store it in the artifacts but they are scoped on project level. This information would need to be available across multiple projects within the organization.

Problem 3 - Navigation/UX > No UI that displays this info on workflow page

If this info were available, it would need to be easily accessible within the workflow page. For example, the team that works on project C would open a project C workflow and should be able to see the dependency tree for that workflow somewhere in UI.

At this moment we don't have any section that would render this.

Problem 4 - Navigation/UX > No links in job logs

Workflow on project A triggers through an API call in a job workflow on project C. The API returns information about child workflow that was triggered in a response, allowing the link to said workflow to be printed out in job log output.

However, we do not render links in the job logs, so the user that wants to open it has to copy the url string and paste it in a browser.

Solutions

Solution 1 - Traceability > Extend the API

We can extend the API to support the additional parameters describing the parent workflow like this:

curl -i -H "Authorization: Token {api_token}" \
     -d "project_id={project_id}" \
     -d "reference={reference}" \
     -d "parent_wf_id=$SEMAPHROE_WORKFLOW_ID" \
     -d "parent_ppl_id=$SEMAPHROE_PIPELINE_ID" \
     -d "parent_job_id=$SEMAPHROE_JOB_ID" \
     -X POST  "https://{org_name}.semaphoreci.com/api/v1alpha/plumber-workflows"

This way we would:

• Know that the triggered workflow is a part of a dependency tree
• Have the information about parent workflow that triggered it
• Be able to export parent workflow information as environment variables in child workflow

Potential extended scope to explore: All this information is available in the OIDC token. User could just pass the token in the API.

Solution 2 - Traceability > Extend the promotions

Alternatively, we could extend the promotions to allow builds on other projects to be triggered like this:

promotions:
  - name: "Start Project C"
    project: project_c
      reference: `refs/heads/master`
    auto_promote:
      when: branch = 'main' AND result = 'passed'

This would make such workflows a first-class citizen, however this idea needs to be discussed with the pipelines team as there are many open questions (eg. permissions for triggering a workflow on a completely different project).

Solution 3 - Storage > S3 bucket on organization level

Similar to artifacts, we could introduce the option of having an S3 bucket scoped on the organization level.

This storage can then be in turn used by the users to generate various reports in their jobs and edit them. We can then render these user-generated artifacts in a special part of the UI if they fit predefined format.

Pros:

• Can be used for multiple other things as user can push any info in these report files.
• It's basically organization level artifacts

Cons:

• A lot of manual work on user end to build reports for the "dependency tree" use-case
• Hard to keep up to date
• Still need to figure out how to connect correct reports to wf/job
• permission & data retention policies have to be considered

Solution 4 - Storage > Introduce a new table

We can introduce a new table for the purpose of tracking these dependencies. Engineering team believes that introducing such table would be faster than Solution 3 and more straightforward. It would also allow us to build better UX.

Pros:

• Straightforward implementation
• No need for user to manually collect data
• Easy to update, easy to pull data from other tables

Cons:

• It would not be as flexible as Solution 3 > it would store only data related to dependency tree
• Needs to be planned carefully > Once introduced it will be hard to change, if we take this route we need to make sure we're collecting all the data that we will need in the future.

Solution 5 - Navigation/UX > Reports tab

App design link: HERE

We can add a new tab on workflow and job pages called Report

Content of this tab would be generated based on the .html or .md files built by user and stored in the bucket described in Solution 3. Front would need to be able to somehow identify the correct file to render in the report tab.

Assuming that the user generates a report that looks like this:

<h3>External reports</h3>
<ul>
    <li><a href="#">Datadog dashboard</a></li>
    <li><a href="#">Codecov report</a></li>
</ul>
<h3>Dependency tree</h3>
<li>project-a: <a href="#">3daddec5-9f12-428a-bf28-1d4366d47aed</a></li>
<ul>
    <li><b>project-c</b> <a href="#">a4af7b4b-b487-4a35-ac17-fb7a88f0bccd</a></li>
    <li>project-b: <a href="#">bebd7dda-b433-4ffa-8604-b7dc7e7cc06f</a></li>
    <ul>
        <li>project-e: <a href="#">01e63dd8-9e5b-43d1-90f3-d6a690876e2b</a></li>
        <li>project-f: <a href="#">d5bf3fe7-8b96-4a41-b9fb-f0e42e16b18e</a></li>
    </ul>
    <li>project-d: <a href="#">b11a87d7-97bb-4222-9470-f405268c3388</a></li>
</ul>

The content of the report would be rendered on the workflow page like this:

Solution 6 - Navigation/UX > Hierarchy tab

App design link: HERE

This solution depends on Solution 4 for data storage.
If we go this route, it would not be too complex to build a new tab on the workflow page called Hierarchy that contains full dependency tree.

This tab would appear only if there is are parent or child workflows exist, and would show the full chain. The current workflow would be marked in the chain by blue outline.

It would look something like this:

Screen.Recording.2024-06-07.at.10.47.19.mov

This is not a final design, there's some additional UI/UX things to figure out (eg. branches, collapsing the boxes a bit like what we do for tasks etc.)

If this solution is combined with Solution 2 we could make multi-project workflows a first-class citizen.

Solution 7 - Navigation/UX > Links in job logs

This is WIP and hasn't been reviewed with engineering yet. Leaving it here as a note, but it will probably be extracted as a separate discussion.

---

Next steps

• Collect feedback from users through this discussion to further refine the idea > @AleksandarCole
• Decide on the approach to pursue > @AleksandarCole @darkofabijan
• Build a full feature spec doc with help from eng. > @AleksandarCole
• Log and kick off the project > @lucianoliberti @darkofabijan