[Feature] Jobs mesh setup

[amir-nazarbeigi-ay]

Describe the feature
Hi,

I'm not entirely sure which category this request falls under, so please advise if this feature already exists or if there's a workaround I'm missing.

We have a mesh setup within our DBT projects, where multiple repositories are involved. In this setup:

• We have common jobs that are identical across all projects and are pushed from our main repository.
• At the same time, each project repository needs the ability to define their own custom jobs that are specific to their project.

The issue we're encountering is that when project repositories create their own jobs, it seems to override (or delete) the common jobs created from the main repository.

We'd like to request support for a feature (or configuration) that enables:

• Preserving common jobs from the main repository.
• Allowing each project to define and maintain their custom jobs without interfering with the common ones.

Let me know if you need additional details or if there's documentation I may have missed that covers this behavior.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Who will this benefit?
What kind of use case will this feature be useful for? Please be specific and provide examples, this will help us prioritize properly.
Organizations with multi project setup with need for common jobs.

Additional context
Add any other context or screenshots about the feature request here.

Are you interested in contributing this feature?
Let us know if you want to write some code, and how we can help!

[b-per]

Hi! Thanks for raising this feature request!

Would adding a new parameter called --exclude-ids-matching for plan and sync work in your use case?

The idea is that if this parameter is provided, we would check if remote jobs are matching it from a regex point of view, and we wouldn't retrieve them when running dbt-jobs-as-code

So, you could create your common jobs with an ID starting with com and if you do a

dbt-jobs-as-code sync myjobs.yml --exclude-ids-matching "^com"

We would not remove the jobs which ID start with com

[amir-nazarbeigi-ay]

Hi,

Thanks for the suggestion! This approach would certainly be helpful, but I foresee some challenges with relying on job IDs in our particular use case.

The main concern with this approach is that job IDs might change over time across our 13 projects that share 7 common jobs each. Managing exclusions based on job IDs could become cumbersome to maintain.

It would be much more manageable if exclusions could instead be based on job names rather than IDs. Job names are consistent across projects and are less likely to change, making it easier for us to filter just the 7 common job names in each project.

Let me know what you think, and thanks again for working on this!

[b-per]

When I say identifier, it is not the same as ID. (i can maybe choose a better parameter name then)

• identifier:
  • the part of the job name between [[ ... ]]
  • the key for the job in the YAML file
• ID
  • the number that identifies a job in the platform e.g. 12345

Would filtering out the identifiers work for you?