Skip to content

Add pre-commit that verifies if shared project are properly structured #53697

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 25, 2025

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Jul 24, 2025

Some basic checks for structure of the shared projects - includig package structure and pyproject.toml contents.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:dev-tools backport-to-v3-0-test Mark PR with this label to backport to v3-0-test branch labels Jul 24, 2025
@potiuk
Copy link
Member Author

potiuk commented Jul 24, 2025

First small pre-commit - for now just checking if shared projects are properly structured.

@potiuk
Copy link
Member Author

potiuk commented Jul 24, 2025

I also moved "test_timezones" to "timezones" subpackage to add a namespace for tests - when we run the tests from IDE, all tests folders are part of the same namespace so we should namespace them too to not get confused with tests from other distributions.

@potiuk
Copy link
Member Author

potiuk commented Jul 24, 2025

Also maybe it's good to also add "airflow_shared" implicit package for "tests" (for even better namespacing?) WDYT?

@potiuk
Copy link
Member Author

potiuk commented Jul 24, 2025

I am also thinking about globally changing other namespaces of ours "the unit, system, integration" at the top level of "tests" is quite confusing so maybe we could introduce a convention similar to:

  • tests/airflow_providers/{unit,system,integration}/provider/
  • tests/airflow_core/{unit,system,integration}/something
  • tests/airflow_shared/{unit}/timezones

That would make some of the cross-package imports much more 'reasonable"

from unit.amazon -> from airflow_providers.unit.amazon

Maybe that would be our target convention?

I am happy to raise a question / proposal at the devlist if we think it is a good idea. We can do it all with implicit namespaces.

@potiuk
Copy link
Member Author

potiuk commented Jul 24, 2025

Also we could potentially do "from airflow_providers/amazon/unit" which seems a bit more logical, but it will complicate a bit the selection of tests to execute (but it's also possible).

@amoghrajesh
Copy link
Contributor

I am also thinking about globally changing other namespaces of ours "the unit, system, integration" at the top level of "tests" is quite confusing so maybe we could introduce a convention similar to:

  • tests/airflow_providers/{unit,system,integration}/provider/
  • tests/airflow_core/{unit,system,integration}/something
  • tests/airflow_shared/{unit}/timezones

That would make some of the cross-package imports much more 'reasonable"

from unit.amazon -> from airflow_providers.unit.amazon

Maybe that would be our target convention?

I am happy to raise a question / proposal at the devlist if we think it is a good idea. We can do it all with implicit namespaces.

I like it. Will follow up on your email.

Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a few comments here, but its looking nice generally.

BTW, no need to backport it.

@potiuk potiuk force-pushed the add-check-shared-projects-pre-commit branch from 80a036a to 7df94c5 Compare July 25, 2025 09:02
@potiuk
Copy link
Member Author

potiuk commented Jul 25, 2025

Nice copilot, good copilot :D

Some basic checks for structure of the shared projects - includig
package structure and pyproject.toml contents.
@potiuk potiuk force-pushed the add-check-shared-projects-pre-commit branch from 7df94c5 to b540e32 Compare July 25, 2025 09:08
Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Niceee! Love it!

@potiuk
Copy link
Member Author

potiuk commented Jul 25, 2025

I merge it for now - we can also restructure tests/structure separately.

@potiuk potiuk merged commit e3ddf97 into apache:main Jul 25, 2025
101 of 102 checks passed
@potiuk potiuk deleted the add-check-shared-projects-pre-commit branch July 25, 2025 14:26
Copy link

Backport failed to create: v3-0-test. View the failure log Run details

Status Branch Result
v3-0-test Commit Link

You can attempt to backport this manually by running:

cherry_picker e3ddf97 v3-0-test

This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dev-tools backport-to-v3-0-test Mark PR with this label to backport to v3-0-test branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants