Speed improvements

[gwdekker]

Hey, for our repo speed starts to become an issue.

In our CI, we run a bunch of linting and static checking tools, including tach, mypy, several pre-commit checks, and poly check. We see that poly check (or more precise: uv run poly check --strict --alias=<bunch of aliases> takes about 30 seconds now.

I did some profiling with some interesting results already:

1. it seems that --strict is slightly quicker than non strict
2. It seems that an easy parallelization to run over projects in parallel makes things a bit quicker! The below is in my makefile now

poly-check:
	@echo "Running poly check in parallel for all projects..."
	@projects=$$(find projects -mindepth 1 -maxdepth 1 -type d -exec basename {} \;); \
	pids=""; \
	for project in $$projects; do \
		echo "🔍 Checking project: $$project"; \
		(uv run poly check --directory projects/$$project --strict --alias=apache-airflow=airflow, <more aliases> || (echo "❌ $$project failed"; exit 1)) & \
		pids="$$pids $$!"; \
	done; \
	failed=0; \
	for pid in $$pids; do \
		if ! wait $$pid; then \
			failed=1; \
		fi; \
	done; \
	if [ $$failed -eq 1 ]; then \
		echo "❌ Some poly checks failed!"; \
		echo "💡 Hint: Try running 'poly sync' and/or `make poly-libs` to fix poly-related issues"; \
		exit 1; \
	else \
		echo "✅ All poly checks passed!"; \
	fi

This seems to indicate that there is room for parallelization as one first way to speed up things.

Could it be an idea to provide a flag to the relevant polylith commands which would internally parallelize over projects? By doing it in pyhton there is maybe even stuff that is in this setup ran redundantly and only parallelize the part that actually needs to be done per project.

[DavidVujic]

Thank you for reporting this @gwdekker! I haven't been aware of any performance issues before and will look into it. Does it behave the same on your local machine?

Meanwhile, to speed things up, I would suggest to run the poly check command only for the projects that has been changed.

[DavidVujic]

Are there a lot of Python modules/files in there? I try by adding many copies of components (and including them in the projects) in a polylith repo, but cannot reproduce any significant speed issues.

Maybe I have missed out on something, such as resource files that should be ignored during the check - or if there are a very large amount of files (the caching is using the relatively low default of 128 hits). Can you add some more details @gwdekker, such as amount of bricks, total files (and LOC if possible), number of projects? Also, if you have anything else than Python files in the bricks would be very useful to know. 🙏

[gwdekker]

Claude code generated summary (looks roughly correct to me): Overall Stats: - Projects: 15 - Components: 79 - Bases: 36 - Development: 1 Lines of Code - Total Python LoC: 146,773 - Python files: 4,930 total In the bricks itself, there are a few non .py files but not many: ⏺ Non-Python File Types in Bases and Components File Type Breakdown (excluding .pyc cache files) | File Type | Count | Purpose | |-----------------|-------|-----------------------------------------------------| | py.typed | 136 | MyPy type checking markers (one per component/base) | | ipynb | 39 | Jupyter notebooks for analysis and prototyping | | md | 28 | Documentation, READMEs for components and bases | | json | 18 | Configuration files, API examples, test data | | yaml/yml | 6 | Configuration files, DAG configs | | png | 3 | Documentation images, diagrams | | nbi/nbc | 6 | Notebook-related cache files | | mo/pymoca_cache | 4 | Model/cache files | | gz | 2 | Compressed test data | | gitignore | 2 | Git ignore files | | svg | 1 | Vector graphics (architecture diagrams) | | sh | 1 | Shell script | | pkl | 1 | Pickled Python object | | joblib | 1 | Joblib serialized object | | gitkeep | 1 | Git placeholder file | Op vr 25 jul 2025 om 17:20 schreef David Vujic ***@***.***>:

…

Are there a lot of Python modules/files in there? I try by adding many copies of components (and including them in the projects) in a polylith repo, but cannot reproduce any significant speed issues. Maybe I have missed out on something, such as resource files that should be ignored during the check - or if there are a very large amount of files (the caching is using the relatively low default of 128 hits). Can you add some more details @gwdekker <https://github.com/gwdekker>, such as amount of bricks, total files (and LOC if possible), number of projects? Also, if you have anything else than Python files in the bricks would be very useful. 🙏 — Reply to this email directly, view it on GitHub <#354 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI3UPMPPEVL6QYZL5MKNBQL3KJDKRAVCNFSM6AAAAACCLK2ITCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGOBZGA2DANI> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[DavidVujic]

Thank you! I have released a new version of the CLI (v 1.31.0) with an increased cache when running the poly check command. It previosly used the defaults of lru_cache but now it is set to grow indefinitely. I think this should shorten the run time!

Implementation details in #355

[gwdekker]

How would you exactly do this in CI, run only for the projects that have been changed? Op vr 25 jul 2025 om 14:43 schreef David Vujic ***@***.***>:

…

Thank you for reporting this @gwdekker <https://github.com/gwdekker>! I haven't been aware of any performance issues before and will look into it. Does it behave the same on your local machine? Meanwhile, to speed things up, I would suggest to run the poly check command only for the projects that has been changed. — Reply to this email directly, view it on GitHub <#354 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI3UPMLVQEH35SVLX357N3T3KIRA5AVCNFSM6AAAAACCLK2ITCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGOBYHEYDAMQ> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[DavidVujic]

I think you can run the poly diff command, and with the --short option for better programmatic usage. This command will print the affected projects and you should be able to use that list in a similar way as your script "for project in projects".

The poly diff command uses git tags to figure out the changes, so you would need to have tags set on your main branch when a feature branch is merged. I use the stable-x format for it, but it is configurable.

Here's some docs about it:
the diff command: https://davidvujic.github.io/python-polylith-docs/commands/#uv_5
the configuration: https://davidvujic.github.io/python-polylith-docs/configuration/

Let me know if I can help you in any way with this of if you have any questions!

[gwdekker]

I tried this and I get it to work locally, by making a temporary tag on the head of main, diffing against that, and then removing the tag again. The problem is that in CI, this gets a bit more tricky: we are there running on a merge commit, so the diff logic does not work out of the box. will keep looking at this a bit more and report back when i either give up or found a solution. Op zo 27 jul 2025 om 10:06 schreef David Vujic ***@***.***>:

…

I think you can run the poly diff command, and with the --short option for better programmatic usage. This command will print the affected projects and you should be able to use that list in a similar way as your script "for project in projects". The poly diff command uses git tags to figure out the changes, so you would need to have tags set on your main branch when a feature branch is merged. I use the stable-x format for it, but it is configurable. Here's some docs about it: the diff command: https://davidvujic.github.io/python-polylith-docs/commands/#uv_5 the configuration: https://davidvujic.github.io/python-polylith-docs/configuration/ Let me know if I can help you in any way with this of if you have any questions! — Reply to this email directly, view it on GitHub <#354 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI3UPMJQG3BD2GVEML64VND3KSCA3AVCNFSM6AAAAACCLK2ITCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGOJQGAZTINQ> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[gwdekker]

Thanks for the quick release David of a new version! This leads to the following interesting results. I ran this locally (the pattern is similar on CI but a bit more cumbersome to test): Summary: | Test | 1.26.0 | 1.31.0 | Change | |-------------------------|--------|--------|----------------| | poly-check (sequential) | 10.45s | 8.72s | 16.5% faster ✅ | | poly-check-parallel | 3.83s | 4.61s | 20% slower ❌ | So your latest release indeed speeds up the default run, but actually slows down the parallel run a bit. In both cases, the parallel run is still much faster than the default run. Op zo 27 jul 2025 om 13:48 schreef Gerben Dekker ***@***.***>:

…

I tried this and I get it to work locally, by making a temporary tag on the head of main, diffing against that, and then removing the tag again. The problem is that in CI, this gets a bit more tricky: we are there running on a merge commit, so the diff logic does not work out of the box. will keep looking at this a bit more and report back when i either give up or found a solution. Op zo 27 jul 2025 om 10:06 schreef David Vujic ***@***.***>: > I think you can run the poly diff command, and with the --short option > for better programmatic usage. This command will print the affected > projects and you should be able to use that list in a similar way as your > script "for project in projects". > > The poly diff command uses git tags to figure out the changes, so you > would need to have tags set on your main branch when a feature branch is > merged. I use the stable-x format for it, but it is configurable. > > Here's some docs about it: > the diff command: > https://davidvujic.github.io/python-polylith-docs/commands/#uv_5 > the configuration: > https://davidvujic.github.io/python-polylith-docs/configuration/ > > Let me know if I can help you in any way with this of if you have any > questions! > > — > Reply to this email directly, view it on GitHub > <#354 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AI3UPMJQG3BD2GVEML64VND3KSCA3AVCNFSM6AAAAACCLK2ITCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGOJQGAZTINQ> > . > You are receiving this because you were mentioned.Message ID: > ***@***.*** > .com> >

[DavidVujic]

Thank you for testing it out! I had expected about 40-50% faster runs, so it's probably more things to improve before going into the parallel/threading route. It's surprising that the parallel run is slower, it should also have been faster 🤔 I'm continuing the work to improve this flow.

[DavidVujic]

I have released another version, that should make the command a little bit faster: moved up the caching to a layer above the AST parsing, to avoid any unnecessary AST walking & parsing that has already been done for the particular Python module.

The new version is out (CLI v. 1.38.1).

[gwdekker]

Thanks a lot for the quick turnaround! Results of a quick benchmark:

Version	Sequential	Parallel	Sequential Δ	Parallel Δ
1.26.0	10.45s	3.83s	baseline	baseline
1.31.0	8.72s	4.61s	+16.5% ✅	-20% ❌
1.31.1	7.43s	3.57s	+29.0% ✅	+6.8% ✅

Let me know if you ever want me to do profiling on our monorepo, more than willing to cooperate and contribute where I can!