[Ideas] Remove Bundled Python Packages in Cloudberry Main Repo

[tuhaihe]

Description

During the code cleanup process, I noticed that the Cloudberry main repo includes source tarballs for several Python packages under gpMgmt/bin/pythonSrc and gpMgmt/bin/pythonSrc/ext, such as:

* PyGreSQL-5.2.tar.gz
* PyYAML-5.3.1.tar.gz
* behave-1.2.6.tar.gz
* mock-1.0.1.tar.gz
* parse-1.8.2.tar.gz
* psutil-5.7.0.tar.gz
* pylint-0.21.0.tar.gz
* setuptools-36.6.0.tar.gz

While most of these packages are under Apache-compatible licenses, just noticed that pylint-0.21.0.tar.gz is licensed under GPL-2, which may raise license compatibility concerns.

Bundling these source packages in the repo has some potential drawbacks:

1. It increases the size of the source code repository.
2. It introduces challenges in managing open-source license compliance, especially for non-Apache-compatible licenses.

Would it be possible to remove these bundled source packages and instead install them via package managers like pip during the build or test process, or install them in our build or test docker image? This would simplify repo management and ensure better compliance with licensing policies.

Maybe we need to do some source code refactoring work to adopt this strategy.

Looking forward to hearing your thoughts!

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[tuhaihe]

One experimental PR has been created: #982.

Love to have your feedback!

[avamingli]

Would it be possible to remove these bundled source packages and instead install them via package managers like pip during the build or test process,

It doesn’t seem like anyone cares about this topic, but I have a question:
How can you guarantee that packages installed through pip will always function properly? If compatibility problems arise, we’d still need to copy and modify the code.

[tuhaihe]

Yes, I understand that not all discussions can engage more people. However, we still need to try to make the major change process open to the public. I'm glad to see your response, and I appreciate your great question.

I cannot guarantee that the pip-installed packages function properly. I see that they are just tarball files, and we haven't altered them at all since they were imported.

I blamed these files' commit history and found that they were imported by the PR #16, which was used in the obsolete CI/CD workflow following the GP compile method. And more, before our Cloudberry init commits, they were referenced as a submodule from here - https://github.com/vmware-archive/pythonsrc-ext/tree/135d649aef8b70eb4ee8ec824f869026ac3f2b28 in the GP repo, then were changed in the commit greenplum-db/gpdb-archive@2419657 - as GP decided to change their building tool on which they were unable to fetch submodules from outside repos. In this commit, GP developers said users can also install these packages from PyPI.

Given the license management for ASF compliance, I want to explore the possibility of installing them by the pip way, but it will certainly be an experiment. If you can provide more information, I would appreciate your help.

Love to hear more feedback on this topic.

[edespino]

@tuhaihe I'm curious whether anyone is currently using pylint during development or testing. Personally, I'm not aware of it being in active use. It might be worth cross-posting this to dev@cloudberry.apache.org to check if anyone is relying on pylint. If it turns out that someone is, we can evaluate whether it's feasible to switch to ruff (a compatible Python linter under the MIT license) or another alternative that aligns with the Apache License policy.

On the other hand, if no one responds or confirms usage, we should consider removing pylint, since its GPv2 license is not compatible. I’m happy to help with the removal and leave a note in the repo explaining the decision, along with a suggestion to adopt a license-compatible linter if needed.

As for the other tools, I suggest leaving them in place for now until someone has the time and interest to review their licensing and relevance to the project.

[tuhaihe]

Here, I can provide more information based on the research work for the reference:

Packages	License	Used by
PyGreSQL-5.2.tar.gz	PostgreSQL License	used by gpMgmt tools
PyYAML-5.3.1.tar.gz	MIT License (MIT)	used by gpload
behave-1.2.6.tar.gz	BSD	used for gpMgmt tools testing
mock-1.0.1.tar.gz	BSD	used for gpMgmt tools testing
parse-1.8.2.tar.gz	MIT	used by behave-1.2.6
psutil-5.7.0.tar.gz	BSD License (BSD-3-Clause)	used by gpMgmt tools
pylint-0.21.0.tar.gz	GPLv2	used for linting python codes
setuptools-36.6.0.tar.gz	MIT	used for installing dependencies

[tuhaihe]

Hey @edespino, thanks for your suggestion on this discussion. I agree that we can replace pylint with a license-compatible component for now, like ruff. I posted a discussion to the Dev mailing list[1] to collect feedback on the use of pylint. As of now, it seems no one has responded to it. We can wait a few days; if no one confirms, we can proceed next.

Thanks again!

[1] https://lists.apache.org/thread/4nmc9kd3krjofro2xkg78vq9f5syop50