psycopg2's code visualization (python part)

[ivanmilevtues]

Hey all,

Me and a friend recently were talking and came to the conclusion that getting to know new code bases is not an easy task for both maintainers and devs. It requires quite an effor to have a nice documentation which is also maintained from the maintainers pov and if such thing doesn't exist it can be quite challenging for new devs to contribute to a project.

This said we decided that we will try to help and make this process nicer for everyone. We believe that the best starting point is high-level visualization of the codebase - to us that is diagrams. In order to have them always up-to-date we are levering static analysis and LLMs to create them (thus only the python part, as we don't have the C static analysis just yet 😅 ).

I would love to hear your opinion on diagram first documentation and also on the one we generated for this repo:
https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/psycopg2/on_boarding.md

Any sort of feedback is more than welcome!

Full disclosure: we're trying to turn this into a startup, but we're still in a very early stage and figuring out what will actually be useful for people.

[dvarrazzo]

Hello,

psycopg2 is in maintenance mode and I don't see ourselves adding more features to it. Because it is a dense hand-written C module I assume that the LLM will not be able to find much more than hallucination in the way of explaining it.

Psycopg 2 is pretty much this big blob of C and a few features written in Python, pretty much ony one-level deep around the C module. If you want to play with something you may try and run this analysis on psycopg 3, which is entirely written in Python (with an optional C module that replaces some Python functionality by offering a 1:1 compatible interface).

Giving a skim of the result documentation, I see these issues which you might try to address:

• The links are not permalinks. I think that a more appropriate target could be a tag, so you can generate documentation based on (the snapshot of) a stable architecture;
• Documenting private objects and public ones at the same level seems confusing. Certain objects might be implemented in one module but exposed by another (to give a clear API but a convenient implementation). What facet do you want to show? I assume the dev one, which also makes sense as the API for end users is (should) be documented by hand IMO.

Good luck with your project!

[ivanmilevtues]

Thank you so much for the response @dvarrazzo!

I have noted down your feedback and will try to address it, when we have that I will run it for psycopg 3. On what you said with the tag generation, we are currenlty looking into making it into a github action. Do you think that this can be useful, then you can setup according to your own heuristic when it is run?

The public and private API makes lots of sense now, we also got some feedback that people want to see what kind of pattern is used in the components, so we will work on that as well!

Thank you and hopefully I will try the generation on the other repo and we can chat again there!