Debugging backend

[szabozoltan69]

Besides the great Sentry issues sometimes it is useful to debug backend processes.

One option is the evergreen pdb, via a simple import pdb; pdb.set_trace() breakpoint in any def-s or locations (which is not a class definition and where program control can be stopped). With simple commands wonders can be done. One of the most useful ones is the pp vars(...) listing of object properties.

Sometimes it is useful to debug in an IDE. PyCharm has very sophisticated options, but mainly in the paid version.

So I tried to fire up the VS Code debugging. It was not so straightforward as it seemed to (or it has to be), but finally it succeeded.

In case of a non-docker local run, the defaults are fine, "Run and Debug" blue button (in the Debug left section), then choose Django, then manage.py. Via ps aux|grep debugpy it turned out that quite a complex system is fired up (4 processes can be seen for the simple "listening").

But the dockerised solution needed some luck to work. I had to install the debugpy package ( debugpy==1.8.14) and remove the typing package (which is not any more needed above python 3.11) via pyproject.toml. The "on the fly, in-container install and remove" did not succeed. Also DebugToolbar is something to be removed for now, due to sometimes it causes double runs.

I had to add 5678 port to the exposed ones, so in docker-compose.yml I had:

    ports:
      - 8000:8000
      - 5678:5678

and instead of the command: python manage.py runserver 0.0.0.0:8000 I could use:
command: python -Xfrozen_modules=off -m debugpy --wait-for-client --listen 0.0.0.0:5678 manage.py runserver 0.0.0.0:8000

The -Xfrozen_modules=off is crucial, otherwise a lot of annoying issues come up.

In the .vscode subdir (of go-api) I have these files:
settings.json

{
    "python.pythonPath": "/usr/local/bin/python"
}

and the most important one, launch.json:

{
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug Django",
            "type": "debugpy",
            "request": "attach",
            "host": "localhost",
            "port": 5678,
//          "processId": "${command:pickProcess}",  # not bad this dropdown, but did not succeed.
            "django": true,
            "justMyCode": false,
            "pathMappings": [
              {
                "localRoot": "${workspaceFolder}",
                "remoteRoot": "/home/ifrc/go-api"
              }
            ],
        }
    ]
}

The strange is that host and port is underlined as erroneous, but it works fine (and I did not find any other better solution).

This way I could insert breakpoints and after pushing the "green play triangle" I could have the desired functionality:

Let's see a practical troubleshooting.
In case of a curl 'http://localhost:8000/api/v2/country-plan/81/' request the CountryPlanViewset gives the response. Unfortunately there is no "def" there in drf_views, only the class itself. But we can insert a

    def get_object(self):
        pass

method there, and can set the breakpoint to pass. If the run pauses there, we can request str(self.queryset.query) – the str() is important, otherwise an overcomplicated structure is to be struggled with – and this way we can check the SQL query that is generated from the default queryset:
SELECT "country_plan_countryplan"."created_by_id", "country_plan_countryplan"."created_at", "country_plan_countryplan"."updated_by_id", "country_plan_countryplan"."updated_at", "country_plan_countryplan"."country_id", "country_plan_countryplan"."internal_plan_file", "country_plan_countryplan"."internal_plan_url", "country_plan_countryplan"."public_plan_file", "country_plan_countryplan"."public_plan_url", "country_plan_countryplan"."requested_amount", "country_plan_countryplan"."people_targeted", "country_plan_countryplan"."is_publish", "country_plan_countryplan"."public_plan_inserted_date", "country_plan_countryplan"."internal_plan_inserted_date" FROM "country_plan_countryplan" WHERE "country_plan_countryplan"."is_publish"

Or, if you want to dive deeper, you can put a breakpoint to def retrieve (in the viewset class) or to def to_representation (in the serializer class). This way, debugging the "instance" it is possible to have a look at the prefetched or annotated entities, e.g. via str(instance.country_plan_sp.all().query) and str(instance.country_plan_mc.all().query).

E.g. this way:

def retrieve(self, request, *args, **kwargs):
        instance = self.get_object()
        # Set your breakpoint here to inspect 'instance'
        serializer = self.get_serializer(instance)
        return Response(serializer.data)

If there is a def get_queryset(self), the following approach can be followed:

    def get_queryset(self):
        a = FormQuestion.objects.all().order_by("component__component_num", "question_num", "question").select_related("component", "component__area").prefetch_related("answers")
        list(a[:1])
        print(str(a.query))
        return a

One more step is to use from django.db import connection, then in get_queryset it is possible use a print(connection.queries), which gives a lot of queries with prefetch_related.

---

I've pushed the changes into vscode branch.