feat: run scheduled tasks after startup

[samhotep]

Done

• Start background tasks after the first request

QA

• Clone this branch, and run the task using:

docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres
docker run -d -p 6379:6379 redis
dotrun build && dotrun
## To execute celery tasks
dotrun exec celery -A webapp.app.celery_app worker -B  --loglevel=DEBUG

• The site should start up normally.
• Open the index page at http://localhost:8104/app.
• The tree should start loading, which also kicks off the background tasks.
• The scheduled tasks will start and should complete without errors. Note that there might be no additional logs besides error logs.
• Once the loading is done on the home page, the tree should be visible.

Fixes

• WD-22462

[webteam-app]

Demo

Jenkins

demos.haus

[muhammad-ali-pk]

@samhotep Ran QA steps with clean project (deleted all repos from /repositories folder, deleted cache). None of the projects were actually cloned.

[samhotep]

@muhammad-ali-pk could you take another look please, i've also updated the qa steps

[muhammad-ali-pk]

Thanks @samhotep, here are my findings from QA testing.

• ✅ I ran the project with clean state (no projects in repositories folder, no cache) and the way you described in QA steps.
• ✅ When I visited http://localhost:8104/app, the website showed projects loading and the background tasks were started and were visible in the terminal
  e.g.
  [2025-06-12 06:53:12,192: INFO/MainProcess] Task webapp.github.async_save_file[e3f18284-3c64-4cab-9516-062d54f43d0d] received
• ✅ In the repositories folder, all the projects were successfully cloned and pages individual pages/directories were present

So far so good, but

• ❌ When the requests were completed, the website still showed empty projects. So I checked the db in postgres container, and there were no projects in the projects table and no pages in the webpages table. Looks like the background tasks are not saving projects/pages in the table.

But here's the good news,

• ✅ The get-tree request like http://localhost:8104/api/get-tree/canonical.com/main/true clones project and pages, creates a project in db and saves pages in db and returns the tree successfully.

What to do next:

• Fix the background tasks so they store the data in postgres db
• Make sure the fixes do not break the /get-tree endpoint flow
• Test all the flows before pushing changes
  - [ ] Clone all the projects solely from background tasks and verify the data in database as well as files/folders in repositories directory
  - [ ] Clone a project using /get-tree endpoint, and verify the data in database as well as files/folders in repositories directory

[muhammad-ali-pk]

Some questions from QA:

1. Seeing this log quite often regarding file_cache
   

2. It would seem like the background task for fetching trees is only triggered upon initiating request in the browser. Is that intentional? From my previous understanding, the background tasks ran as soon as the flask app was run? Can you please clarify a bit on the flow here?

[samhotep]

Hey Ali! Thanks for the quick review;

1. I haven't run into this warning, but from some investigation it seems to be unrelated to our changes, but I will make a separate PR to disable cache discovery with the google SDK.
2. Yes, this is intentional. The reason for this is that the background tasks also instantiate separate flask apps when they start, which cause memory-related crashes on startup. So for now we let the app start and instantiate projects first, and then trigger the background tasks on the first request.

[muhammad-ali-pk]

Thanks for the fixes, Sam. LGTM!

[github-actions]

🎉 This PR is included in version 1.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀