Speed up the build process.

[cristim]

Builds take some 15-20min, mostly because of the scraper that fetches all the data sequentially.

A while ago I started to parallelize the scraping but it's not finished yet.

I'm thinking maybe about rewriting the scraper to a faster language like Go, which also has nicer way to do concurrency.

Another way could be to only do the scraping once a day and reuse the results for software update builds that only render the site without running the scraper.

[syedaffanhamdani]

Building on your ideas I would suggest following process:

1. Conceptually keep "Data Collection System" a separate step than "Website Generation System"
2. Data is collected once per day and stored somewhere.
3. Add an optional flag in build process to fetch fresh data. If flag is not set then last stored data is used

Orthogonal to this, one can parallelize scraping.

What are your thoughts @cristim ? Once we agree on some approach I can start implementing it.

[cristim]

Yes that's exactly what I meant in the last paragraph I wrote, I just didn't get to articulate the implementation as well as you did.

I'm thinking to use the same bucket we use for hosting the website for storing the scraped files, just have a timestamp object that keeps track of when we did the last update.

We can have a new github action that checks the timestamp and runs the build process from cron when the timestamp is older than a few hours but only pushes the scraping results without rendering the html.

And then we update the normal build process to just use those scraper results, assumed to be fresh.

[syedaffanhamdani]

If it is okay then as a first step, I shall create the PR for:

• Add an optional flag in build process to fetch fresh data. If flag is not set then last stored data is used

[cristim]

I want that data refresh flag to be off by default and only be set for the data refresh builds which we run triggered by a cron expression.

I'm also thinking to extract those cron triggered runs out of the current deploy github action.

My only concern is that we're now duplicating the deployment code over too many github actions so I'm thinking of extracting the run steps into a script to deduplicate it as much as possible.

[syedaffanhamdani]

In order to avoid duplication we can extract the common steps to a reusable workflow, that can be then just called in other workflows.