Automatically populate new data during migrations

[john-gom]

Story

Currently the logic to detect whether a query is allowed is quite complex as we need to check to see if newly added fields have been fully loaded. Instead we could populate these during migrations

One risk is that the old service might be processing updates and not setting the new fields while the migration is running. To mitigate this we can capture a "pre_migration_message_id" before the migration starts and reset the "last_message_id" field to this when the new version of the service starts, so that any product updates that happened during the migration are re-played

If this works we can potentially get rid of a lot of the loaded_tags logic, although still need this for scans as off-query is currently not able to actively pull this data itself.

[john-gom]

Note that imports from MongoDB will commit the migration transaction which can then cause issues if the import fails as earlier parts of the migration will have been committed and will potentially fail if re-run. Might want to create a nested transaction (save point) or something, but need to check how this works with performance as don't want to keep a huge transaction open for a full MongoDB load.

Maybe the simplest option is to always do the imports in a separate migration script

[john-gom]

Decided to just put in separate scripts

[john-gom]

Also need to add a command_timeout to the Start services step in the container-deploy.yml workflow as a full import (with partial fields) can take about 30 minutes

TIP: If it does timeout you can ssh onto the host and look at the logs of the temporary contaner, e.g.

docker logs --tail=100 --follow off-query-query-run-260a3b6c960a

[john-gom]

Now fixed in main

[john-gom]

Might want to introduce a new source type so we know a product was only partially updated by a migration. Also probably don't want to delete products during migrations as Redis should be handling this

[john-gom]

Done