[FEATURE] Create a `ucx.workflow_runs` table

[JCZuurmond]

Summary

Create a ucx.workflow_runs table to persist UCX workflow runs, e.g. assessment or migration-process.

Description

Table schema

Column name	Data Type	Comment
started_at	dt.datetime	The timestamp of the workflow run start.
finished_at	dt.datetime	The timestamp of the workflow run end.
workspace_id	int	The workspace id in which the workflow job ran.
workflow_name	str	The workflow name that ran, e.g. "migration-progress" or "assessment"
workflow_id	int	The workflow id of the workflow that ran.
workflow_run_id	int	The workflow run id.
run_as	str	The identity the workflow was run as`
status	str	The workflow status

[JCZuurmond]

@nfx and @asnare : In this table, we want to persist when a crawler did not return any objects to differentiate between a crawler not ran and no objects returned. How do you want to persist this information, i.e. what column schema?

[JCZuurmond]

Also, do we need to keep this in the ucx catalog? Or should we keep this in the hive_metastore?

	ucx	hive_metastore
Pro	Contains data over multiple workspace scans to which this is relevant.	Available during assessment
Con	Growing the new ucx catalog; we might want to be conservative with this.	Assessment should only be ran once, therefore irrelevant as it is only used when rerunning.

[asnare]

When we discussed this at the office the intent was: This will be in the ucx catalog and not the hive_metastore. (So yes, it's not available during assessment.)

I think a technical driver for this choice is that this table will be updated quite often with small updates, and that will behave better on the ucx catalog.

The purpose of the table is also to allow for interpretation of the history table when a crawl produces no records. Without this table we won't be able to handle that situation.

I was going to update the snapshot() functionality of the crawlers to use this table as an optimisation: if the loader returns 0 rows, consult this table if available to determine whether to actually return 0 rows or (as is currently the case) perform a crawl.

[JCZuurmond]

Updating the table above according to: #2744 (comment)

[JCZuurmond]

The workflow_run_id is to be used for joining with historical records, create a view to pre-join these. i think you called it "snapshot" or something:

WITH level0 AS (
  SELECT workspace_id, workflow_id, MAX(finished_at) AS finished_at FROM workflow_runs GROUP BY 1,2
), level1 AS (
  SELECT * FROM level0 LEFT JOIN workflow_runs USING (workspace_id, workflow_id, finished_at)
)
SELECT * FROM level1