Provide Optimus Job explain similar to DB query explain

[sravankorumilli]

Provide a command like

• optimus job explain --schedule-date --location
  which does
• basic validations { source & destination exists, query is valid with all the macros resolved}
• dependency resolution and show all the upstream jobs, any static dependencies that doesn't exist.
• upstream job basic details { owner, schedule, paused/scheduled, window configuration, historical job finish trend times}
• upstream job runs the job will depend on given schedule date.
• dump compiled specs

Dependent Job Details that can be fetched

• Dependency Job name
• Dependency job namespace
• Dependency job Project
• Dependency Owner
• Dependency job Destination table name
• Dependency job Source tablename
• upstream job runs the job will depend on given schedule date.

Option to fetch even more details with --analyze flag
-- upstream jobs last few runs details
-- source location DQ and input data stats

Out of Scope

• analyze & dry-run of job will be out of scope in the first implementation of this command
• considering other modified jobs in local.

[sravankorumilli]

@vianhazman @Mryashbhardwaj

[sravankorumilli]

@maztohir fyi

[vianhazman]

Thanks! Will we move the capabilities to the client side or API call will be acceptable for this for cases such as plugin specific validation and asset rendering?

[vianhazman]

Would it be possible and make sense to consider a scenario where --schedule-date is optional and and use the next scheduled time based on cron notation by default? This will be useful for users that is not familiar with cron scheduling or unsure with the schedule date that they should enter, so from user perspective they informed on how the job will run on the next scheduled time. We can call schedule.Next(time.Now()) from here with cron notation from the job spec.

[sravankorumilli]

We will do things through an api instead of doing things on client side.
And all the configs including schedule_date will have defaults the default we will be considering is next schedule_date of the job if schedule is configured otherwise it will be time.now

[sravankorumilli]

Below requirements we can consider in the later implementations of the command.

• analyze & dry-run of job will be out of scope in the first implementation of this command
• considering other modified jobs in local.