-
Notifications
You must be signed in to change notification settings - Fork 3.2k
[Feature] Allow to set job name in OpenLineage events #25535
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Unfortunately, only system and catalog properties can be overridden this way, but OpenLineage is neither of that - it's an EventListener with static configuration. Also, using some query context fields as a new default for job name could lead to new issues:
In #25704 I've implemented an different approach - configurable job name with several substitutions (
This doesn't sound ideal for me, but it can solve this particular issue. Maybe some other source of job name should be used instead - I'm new to Trino, and may not know something yet. |
@dolfinus Thanks for raising this. Having timestamp based query id as a job name seems more like a bug to me. Being able to configure would mitigate the issue. |
Currently, OpenLineage integration uses
queryId
asjobName
field value:https://github.com/trinodb/trino/blob/474/plugin/trino-openlineage/src/main/java/io/trino/plugin/openlineage/OpenLineageListener.java#L249
This is not very convenient, as each queryId is unique, and it also doesn't mean anything for end user. Instead, consider allowing user to set custom
jobName
using session variables:Another option is to use
X-Trino-Client-Info
orX-Trino-Source
, but it can contain data populated by low-level clients (Python client, HTTP client, JDBC driver and so on), and it usually used to identify particular client/software, not a session.The text was updated successfully, but these errors were encountered: