Add `get_job_run_error` to the Admin API tools

[jairus-m]

The ability to retrieve dbt Cloud artifacts/details via get_job_run_details and get_job_run_artifacts is great! However, for projects with many nodes, the context that is read in by these tools often exceed the LLM agent's token limit.

In addition, a good majority of the use of these artifacts (in my experience) have to do with understanding errors that caused a run job to fail.

My hope is to implement and add a tool get_job_run_error that will retrieve relevant errors without having to read-in the rest of the details/artifacts into the agent's context window.

May be worth a discussion!

Here is an example of hitting the token limit:

[bhodsonAmplify]

Please implement this.

The current dbt Model Context Protocol (MCP) implementation returns comprehensive run results, requiring downstream systems (particularly LLMs) to parse through extensive logs to identify and analyze errors. This approach creates unnecessary overhead and reduces operational efficiency.

Recommended Enhancement:

• Implement a filtered logging endpoint in the dbt MCP that specifically exposes:
• Error-only log streams with configurable severity levels
• Detailed debug context for each error (stack traces, affected models, dependencies)
• Structured error metadata (error codes, categories, timestamps)

Business Benefits:

1. Reduced Token Consumption: LLMs can focus on relevant error data rather than processing verbose success logs, significantly reducing API costs and processing time.
2. Faster Error Resolution: Direct access to contextual error information enables rapid troubleshooting without manual log parsing or filtering.
3. Enhanced Developer Experience: Developers and automated systems can immediately access actionable error information, improving Mean Time to Resolution (MTTR).
4. Improved System Performance: Reduced payload sizes decrease network overhead and improve response times across the integration.
5. Better Observability: Structured error logging enables more sophisticated monitoring, alerting, and analytics capabilities.

Technical Implementation:
A filtered logging endpoint (/logs/errors) with parameters for severity level, time range, and model scope would provide precise error context while maintaining the existing comprehensive logging for audit purposes.

[jairus-m]

Creating a net-new logging endpoint that is specific to errors is probably not necessary. The artifacts that are retrieved are already structured JSON that can be parsed for errors.

As @b-per mentioned to me, the complexity here is that Fusion and Core logs differ. I'm hoping to play with the core-side just to learn and get a better understanding of what this implementation would take and plan on having more discussion about this in the coming weeks!

[DevonFulcher]

JSON that can be parsed for errors

Yes, I think the solution should probably look like that. There are many possible implementations, ranging from something simple with regex to something more complicated, like in-memory keyword, semantic, or hybrid search.

With those more advanced options, the difference between Fusion & Core logs shouldn't matter too much. That difference between Fusion & Core logs is only relevant if we want to search for very specific string matches. Maybe we start with that, but I could imagine that becoming a bit flaky and a bit tightly coupled to how Fusion & Core format their logs.

Also, I think this is a problem that is generally applicable across many of the tools, not just the new Admin tools. For instance, list_metrics, get_all_models, etc.

Additionally, we should consider when we should revert to search techniques rather than giving the LLM the whole output. Should we always use search for certain tool calls? Should we count tokens and divert toward a search path only when the token count is sufficiently large? Should we implement separate search tools that are distinct from the existing ones?

So, there are some things to consider, but that shouldn't prevent us from implementing something simple and limited in scope to begin with and iterate from there.

[jairus-m]

@DevonFulcher I'm going to put in some work to better understand the approaches and implications of what you just shared. Before I start any work/exploration, would it be possible to get more of your insights on this in general?

I want to understand as much as I can and align with you/Benoit before playing around with anything. But as you said, maybe a very limited scope to just get_job_run_error is a fine exploratory start!