When to use Outputparsers, tools, and/or LangSmith Evaluators to test LLM output?

[jameswpm]

I was working on a simple LCEL chain for a simple task, and this question came to my mind.

Imagine I have a straightforward LCEL chain containing 2 prompts and 2 output parsers that "force" the output to be a JSON.

llm = ChatOpenAI(temperature=0)

joke_for_topic_prompt = ChatPromptTemplate.from_template("Tell me a short joke about {topic}.")
output_parser1 = StrOutputParser()

analyze_joke_prompt = ChatPromptTemplate.from_template("What is the joke below about?\n{joke}. Answer it with a JSON object.")
output_parser2 = JsonOutputParser()

# Chain
chain = (
    {"topic": RunnablePassthrough()}
    | joke_for_topic_prompt
    | llm
    | {"joke": output_parser1}
    | analyze_joke_prompt
    | llm
    | output_parser2
)

# Invoke the chain
result = chain.invoke({"topic": "ice cream"})

print(result)

Example output I got on a simple test

{'joke': "Why did the ice cream truck break down? It had too many 'scoops'!", 'punchline': "It's a pun on the word 'scoops', which can mean both the amount of ice cream served and a problem or mishap." 'theme': 'Ice cream', 'type': 'Pun'}

Now, I can see 3 main ways to ensure that the output is a JSON object and that I can use it in other parts of my code.

1. The output parser I used (JsonOutputParser) can be encapsulated into a RetryOutputParser (or similar) to ensure that the output will be well formatted JSON;
2. I can use a Python function that parses the output as the final step of my chain to ensure that this output is a valid JSON. Illustrative example:

# Function to print JSON keys
def print_json_keys(json_output):
    r = json.dumps(json_output)
    data = json.loads(r)
    print("Keys:", data.keys())
    return data

#Same code as before

# Chain
chain = (
    {"topic": RunnablePassthrough()}
    | joke_for_topic_prompt
    | llm
    | {"joke": output_parser1}
    | analyze_joke_prompt
    | llm
    | output_parser2
    | print_json_keys
)

3. I can create a dataset with many inputs for my chain in LangSmith and then use an Evaluator like JsonValidityEvaluator. Code example:

def construct_chain():
    return (
        {"topic": RunnablePassthrough()}
        | joke_for_topic_prompt
        | llm
        | {"joke": output_parser1}
        | analyze_joke_prompt
        | llm
        | output_parser2
        | print_json_keys
    )

# Evaluation config
evaluation_config = RunEvalConfig(
    evaluators=[
        JsonValidityEvaluator(), 
    ]
)

# LangSmith client
client = Client()

# Run on the dataset
run_on_dataset(
    client,
    "dataset_of_topics",
    construct_chain,
    evaluation=evaluation_config,
)

I know these 3 options are not equivalents, but I'd like to know which one is recommended by the ones working with production LLM apps. Is it better to use an Output parser or a pure Python function to test the output? If I'm using one of these 2, is it worth it to use a Evaluator?

This is more of an opinion-based discussion, but it would be nice to have some good explanations.

Note: I'm using JSON to explain the whole thing as a simple example, but it can be a more complicated structured output.

[eyurtsev]

If you're using OpenAI you should probably be using tool calling capabilities.

https://python.langchain.com/docs/modules/model_io/chat/function_calling/

or

https://python.langchain.com/docs/modules/model_io/chat/structured_output/

[jameswpm]

Indeed. The option of getting the tool selected by the model adds a 4th option to my example scenario.

Apparently, my thread didn't attract much attention, but I'm interested in learning about the LLM-powered apps in production. Which one is the best option? How do we choose between a "simple" Outputparser, a tool (being selected by the model through tool calling capabilities or not), and a final LangSmith evaluator?

Again, I know they are not equivalent, but I'm curious to know which is the go-to option for those working with LangChain.

[drew-wks]

I wrote code myself to fix parsing errors downstream. Rationale: 1) I want to be able to switch LLMs and don't want to depend on whether a particular one has a good function-calling capability. 2) Langchain has OutputFixingParser, but that involves yet another LLM call, introducing more cost and latency.