Skip to content

ignore: Preview environment with customizable temperature #319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

ignore: Preview environment with customizable temperature #319

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6c28948
Impl
KevinJBoyer Jun 16, 2025
5b56b79
Use Georgetown API key
KevinJBoyer Jun 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
234 changes: 234 additions & 0 deletions app/notebooks/StreamingComparison.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,234 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 6,
"id": "129dcf7d-b429-4a0f-a611-2ab1c2896af6",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"Sure, I'm here to help! What would you like to test or discuss?\""
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from litellm import completion\n",
"\n",
"temperature = 0.0\n",
"\n",
"def nonstreaming_completion(input):\n",
" return completion(model=\"gpt-4o\", messages=[{\"role\": \"user\", \"content\": input}], temperature=temperature).choices[0].message.content\n",
"\n",
"def streaming_completion(input):\n",
" response_stream = completion(model=\"gpt-4o\", messages=[{\"role\": \"user\", \"content\": input}], temperature=temperature, stream=True)\n",
" \n",
" response = \"\"\n",
" for chunk in response_stream:\n",
" if content := chunk.choices[0].delta.content:\n",
" response += content\n",
"\n",
" return response"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "ee5da024-d893-4b20-aa5e-26e74f2ffce4",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The client qualifies for the Standard Utility Allowance (SUA) for heating expenses because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling costs separately from their rent or mortgage. The Telephone Utility Allowance (TUA) is specifically for phone bills, but since the client incurs heating costs, the SUA would be the applicable allowance. Therefore, the client would not qualify for both SUA and TUA together, nor would they be disqualified from utility allowances altogether, as they do have a heating expense.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) for heating expenses because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling costs separately from their rent. The Telephone Utility Allowance (TUA) is specifically for phone bills, but since the client incurs heating costs, the SUA would be the applicable allowance. Therefore, the client would not qualify for both SUA and TUA together, nor would they be disqualified from utility allowances altogether, as they do have a qualifying heating expense.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. The fact that the client also pays a monthly phone bill does not change the qualification for the SUA, as the SUA covers heating expenses, which are considered more significant. Therefore, the client qualifies for the SUA for heating expenses.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is applicable when a household incurs heating or cooling expenses separately from their rent or mortgage. Since the client is paying for kerosene, which is a heating expense, they are eligible for the SUA. The Telephone Utility Allowance (TUA) is not applicable in this case because the SUA covers all utilities, including telephone expenses, when heating or cooling costs are incurred. Therefore, the client does not need to separately qualify for the TUA.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses, regardless of the specific utility used for heating. Since the client is paying for kerosene to heat their home, they meet the criteria for the SUA. The Telephone Utility Allowance (TUA) is specifically for phone bills, but since the client qualifies for the SUA due to heating expenses, the SUA would cover the utility allowance needs. Therefore, the client does not need to separately qualify for the TUA. \n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. The fact that the client also pays a monthly phone bill does not change the qualification for SUA, as the heating expense is sufficient to qualify for it. Therefore, the client qualifies for the SUA for heating expenses.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. The fact that the client pays a monthly phone bill is not relevant for the SUA, but it does qualify them for the Telephone Utility Allowance (TUA). However, since the SUA covers heating expenses, which is a more significant utility cost, the client would qualify for the SUA rather than just the TUA. Therefore, the correct answer is that the client qualifies for the SUA for heating expenses.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. The fact that the client also pays a phone bill does not change the qualification for the SUA, as the SUA covers heating expenses, which is a broader category than just the phone bill. Therefore, the client qualifies for the SUA due to their heating expenses.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. Although the client also pays a monthly phone bill, the SUA generally covers all utilities, including telephone expenses, when heating or cooling costs are involved. Therefore, the client would not need to separately qualify for the Telephone Utility Allowance (TUA) since the SUA encompasses it. \n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. The fact that the client also pays a phone bill does not change the qualification for the SUA, as the SUA covers heating expenses, which is a broader category than just the phone bill. Therefore, the client qualifies for the SUA due to their heating expenses.\n",
"\n",
"### \n",
"A\n",
"===================================\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. The fact that the client also pays a monthly phone bill does not change the qualification for the SUA, as the SUA covers heating expenses, which are considered more significant. Therefore, the client qualifies for the SUA for heating expenses, not the TUA, which is specifically for phone bills. \n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. Although the client also pays a monthly phone bill, the SUA generally covers all utilities, including telephone expenses, when heating or cooling costs are present. Therefore, the client would not need to separately qualify for the Telephone Utility Allowance (TUA) since the SUA encompasses it. \n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. The fact that the client pays for heating with kerosene makes them eligible for the SUA. The Telephone Utility Allowance (TUA) is specifically for phone bills, but since the client qualifies for the SUA due to heating expenses, they would not separately qualify for the TUA. Therefore, the correct utility allowance for the client is the SUA for heating expenses.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) for heating expenses because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling costs separately from their rent or mortgage. The Telephone Utility Allowance (TUA) is specifically for phone bills, but since the client incurs heating costs, the SUA would be the appropriate allowance. Therefore, the client would not qualify for both SUA and TUA together, nor would they be disqualified from utility allowances altogether, as they do have a heating expense.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) for heating expenses because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling costs separately from their rent or mortgage. The Telephone Utility Allowance (TUA) is specifically for phone bills, but since the client incurs heating costs, the SUA would be the appropriate allowance. Therefore, the client would not qualify for both SUA and TUA together, nor would they be disqualified from utility allowances altogether, as they do have a qualifying heating expense.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. The fact that the client pays a monthly phone bill would qualify them for the Telephone Utility Allowance (TUA) if they did not have heating expenses. However, since the client has heating expenses, they qualify for the SUA, which generally covers all utilities, including phone expenses. Therefore, the client qualifies for the SUA, not the TUA separately.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses, regardless of the specific utility used for heating. The fact that the client also pays a phone bill does not change the qualification for SUA, as the SUA covers heating expenses, which is a broader category than just the phone bill. Therefore, the client qualifies for the SUA due to their heating expenses.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. The fact that the client pays a monthly phone bill would qualify them for the Telephone Utility Allowance (TUA) if they did not qualify for the SUA. However, since the client incurs heating expenses, they qualify for the SUA, which generally covers all utilities, including telephone expenses. Therefore, the client qualifies for the SUA, not the TUA separately.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. The fact that the client also pays a monthly phone bill does not change the qualification for the SUA, as the SUA covers heating expenses, which are considered more significant. Therefore, the client qualifies for the SUA for heating expenses.\n",
"\n",
"### \n",
"A\n",
"The client qualifies for the Standard Utility Allowance (SUA) because they report purchasing kerosene to heat their home. The SUA is typically applicable when a household incurs heating or cooling expenses. The fact that the client also pays a monthly phone bill does not change the qualification for the SUA, as the SUA covers heating expenses, which are considered more significant. Therefore, the client qualifies for the SUA for heating expenses.\n",
"\n",
"### \n",
"A\n"
]
}
],
"source": [
"test_input = \"\"\"Client Attributes\n",
"Employment status: Not in labor force\n",
"Age: Senior (60+)\n",
"Gender: Male\n",
"Household immigration status: All U.S. Citizens\n",
"Disability status: Receives SSI/RSDI\n",
"Program: CalFresh\n",
"The client reports purchasing kerosene to heat their home and paying a monthly phone bill. The client does not pay for any other utilities. Which utility allowance does the client qualify for? Select one correct answer.\n",
"Please provide your rationale without any reference to the multiple choice letters, followed by ###, and then provide only the single letter choice that you believe is correct.\n",
"A. SUA (Standard Utility Allowance) for heating expenses\n",
"B. TUA (Telephone Utility Allowance) for the phone bill\n",
"C. SUA and TUA together, as both utilities are eligible\n",
"D. No utility allowances, as the client does not pay for electricity or water\"\"\"\n",
"\n",
"for i in range(10):\n",
" print(nonstreaming_completion(test_input))\n",
"\n",
"print(\"===================================\")\n",
"\n",
"for i in range(10):\n",
" print(streaming_completion(test_input))"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "332c1da0-bd17-4c81-8b72-17937638d6a9",
"metadata": {},
"outputs": [],
"source": [
"from src.chat_engine import ImagineLaEngine\n",
"\n",
"engine = ImagineLaEngine()"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "f8ef5aea-ccb0-4baa-9c15-d3e61eed01ca",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'The client qualifies for the Standard Utility Allowance (SUA) for heating expenses because they purchase kerosene to heat their home. The SUA is applicable for heating expenses, which includes costs for heating fuel like kerosene. The Telephone Utility Allowance (TUA) is for phone bills, but since the client already qualifies for the SUA due to heating expenses, they would not separately qualify for the TUA. Therefore, the correct choice is the SUA for heating expenses.\\n\\n### A'"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"engine.on_message(test_input).response"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "4ea19107-dcae-4053-aa6f-70ab260bd60b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'2c86bce760b09cb9d35c4256dea60879'"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from hashlib import md5\n",
"md5(test_input.encode(\"utf-8\")).hexdigest()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
8 changes: 8 additions & 0 deletions app/src/chainlit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,14 @@ async def update_settings(settings: dict[str, Any]) -> Any:
initial_value=default_value,
items=get_models(),
),
"temperature": lambda default_value: Slider(
id="temperature",
label="Temperature for LLM response generation",
initial=default_value,
min=0,
max=2,
step=.5,
),
"retrieval_k": lambda default_value: Slider(
id="retrieval_k",
label="Number of citations to retrieve for generating LLM response",
Expand Down
5 changes: 5 additions & 0 deletions app/src/chat_engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ class ChatEngineInterface(ABC):
engine_id: str
name: str
llm: str = "gpt-4o" # Default LLM to use
temperature: float = 0.0 # Default temperature for LLM responses

# Configuration for formatting responses
formatting_config: FormattingConfig
Expand Down Expand Up @@ -131,6 +132,7 @@ class BaseEngine(ChatEngineInterface):

user_settings = [
"llm",
"temperature",
"retrieval_k",
"retrieval_k_min_score",
"show_msg_attributes",
Expand Down Expand Up @@ -182,6 +184,7 @@ def _build_response(
start_time = time.perf_counter()
response = generate(
self.llm,
self.temperature,
self.system_prompt_2,
question,
None,
Expand Down Expand Up @@ -222,6 +225,7 @@ def _build_response_with_context(
start_time = time.perf_counter()
response = generate(
self.llm,
self.temperature,
self.system_prompt_2,
question,
context_text,
Expand Down Expand Up @@ -327,6 +331,7 @@ class ImagineLaEngine(BaseEngine):

user_settings = [
"llm",
"temperature",
"retrieval_k",
"retrieval_k_min_score",
"show_msg_attributes",
Expand Down
11 changes: 9 additions & 2 deletions app/src/generate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ def get_models() -> dict[str, str]:
"""
models: dict[str, str] = {}
if "OPENAI_API_KEY" in os.environ:
models |= {"OpenAI GPT-4o": "gpt-4o"}
models |= {"OpenAI GPT-4o": "gpt-4o", "OpenAI GPT-4o-mini": "gpt-4o-mini"}
if "ANTHROPIC_API_KEY" in os.environ:
models |= {"Anthropic Claude 3.5 Sonnet": "claude-3-5-sonnet-20240620"}
if _has_aws_access():
Expand Down Expand Up @@ -100,6 +100,7 @@ def _prepare_messages(

def generate(
llm: str,
temperature: float,
system_prompt: str,
query: str,
context_text: str | None = None,
Expand All @@ -111,8 +112,12 @@ def generate(
messages = _prepare_messages(system_prompt, query, context_text, chat_history)
logger.debug("Calling %s for query: %s with context:\n%s", llm, query, context_text)

logger.info("========= LLM: %s, temperature: %s", llm, temperature)

os.environ["OPENAI_API_KEY"] = os.environ.get("GEORGETOWN_OPENAI_API_KEY", "")

response = completion(
model=llm, messages=messages, **completion_args(llm), temperature=app_config.temperature
model=llm, messages=messages, **completion_args(llm), temperature=temperature
)

return response["choices"][0]["message"]["content"]
Expand All @@ -133,6 +138,7 @@ async def generate_streaming_async(
"Async streaming from %s for query: %s with context:\n%s", llm, query, context_text
)

os.environ["OPENAI_API_KEY"] = os.environ.get("GEORGETOWN_OPENAI_API_KEY", "")
response_stream = completion(
model=llm,
messages=messages,
Expand Down Expand Up @@ -167,6 +173,7 @@ class MessageAttributes(BaseModel):
def analyze_message(
llm: str, system_prompt: str, message: str, response_format: type[MessageAttributesT]
) -> MessageAttributesT:
os.environ["OPENAI_API_KEY"] = os.environ.get("GEORGETOWN_OPENAI_API_KEY", "")
response = (
completion(
model=llm,
Expand Down
Loading
Loading