overriding model length for zephyr 7b alpha (#398)

ian-scale · web-flow · commit 0b970231a817 · 2023-11-30T16:16:27.000-08:00
diff --git a/model-engine/model_engine_server/domain/use_cases/llm_model_endpoint_use_cases.py b/model-engine/model_engine_server/domain/use_cases/llm_model_endpoint_use_cases.py
@@ -194,6 +194,7 @@
     # Can also see 13B, 34B there too
     "llama-2": {"max_model_len": None, "max_num_batched_tokens": 4096},
     "mistral": {"max_model_len": 8000, "max_num_batched_tokens": 8000},
+    "zephyr": {"max_model_len": 32768, "max_num_batched_tokens": 32768},
 }
 
 

Original file line number	Diff line number	Diff line change
`@@ -194,6 +194,7 @@`
`194`	`194`	`# Can also see 13B, 34B there too`
`195`	`195`	`"llama-2": {"max_model_len": None, "max_num_batched_tokens": 4096},`
`196`	`196`	`"mistral": {"max_model_len": 8000, "max_num_batched_tokens": 8000},`
	`197`	`+ "zephyr": {"max_model_len": 32768, "max_num_batched_tokens": 32768},`
`197`	`198`	`}`
`198`	`199`
`199`	`200`