Generate.json Takes >45 minutes on a Complex Schema

[zkalson]

I'm using outlines with vLLM as below:

self.model=models.vllm(
                    model_name=model_checkpoint,
                    enable_prefix_caching=True,
                    gpu_memory_utilization=0.90,
                    tensor_parallel_size=gpu_count,
                )
sampling_params = self.model.model.get_default_sampling_params()
sampling_params.max_tokens = self.model.model.llm_engine.model_config.max_model_len
generator = generate.json(self.model, schema)
outputs = generator(prompts, sampling_params=sampling_params)

generate.json takes more than 45 minutes for some of my more complex schemas.

I'm running in a serverless environment, and my understanding is that I can set OUTLINES_CACHE_DIR to a persistent storage volume to reduce this time. Nonetheless, that initial run is going to be very costly, especially if I have an arbitrary number of complex schemas.

[rlouf]

Could you paste the schema here? Hard to say what's happening otherwise.

[zkalson]

It's a customer's schema. I can get their permission, but would it be possible for me to email you instead? Sorry for the inconvenience.

[rlouf]

Sure, remi at dottxt.co

[zkalson]

This has been solved (thanks so much Remi)! I had a string field with minLength and maxLength arguments:
"a_field_name": {
"maxLength": 2500,
"minLength": 150,
"title": "A Field Name",
"type": "string"
},

After removing minLength and maxLength from my schema, generate.json functions normally (sub-minute)!

[juanevenegas]

Hi, how did you know what to remove? Are there any best practices in terms of how to define the schema?

[rlouf]

minLength and maxLength with large values will create many states in the finite state machine we create from the JSON Schema. We know because we've hit this problem repeatedly :)