🐞 Bug Report: model.respond() hangs indefinitely without timeout support

[0xIslamTaha]

Summary

In certain requests, the model.respond(code_chat, response_format=response_format) call in the LM Studio Python SDK hangs indefinitely. The model appears to be running, but it never returns a response, and the call never raises any exception either.

Steps to Reproduce

1. Load a model via LM Studio's Python SDK.

2. Construct a prompt and call model.respond(...) with a Chat object and a response_format.

3. Occasionally, the call does not return or raise an error.

4. The Python process remains stuck, and the model engine keeps running in the background.

5. The only recovery options are:

   • Killing the Python process.
   • Manually stopping the running model in LM Studio.

Expected Behavior

There should be a way to set a timeout for the synchronous API (model.respond(...)). If the model takes too long to respond, the SDK should either:

• Raise a timeout exception, or
• Provide a configurable timeout parameter (e.g., timeout=180).

Actual Behavior

• No timeout mechanism exists.
• The process hangs indefinitely on certain inputs.
• This creates challenges for production or automated workflows where stuck calls block the system.

Environment

• OS: Windows 11
• Python: 3.11.9
• LMSTUDIO Version: 0.3.17 build 11
• Model: qwen2.5-14b-instruct

[ncoghlan]

True, the only current general workaround for this is to use the async API instead, which is still formally considered unstable (however, as discussed in #47 there probably aren't any real blockers to simply dropping that API stability warning at this point...).

[0xIslamTaha]

@ncoghlan thank you for your replay. As I got stuck into this bug, I managed to do a work around to be able to set a timeout manually

from concurrent.futures import ThreadPoolExecutor, TimeoutError

       def lm_studio(self):

        def do_request():
            code_chat.add_user_message(f"{prompt['user']}{json.dumps(candidate_list, indent=2)}")
            return model.respond(code_chat, response_format=response_format)

        with ThreadPoolExecutor(max_workers=1) as executor:
            future = executor.submit(do_request)
            try:
                # Wait for the result with a timeout
                prediction = future.result(timeout=self.TIMEOUT + 10)
                return prediction
            except TimeoutError:
                logger.error(f"Request to lm studio timed out after {self.TIMEOUT} seconds.")
                raise 
            except Exception as e:
                logger.error(f"Failed to send data to lm studio: {e}")
                return None

[ncoghlan]

When it comes to actually adding the missing timeouts, the most reasonable place to put them would be in the synchronous facade that submits requests to the background async thread (since those blocking calls are what are actually blocking the main thread).

Each affected blocking call would then need a way to terminate the work that was previously scheduled in the background thread.

The other design question would be to decide on the level of configurability to offer:

• a global default timeout?
• a per-client default timeout?
• a per-model-handle default timeout?
• a per-call timeout parameter?

Making the general default something other than "never time out" would probably be a good idea.

If there are multiple levels of timeout configuration, this would require a sentinel value that means "use the relevant default value", since None is already used in this context to mean "don't time out".

[0xIslamTaha]

@ncoghlan yes, great demonstration of the expected flow.

Also, I noticed that in sometime, even I managed to kill the blocked call (timeout it), The model is still trying to "Generate" the response, so Im expecting that whenever the request being timeout, the model should stop handling the timeout request.

[ncoghlan]

The server getting stuck like that would be the reason why the client didn't get the response it expected, so I wouldn't expect the client to be able to get the server to sort out its own misbehaviour.

On the client side, I'm going to start out with just a global timeout setting, since that's the least intrusive change (just two new APIs to read and set the default timeout).