Feature Request: Ability to scale ingestion pipeline in distributed manner over multiple workers rather than being on the same system.

[rnbokade]

Individual files being processed could be distributed to different workers or queued to be processed with said params. We might use celery or something like that

[onestardao]

Hi! Yes, you can absolutely build a similar code interpreter agent using Qwen3 instead of OpenAI.

Here’s how you can approach it:

Model Setup
Make sure Qwen3 is available either through HuggingFace Transformers or via an API endpoint you can call (e.g., through DashScope or your own inference server).

LLM Predictor Layer
In your LlamaIndex setup, just replace the OpenAI LLMPredictor with a custom one using Qwen3. If you're using the llama-index-llms module, define a custom class or use CustomLLM with Qwen-compatible completions.

Code Interpreter Logic
The core idea behind a code interpreter agent is chaining:

LLM → generates analysis plan

Python executor → runs code

Memory / context loop → feeds result back to the LLM for the next step
This structure doesn’t depend on OpenAI — it only needs the LLM to support basic I/O.

JSON Mode / Tool Use
If Qwen3 supports function-calling (or structured output), you can use it for more precise control. If not, you’ll need to rely on prompt formatting and regex parsing to extract code blocks.

Let me know if you need help wiring the Qwen model into llama-index or managing the tool callbacks — happy to help further.