OpenAI Batch API support

[zigzag312]

Hi!

OpenAI offers special batch API that is 50% cheaper than the normal (synchronous) API.

With batch API all requests are combined into a single jsonl (where json requests are newline-delimited). The reason batch API is cheaper is that the response is not immediate. It can take up to 24h to process the batch.

https://platform.openai.com/docs/guides/batch
https://platform.openai.com/docs/api-reference/batch

Book translations seems to be a perfect fit for the batch API. Very high token counts and results usually do not need to be immediate.

Single batch can contain up to 50'000 requests or up to 100MB.

[bookfere]

Thank you for the suggestion! It's a fantastic choice for our translation needs, and I'm excited to implement it. You can create an issue from this discussion to keep track of this requirement.

[zigzag312]

Thank you for a kind and welcoming response! As you suggested I created an issue for this #319.

[bookfere]

I initially implemented the ChatGPT batch translation feature based on OpenAI's API documentation (commit d90fcbf). Since I do not have access to the actual API, there may be many faults within the functionality. To improve the feature's usability, ongoing feedback is needed.

[bookfere]

Thank you for your interest in the feature. At the moment, I need to prioritize taking care of a family member who is in the hospital. Once I have some free time available, I will resume the debugging process. I truly appreciate your patience during this time.

[zigzag312]

@bookfere Thank you for the update. Wishing your family member a smooth recovery!

[bookfere]

I've updated the code and tried to fix the error you encountered. Please give it a try.

[DomBedos]

Thanks a lot...
I just tried and got :

[zigzag312]

I get same message as @DomBedos.

On OpenAI's website (https://platform.openai.com/batches) I can see the failed batch. There seems to be an an error for each line:

Line 1 Invalid type for 'body': expected an object, but got a string instead.
Line 2 Invalid type for 'body': expected an object, but got a string instead.
Line 3 Invalid type for 'body': expected an object, but got a string instead.
...

Example of a line with an error (I truncated few values to make it shorter):

{"custom_id": "1d..", "method": "POST", "url": "/v1/chat/completions", "body": "{\"model\": \"gpt-3.5-turbo\", \"messages\": [{\"role\": \"system\", \"content\": \"Translate...\"}, {\"role\": \"user\", \"content\": \"Content...\"}], \"stream\": true, \"temperature\": 0.0}"}

I think the issue is that body value is wrapped in quotes.

[bookfere]

Thank you for your information and details. I tried to fix the error and uploaded the revised version here. Feel free to give me further feedback.

ebook-translator_202407040105.zip

[zigzag312]

Hmm now I get, what seems to be same error, displayed in plugin's as "Batch Translation Details":

{'object': 'list', 'data': [{'code': 'invalid_type', 'message': "Invalid type for 'body': expected an object, but got a string instead.", 'param': 'body', 'line': 1}

Could this details be from the previous/old batch?

If I then click "Cancel Batch Translation" I get an error:

Traceback (most recent call last):
  File "calibre_plugins.ebook_translator.components.chatgpt", line 28, in wrapper
  File "calibre_plugins.ebook_translator.components.chatgpt", line 109, in cancel_batch
  File "calibre_plugins.ebook_translator.engines.openai", line 295, in cancel
  File "calibre_plugins.ebook_translator.lib.utils", line 170, in request
  File "mechanize\_mechanize.py", line 257, in open
  File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 409: Conflict

[bookfere]

It likely needs to create a new batch. I updated the code. Please click the "Cancel Batch Translation" button to remove the old batch identifier from the cache, and then recreate a batch to try again.

ebook-translator_202407042238.zip

[zigzag312]

Great work! This version is almost complete. Canceling translation now works. Starting new batch translation also works. And refreshing status after some time shows that the batch translation has completed successfully!

But clicking on "Apply" button, returns this error:

Traceback (most recent call last):
  File "calibre_plugins.ebook_translator.components.chatgpt", line 30, in wrapper
  File "calibre_plugins.ebook_translator.components.chatgpt", line 127, in apply_batch
  File "calibre_plugins.ebook_translator.engines.openai", line 233, in retrieve
  File "calibre_plugins.ebook_translator.lib.utils", line 170, in request
  File "mechanize\_mechanize.py", line 257, in open
  File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 404: Not Found

[bookfere]

I used the wrong endpoint to retrieve the batch translation result. It is now fixed. Please try again.

ebook-translator_202407052102.zip

[DomBedos]

Thanks a lot for your efforts !
When clicking "Apply" I get :
Traceback (most recent call last):
File "calibre_plugins.ebook_translator.components.chatgpt", line 30, in wrapper
func(self)
File "calibre_plugins.ebook_translator.components.chatgpt", line 127, in apply_batch
translations = self._batch_translator.retrieve(output_file_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "calibre_plugins.ebook_translator.engines.openai", line 238, in retrieve
for line in io.BytesIO(response):
^^^^^^^^^^^^^^^^^^^^
TypeError: a bytes-like object is required, not 'str'

[bookfere]

Wrongly processed the response. Please try this one.

ebook-translator_202407060120.zip

[zigzag312]

Everything works now. Translation completed successfully. Thank you a lot!

[bookfere]

@zigzag312 @DomBedos Thank you for your continued feedback! I will continuously refine the feature, and this discussion will remain open in case any related issues arise.

[DomBedos]

Hi !
GPT-4o mini has just been released and is recommended as a replacement for GPT-3.5 turbo. Would it be possible to implement this function? Also, I systematically get errors when using the advanced translation function, using a text selection.
Any ideas ?
I also get this error when going to the settings to test the engine:
Traceback (most recent call last):
File "calibre_plugins.ebook_translator.components.engine", line 76, in translate_text
for text in translation:
File "calibre_plugins.ebook_translator.engines.openai", line 116, in _parse_stream
raise Exception(
Exception: Ne peux analyser la réponse retournée. Data non-traité: 'bytes' object has no attribute 'readline'

[translatefree]

Hi,

I encountered the same error in both Anthropic and OpenAI implementations:

For OpenAI:

Traceback (most recent call last):
  File "calibre_plugins.ebook_translator.components.engine", line 76, in translate_text
  File "calibre_plugins.ebook_translator.engines.openai", line 116, in *parse*stream
Exception: Can not parse returned response. Raw data: 'bytes' object has no attribute 'readline'

For Anthropic:

Traceback (most recent call last):
  File "calibre_plugins.ebook_translator.components.engine", line 76, in translate_text
  File "calibre_plugins.ebook_translator.engines.anthropic", line 104, in *parse*stream
Exception: Can not parse returned response. Raw data: 'bytes' object has no attribute 'readline'

After unsuccessfully trying to resolve this on my own, I sought help from AI. ChatGPT wasn't able to provide a solution, but Claude helped me solve the problem. I'd like to share this solution as it might be useful to others and to the author, given the recent family issues (I hope they resolve soon).

The issue stems from the _parse_stream method attempting to use readline() on a bytes object, which doesn't support this method. Here's the fix for both the Anthropic and OpenAI implementations:

For the ClaudeTranslate class in anthropic.py:

def _parse_stream(self, data):
    if isinstance(data, bytes):
        # If data is a bytes object, convert it to an iterator of lines
        lines = data.decode('utf-8').splitlines()
    else:
        # Otherwise, assume it's a file-like object
        lines = data

    for line in lines:
        try:
            line = line.strip()
            if not line or not line.startswith('data:'):
                continue
            chunk = json.loads(line.split('data: ', 1)[1])
            if chunk.get('type') == 'message_stop':
                break
            if chunk.get('type') == 'content_block_delta':
                yield str(chunk.get('delta', {}).get('text', ''))
        except json.JSONDecodeError:
            continue
        except Exception as e:
            raise Exception(
                _('Cannot parse returned response. Raw data: {}')
                .format(str(e)))

For the ChatgptTranslate class in openai.py:

def _parse_stream(self, data):
    if isinstance(data, bytes):
        # If data is a bytes object, convert it to an iterator of lines
        lines = data.decode('utf-8').splitlines()
    else:
        # Otherwise, assume it's a file-like object
        lines = data

    for line in lines:
        try:
            line = line.strip()
            if not line or not line.startswith('data:'):
                continue
            chunk = line.split('data: ', 1)[1]
            if chunk == '[DONE]':
                break
            delta = json.loads(chunk)['choices'][0]['delta']
            if 'content' in delta:
                yield str(delta['content'])
        except json.JSONDecodeError:
            continue
        except Exception as e:
            raise Exception(
                _('Can not parse returned response. Raw data: {}')
                .format(str(e)))

These changes make the _parse_stream method more robust by handling both bytes objects and file-like objects. It also improves error handling and maintains the specific parsing logic for each API.

To implement these fixes:

1. Open the respective files (anthropic.py and openai.py) in the Calibre plugin.
2. Replace the existing _parse_stream methods with the ones provided above.
3. Ensure proper indentation within the respective classes.
4. Save the files and restart Calibre to apply the changes.

I hope this helps resolve the issue for others experiencing similar problems. Thank you for maintaining this useful plugin!
ebook-translator_202408240120.zip

[seh-len]

Would it be possible to split a source into several batches? I can only process 90,000 tokens per batch in my current tier (I don't know if more would be possible in a higher tier), if I put more tokens in a batch, the batch aborts immediately with an error during processing because the TPD per batch is exceeded. Or should i use a different model for translations than gpt-4o?

Enqueued token limit reached for gpt-4o in organization org-xxx. Limit: 90,000 enqueued tokens. Please try again once some in_progress batches have been completed.