Audiobooks not downloading as the URL is now too long

[geekylou]

Before you open an issue
Please make sure you've upgraded to the latest version of kobodl on pypi or docker. Also read through the troubleshooting guide. Try running with --debug enabled and see what comes up in the debug log.

Do not erase this template, it's here to help you.

Describe the issue
It looks like now kobodl is failing due to the URL returned from the kobo servers being too long for something in the python libraries. I receive a 403 Client error in __DownloadAudiobook but copying the URL manually to firefox loads the manifest. I can then copy then also copy the audiotracks from the manifest into firefox and the audiofile will load correctly.

To Reproduce
Steps to reproduce the behavior:
just call kobodl book get

[subdavis]

You erased some important parts of the issue template, so I can't help as much as I'd like to. To make sure this isn't an issue with the file path being too long:

kobodl book get -f '{ShortRevisionId}' UUID

If that doesn't work, please share the error you're seeing.

Possibly related: #95

[geekylou]

With -f '{ShortRevisionId}' I get:

ownloading f2e31e75-864d-4eb4-b105-0559fe9e7f27 to /home/louise/kobo_downloads/f2e31e75
Traceback (most recent call last):
  File "/home/louise/kobodl/bin/kobodl", line 8, in <module>
    sys.exit(cli())
  File "/home/louise/kobodl/lib/python3.10/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/louise/kobodl/lib/python3.10/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/louise/kobodl/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/louise/kobodl/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/louise/kobodl/lib/python3.10/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/louise/kobodl/lib/python3.10/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/louise/kobodl/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/louise/kobodl/lib/python3.10/site-packages/kobodl/commands/book.py", line 83, in get
    actions.GetBookOrBooks(usercls, output_dir, formatStr=format_str, productId=pid)
  File "/home/louise/kobodl/lib/python3.10/site-packages/kobodl/actions.py", line 231, in GetBookOrBooks
    kobo.Download(bookMetadata, book_type == BookType.AUDIOBOOK, outputFilePath)
  File "/home/louise/kobodl/lib/python3.10/site-packages/kobodl/kobo.py", line 349, in Download
    self.__DownloadAudiobook(downloadUrl, outputPath)
  File "/home/louise/kobodl/lib/python3.10/site-packages/kobodl/kobo.py", line 277, in __DownloadAudiobook
    response.raise_for_status()
  File "/home/louise/kobodl/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://storedownloads.kobo.com/download?downloadToken= \<redacted\>

Loading the url into firefox though gives me the metadata and I can then download it manually. Curiously using curl gives me cloudflare error about being blocked so maybe something else is going on.

[jljusten]

Loading the url into firefox though gives me the metadata and I can then download it manually. Curiously using curl gives me cloudflare error about being blocked so maybe something else is going on.

Here's a potentially related comment saying that using the urllib module instead of the requests module might help some 403 responses: https://stackoverflow.com/a/74674276

[leela1255]

I've also run into this problem - the original stack looks like this:

Traceback (most recent call last):
  File "kobodl\__main__.py", line 5, in <module>
  File "click\core.py", line 829, in __call__
  File "click\core.py", line 782, in main
  File "click\core.py", line 1259, in invoke
  File "click\core.py", line 1259, in invoke
  File "click\core.py", line 1066, in invoke
  File "click\core.py", line 610, in invoke
  File "click\decorators.py", line 33, in new_func
  File "kobodl\commands\book.py", line 83, in get
  File "kobodl\actions.py", line 231, in GetBookOrBooks
  File "kobodl\kobo.py", line 346, in Download
  File "kobodl\kobo.py", line 276, in __DownloadAudiobook
  File "requests\models.py", line 1021, in raise_for_status
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://storedownloads.kobo.com/download?downloadToken=
[11580] Failed to execute script '__main__' due to unhandled exception!

This appears to match the stack above after using -f '{ShortRevisionId}'. I also retried with -f '{ShortRevisionId}' and got the same stack again. This happened for 3 separate audiobooks I was attempting to download, so definitely seems to be a problem with them.

As noted, I could load the json and download each track individually so the ability to get the content still seems to be there, just needs some tweaking.

[geekylou]

Using cloudscraper fixed this issue for me. Replacing all references to self.Session.get(url) with scraper.get(url) in __DownloadAudiobook stopped the forbidden 403 error from being generated.