Merge pull request #163 from Ljzd-PRO/devel

Bump to v0.9.0

Author: Ljzd-PRO

CHANGELOG.md

@@ -2,46 +2,28 @@
 
 ### 💡 Feature
 
-- Stop using binary prefix (IEEE 1541-2002) in download speed unit (use `KB`, `MB`, ... instead of `KiB`, `MiB`, ...)
-- Stop downloading when failing to retrieve the creator's name, instead of using the creator ID as the directory name to continue downloading
-- In addition to the `prod.env` file, KToolBox also reads configurations from the **`.env`** file
-- When KToolBox starts, it will output the configuration details for user inspection
+- Show the download job status each 30s (waiting, running, completed%)
+- Shortened the log length
+  - E.g. `2024-10-05 20:12:37 | WARNING  | ktoolbox.job.runner - Download file already exists, skipping ...`
 
 ### 🪲 Fix
 
-- Fix the issue where the log output interrupts the download progress bar
-- Fix the `job.filename_format` configuration, where `{}` is simply replaced with the filename and extension without considering its position
-  - 📖More information: [Configuration-Reference-JobConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.JobConfiguration)
-      ```dotenv
-      # Rename attachments in numerical order, e.g. `1.png`, `2.png`, ...
-      KTOOLBOX_JOB__SEQUENTIAL_FILENAME=True
-
-      # `{}`: Basic filename
-      # Rename attachments to `1_[2024-1-1].png`, `2_[2024-1-1].png`, ...
-      KTOOLBOX_JOB__FILENAME_FORMAT="{}_[{published}]"
-      ```
+- Fix error when attempting to download files which posses too long names (invalid names) (#150)
+  - For example the wrong filename like this: `https://www.patreon.com/media-u/Z0FBQUFBQm........=#12345678_` \
+    KToolBox can get the correct filename: `6edd5bdae......0e7f913.png`
 
 - - -
 
 ### 💡 新特性
 
-- 停止在下载速度单位中使用二进制前缀（IEEE 1541-2002）（使用 `KB`, `MB`, ... 而不是 `KiB`, `MiB`, ...）
-- 获取作者名称失败时停止下载，而不是采用作者ID作为目录名继续下载
-- 除了 `prod.env` 文件以外，KToolBox 也会从 **`.env`** 文件读取配置
-- KToolBox 启动时将会输出配置详情，以便用户检查
+- 每隔 30 秒显示下载任务状态（等待中、运行中、已完成%）
+- 缩短了日志长度
+  - 例如 `2024-10-05 20:12:37 | WARNING  | ktoolbox.job.runner - Download file already exists, skipping ...`
 
 ### 🪲 修复
 
-- 修复下载进度条被输出的日志打断的问题
-- 修复 job.filename_format 文件名格式配置中的 {} 被简单地替换成文件名和后缀，而没有考虑其所在位置的问题
-  - 📖更多信息: [配置-参考-JobConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.JobConfiguration)
-      ```dotenv
-      # 按照数字顺序重命名附件, 例如 `1.png`, `2.png`, ...
-      KTOOLBOX_JOB__SEQUENTIAL_FILENAME=True
-
-      # `{}`：基本文件名
-      # 附件将被重命名为 `1_[2024-1-1].png`, `2_[2024-1-1].png`, ...
-      KTOOLBOX_JOB__FILENAME_FORMAT="{}_[{published}]"
-      ```
+- 修复下载过长文件名（非法文件名）的文件时报错的问题 (#150)
+  - 例如这样的错误文件名：`https://www.patreon.com/media-u/Z0FBQUFBQm........=#12345678_` \
+    KToolBox 可以获取到正确的文件名：`6edd5bdae......0e7f913.png`
 
-**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.7.0...v0.8.0
+**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.8.0...v0.9.0

docs/en/configuration/guide.md

@@ -1,11 +1,11 @@
 # Guide
 
-- KToolBox read **`prod.env` file** in the work folder or **environment variables** to store configuration
+- KToolBox read **`.env` or `prod.env` file** in the work folder or **environment variables** to store configuration
 - Check [Reference](reference.md) for all configuration options
 - Use `__` to specify the sub option, like `KTOOLBOX_API__SCHEME` means `api.scheme`
 - All configuration options are optional
 
-## `prod.env` file example
+## `.env` / `prod.env` file example
 
 ```dotenv
 # Download 10 files at the same time.
@@ -27,11 +27,4 @@ KTOOLBOX_JOB__FILENAME_FORMAT=[{published}]_{}
 
 # Prefix the post directory name with its release/publish date, e.g. `[2024-1-1]HelloWorld`
 KTOOLBOX_JOB__POST_DIRNAME_FORMAT=[{published}]{title}
-
-# Allocate 102400 Bytes as buffer for each download job
-KTOOLBOX_DOWNLOADER__BUFFER_SIZE=102400
-
-# Disable SSL certificate verification for Kemono API server and download server
-# It's useful when certificate on Kemono server expired. (SSL: CERTIFICATE_VERIFY_FAILED)
-KTOOLBOX_SSL_VERIFY=False
 ```

docs/en/coomer.md

@@ -2,9 +2,7 @@
 
 KToolBox support downloading from Coomer.su / Coomer.party
 
-Files of Coomer.su can be downloaded from Kemono.su / Kemono.party server, so you can download Coomer posts **without modifying configuration**.
-
-If it did not work, you can modify the configuration by `prod.env` dotenv file or system environment variables and then try again:
+You need to modify the configuration by `.env` or `prod.env` dotenv file or system environment variables:
 ```dotenv
 # For Coomer API
 KTOOLBOX_API__NETLOC=coomer.su

docs/en/index.md

@@ -106,3 +106,12 @@ ktoolbox sync-creator https://kemono.su/fanbox/user/9016 --offset=10 --length=5
 # Download posts from the creator/artist from 2024-1-1 to 2024-3-1
 ktoolbox sync-creator https://kemono.su/fanbox/user/9016 --start-time=2024-1-1 --end-time=2024-3-1
 ```
+
+### Configuration
+
+- Download 10 files at the same time
+- Rename attachments in numerical order
+- Prefix the post directory name with its release/publish date
+- ...
+
+Goto [Configuration-Guide](https://ktoolbox.readthedocs.io/latest/configuration/guide/) page for more details.

docs/zh/configuration/guide.md

@@ -1,11 +1,11 @@
 # 向导
 
-- KToolBox 读取工作目录下的 **`prod.env` 文件** 或 **环境变量** 来设定配置
+- KToolBox 读取工作目录下的 **`.env` 或 `prod.env` 文件** 或 **环境变量** 来设定配置
 - 前往 [参考](./reference.md) 查看所有配置选项
 - 用 `__` 来指定子选项, 例如 `KTOOLBOX_API__SCHEME` 相当于 `api.scheme`
 - 所有配置选项都是可选的
 
-## `prod.env` 文件示例
+## `.env` / `prod.env` 文件示例
 
 ```dotenv
 # 可同时下载10个文件
@@ -27,11 +27,4 @@ KTOOLBOX_JOB__FILENAME_FORMAT=[{published}]_{}
 
 # 将发布日期作为作品目录名的开头，例如 `[2024-1-1]HelloWorld`
 KTOOLBOX_JOB__POST_DIRNAME_FORMAT=[{published}]{title}
-
-# 为每个下载任务分配 102400 字节内存作为缓冲区
-KTOOLBOX_DOWNLOADER__BUFFER_SIZE=102400
-
-# 为Kemono API服务器和下载服务器禁用SSL证书检查
-# 在Kemono服务器的证书过期时很有用 （SSL: CERTIFICATE_VERIFY_FAILED）
-KTOOLBOX_SSL_VERIFY=False
 ```

docs/zh/coomer.md

@@ -2,9 +2,7 @@
 
 KToolBox 支持从 Coomer.su / Coomer.party 下载
 
-Coomer.su 的文件可以从 Kemono.su / Kemono.party 服务器下载，所以 **无需编辑配置** 即可下载 Coomer 里的作品
-
-如果无效，你可以通过 dotenv文件 `prod.env` 或系统环境变量来修改配置，然后再次尝试：
+你需要通过 dotenv文件 `.env` 或 `prod.env` 或系统环境变量来修改配置：
 ```dotenv
 # Coomer API
 KTOOLBOX_API__NETLOC=coomer.su

docs/zh/index.md

@@ -105,3 +105,12 @@ ktoolbox sync-creator https://kemono.su/fanbox/user/9016 --offset=10 --length=5
 # 下载作者/画师从 2024-1-1 到 2024-3-1 的作品
 ktoolbox sync-creator https://kemono.su/fanbox/user/9016 --start-time=2024-1-1 --end-time=2024-3-1
 ```
+
+### 配置
+
+- 同时下载10个文件
+- 按照数字顺序重命名附件, 例如 `1.png`, `2.png`, ...
+- 将发布日期作为作品目录名的开头，例如 `[2024-1-1]HelloWorld`
+- ...
+
+前往 [配置-向导](https://ktoolbox.readthedocs.io/latest/zh/configuration/guide/) 页面查看更多详情。

ktoolbox/__init__.py

@@ -1,4 +1,4 @@
 __title__ = "KToolBox"
 # noinspection SpellCheckingInspection
 __description__ = "A useful CLI tool for downloading posts in Kemono.party / .su"
-__version__ = "0.8.0"
+__version__ = "0.9.0"

ktoolbox/action/job.py

@@ -81,22 +81,24 @@ async def create_job_from_post(
 
     # Filter and create jobs for ``Post.file``
     if post.file and post.file.path:
-        post_file_name = post.file.name or Path(post.file.path).name
+        post_file_name = Path(post.file.name) if is_valid_filename(post.file.name) else Path(
+            urlparse(post.file.path).path
+        )
         if (not config.job.allow_list or any(
                 map(
-                    lambda x: fnmatch(post_file_name, x),
+                    lambda x: fnmatch(post_file_name.name, x),
                     config.job.allow_list
                 )
         )) and not any(
             map(
-                lambda x: fnmatch(post_file_name, x),
+                lambda x: fnmatch(post_file_name.name, x),
                 config.job.block_list
             )
         ):
             jobs.append(
                 Job(
                     path=post_path,
-                    alt_filename=f"{post.id}_{post_file_name}",
+                    alt_filename=f"{post.id}_{post_file_name.name}",
                     server_path=post.file.path,
                     type=PostFileTypeEnum.File
                 )

ktoolbox/job/runner.py

@@ -141,6 +141,17 @@ async def processor(self) -> int:
         await self._job_queue.join()
         return failed_num
 
+    async def _watch_status(self):
+        """
+        Watch running, completed, failed jobs
+        """
+        while not self._job_queue.empty():
+            await asyncio.sleep(30)
+            logger.info(f"Waiting: {self.waiting_size} / "
+                        f"Running: {self.processing_size} / "
+                        f"Completed: {self.done_size} "
+                        f"({(self.done_size / (self.waiting_size + self.processing_size + self.done_size)) * 100:.2f}%)")
+
     async def start(self):
         """
         Start processing jobs concurrently
@@ -154,7 +165,10 @@ async def start(self):
                 task = asyncio.create_task(self.processor())
                 self._concurrent_tasks.add(task)
                 task.add_done_callback(self._concurrent_tasks.discard)
-            task_done_set, _ = await asyncio.wait(self._concurrent_tasks)
+            _, (task_done_set, _) = await asyncio.gather(
+                self._watch_status(),
+                asyncio.wait(self._concurrent_tasks)
+            )
             for task in task_done_set:
                 try:
                     failed_num += task.result()

ktoolbox/utils.py

@@ -63,6 +63,9 @@ def logger_init(cli_use: bool = False, disable_stdout: bool = False):
         logger.add(
             tqdm.write,
             colorize=True,
+            format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | "
+                   "<level>{level: <8}</level> | "
+                   "<cyan>{name}</cyan> - <level>{message}</level>",
             level=logging.INFO,
             filter=lambda record: record["level"].name != "SUCCESS"
         )