Merge pull request #187 from Ljzd-PRO/devel

Ljzd-PRO · web-flow · commit cf5c12c631fd · 2024-11-19T03:19:10.000+08:00
Bump to v0.11.0
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,24 +2,21 @@
 
 ### 💡 Feature
 
-- Added a **graphical configuration editor**, making configuration editing simple and convenient.
-  - Run `ktoolbox config-editor`
-- Added a command to generate a complete sample configuration file.
-  - Run `ktoolbox example-env`
-- **Python** versions below 3.8.1 (**<3.8.1**) are no longer supported
+- The `search-creator` command will include search results with similar names.
+  - For example, the search parameter `--name abc` might return author information such as: `abc, abcd, hi-abc`
+- Share an HTTPX client to reuse underlying TCP connections through an HTTP connection pool when calling APIs and downloading, 
+**significantly improving query and download speeds as well as connection stability**
 
 [//]: # (### 🪲 Fix)
 
 - - -
 
 ### 💡 新特性
 
-- 新增 **图形化配置编辑器**，配置编辑将变得简单方便
-  - 运行 `ktoolbox config-editor`
-- 新增命令可生成完整的配置文件样例
-  - 运行 `ktoolbox example-env`
-- **Python** 3.8.1 以下 (**<3.8.1**) 的版本不再受支持
+- search-creator 搜索作者的命令将包含那些名字相近的搜索结果
+  - 如搜索参数 `--name abc` 可能得到如下作者信息：`abc, abcd, hi-abc`
+- 共享 HTTPX 客户端，调用 API 和下载时将通过 HTTP 连接池重用底层 TCP 连接，**显著提升查询、下载速度和连接稳定性**
 
 [//]: # (### 🪲 修复)
 
-**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.9.0...v0.10.0
+**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.10.0...v0.11.0
diff --git a/README.md b/README.md
@@ -8,7 +8,7 @@
 
 <p align="center">
   KToolBox is a useful CLI tool for downloading posts content in
-  <a href="https://kemono.su/">Kemono.party / Kemono.su</a>
+  <a href="https://kemono.su/">Kemono.su / Kemono.party</a>
 </p>
 
 <p align="center">
@@ -51,19 +51,20 @@
 
 ## Features
 
-- Support for **multi-thread** downloads (technically, coroutine)
-- **Retry** after download failed
-- Ability to download individual post as well as **all post** by a specified creator/artist
-- **Update downloaded** creator/artist directories to the latest status
-- Customize the **structure** of downloaded post/creator **directories**
-- Search for creators/artists and posts, and **export the results**
-- **Cross-platform** support & **iOS shortcuts** available
-- For Coomer.su / Coomer.party support, check document [Coomer](https://ktoolbox.readthedocs.io/latest/coomer/) for more.
+- Supports concurrent downloads  
+- Automatically retries API calls and downloads after failures  
+- Allows downloading individual posts or **all posts** of a specified artist  
+- Can **update downloaded** artist directories to the latest state  
+- Supports customizable **file and directory naming formats** and **directory structures** for downloaded posts/artists  
+- Enables excluding **specified file formats** or downloading only specified formats  
+- Allows searching for artists and posts, with options to export results  
+- Compatible with all platforms, with iOS shortcuts provided  
+- For support related to _Coomer.su / Coomer.party_, please refer to the documentation: [Coomer](https://ktoolbox.readthedocs.io/latest/coomer/)
 
 ## Dev Plan
 
 - [ ] GUI
-- [x] Add uvloop support for Unix platform
+- [ ] Discord support
 
 ## Tutorial
 
diff --git a/README_zh-CN.md b/README_zh-CN.md
@@ -8,7 +8,7 @@
 
 <p align="center">
   KToolBox 是一个用于下载
-  <a href="https://kemono.su/">Kemono.party / Kemono.su</a>
+  <a href="https://kemono.su/">Kemono.su / Kemono.party</a>
   中作品内容的实用命令行工具
 </p>
 
@@ -52,19 +52,20 @@
 
 ## 功能
 
-- 支持 **多线程** 下载（技术上是协程）
-- 下载失败后进行 **重试**
+- 支持并发下载
+- API 调用和下载失败后 **自动重试**
 - 支持下载单个作品以及指定的画师的 **所有作品**
 - 可 **更新已下载** 的画师目录至最新状态
-- 可自定义下载的作品/画师 **目录结构**
-- 可搜索画师和作品，并 **导出结果**
-- 支持全平台，并提供 **iOS 快捷指令**
-- 对于 Coomer.su / Coomer.party 的支持，请查看文档 [Coomer](https://ktoolbox.readthedocs.io/latest/zh/coomer/)。
+- 支持自定义下载的作品/画师 **文件和目录名格式**、**目录结构**
+- 支持排除 **指定格式** 的文件或仅下载指定格式的文件
+- 可搜索画师和作品，并导出结果
+- 支持全平台，并提供 iOS 快捷指令
+- 对于 Coomer.su / Coomer.party 的支持，请查看文档 [Coomer](https://ktoolbox.readthedocs.io/latest/zh/coomer/)
 
 ## 开发计划
 
 - [ ] GUI
-- [x] 对 Unix 平台增加 uvloop 支持
+- [ ] Discord 下载支持
 
 ## 使用方法
 
diff --git a/docs/en/index.md b/docs/en/index.md
@@ -39,13 +39,15 @@
 
 ## Features
 
-- Support for **multi-thread** downloads (technically, coroutine)
-- **Retry** after download failed
-- Ability to download individual post as well as **all post** by a specified creator/artist
-- **Update downloaded** creator/artist directories to the latest status
-- Customize the **structure** of downloaded post/creator **directories**
-- Search for creators/artists and posts, and **export the results**
-- **Cross-platform** support & **iOS shortcuts** available
+- Supports concurrent downloads  
+- Automatically retries API calls and downloads after failures  
+- Allows downloading individual posts or **all posts** of a specified artist  
+- Can **update downloaded** artist directories to the latest state  
+- Supports customizable **file and directory naming formats** and **directory structures** for downloaded posts/artists  
+- Enables excluding **specified file formats** or downloading only specified formats  
+- Allows searching for artists and posts, with options to export results  
+- Compatible with all platforms, with iOS shortcuts provided  
+- For support related to _Coomer.su / Coomer.party_, please refer to the documentation: [Coomer](https://ktoolbox.readthedocs.io/latest/coomer/)
 
 ## Tutorial
 
diff --git a/docs/zh/index.md b/docs/zh/index.md
@@ -8,7 +8,7 @@
 
 <p style="text-align: center">
   KToolBox 是一个用于下载
-  <a href="https://kemono.su/">Kemono.party / Kemono.su</a>
+  <a href="https://kemono.su/">Kemono.su / Kemono.party</a>
   中作品内容的实用命令行工具
 </p>
 
@@ -40,13 +40,15 @@
 
 ## 功能
 
-- 支持 **多线程** 下载（技术上是协程）
-- 下载失败后进行 **重试**
+- 支持并发下载
+- API 调用和下载失败后 **自动重试**
 - 支持下载单个作品以及指定的画师的 **所有作品**
 - 可 **更新已下载** 的画师目录至最新状态
-- 可自定义下载的作品/画师 **目录结构**
-- 可搜索画师和作品，并 **导出结果**
-- 支持全平台，并提供 **iOS 快捷指令**
+- 支持自定义下载的作品/画师 **文件和目录名格式**、**目录结构**
+- 支持排除 **指定格式** 的文件或仅下载指定格式的文件
+- 可搜索画师和作品，并导出结果
+- 支持全平台，并提供 iOS 快捷指令
+- 对于 Coomer.su / Coomer.party 的支持，请查看文档 [Coomer](https://ktoolbox.readthedocs.io/latest/zh/coomer/)
 
 ## 使用方法
 
diff --git a/ktoolbox/__init__.py b/ktoolbox/__init__.py
@@ -1,4 +1,4 @@
 __title__ = "KToolBox"
 # noinspection SpellCheckingInspection
 __description__ = "A useful CLI tool for downloading posts in Kemono.party / .su"
-__version__ = "v0.10.0"
+__version__ = "v0.11.0"
diff --git a/ktoolbox/action/search.py b/ktoolbox/action/search.py
@@ -19,23 +19,23 @@ async def search_creator(id: str = None, name: str = None, service: str = None)
     :param service: The service for the creator
     """
 
-    async def inner(**kwargs):
-        def filter_func(creator: Creator):
-            """Filter creators with attributes"""
-            for key, value in kwargs.items():
-                if value is None:
-                    continue
-                elif creator.__getattribute__(key) != value:
-                    return False
-            return True
-
-        ret = await get_creators()
-        if not ret:
-            return ret
-        creators = ret.data
-        return ActionRet(data=iter(filter(filter_func, creators)))
+    def filter_func(creator: Creator):
+        """Filter creators with attributes"""
+        if id is not None and creator.id != id:
+            return False
+        if name is not None and name not in creator.name:
+            return False
+        if service is not None and creator.service != service:
+            return False
+        return True
 
-    return await inner(id=id, name=name, service=service)
+    ret = await get_creators()
+    if not ret:
+        base_ret = BaseRet.model_validate(ret.model_dump())
+        base_ret.data = iter([])
+        return base_ret
+    creators = ret.data
+    return ActionRet(data=iter(filter(filter_func, creators)))
 
 
 # noinspection PyShadowingBuiltins
diff --git a/ktoolbox/api/base.py b/ktoolbox/api/base.py
@@ -68,6 +68,7 @@ class BaseAPI(ABC, Generic[_T]):
     path: str = "/"
     method: Literal["get", "post"]
     extra_validator: Optional[Callable[[str], BaseModel]] = None
+    client = httpx.AsyncClient(verify=config.ssl_verify)
 
     Response = BaseModel
     """API response model"""
@@ -110,14 +111,13 @@ async def request(cls, path: str = None, **kwargs) -> APIRet[_T]:
         url_parts = [config.api.scheme, config.api.netloc, f"{config.api.path}{path}", '', '', '']
         url = str(urlunparse(url_parts))
         try:
-            async with httpx.AsyncClient(verify=config.ssl_verify) as client:
-                res = await client.request(
-                    method=cls.method,
-                    url=url,
-                    timeout=config.api.timeout,
-                    follow_redirects=True,
-                    **kwargs
-                )
+            res = await cls.client.request(
+                method=cls.method,
+                url=url,
+                timeout=config.api.timeout,
+                follow_redirects=True,
+                **kwargs
+            )
         except Exception as e:
             return APIRet(
                 code=RetCodeEnum.NetWorkError,
diff --git a/ktoolbox/api/model/creator.py b/ktoolbox/api/model/creator.py
@@ -1,3 +1,5 @@
+from datetime import datetime
+
 from pydantic import BaseModel
 
 __all__ = ["Creator"]
@@ -10,11 +12,11 @@ class Creator(BaseModel):
     """The number of times this creator has been favorited"""
     id: str
     """The ID of the creator"""
-    indexed: float
+    indexed: datetime
     """Timestamp when the creator was indexed, Unix time as integer"""
     name: str
     """The name of the creator"""
     service: str
     """The service for the creator"""
-    updated: float
+    updated: datetime
     """Timestamp when the creator was last updated, Unix time as integer"""
diff --git a/ktoolbox/downloader/downloader.py b/ktoolbox/downloader/downloader.py
@@ -29,6 +29,7 @@ class Downloader:
     """
     :ivar _save_filename: The actual filename for saving.
     """
+    client = httpx.AsyncClient(verify=config.ssl_verify)
 
     def __init__(
             self,
@@ -172,56 +173,55 @@ async def run(
 
         tqdm_class: Type[std_tqdm] = tqdm_class or tqdm.asyncio.tqdm
         async with self._lock:
-            async with httpx.AsyncClient(verify=config.ssl_verify) as client:
-                async with client.stream(
-                        method="GET",
-                        url=self._url,
-                        follow_redirects=True,
-                        timeout=config.downloader.timeout
-                ) as res:  # type: httpx.Response
-                    if res.status_code != httpx.codes.OK:
-                        return DownloaderRet(
-                            code=RetCodeEnum.GeneralFailure,
-                            message=generate_msg(
-                                "Download failed",
-                                status_code=res.status_code,
-                                filename=save_filepath
-                            )
+            async with self.client.stream(
+                    method="GET",
+                    url=self._url,
+                    follow_redirects=True,
+                    timeout=config.downloader.timeout
+            ) as res:  # type: httpx.Response
+                if res.status_code != httpx.codes.OK:
+                    return DownloaderRet(
+                        code=RetCodeEnum.GeneralFailure,
+                        message=generate_msg(
+                            "Download failed",
+                            status_code=res.status_code,
+                            filename=save_filepath
                         )
-
-                    # Get filename for saving and check if file exists (Second-time duplicate file check)
-                    # Priority order can be referenced from the constructor's documentation
-                    self._save_filename = self._designated_filename or sanitize_filename(
-                        filename_from_headers(res.headers)
-                    ) or server_path_filename
-                    save_filepath = self._path / self._save_filename
-                    file_existed, ret_msg = duplicate_file_check(save_filepath, bucket_file_path)
-                    if file_existed:
-                        return DownloaderRet(
-                            code=RetCodeEnum.FileExisted,
-                            message=generate_msg(
-                                ret_msg,
-                                path=save_filepath
-                            )
-                        )
-
-                    # Download
-                    temp_filepath = Path(f"{save_filepath}.{config.downloader.temp_suffix}")
-                    total_size = int(length_str) if (length_str := res.headers.get("Content-Length")) else None
-                    async with aiofiles.open(str(temp_filepath), "wb", self._buffer_size) as f:
-                        chunk_iterator = res.aiter_bytes(self._chunk_size)
-                        t = tqdm_class(
-                            desc=self._save_filename,
-                            total=total_size,
-                            disable=not progress,
-                            unit="B",
-                            unit_scale=True
+                    )
+
+                # Get filename for saving and check if file exists (Second-time duplicate file check)
+                # Priority order can be referenced from the constructor's documentation
+                self._save_filename = self._designated_filename or sanitize_filename(
+                    filename_from_headers(res.headers)
+                ) or server_path_filename
+                save_filepath = self._path / self._save_filename
+                file_existed, ret_msg = duplicate_file_check(save_filepath, bucket_file_path)
+                if file_existed:
+                    return DownloaderRet(
+                        code=RetCodeEnum.FileExisted,
+                        message=generate_msg(
+                            ret_msg,
+                            path=save_filepath
                         )
-                        async for chunk in chunk_iterator:
-                            if self._stop:
-                                raise CancelledError
-                            await f.write(chunk)
-                            t.update(len(chunk))  # Update progress bar
+                    )
+
+                # Download
+                temp_filepath = Path(f"{save_filepath}.{config.downloader.temp_suffix}")
+                total_size = int(length_str) if (length_str := res.headers.get("Content-Length")) else None
+                async with aiofiles.open(str(temp_filepath), "wb", self._buffer_size) as f:
+                    chunk_iterator = res.aiter_bytes(self._chunk_size)
+                    t = tqdm_class(
+                        desc=self._save_filename,
+                        total=total_size,
+                        disable=not progress,
+                        unit="B",
+                        unit_scale=True
+                    )
+                    async for chunk in chunk_iterator:
+                        if self._stop:
+                            raise CancelledError
+                        await f.write(chunk)
+                        t.update(len(chunk))  # Update progress bar
 
             # Download finished
             if config.downloader.use_bucket:
diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
