Merge pull request #217 from Ljzd-PRO/devel

Bump to v0.13.0

Author: Ljzd-PRO

CHANGELOG.md

@@ -1,19 +1,61 @@
 ## Changes
 
-![Downloads](https://img.shields.io/github/downloads/Ljzd-PRO/KToolBox/v0.12.0/total)
+![Downloads](https://img.shields.io/github/downloads/Ljzd-PRO/KToolBox/v0.13.0/total)
 
 ### 💡 Feature
 
-- Supports resuming downloads using temporary files to restore the download process - (#205) by @kgbq3nu6s9ptn3jr
+- Add support for customizing the `Post.file` filename format (not `Post.attachments`)
+  - Run `ktoolbox config-editor` to edit the configuration (`Job -> post_structure -> file`)
+  - Or manually edit `KTOOLBOX_JOB__POST_STRUCTURE__FILE` in `.env` file or environment variables to set this option
+    ```dotenv
+    # Example
+    # Result filename: [2023-01-01]_TheTitle_12345_UxTleZ3zP6LHA7BPNxlEWDzX.jpg
+    KTOOLBOX_JOB__POST_STRUCTURE__FILE=[{published}]_{title}_{id}_{}
+    ```
+  - 📖More information: [Configuration-Reference-JobConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.JobConfiguration)
+
+- Add support for customizing reverse proxy for download URLs - (#216)
+  - Run `ktoolbox config-editor` to edit the configuration (`Downloader -> reverse_proxy`)
+  - Or manually edit `KTOOLBOX_DOWNLOADER__REVERSE_PROXY` in `.env` file or environment variables to set this option
+    ```dotenv
+    # Example
+    # Result download URL: https://example.com/https://n1.kemono.su/data/66/83/xxxx.jpg
+    KTOOLBOX_DOWNLOADER__REVERSE_PROXY="https://example.com/{}"
+    ```
+  - 📖More information: [Configuration-Reference-DownloaderConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.DownloaderConfiguration)
+
+- Rename `PostStructure.content_filepath` to `PostStructure.content`, the old configuration is still available, but it will be removed in the future. If you use this option, you will receive a warning message.
+  - 📖More information: [Configuration-Reference-PostStructureConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.PostStructureConfiguration)
 
 [//]: # (### 🪲 Fix)
 
 - - -
 
 ### 💡 新特性
 
-- 支持断点续传，借助临时下载文件恢复下载 - (#205) by @kgbq3nu6s9ptn3jr
+- 增加支持自定义 `Post.file` 的文件名格式（并非 `Post.attachments`）
+  - 执行 `ktoolbox config-editor` 来编辑这项配置 (`Job -> post_structure -> file`)
+  - 或手动编辑 `.env` 文件中的 `KTOOLBOX_JOB__POST_STRUCTURE__FILE` 或环境变量来设置这项配置
+    ```dotenv
+    # 示例
+    # 文件名：[2023-01-01]_TheTitle_12345_UxTleZ3zP6LHA7BPNxlEWDzX.jpg
+    KTOOLBOX_JOB__POST_STRUCTURE__FILE=[{published}]_{title}_{id}_{}
+    ```
+  - 📖更多信息：[Configuration-Reference-JobConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.JobConfiguration)
+
+- 增加支持自定义下载 URL 的反向代理 - (#216)
+  - 执行 `ktoolbox config-editor` 来编辑这项配置 (`Downloader -> reverse_proxy`)
+  - 或手动编辑 `.env` 文件中的 `KTOOLBOX_DOWNLOADER__REVERSE_PROXY` 或环境变量来设置这项配置
+    ```dotenv
+    # 示例
+    # 下载 URL：https://example.com/https://n1.kemono.su/data/66/83/xxxx.jpg
+    KTOOLBOX_DOWNLOADER__REVERSE_PROXY="https://example.com/{}"
+    ```
+  - 📖更多信息：[Configuration-Reference-DownloaderConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.DownloaderConfiguration)
+
+- `PostStructure.content_filepath` 更名为 `PostStructure.content`，原先的配置仍可继续使用，但未来将会被移除。如果您使用了这项配置，将会收到警告信息。
+  - 📖更多信息：[Configuration-Reference-PostStructureConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.PostStructureConfiguration)
 
 [//]: # (### 🪲 修复)
 
-**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.11.1...v0.12.0
+**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.12.0...v0.13.0

ktoolbox/__init__.py

@@ -1,4 +1,4 @@
 __title__ = "KToolBox"
 # noinspection SpellCheckingInspection
 __description__ = "A useful CLI tool for downloading posts in Kemono.party / .su"
-__version__ = "v0.12.0"
+__version__ = "v0.13.0"

ktoolbox/action/job.py

@@ -43,7 +43,7 @@ async def create_job_from_post(
     if post_structure:
         attachments_path = post_path / post_structure.attachments  # attachments
         attachments_path.mkdir(exist_ok=True)
-        content_path = post_path / post_structure.content_filepath  # content
+        content_path = post_path / post_structure.content  # content
         content_path.parent.mkdir(exist_ok=True)
     else:
         attachments_path = post_path
@@ -69,7 +69,7 @@ async def create_job_from_post(
             )
         ):
             basic_filename = f"{i + 1}{file_path_obj.suffix}" if config.job.sequential_filename else file_path_obj.name
-            alt_filename = generate_filename(post, basic_filename)
+            alt_filename = generate_filename(post, basic_filename, config.job.filename_format)
             jobs.append(
                 Job(
                     path=attachments_path,
@@ -84,6 +84,7 @@ async def create_job_from_post(
         post_file_name = Path(post.file.name) if is_valid_filename(post.file.name) else Path(
             urlparse(post.file.path).path
         )
+        post_file_name = Path(generate_filename(post, post_file_name.name, config.job.post_structure.file))
         if (not config.job.allow_list or any(
                 map(
                     lambda x: fnmatch(post_file_name.name, x),
@@ -98,7 +99,7 @@ async def create_job_from_post(
             jobs.append(
                 Job(
                     path=post_path,
-                    alt_filename=f"{post.id}_{post_file_name.name}",
+                    alt_filename=post_file_name.name,
                     server_path=post.file.path,
                     type=PostFileTypeEnum.File
                 )

ktoolbox/action/utils.py

@@ -36,13 +36,13 @@ def generate_post_path_name(post: Post) -> str:
             exit(1)
 
 
-def generate_filename(post: Post, basic_name: str) -> str:
+def generate_filename(post: Post, basic_name: str, filename_format: str) -> str:
     """Generate download filename"""
-    basic_name_suffix = Path(basic_name).suffix
-    basic_name_filename = basic_name.split(basic_name_suffix)[0] if basic_name_suffix else basic_name
+    basic_name_path = Path(basic_name)
+    basic_name_filename = basic_name.replace(basic_name_path.suffix, "")
     try:
         return sanitize_filename(
-            config.job.filename_format.format(
+            filename_format.format(
                 basic_name_filename,
                 id=post.id,
                 user=post.user,
@@ -51,10 +51,11 @@ def generate_filename(post: Post, basic_name: str) -> str:
                 added=post.added.strftime(TIME_FORMAT) if post.added else "",
                 published=post.published.strftime(TIME_FORMAT) if post.published else "",
                 edited=post.edited.strftime(TIME_FORMAT) if post.edited else ""
-            ) + basic_name_suffix
+            ) + basic_name_path.suffix
         )
     except KeyError as e:
-        logger.error(f"`JobConfiguration.filename_format` contains invalid key: {e}")
+        logger.error(
+            f"`JobConfiguration.filename_format` or `PostStructureConfiguration.file` contains invalid key: {e}")
         exit(1)
 
 

ktoolbox/configuration.py

@@ -2,11 +2,12 @@
 import logging
 import os
 import tempfile
+import warnings
 from pathlib import Path
-from typing import Literal, Union, Optional, Set
+from typing import Literal, Union, Optional, Set, ClassVar
 
 from loguru import logger
-from pydantic import BaseModel, model_validator
+from pydantic import BaseModel, model_validator, Field, field_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
 __all__ = [
@@ -60,6 +61,10 @@ class DownloaderConfiguration(BaseModel):
     :ivar retry_interval: Seconds of downloader retry interval
     :ivar use_bucket: Enable local storage bucket mode
     :ivar bucket_path: Path of local storage bucket
+    :ivar reverse_proxy: Reverse proxy format for download URL. \
+    Customize the filename format by inserting an empty ``{}`` to represent the original URL. \
+    For example: ``https://example.com/{}`` will be ``https://example.com/https://n1.kemono.su/data/66/83/xxxxx.jpg``;  \
+    ``https://example.com/?url={}`` will be ``https://example.com/?url=https://n1.kemono.su/data/66/83/xxxxx.jpg``
     """
     scheme: Literal["http", "https"] = "https"
     timeout: float = 30.0
@@ -72,6 +77,7 @@ class DownloaderConfiguration(BaseModel):
     retry_interval: float = 3.0
     use_bucket: bool = False
     bucket_path: Path = Path("./.ktoolbox/bucket_storage")
+    reverse_proxy: str = "{}"
 
     @model_validator(mode="after")
     def check_bucket_path(self) -> "DownloaderConfiguration":
@@ -107,18 +113,48 @@ class PostStructureConfiguration(BaseModel):
        └─ 2.png
     ```
 
+    - Available properties for ``file``
+
+        | Property      | Type   |
+        |---------------|--------|
+        | ``id``        | String |
+        | ``user``      | String |
+        | ``service``   | String |
+        | ``title``     | String |
+        | ``added``     | Date   |
+        | ``published`` | Date   |
+        | ``edited``    | Date   |
+
     :ivar attachments: Sub path of attachment directory
-    :ivar content_filepath: Sub path of post content file
+    :ivar content: Sub path of post content file
+    :ivar content_filepath: (**Deprecated**, Use ``content`` instead) Sub path of post content file
+    :ivar file: The format of the post `file` filename (`file` is not `attachment`, each post has only one `file`, usually the cover image) \
+    Customize the filename format by inserting an empty ``{}`` to represent the basic filename. \
+    You can use some of the [properties][ktoolbox.configuration.JobConfiguration] \
+    in Post. For example: ``{title}_{}`` could result in filenames like \
+    ``TheTitle_Stelle_lv5_logo.gif``, ``TheTitle_ScxHjZIdxt5cnjaAwf3ql2p7.jpg``, etc.
     """
     attachments: Path = Path("attachments")
+    content: Path = Path("content.txt")
     content_filepath: Path = Path("content.txt")
+    file: str = "{id}_{}"
+
+    @field_validator("content_filepath")
+    def content_filepath_validator(cls, v):
+        # noinspection PyUnresolvedReferences
+        if v != cls.model_fields["content_filepath"].default:
+            warnings.warn(
+                "`PostStructureConfiguration.content_filepath` is deprecated and is scheduled for removal in further version. "
+                "Use `PostStructureConfiguration.content` instead",
+                FutureWarning
+            )
 
 
 class JobConfiguration(BaseModel):
     """
     Download jobs Configuration
 
-    - Available properties for ``post_dirname_format``
+    - Available properties for ``post_dirname_format`` and ``filename_format``
 
         | Property      | Type   |
         |---------------|--------|
@@ -133,15 +169,15 @@ class JobConfiguration(BaseModel):
     :ivar count: Number of coroutines for concurrent download
     :ivar post_dirname_format: Customize the post directory name format, you can use some of the \
     [properties][ktoolbox.configuration.JobConfiguration] in ``Post``. \
-    e.g. ``[{published}]{id}`` > ``[2024-1-1]123123``, ``{user}_{published}_{title}`` > ``234234_2024-1-1_HelloWorld``
+    e.g. ``[{published}]{id}`` > ``[2024-1-1]123123``, ``{user}_{published}_{title}`` > ``234234_2024-1-1_TheTitle``
     :ivar post_structure: Post path structure
     :ivar mix_posts: Save all files from different posts at same path in creator directory. \
     It would not create any post directory, and ``CreatorIndices`` would not been recorded.
     :ivar sequential_filename: Rename attachments in numerical order, e.g. ``1.png``, ``2.png``, ...
     :ivar filename_format: Customize the filename format by inserting an empty ``{}`` to represent the basic filename.
     Similar to post_dirname_format, you can use some of the [properties][ktoolbox.configuration.JobConfiguration] \
     in Post. For example: ``{title}_{}`` could result in filenames like \
-    ``HelloWorld_b4b41de2-8736-480d-b5c3-ebf0d917561b``, ``HelloWorld_af349b25-ac08-46d7-98fb-6ce99a237b90``, etc. \
+    ``TheTitle_b4b41de2-8736-480d-b5c3-ebf0d917561b``, ``TheTitle_af349b25-ac08-46d7-98fb-6ce99a237b90``, etc. \
     You can also use it with ``sequential_filename``. For instance, \
     ``[{published}]_{}`` could result in filenames like ``[2024-1-1]_1.png``, ``[2024-1-1]_2.png``, etc.
     :ivar allow_list: Download files which match these patterns (Unix shell-style), e.g. ``["*.png"]``
@@ -153,8 +189,10 @@ class JobConfiguration(BaseModel):
     mix_posts: bool = False
     sequential_filename: bool = False
     filename_format: str = "{}"
-    allow_list: Set[str] = set()
-    block_list: Set[str] = set()
+    # noinspection PyDataclass
+    allow_list: Set[str] = Field(default_factory=set)
+    # noinspection PyDataclass
+    block_list: Set[str] = Field(default_factory=set)
 
 
 class LoggerConfiguration(BaseModel):
@@ -195,13 +233,13 @@ class Configuration(BaseSettings):
     use_uvloop: bool = True
 
     # noinspection SpellCheckingInspection
-    model_config = SettingsConfigDict(
+    model_config: ClassVar[SettingsConfigDict] = SettingsConfigDict(
         env_prefix='ktoolbox_',
         env_nested_delimiter='__',
-        env_file='.env',
+        env_file=['.env', 'prod.env'],
         env_file_encoding='utf-8',
         extra='ignore'
     )
 
 
-config = Configuration(_env_file=['.env', 'prod.env'])
+config = Configuration()

ktoolbox/downloader/downloader.py

@@ -59,7 +59,7 @@ def __init__(
         it will be used as the save path.
         """
 
-        self._url = url
+        self._url = config.downloader.reverse_proxy.format(url)
         self._path = path
         self._buffer_size = buffer_size or config.downloader.buffer_size
         self._chunk_size = chunk_size or config.downloader.chunk_size