Merge branch 'r/3.3.0'

Author: bitdruid@vbox

README.md

@@ -16,16 +16,16 @@ This tool allows you to download content from the Wayback Machine (archive.org).
 ### Pip
 
 1. Install the package <br>
-   ```pip install pywaybackup```
+   `pip install pywaybackup`
 2. Run the tool <br>
-   ```waybackup -h```
+   `waybackup -h`
 
 ### Manual
 
 1. Clone the repository <br>
-   ```git clone https://github.com/bitdruid/python-wayback-machine-downloader.git```
+   `git clone https://github.com/bitdruid/python-wayback-machine-downloader.git`
 2. Install <br>
-   ```pip install .```
+   `pip install .`
    - in a virtual env or use `--break-system-package`
 
 ## notes / issues / hints
@@ -49,6 +49,7 @@ This tool allows you to download content from the Wayback Machine (archive.org).
   The URL of the web page to download. This argument is required.
 
 #### Mode Selection (Choose One)
+
 - **`-a`**, **`--all`**:<br>
   Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
 - **`-l`**, **`--last`**:<br>
@@ -63,66 +64,67 @@ This tool allows you to download content from the Wayback Machine (archive.org).
 - **`-e`**, **`--explicit`**:<br>
   Only download the explicit given URL. No wildcard subdomains or paths. Use e.g. to get root-only snapshots. This is recommended for explicit files like `login.html` or `?query=this`.
 
-- **`--filetype`** `<filetype>`:<br>
-  Specify filetypes to download. Default is all filetypes. Separate multiple filetypes with a comma. Example: `--filetype jpg,css,js`. A filter will result in a filtered cdx-file. So if you want to download all files later, you need to query again without the filter. Filetypes are filtered as they are in the snapshot. So if there is no explicit `html` file in the path (common practice) then you cant filter them.
-
 - **`--limit`** `<count>`:<br>
-Limits the amount of snapshots to query from the CDX server. If an existing CDX file is injected, the limit will have no effect. So you would need to set `--keep`.
+  Limits the amount of snapshots to query from the CDX server. If an existing CDX file is injected, the limit will have no effect. So you would need to set `--keep`.
 
 - **Range Selection:**<br>
   Specify the range in years or a specific timestamp either start, end, or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
   (year 2019, year 2019, year+month+day 20190101, year+month+day+hour 2019010112)
-   - **`-r`**, **`--range`**:<br>
-     Specify the range in years for which to search and download snapshots.
-   - **`--start`**:<br>
-     Timestamp to start searching.
-   - **`--end`**:<br>
-     Timestamp to end searching.
+
+  - **`-r`**, **`--range`**:<br>
+    Specify the range in years for which to search and download snapshots.
+  - **`--start`**:<br>
+    Timestamp to start searching.
+  - **`--end`**:<br>
+    Timestamp to end searching.
+
+- **Filtering:**<br>
+  A filter will result in a filtered cdx-file. So if you want to download all files later, you need to query again without the filter.
+
+  - **`--filetype`** `<filetype>`:<br>
+    Specify filetypes to download. Default is all filetypes. Separate multiple filetypes with a comma. Example: `--filetype jpg,css,js`. Filetypes are filtered as they are in the snapshot. So if there is no explicit `html` file in the path (common practice) then you cant filter them.
+
+  - **`--statuscode`** `<statuscode>`:<br>
+    Specify HTTP status codes to download. Default is all statuscodes. Separate multiple status codes with a comma. Example: `--statuscode 200,301`. Pywaybackup will try to download any snapshot regardless of it's statuscode. For 404 of course this means logged errors and corresponding entries in the csv. However, you may want to get a csv that includes these negative attempts for your needs.<br>
+    Common status codes you may want to handle/filter:
+      - `200` (OK)
+      - `301` (Moved Permanently - will redirect snapshot)
+      - `404` (Not Found - snapshot seems to be empty)
+      - `500` (Internal Server Error - snapshot is at least for now not available)
 
 ### Optional
 
 #### Behavior Manipulation
 
 - **`-o`**, **`--output`**:<br>
-Defaults to `waybackup_snapshots` in the current directory. The folder where downloaded files will be saved.
+  Defaults to `waybackup_snapshots` in the current directory. The folder where downloaded files will be saved.
 
 - **`-m`**, **`--metadata`**<br>
-Change the folder where metadata will be saved (`cdx`/`db`/`csv`/`log`). Especially if you are downloading into a network share, you SHOULD set this to a local path because sqlite locking mechanism may cause issues with network shares.
+  Change the folder where metadata will be saved (`cdx`/`db`/`csv`/`log`). Especially if you are downloading into a network share, you SHOULD set this to a local path because sqlite locking mechanism may cause issues with network shares.
+
+- **`--verbose`**:<br>
+  Increase output verbosity.
 
 <!-- - **`--verbosity`** `<level>`:<br>
 Sets verbosity level. Options are `info`and `trace`. Default is `info`. -->
 
 - **`--log`** <!-- `<path>` -->:<br>
-Saves a log file into the output-dir. Named as `waybackup_<sanitized_url>.log`.
+  Saves a log file into the output-dir. Named as `waybackup_<sanitized_url>.log`.
 
 - **`--progress`**:<br>
-Shows a progress bar instead of the default output.
+  Shows a progress bar instead of the default output.
 
 - **`--workers`** `<count>`:<br>
-Sets the number of simultaneous download workers. Default is 1, safe range is about 10. Be cautious as too many workers may lead to refused connections from the Wayback Machine.
+  Sets the number of simultaneous download workers. Default is 1, safe range is about 10. Be cautious as too many workers may lead to refused connections from the Wayback Machine.
 
 - **`--no-redirect`**:<br>
-Disables following redirects of snapshots. Useful for preventing timestamp-folder mismatches caused by Archive.org redirects.
+  Disables following redirects of snapshots. Useful for preventing timestamp-folder mismatches caused by Archive.org redirects.
 
 - **`--retry`** `<attempts>`:<br>
-Specifies number of retry attempts for failed downloads.
+  Specifies number of retry attempts for failed downloads.
 
 - **`--delay`** `<seconds>`:<br>
-Specifies delay between download requests in seconds. Default is no delay (0).
-
-- **`--verbose`**:<br>
-Increase output verbosity.
-  - verbose:
-  ```
-  -----> Worker: 2 - Attempt: [1/1] Snapshot ID: [23/81]
-  SUCCESS   -> 200 OK
-            -> URL:  https://web.archive.org/web/20240225193302id_/https://example.com/assets/css/custom-styles.css
-            -> FILE: /home/manjaro/Stuff/python-wayback-machine-downloader/waybackup_snapshots/example.com/20240225193302id_/assets/css/custom-styles.css
-  ```
-  - non-verbose:
-  ```
-  55/81 - W:2 - SUCCESS - 20240225193302 - https://example.com/assets/css/custom-styles.css
-  ```
+  Specifies delay between download requests in seconds. Default is no delay (0).
 
 <!-- - **`--convert-links`**:<br>
 If set, all links in the downloaded files will be converted to local links. This is useful for offline browsing. The links are converted to the local path structure. Show output with `--verbosity trace`. -->
@@ -147,14 +149,16 @@ If set, all links in the downloaded files will be converted to local links. This
 - Detects existing `.cdx` and `.db` files in an `output dir` to resume downloading from the last successful point.
 - Compares `URL`, `mode`, and `optional query parameters` to ensure automatic resumption.
 - Skips previously downloaded files to save time.
-> **Note:** Changing URL, mode selection, query parameters or output prevents automatic resumption.
+  > **Note:** Changing URL, mode selection, query parameters or output prevents automatic resumption.
 
 #### Resetting a Job (`--reset`)
+
 - Deletes `.cdx` and `.db` files and restarts the process from scratch.
 - Does **not** remove already downloaded files.
 - `waybackup -u https://example.com -a --reset`
 
 #### Keeping Job Data (`--keep`)
+
 - Normally, `.cdx` and `.db` files are deleted after a successful job.
 - `--keep` preserves them for future re-analysis or extending the query.
 - `waybackup -u https://example.com -a --keep`
@@ -165,13 +169,13 @@ If set, all links in the downloaded files will be converted to local links. This
 ## Examples
 
 1. Download a specific single snapshot of all available files (starting from root):<br>
-`waybackup -u https://example.com -a --start 20210101000000 --end 20210101000000`
+   `waybackup -u https://example.com -a --start 20210101000000 --end 20210101000000`
 2. Download a specific single snapshot of all available files (starting from a subdirectory):<br>
-`waybackup -u https://example.com/subdir1/subdir2/assets/ -a --start 20210101000000 --end 20210101000000`
+   `waybackup -u https://example.com/subdir1/subdir2/assets/ -a --start 20210101000000 --end 20210101000000`
 3. Download a specific single snapshot of the exact given URL (no subdirs):<br>
-`waybackup -u https://example.com -a --start 20210101000000 --end 20210101000000 --explicit`
+   `waybackup -u https://example.com -a --start 20210101000000 --end 20210101000000 --explicit`
 4. Download all snapshots of all available files in the given range:<br>
-`waybackup -u https://example.com -a --start 20210101000000 --end 20231122000000`
+   `waybackup -u https://example.com -a --start 20210101000000 --end 20231122000000`
 
 <br>
 <br>
@@ -184,7 +188,9 @@ The output path is currently structured as follows by an example for the query:<
 `http://example.com/subdir1/subdir2/assets/`
 <br><br>
 For the first and last version (`-f` or `-l`):
+
 - Will only include all files/folders starting from your query-path.
+
 ```
 your/path/waybackup_snapshots/
 └── the_root_of_your_query/ (example.com/)
@@ -195,8 +201,11 @@ your/path/waybackup_snapshots/
                 ├── style.css
                 ...
 ```
+
 For all versions (`-a`):
+
 - Will create a folder named as the root of your query. Inside this folder, you will find all timestamps and per timestamp the path you requested.
+
 ```
 your/path/waybackup_snapshots/
 └── the_root_of_your_query/ (example.com/)
@@ -237,6 +246,23 @@ For download queries:
 ]
 ```
 
+### Log
+
+Verbose:
+
+```
+-----> Worker: 2 - Attempt: [1/1] Snapshot ID: [23/81]
+SUCCESS   -> 200 OK
+          -> URL:  https://web.archive.org/web/20240225193302id_/https://example.com/assets/css/custom-styles.css
+          -> FILE: /home/manjaro/Stuff/python-wayback-machine-downloader/waybackup_snapshots/example.com/20240225193302id_/assets/css/custom-styles.css
+```
+
+Non-verbose:
+
+```
+55/81 - W:2 - SUCCESS - 20240225193302 - https://example.com/assets/css/custom-styles.css
+```
+
 ### Debugging
 
 Exceptions will be written into `waybackup_error.log` (each run overwrites the file).

pyproject.toml

@@ -7,7 +7,7 @@ packages = ["pywaybackup"]
 
 [project]
 name = "pywaybackup"
-version = "3.2.0"
+version = "3.3.0"
 description = "Query and download archive.org as simple as possible."
 authors = [
     { name = "bitdruid", email = "bitdruid@outlook.com" }

pywaybackup/Arguments.py

@@ -3,16 +3,19 @@
 import os
 import argparse
 
+from argparse import RawTextHelpFormatter
+
 from importlib.metadata import version
 
 from pywaybackup.helper import url_split, sanitize_filename
 
 class Arguments:
 
     def __init__(self):
-
-        parser = argparse.ArgumentParser(description='Download from wayback machine (archive.org)')
-        parser.add_argument('-v', '--version', action='version', version='%(prog)s ' + version("pywaybackup") + ' by @bitdruid -> https://github.com/bitdruid')
+        parser = argparse.ArgumentParser(
+            description=f"<<< python-wayback-machine-downloader v{version('pywaybackup')} >>>\nby @bitdruid -> https://github.com/bitdruid",
+            formatter_class=RawTextHelpFormatter,
+        )
 
         required = parser.add_argument_group('required (one exclusive)')
         required.add_argument('-u', '--url', type=str, metavar="", help='url (with subdir/subdomain) to download')
@@ -27,20 +30,21 @@ def __init__(self):
         optional.add_argument('-r', '--range', type=int, metavar="", help='range in years to search')
         optional.add_argument('--start', type=int, metavar="", help='start timestamp format: YYYYMMDDhhmmss')
         optional.add_argument('--end', type=int, metavar="", help='end timestamp format: YYYYMMDDhhmmss')
-        optional.add_argument('--filetype', type=str, metavar="", help='filetypes to download comma separated (e.g. "html,css")')
         optional.add_argument('--limit', type=int, nargs='?', const=True, metavar='int', help='limit the number of snapshots to download')
+        optional.add_argument('--filetype', type=str, metavar="", help='filetypes to download comma separated (js,css,...)')
+        optional.add_argument('--statuscode', type=str, metavar="", help='statuscodes to download comma separated (200,404,...)')
 
         behavior = parser.add_argument_group('manipulate behavior')
         behavior.add_argument('-o', '--output', type=str, metavar="", help='output for all files - defaults to current directory')
         behavior.add_argument('-m', '--metadata', type=str, metavar="", help='change directory for db/cdx/csv/log files')
+        behavior.add_argument('-v', '--verbose', action='store_true', help='overwritten by progress - gives detailed output')
         behavior.add_argument('--log', action='store_true', help='save a log file into the output folder')
         behavior.add_argument('--progress', action='store_true', help='show a progress bar')
         behavior.add_argument('--no-redirect', action='store_true', help='do not follow redirects by archive.org')
         behavior.add_argument('--retry', type=int, default=0, metavar="", help='retry failed downloads (opt tries as int, else infinite)')
         behavior.add_argument('--workers', type=int, default=1, metavar="", help='number of workers (simultaneous downloads)')
         # behavior.add_argument('--convert-links', action='store_true', help='Convert all links in the files to local paths. Requires -c/--current')
         behavior.add_argument('--delay', type=int, default=0, metavar="", help='delay between each download in seconds')
-        behavior.add_argument('--verbose', action='store_true', help='overwritten by progress - gives detailed output')
 
         special = parser.add_argument_group('special')
         special.add_argument('--reset', action='store_true', help='reset the job and ignore existing cdx/db/csv files')
@@ -61,6 +65,52 @@ def get_args(self):
         return self.args
 
 class Configuration:
+
+    # def __init__(self):
+    #     self.args = Arguments().get_args()
+    #     for key, value in vars(self.args).items():
+    #         setattr(Configuration, key, value)
+
+    #     self.set_config()
+
+    # def set_config(self):
+    #             # args now attributes of Configuration // Configuration.output, ...
+    #     self.command = ' '.join(sys.argv[1:])
+    #     self.domain, self.subdir, self.filename = url_split(self.url)
+        
+    #     if self.output is None:
+    #         self.output = os.path.join(os.getcwd(), "waybackup_snapshots")
+    #     if self.metadata is None:
+    #         self.metadata = self.output
+    #     os.makedirs(self.output, exist_ok=True) if not self.save else None
+    #     os.makedirs(self.metadata, exist_ok=True) if not self.save else None
+                
+    #     if self.all:
+    #         self.mode = "all"
+    #     if self.last:
+    #         self.mode = "last"
+    #     if self.first:
+    #         self.mode = "first"
+    #     if self.save:
+    #         self.mode = "save"
+        
+    #     if self.filetype:
+    #         self.filetype = [f.lower().strip() for f in self.filetype.split(",")]
+    #     if self.statuscode:
+    #         self.statuscode = [s.lower().strip() for s in self.statuscode.split(",")]
+
+    #     base_path = self.metadata
+    #     base_name = f"waybackup_{sanitize_filename(self.url)}"
+    #     self.cdxfile = os.path.join(base_path, f"{base_name}.cdx")
+    #     self.dbfile = os.path.join(base_path, f"{base_name}.db")
+    #     self.csvfile = os.path.join(base_path, f"{base_name}.csv")
+    #     self.log = os.path.join(base_path, f"{base_name}.log") if self.log else None
+        
+    #     if self.reset:
+    #         os.remove(self.cdxfile) if os.path.isfile(self.cdxfile) else None
+    #         os.remove(self.dbfile) if os.path.isfile(self.dbfile) else None
+    #         os.remove(self.csvfile) if os.path.isfile(self.csvfile) else None
+        
 
     @classmethod
     def init(cls):
@@ -90,7 +140,9 @@ def init(cls):
             cls.mode = "save"
 
         if cls.filetype:
-            cls.filetype = [ft.lower().strip() for ft in cls.filetype.split(",")]
+            cls.filetype = [f.lower().strip() for f in cls.filetype.split(",")]
+        if cls.statuscode:
+            cls.statuscode = [s.lower().strip() for s in cls.statuscode.split(",")]
 
         base_path = cls.metadata
         base_name = f"waybackup_{sanitize_filename(cls.url)}"