Merge branch 'r/1.0.0'

Author: bitdruid

README.md

@@ -2,7 +2,6 @@
 
 [![PyPI](https://img.shields.io/pypi/v/pywaybackup)](https://pypi.org/project/pywaybackup/)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/pywaybackup)](https://pypi.org/project/pywaybackup/)
-![Release](https://img.shields.io/badge/Release-beta-orange)
 ![Python Version](https://img.shields.io/badge/Python-3.6-blue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
@@ -31,43 +30,41 @@ Internet-archive is a nice source for several OSINT-information. This script is
 
 This script allows you to download content from the Wayback Machine (archive.org). You can use it to download either the latest version or all versions of web page snapshots within a specified range.
 
-<!-- ## Info -->
-
 ### Arguments
 
 - `-h`, `--help`: Show the help message and exit.
 - `-a`, `--about`: Show information about the script and exit.
 
 #### Required Arguments
 
-- `-u URL`, `--url URL`: The URL of the web page to download. This argument is required.
+- `-u`, `--url`: The URL of the web page to download. This argument is required.
 
 #### Mode Selection (Choose One)
 
-- `-c`, `--current`: Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files.
+- `-c`, `--current`: Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files (but not any original state because new and old versions are mixed).
 - `-f`, `--full`: Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
 - `-s`, `--save`: Save a page to the Wayback Machine. (beta)
 
 #### Optional Arguments
 
 - `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
 - `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths.
-- `-o OUTPUT`, `--output OUTPUT`: The folder where downloaded files will be saved.
+- `-o`, `--output`: The folder where downloaded files will be saved.
 
 - **Range Selection:**<br>
 Specify the range in years or a specific timestamp either start, end or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
 (year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
-   - `-r RANGE`, `--range RANGE`: Specify the range in years for which to search and download snapshots.
+   - `-r`, `--range`: Specify the range in years for which to search and download snapshots.
    - `--start`: Timestamp to start searching.
    - `--end`: Timestamp to end searching.
 
 #### Additional
 
-- `--csv`: Save a csv file with the list of snapshots inside the output folder.
+- `--csv`: Save a csv file with the list of snapshots inside the output folder or a specified folder. If you set `--list` the csv will contain the cdx list of snapshots. If you set either `--current` or `--full` the csv will contain the downloaded files.
 - `--no-redirect`: Do not follow redirects of snapshots. Archive.org sometimes redirects to a different snapshot for several reasons. Downloading redirects may lead to timestamp-folders which contain some files with a different timestamp. This does not matter if you only want to download the latest version (`-c`).
-- `--verbosity [LEVEL]`: Set the verbosity: json (print json response), progress (show progress bar) or standard (default).
-- `--retry [RETRY_FAILED]`: Retry failed downloads. You can specify the number of retry attempts as an integer.
-- `--worker [AMOUNT]`: The number of worker to use for downloading (simultaneous downloads). Default is 1. A safe spot is about 10 workers. Beware: Using too many worker will lead into refused connections from the Wayback Machine. Duration about 1.5 minutes.
+- `--verbosity`: Set the verbosity: json (print json response), progress (show progress bar).
+- `--retry`: Retry failed downloads. You can specify the number of retry attempts as an integer.
+- `--workers`: The number of workers to use for downloading (simultaneous downloads). Default is 1. A safe spot is about 10 workers. Beware: Using too many workers will lead into refused connections from the Wayback Machine. Duration about 1.5 minutes.
 
 ### Examples
 
@@ -77,14 +74,20 @@ Download latest snapshot of all files:<br>
 Download latest snapshot of all files with retries:<br>
 `waybackup -u http://example.com -c --retry 3`
 
-Download all snapshots sorted per timestamp with a specified range and follow redirects:<br>
-`waybackup -u http://example.com -f -r 5 --redirect`
+Download all snapshots sorted per timestamp with a specified range and do not follow redirects:<br>
+`waybackup -u http://example.com -f -r 5 --no-redirect`
+
+Download all snapshots sorted per timestamp with a specified range and save to a specified folder with 3 workers:<br>
+`waybackup -u http://example.com -f -r 5 -o /home/user/Downloads/snapshots --workers 3`
+
+Download all snapshots from 2020 to 12th of December 2022 with 4 workers, save a csv and show a progress bar:
+`waybackup -u http://example.com -f --start 2020 --end 20221212 --workers 4 --csv --verbosity progress`
 
-Download all snapshots sorted per timestamp with a specified range and save to a specified folder with 3 worker:<br>
-`waybackup -u http://example.com -f -r 5 -o /home/user/Downloads/snapshots --worker 3`
+Download all snapshots and output a json response:<br>
+`waybackup -u http://example.com -f --verbosity json`
 
-List available snapshots per timestamp without downloading:<br>
-`waybackup -u http://example.com -f -l`
+List available snapshots per timestamp without downloading and save a csv file to home folder:<br>
+`waybackup -u http://example.com -f -l --csv /home/user/Downloads`
 
 ## Contributing
 

pywaybackup/__version__.py

@@ -1 +1 @@
-__version__ = "0.8.1"
+__version__ = "1.0.0"

pywaybackup/archive.py

@@ -107,17 +107,17 @@ def query_list(url: str, range: int, start: int, end: int, explicit: bool, mode:
 
 
 # example download: http://web.archive.org/web/20190815104545id_/https://www.google.com/
-def download_list(output, retry, no_redirect, worker):
+def download_list(output, retry, no_redirect, workers):
     """
     Download a list of urls in format: [{"timestamp": "20190815104545", "url": "https://www.google.com/"}]
     """
     if sc.count_list() == 0: 
         v.write("\nNothing to download");
         return
     v.write("\nDownloading snapshots...", progress=0)
-    if worker > 1:
-        v.write(f"\n-----> Simultaneous downloads: {worker}")
-        batch_size = sc.count_list() // worker + 1
+    if workers > 1:
+        v.write(f"\n-----> Simultaneous downloads: {workers}")
+        batch_size = sc.count_list() // workers + 1
     else:
         batch_size = sc.count_list()
     sc.create_collection()
@@ -126,7 +126,7 @@ def download_list(output, retry, no_redirect, worker):
     worker = 0
     for batch in batch_list:
         worker += 1
-        thread = threading.Thread(target=download_loop, args=(batch, output, worker, retry, no_redirect))
+        thread = threading.Thread(target=download_loop, args=(batch, output, workers, retry, no_redirect))
         threads.append(thread)
         thread.start()
     for thread in threads:
@@ -256,15 +256,18 @@ def parse_response_code(response_code: int):
         return RESPONSE_CODE_DICT[response_code]
     return "Unknown response code"
 
-def save_csv(csv_path: str):
+def save_csv(csv_path: str, url: str):
     """
     Write a CSV file with the list of snapshots.
     """
     import csv
+    disallowed = ['<', '>', ':', '"', '/', '\\', '|', '?', '*']
+    for char in disallowed:
+        url = url.replace(char, '.')
     if sc.count_list() > 0:
         v.write("\nSaving CSV file...")
         os.makedirs(os.path.abspath(csv_path), exist_ok=True)
-        with open(os.path.join(csv_path, "waybackup.csv"), mode='w') as file:
+        with open(os.path.join(csv_path, f"waybackup_{url}.csv"), mode='w') as file:
             row = csv.DictWriter(file, sc.SNAPSHOT_COLLECTION[0].keys())
             row.writeheader()
             for snapshot in sc.SNAPSHOT_COLLECTION:

pywaybackup/arguments.py

@@ -1,3 +1,4 @@
+import sys
 import argparse
 from pywaybackup.__version__ import __version__
 
@@ -7,7 +8,7 @@ def parse():
     parser.add_argument('-a', '--about', action='version', version='%(prog)s ' + __version__ + ' by @bitdruid -> https://github.com/bitdruid')
 
     required = parser.add_argument_group('required')
-    required.add_argument('-u', '--url', type=str, help='URL to use')
+    required.add_argument('-u', '--url', type=str, metavar="", help='URL to use')
     exclusive_required = required.add_mutually_exclusive_group(required=True)
     exclusive_required.add_argument('-c', '--current', action='store_true', help='Download the latest version of each file snapshot (opt range in y)')
     exclusive_required.add_argument('-f', '--full', action='store_true', help='Download snapshots of all timestamps (opt range in y)')
@@ -16,18 +17,18 @@ def parse():
     optional = parser.add_argument_group('optional')
     optional.add_argument('-l', '--list', action='store_true', help='Only print snapshots (opt range in y)')
     optional.add_argument('-e', '--explicit', action='store_true', help='Search only for the explicit given url')
-    optional.add_argument('-o', '--output', type=str, help='Output folder defaults to current directory')
-    optional.add_argument('-r', '--range', type=int, help='Range in years to search')
-    optional.add_argument('--start', type=int, help='Start timestamp format: YYYYMMDDhhmmss')
-    optional.add_argument('--end', type=int, help='End timestamp format: YYYYMMDDhhmmss')
+    optional.add_argument('-o', '--output', type=str, metavar="", help='Output folder defaults to current directory')
+    optional.add_argument('-r', '--range', type=int, metavar="", help='Range in years to search')
+    optional.add_argument('--start', type=int, metavar="", help='Start timestamp format: YYYYMMDDhhmmss')
+    optional.add_argument('--end', type=int, metavar="", help='End timestamp format: YYYYMMDDhhmmss')
 
     special = parser.add_argument_group('special')
-    special.add_argument('--csv', type=str, nargs='?', const=True, help='Save a csv file on a given path or defaults to the output folder')
+    special.add_argument('--csv', type=str, nargs='?', metavar='', help='Save a csv file on a given path or defaults to the output folder')
     special.add_argument('--no-redirect', action='store_true', help='Do not follow redirects by archive.org')
-    special.add_argument('--verbosity', type=str, default="standard", choices=["standard", "progress", "json"], help='Verbosity level')
-    special.add_argument('--retry', type=int, default=0, metavar="X-TIMES", help='Retry failed downloads (opt tries as int, else infinite)')
-    special.add_argument('--worker', type=int, default=1, metavar="AMOUNT", help='Number of worker (simultaneous downloads)')
+    special.add_argument('--verbosity', type=str, default="standard", metavar="", help='["progress", "json"] Verbosity level')
+    special.add_argument('--retry', type=int, default=0, metavar="", help='Retry failed downloads (opt tries as int, else infinite)')
+    special.add_argument('--workers', type=int, default=1, metavar="", help='Number of workers (simultaneous downloads)')
 
-    args = parser.parse_args()
+    args = parser.parse_args(args=None if sys.argv[1:] else ['--help']) # if no arguments are given, print help
 
     return args

pywaybackup/main.py

@@ -15,7 +15,7 @@ def main():
 
     if args.output is None:
         args.output = os.path.join(os.getcwd(), "waybackup_snapshots")
-    if args.csv is True:
+    if args.csv == "":
         args.csv = args.output
 
     if args.save:
@@ -25,9 +25,9 @@ def main():
         if args.list:
             archive.print_list()
         else:
-            archive.download_list(args.output, args.retry, args.no_redirect, args.worker)
+            archive.download_list(args.output, args.retry, args.no_redirect, args.workers)
         if args.csv:
-            archive.save_csv(args.csv)
+            archive.save_csv(args.csv, args.url)
     v.close()
 
 if __name__ == "__main__":