[Optimization] File integrity verification

[jien37]

Background:
Currently, during the download of segments, if a network issue causes a segment to fail, the entire video download fails, requiring the command to be re-executed and restarted from the beginning. In unfortunate scenarios where a segment consistently fails to download, the video cannot be successfully retrieved.

Optimization:
After completing all segment download attempts, verify integrity and retry failed segments.
(Since network issues typically persist for a certain period, immediate retries at short intervals might still fail. This method serves as a supplement to the previously implemented approach of quickly retrying failed download segments.)

Temporary Modification:

Case hulu: ext/utils/hulu_jp.py#download_segment

            with ThreadPoolExecutor(max_workers=8) as executor:
                        ......
                        pbar.update(1)
            
            # MOD START:
            print("Starting file integrity verification......")
            for i, url in enumerate(segment_links):
                temp_path = os.path.join(base_temp_dir, f"{i:05d}.ts")
                if not os.path.exists(temp_path):
                    fetch_and_save((i, url))
                    print(f"Successfully downloaded segment {i}: {url}")
                    time.sleep(2)

            print("Completed file integrity verification.")
            # MOD END

            # 結合処理
            output_path = os.path.join(base_temp_dir, name)
            with open(output_path, 'wb') as out_file:
                for i in range(len(segment_links)):
                    temp_path = os.path.join(base_temp_dir, f"{i:05d}.ts")
                    with open(temp_path, 'rb') as f:
                        out_file.write(f.read())

Case Lemino: ext/global_func/util/download_util.py#segment_downloader#download

            with ThreadPoolExecutor(max_workers=8) as executor:
                        ......
                        pbar.update(1)
            
            # MOD START:
            print("Starting file integrity verification......")
            for i, url in enumerate(segment_links):
                temp_path = os.path.join(output_temp_directory, f"{i:05d}.ts")
                if not os.path.exists(temp_path):
                    fetch_and_save((i, url))
                    print(f"Successfully downloaded segment {i}: {url}")
                    time.sleep(2)

            print("Completed file integrity verification.")
            # MOD END

            output_path = os.path.join(output_temp_directory, output_file_name)
            with open(output_path, 'wb') as out_file:
                for i in range(len(segment_links)):
                    temp_path = os.path.join(output_temp_directory, f"{i:05d}.ts")
                    with open(temp_path, 'rb') as f:
                        out_file.write(f.read())

Failed log:

Successful log:

Please help review this optimization, thanks.

[NyaShinn1204]

In v2.0.0, all download logic will be integrated into download_util, so I'll make some modifications to that part of the file. Anyway thanks issue!

[NyaShinn1204]

Added file verification to v2.0.0 dev. I think it will take a while to release...