Researchers' PowerTools

Fangyuan Zhou uimicro@outlook.com

FastTar

FastTar is a tool to quickly archieve files to a tar file. It is designed to be fast and simple for scenearios where a large number of small files need to be archived (e.g. transferring an image dataset or codebase).

Usage

python fast_tar.py <input_dir> -o <output_tar> -w <workers> -q <queue_ratio> -b <buffer_size>

If you encounter memory overflow, consider:

• Decrease Queue size ratio (default is 256)
• Decrease the buffer size (default is 1GB)
• Decrease the number of processes (default is min(CPU count, 16))